I can recall the excitement when I heard the first digitized song (not MP3 yet) on my Amiga, when I sent my first email using vi in a UNIX terminal, the ICQ chat channels, the touch screen on my Compaq PocketPC, my first blog post, the Virtual Reality experience with an Oculus device. In every technological advancement, I could imagine the future with optimism and decent accuracy.

This is not the case with Artificial Intelligence though. I know that AI is going to be the most exciting technology that has ever been built by humanity and maybe the last one too, as the next could be “AI built”. Not to mention that AI absorbs 90% of our R&D budget in Pobuca. Still, I cannot say how AI will change the society, the humanity, our world. I have more or less ten different scenarios in my mind and with all the branches make around a hundred alternatives…

Last March I was invited as a speaker to The Economist event to talk about the AI research that will be tomorrow’s technology. One of the questions I got asked was “If you had to select who will control AI, politicians or businesses, which would it be?” and I answered “politicians” without a second thought. My explanation was that AI is so important for humanity that it shouldn’t be controlled by profit organizations. Now that I am thinking about my answer again, it could also be that I don’t want the business world (thinking of myself as part of it) to mess with this “hot potato”! I admire in advance the politicians that will have the courage to build an AI framework for the society. They should be bright and brave at the same time, since their decisions may have a permanent impact – positive or negative – on humanity as a whole. Moreover, what I think is missing from the ongoing debates among the European election candidates is a deep conversation about AI and its social implications, since this is definitely an EU-wide (at least) matter.

Dear Candidates, do you want to have my vote and admiration (if that matters at all)? Please discuss, do not answer, just discuss the following:

1. The effect of AI on tomorrow’s jobs will be immense and no country is ready for it. For the first time in history, AI will displace high-skilled white-collar jobs. Until now, although innovation killed some jobs, it created new and better ones; I believe this will not be the case with AI though. This time jobs will be killed, period.Until now we are used to the fact that machines replace routine jobs with repetitive tasks. However, thanks to AI, computers are increasingly able to perform complicated tasks more cheaply and effectively than people. A recent study of Oxford University suggests that 47% of today’s jobs could be automated in the next two decades and this is a huge number that cannot be replaced. Dear European Elections Candidate, what should we do? Maybe work for 3 days per week instead of 5 to have jobs for everybody?
2. Even if we do have enough jobs for everybody, people abilities are by default unequal. In a world that is polarized economically, many will not be able to find work prospects and in most of the cases their incomes will be squeezed. To put it simply, some people may not be capable of doing high-skilled jobs. Dear Candidate, do you recognize this problem and what could be the solution? Maybe we should change the education system by focusing on cognitive abilities and social skills that are prerequisites for any human-centric job of the future.
3. Rising inequality will bring anger and income gaps will broaden, causing huge social disorder. However, optimists say that this will not be a long-term problem, since AI will offer a more productive overall society that will be richer, and its wealthier inhabitants will have access to more goods and services. Dear Candidate, how should we share the increased wealth produced by AI and only part of the society, to all people in the world? Many on the left think of a Universal Income. In other words, anyone was born to have a reasonable income even without working. This also means raising a lot taxes on the rich and somehow to control the mobility of capital and highly skilled labor in a global level.
4. Let’s say now we solve the Universal Income problems and manage to apply it to society. Still, will the people be really happy? AI’s impact will feel like a tornado, and having an income solves only a part of the problem. How would a society react if 50% of its people are actually unemployed living by a Universal Income? What about the fulfillment you feel when you do something important (or at least think so)? Dear Candidate, what we saw with Brexit was that making money isn’t enough for a society to be happy. Most of the Brexiters make more money than ever in Britain’s history, but also feel that they do not participate “in the party” and want to punish those who do. Apart from work careers, what will be the new social rewards that will be enough to give people meaning in their lives?

Instead of answering the last question directly, I will close my article with a story:

Milton Friedman travelled to China in the 1960s and visited a worksite where a new canal was being built. He was shocked to see that, instead of modern tractors and earth movers, the workers had shovels. He asked why there were so few machines. The government bureaucrat explained: “You don’t understand. This is a jobs program.” To which Milton replied: “Oh, I thought you were trying to build a canal. If it’s jobs you want, then you should give these workers spoons, not shovels.”

Maybe because of AI we should invent the “spoons” of the 21st century. The most radical idea that I heard was to officially set targets and rewards through video games, so that people could give a meaning to their lives. In the beginning I was shocked, but then I thought of my childhood hero/god Michael Jordan: he certainly felt very important and had a purpose in his life, while the only thing he did was put a ball in a basket!

Having this as our baseline model we started experimenting with various possible improvements. You can find below what worked well and what didn’t work that great for us:

1. We created domain-specific word embeddings based on the 30.000 similar unlabeled documents that were given to us. These word embeddings were much better than generic Greek embedding.
2. We tried 3 alternatives for word embeddings: don ‘to use pre-trained word embeddings, use pretrained but fine-tune them during training, use pretrained, do not fine-tune them, but freeze their weights. The third alternative produced the best results, then the second and lastly the first. As professor Manning says in his online NLP course, do not fine-tune the word embeddings if you don’t have enough training data.
3. Multi-head attention helped; we used a number of heads equal to the number of output classes * 4.
4. Adding another fully connected layer was also a good idea. Of course, increasing the parameters required regularization, so we used both dropouts with 0.5 probability and weight decay in all possible places.
5. We couldn’t find a way to use the actual evaluation metric as a loss function. The evaluation metric was ROUGE-4 which counts the overlap of 4-grams between the actual answer and our prediction, but we used PyTorch BCEWithLogitsLoss which combines a sigmoid activation and Binary Cross-Entropy loss.
6. We couldn’t resist the temptation to add special features based on specific words that we know that characterize a phrase, as an answer to the question we were given. For example, the word “child” and “kid” most probably would exist in a phrase about family status. We tried to have diversity in these hand-crafted features, as we knew that they were going to add more, unknown questions in the final evaluation phase, so we wanted our model to be effective even in new questions.
7. We tried to also incorporate a Greek BERT model created by AUEB’s Natural Language Processing Group but it increased the training time without improving significantly the results.

 The final model after these aforementioned improvements is:

Wrapping it up, one has to point out that it’s very important to have a solid baseline model and make many experiments to find out what works best. Furthermore, try to find a metric, a single number, that evaluates your model. It’s not always that easy but it’s crucial to be able to compare your models, especially after creating hundreds of models.

So, if there is a takeaway for me, is to stop looking for that perfect, single is that there isn’t a “silver bullet” that can kill the beast and solve any NLP problem, you‘d better prepare yourself to try a great many of them to get the job done!