European elections: deep conversation about AI is missing
FEBRUARY 10, 2021
I am definitely a tech enthusiast. I got my first computer, a Commodore Vic20, at the age of 10 and ever since I have been thrilled with every new technology that changes the world.
- The effect of AI on tomorrow’s jobs will be immense and no country is ready for it. For the first time in history, AI will displace high-skilled white-collar jobs. Until now, although innovation killed some jobs, it created new and better ones; I believe this will not be the case with AI though. This time jobs will be killed, period.Until now we are used to the fact that machines replace routine jobs with repetitive tasks. However, thanks to AI, computers are increasingly able to perform complicated tasks more cheaply and effectively than people. A recent study of Oxford University suggests that 47% of today’s jobs could be automated in the next two decades and this is a huge number that cannot be replaced. Dear European Elections Candidate, what should we do? Maybe work for 3 days per week instead of 5 to have jobs for everybody?
- Even if we do have enough jobs for everybody, people abilities are by default unequal. In a world that is polarized economically, many will not be able to find work prospects and in most of the cases their incomes will be squeezed. To put it simply, some people may not be capable of doing high-skilled jobs. Dear Candidate, do you recognize this problem and what could be the solution? Maybe we should change the education system by focusing on cognitive abilities and social skills that are prerequisites for any human-centric job of the future.
- Rising inequality will bring anger and income gaps will broaden, causing huge social disorder. However, optimists say that this will not be a long-term problem, since AI will offer a more productive overall society that will be richer, and its wealthier inhabitants will have access to more goods and services. Dear Candidate, how should we share the increased wealth produced by AI and only part of the society, to all people in the world? Many on the left think of a Universal Income. In other words, anyone was born to have a reasonable income even without working. This also means raising a lot taxes on the rich and somehow to control the mobility of capital and highly skilled labor in a global level.
- Let’s say now we solve the Universal Income problems and manage to apply it to society. Still, will the people be really happy? AI’s impact will feel like a tornado, and having an income solves only a part of the problem. How would a society react if 50% of its people are actually unemployed living by a Universal Income? What about the fulfillment you feel when you do something important (or at least think so)? Dear Candidate, what we saw with Brexit was that making money isn’t enough for a society to be happy. Most of the Brexiters make more money than ever in Britain’s history, but also feel that they do not participate “in the party” and want to punish those who do. Apart from work careers, what will be the new social rewards that will be enough to give people meaning in their lives?
Having this as our baseline model we started experimenting with various possible improvements. You can find below what worked well and what didn’t work that great for us:
- We created domain-specific word embeddings based on the 30.000 similar unlabeled documents that were given to us. These word embeddings were much better than generic Greek embedding.
- We tried 3 alternatives for word embeddings: don ‘to use pre-trained word embeddings, use pretrained but fine-tune them during training, use pretrained, do not fine-tune them, but freeze their weights. The third alternative produced the best results, then the second and lastly the first. As professor Manning says in his online NLP course, do not fine-tune the word embeddings if you don’t have enough training data.
- Multi-head attention helped; we used a number of heads equal to the number of output classes * 4.
- Adding another fully connected layer was also a good idea. Of course, increasing the parameters required regularization, so we used both dropouts with 0.5 probability and weight decay in all possible places.
- We couldn’t find a way to use the actual evaluation metric as a loss function. The evaluation metric was ROUGE-4 which counts the overlap of 4-grams between the actual answer and our prediction, but we used PyTorch BCEWithLogitsLoss which combines a sigmoid activation and Binary Cross-Entropy loss.
- We couldn’t resist the temptation to add special features based on specific words that we know that characterize a phrase, as an answer to the question we were given. For example, the word “child” and “kid” most probably would exist in a phrase about family status. We tried to have diversity in these hand-crafted features, as we knew that they were going to add more, unknown questions in the final evaluation phase, so we wanted our model to be effective even in new questions.
- We tried to also incorporate a Greek BERT model created by AUEB’s Natural Language Processing Group but it increased the training time without improving significantly the results.
The final model after these aforementioned improvements is:
Wrapping it up, one has to point out that it’s very important to have a solid baseline model and make many experiments to find out what works best. Furthermore, try to find a metric, a single number, that evaluates your model. It’s not always that easy but it’s crucial to be able to compare your models, especially after creating hundreds of models.
So, if there is a takeaway for me, is to stop looking for that perfect, single is that there isn’t a “silver bullet” that can kill the beast and solve any NLP problem, you‘d better prepare yourself to try a great many of them to get the job done!