GPT-2 A nascent transfer learning method that could eliminate supervised learning in some NLP tasks

Many of the current state-of-art models for supervised NLP tasks are models pre-trained on language modeling (which is an unsupervised task), and then fine tuned (supervised) with labeled data specific to a task.

Fig 1. Transfer learning to downstream tasks started around 2013 with using context