It is already in motion… — The gist A paper published a few days ago (15 June 2022) offers a key insight into the impressive task-solving capabilities of the recent spate of billion+ parameter language models. Large language models exihibit the ability to solve specific tasks when two conditions are met — the model parameter size exceeds 100…

An image representation that’s a close analog for words in generative tasks — Overview The search for an answer to the question “what is the analog of words in images?” appears to be broadly proceeding along two paths (with a few exceptions) driven by multiple factors — the nature of the task being solved (discriminative or generative ) how a model is trained (self-supervised…

For select supervised tasks with self-supervised learning(SSL) models satisfying certain properties — TL;DR Self-supervised learning (SSL) could be used to avoid supervised learning for some tasks leveraging self-supervised models like BERT, as is, without fine-tuning (supervision). For instance, this post describes an approach to perform named-entity recognition without fine-tuning a model on sentences. Instead, a small subset of BERT’s learned vocabulary is manually…

A prerequisite to use a pre-trained model as is, without fine tuning — TL;DR Self-supervised learning is being leveraged off at scale using transformers, not only for text, but lately also for images(CLIP, ALIGN), to solve traditionally supervised tasks (e.g. classification), either as is, or with subsequent fine tuning. While most, if not all, downstream NLP tasks are performed, to date, with subsequent fine-tuning…

BERT is a prize addition to the practitioner’s toolbox — TL;DR Natural language processing tasks traditionally requiring labeled data could be solved entirely or in part, subject to a few constraints, without the need for labeled data by leveraging the self-supervised learning of a BERT model, provided those tasks lend themselves to be viewed entirely or in part, as a similarity…

A hybrid approach combining symbolic processing with distributed representations — TL;DR Extracting all the different ways a particular term can be referred to (synonym harvesting) is key for applications in biomedical domain where drugs, genes etc. have many synonyms. While there are human curated knowledge bases for synonyms in the biomedical domain, they are generally incomplete, continually trying to play catchup…

An approach to evaluate a pre-trained BERT model to increase performance — TL;DR Training a BERT model from scratch on a domain specific corpus such as biomedical space with a custom vocabulary generated specific to that space has proven to be critical to maximize model performance in biomedical domain. This is largely because of language characteristics that are unique to biomedical space which…

The data driven argument David Attenborough makes about our planet’s future, using his own lifetime as the scale of measure, in the recently released film on Netflix “A life on our planet”, is hard to ignore, despite the fatigue that may have desensitized us to this topic by the relentless…

For sentence similarity/document search applications — TL;DR To date, models learn fixed size representation of sentences, typically with some form of supervision, which are then used for sentence similarity or other downstream tasks. Examples of this are Google’s Universal sentence encoder (2018) and Sentence transformers (2019). Supervised learning of fixed size representations tends to outperform unsupervised creation…