For select supervised tasks with self-supervised learning(SSL) models satisfying certain properties

Introduction

Self-supervised learning (SSL) is increasingly used to solve language and vision tasks that are traditionally solved with supervised learning.

Supervised learning has been the predominant approach to date in state-of-art approaches for ascribing labels to either whole or parts of an input. The labels in many cases are synthetic in…

Thoughts and Theory

A prerequisite to use a pre-trained model as is, without fine tuning

TL;DR

Self-supervised learning is being leveraged off at scale using transformers, not only for text, but lately also for images(CLIP, ALIGN), to solve traditionally supervised tasks (e.g. classification), either as is, or with subsequent fine tuning. While most, if not all, downstream NLP tasks are performed, to date, with subsequent fine-tuning…

BERT is a prize addition to the practitioner’s toolbox

TL;DR

Natural language processing tasks traditionally requiring labeled data could be solved entirely or in part, subject to a few constraints, without the need for labeled data by leveraging the self-supervised learning of a BERT model, provided those tasks lend themselves to be viewed entirely or in part, as a similarity…

A hybrid approach combining symbolic processing with distributed representations

TL;DR

Extracting all the different ways a particular term can be referred to (synonym harvesting) is key for applications in biomedical domain where drugs, genes etc. have many synonyms. While there are human curated knowledge bases for synonyms in the biomedical domain, they are generally incomplete, continually trying to play catchup…

An approach to evaluate a pre-trained BERT model to increase performance

TL;DR

Training a BERT model from scratch on a domain specific corpus such as biomedical space with a custom vocabulary generated specific to that space has proven to be critical to maximize model performance in biomedical domain. This is largely because of language characteristics that are unique to biomedical space which…

For sentence similarity/document search applications

TL;DR

To date, models learn fixed size representation of sentences, typically with some form of supervision, which are then used for sentence similarity or other downstream tasks. Examples of this are Google’s Universal sentence encoder (2018) and Sentence transformers (2019). Supervised learning of fixed size representations tends to outperform unsupervised creation…

Viruses (COVID-19) — from a computational perspective

Dr. Britt Glaunsinger (virologist) offers an in-depth biological perspective of COVID-19, in her recent video and talks. This post is a computational perspective largely based on the substance of her talks. A computational perspective of an evolving self-replicating linear sequence of data, particularly its manifestations in three dimensions both structurally…

COVID-19 questions — a use case for improving sentence fragment search

TL;DR

Embeddings for sentence fragments harvested from a document can serve as extractive summary facets of that document and potentially accelerate its discovery, particularly when user input is a sentence fragment. These fragment embeddings not only yield better quality results than traditional text matching systems, but also circumvent a problem inherent…

TL;DR

An improved version of this approach published in Jan 2022 describes how to scale it to a large number of entity types (e.g. 68 entity types spanning the domain of biology and PHI entities such as person, location, organization).

In natural language processing, identifying entities of interest (NER) in a…

Ajit Rajasekharan

Machine learning practitioner

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store