How does BERT produce contextualized rich text representations?

BERT (Bidirectional Encoder Representations from Transformer) published in 2018. It uses Transformer encoder (only, without decoder) to attain deep understanding of input sequence representation. It is used to obtain contextual word representations. If you familiar with something so-called word embedding, BERT can produce contextualized word embedding (the other type is static word embedding, e.g. fastText).

While in static word embedding, each word will always be converted into the same exact word embedding regardless the meaning of the words within a context. In contrast, contextualized word embedding will produce a different word embeddings for a word depending on the sentence context. Here, I will explain briefly how does BERT can learn the sequence and produce deep representations of the input. My explanation won’t include mathematical expression and coding example, it is just the concept explained.

Angry Birds (bird = bert)

BERT’s model architecture is based on the Transformer encoder blocks. It has two size of models, base and large. BERT base stacks 12 Transformer’s encoder, while BERT large stacks 24 Transformer’s encoder. There is no difference between them unless the size of the model. Originally in Transformer, they use left-to-right flow in the training. This restricts the model to only learn a word with the previous context only (e.g. in OpenAI Transformer/GPT1). Meanwhile sometimes, to fully grasp a meaning of the word, we need to know the right context as well.

Let’s say I want to predict the blank word in the sentence “I went to the ___ to borrow a book”. The correct answer is “library”. To correctly predict the blank word, I need to see not only the left context “I went to the” but also the right context “to borrow a book”. Without the right context, we might predict the blank word with store, class, etc.

The restriction also applies to the word embedding models. Here, traditional Transformer left-to-right flow would fail learning the entire sequence rich meaning. That’s why BERT propose a new way learning the sentence by enabling two way flow, left-to-right as well as right-to-left (bidirectional). This is why BERT is called Bidirectional Encoder Representations from Transformers.

However, to train the Transformer’s encode such that it applies the bidirectional flow learning is not easy. The paper stated

The problem is bidirectional conditioning would allow each word to indirectly “see itself”, and the model could trivially predict the target word in a multi-layered context.

It will happen because we give the entire input sequence information to the Transformer’s encoder blocks. Transformer allows every word to understand every other words in the sequence. That’s why, bidirectional conditioning in training will “indirectly” leaked the information of the current word that the model want to predict during the training. This is not a good thing to happen, it is like the model is “cheating” and it won’t learn the representations well.

To solve this issue, BERT introduce a training technique to prevent the “indirect leak” happen in bidirectional conditioning. It is Masked Language Model.

Masked Language Model (Cloze task)

It simply mask some percentage of the input tokens randomly. Then the training goal is to predict this masked token. BERT use [MASK] to indicate it is a masked token. With this technique, the model (Transformer’s encoder) doesn’t have any information for this masked token. This way, the bidirectional conditioning will work.

By applying this masked bidirectional conditioning, BERT can attain the deep understanding of the sequences. This learned representations is useful for the downstream tasks.

However, many downstream tasks such as Question-answering and Natural Language Inference are based on the relationship between two sentences. Which is currently our BERT haven’t learn it yet. In order to learn the relationship between two sentences, BERT also introduce another training technique, Next Sentence Prediction.

Next Sentence Prediction

Let’s say we have two sentences, sentence A and B. How does BERT learn the relationship between A and B? They combine the input, sentence A followed by sentence B and create a corresponding label IsNext if B is the actual next sentence that follows a. Otherwise the label is NotNext . This way allows BERT to learn the relationship between sentences that will be useful for downstream tasks.

Yeah just like that, by applying Masked Language Model and Next Sence Prediction training techniques, BERT successfully produced a rich representations and achieve state-of-the-art results in the downstream tasks.

You can use BERT as feature extractor only (just like ELMo) or fine-tune the BERT for specific downstream task by adding a head (linear layer and softmax for that specific task).

That’s all for now. Hopefully this will help you to understand more about BERT. Thank you for reading :)

References

[embed]BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations…arxiv.org

[embed]The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) Discussions: Hacker News (98 points, 19 comments), Reddit r/MachineLearning (164 points, 20 comments) Translations…jalammar.github.io