What are Transformers?

You all might have heard of this word transformers going around ever since GPT came up. GPT stands for Generative Pre-trained Transformer. That is another story which I will write about.

So what are transformers? Transformers are basically a neural network architecture which takes in an input sequence and tries to produce an output based on that input. For example, I ask “ Who is the first man on the moon” it replies “Neil Armstrong is the first man on the moon”. That is transformer.

Architecture

Transformer Architecture

So this is the generalized transformer architecture. It is not very accurate there is more processing happening inside these but I made a rough sketch of how it works.

1. Input

This is the input that we give to the model. Like I previously mentioned “Who is the first man on the moon?”. A tokenizer exists between them to convert the input to tokens.

A token can be anything a word, a character, a full stop, a comma basically anything, the tokenizer decides what a token is.

2. Embeddings

The embeddings layer converts the tokens to embeddings.

Embeddings are high-dimensional numerical vectors. We heard of 2-Dimensional, 3-Dimensional but these embeddings are usually more than that, they have high number of dimensions and represent the input using numbers.

3. Encoder

The encoder will receive these embeddings and convert them into more meaningful embeddings capturing their similarity and relationship so that the model is context aware and not just input aware.

4. Decoder

Decoder will take these embeddings and try to generate an output sequentially by generating the response token by token.

The generation is handled by something called Self-attention. There are more types to it but we will cover the basic concept in this article.

5. Output

This is the final response that is generated by the decoder and is displayed to the user.

Self-Attention

This is the interesting part. Remember Self-attention from the decoder part. This is the core part that actually generates your output. Any question you ask the decoder produces the output using this concept.

So what is self-attention?

So in simple terms

The selection of the next token depends on all the previous tokens.

So how does it actually decide which word to choose?

For example lets assume:

Input: She was ____ that is why she wasn’t able to come to office.

So it will assign a probability score for each of the words that can be the next word that is chosen to complete that sentence.

Example : sick, dancing, partying, go-karting.

Let’s assume the following

sick — 0.98, dancing — 0.2, partying — 0.01, go-karting — 0.001

So like us humans it will analyse which word is better but it does it through mathematical calculations. It will select a word and see which has the highest similarity to the current sentence.

For example if the decoder selects sick it will find out that it has the highest probability assigned amongst all. Other words have lesser probability so it will not select them. That is all it is, based on the input it will see how much similar the word is to the input and generate the next token.

Selecting one word will also affect the next set of response tokens and their probabilities are again reassigned based on what word is selected. Basically the selected word becomes part of the input and decoder is trying to generate the next token.

Self-attention does not exist only in Decoder but it also exists in Encoder I will try to cover it in the next article if this one gets a good number of reach.