How to Train Your Dragon? Try Training an LLM!

Pretrain your very own LLM model at a super small scale.

Photo by Maximilian Müller on Unsplash

Yes! You read it right. I am gonna teach you the basics and put you on the process to train your very own personal LLM model.

Surely it won't outperform the might beasts like GPT or Gemini. But hey, we learn something at least.

Before jumping into the code and crusts, let's understand what is the recipe for training an LLM.

In correct terminology, we would call it pretraining.

The time before GPT

Good old times. When the world was simple and not this complicated. Every week these days, there is a new hype, a new buzzword that gives us a lot of FOMO.

Let's rewind back and learn how the very first LLM has been trained or in this case pretrained.

To understand the pretraining process, you need to know what the LLM is about and the context.

To make it very simple: Our large language model learns the internet and recites it to us whenever we ask.

In a few technical bits, LLM is about predicting the next token from the distribution of tokens it learned during the pretraining process.

Let's start from ground zero.

The data

Open source data for pretraining LLM

Here, the data is simply the whole internet: usually, the open web, like Wikipedia, Reddit, forums, and whatever exists on the internet.

The first step is to find a way to download everything on the internet. Some commonly used ways for this are crawling.

You deploy bots that will visit a website, then save it and click on the links on the website to move to another. They kept doing it until they captured everything we wanted.

If you are too lazy to crawl, you can use the FineWeb open-source website available on HuggingFace. It is a large dataset with 15 trillion tokens taking up 44 TB of disk space. Not too shabby, the datasets these days are wayyyy bigger than this (estimates put it in the range of petabytes!).

For our use case, let's consider a small dataset: our very own and personal 11-word universe.

Our 11-word universe

Here is our entire “internet”. Let’s assume the complete training corpus for our “baby LLM”:

“the robot likes code the robot writes code everyday human rules”

There are 11 words in total, but if we count the unique words, our vocabulary size is just 7 words:

['the', 'robot', 'likes', 'code', 'writes', 'everyday', 'human', 'rules']

Now, we map each word to a number:

• the → 0, robot → 1, likes → 2, code → 3, writes → 4, everyday → 5, human → 6, rules → 7

Our entire training dataset is now just an array of integers:

[0, 1, 2, 3, 0, 1, 4, 3, 5, 6, 7]

We do this as neural networks can only train on numerical data.

Slide the window

In a real corpus, the vocabulary length is super looong. So we “learn” from this long corpus using a window. That is, we randomly select a subset of the corpus of a specific length for training.

You can call this as “context length” or “window size”. For our baby GPT, let’s set the window size to 3.

This means our model can only look at a maximum of 3 words at a time. The goal is to guess the next one using this context. We slide this window across our data to create training pairs:

• Input: What the model sees (X): has 3 words
• Target: What it must predict (Y): will predict one word

[the, robot, likes]→ code

[robot, likes, code]→ the

[likes, code, the]→ robot

[code, the, robot]→ writes

This prepares our training dataset. Our input is 3 words long, and the prediction is the next word.

A sneak peek under the hood

When the model looks at an input during the training process, it converts the tokens into numbers.

After the input passes through the network, the model outputs raw values for each word in our vocabulary. We call these scores logits.

Example:

Vocabulary:  [ the,  robot,  likes,  code,  writes,  everyday,  human,  rules ]
Logits:      [ -1.2,   0.5,    -3.1,   4.2,    1.1,      -0.5,    0.2,   -2.0 ]

We can’t interpret logits directly, so we use a softmax function that converts these arbitrary scores into actual probabilities (that add up to 100%)

After running the logits into Softmax, we get a nice probability distribution:

• code: 72%
• writes: 15%
• robot: 8%
• and the rest: 5%

[the, robot, likes]→ code

For this case in the first example, the model understands that it code comes after, [the, robot, likes] so it gives it a higher probability.

The goal of the LLM model now is to predict the next token using the probability distribution it learned during the training process.

This “pretraining” process leaves us with something called a “Base model”.

A base model just remembers the internet (the data we trained it on) as a memory. Whenever you ask it to retrieve something, it predicts the sequence it remembered.

Inference: make it talk

We have the base model in our hands. This is not much useful if it cannot answer our questions like ChatGPT or Gemini.

The process of making the model generate new text is called inference. This way, we can check if the model performs as expected.

You might have remembered how the GPT got better over the years. In the very beginning, it sounded a lot robotic. Currently, the text made by GPT is super convincing. I sometimes purposefully dumb down my sentences to make sure they sound human (sigh….to err is human).

To watch the inference in its glory, we can give the model a prompt, say “the robot”.

Step 1:

• The model takes [the, robot] (length 2, which fits perfectly in our window size of 3).
• It runs the math and calculates that likes has the highest probability.
• It samples likes and adds it at the end of the text.
• It predicts: “the robot …”
• As: “the robot likes”

Step 2:

• Next, the model takes [the, robot, likes] (length 3, which still fits our window).
• It processes it and predicts that code is next.
• It adds it again to the end of text.
• It predicts: “the robot likes …”
• As: “the robot likes code”

Step 3 (window shift):

• Now the text is “the robot likes code” (4 words).
• Note that our window size is strictly 3, so the model has to drop the oldest word. It drops and forgets the.
• It only looks at the window with these words: [robot, likes, code].
• Based on this history, it predicts the next word, maybe everyday.
• It predicts: “the robot likes code …”
• As: “the robot likes code everyday”

This is the simple inference loop, which looks at the window, predicts the next token, appends it, shifts the window to the right, and does everything again.

This is exactly how ChatGPT used to answer your questions in the very beginning (now the animations have changed, and the models have become way faster to notice this).

Conclusion

There you have it. Now you know the basics of the origin of LLMs. Reality is more complicated than how I explained, but it is a good start. LLMs have gotten better over the years with a lot of new concepts and tweaks by some of the brightest minds in the world.

In my upcoming articles, I will explain the evolution of LLMs (of course, in very simple words).

Thanks for reading, and cheers! Follow for more.

Want to Connect? Reach me at LinkedIn, X, GitHub, or my Website!