Natural Language Processing

Byte-Sized Content

Photo by Kurt Cotoaga on Unsplash

In the previous article, we saw what language models are and how they work. We defined tokens as words or sub-words or even characters.

For example, consider the sentence:

I saw a cub roaring as loud as a huge lion would.

A language model may tokenize it into units like:

["I", "saw", "a", "cub", "roaring", "as", "loud", "as", "a", "huge", "lion"]

These tokens are then converted into numerical representations that the machine can process. Humans can effortlessly understand meaning, tone, and context in language, even though conversations may sometimes feel confusing. Machines, however, do not naturally understand language at all. What machines are exceptionally good at is processing numbers.

Therefore, natural language must first be converted into a numerical form that machines can interpret. The field that focuses on enabling computers to process and understand human language is called Natural Language Processing (NLP).

A huge part of NLP is tokenization.

However, it is not always done word-by-word. Modern systems often use sub-word tokenization, where frequently repeating patterns are compressed into reusable units.

Consider the string:

*abcabdabfaaabaaabi*

We notice that "ab" appears many times. So we can define a new token:

k → ab

The sequence now becomes:

*kckdkfaakaaki*

Next, we observe that "aa" is also frequent. Define:

c → aa

The sequence becomes:

*kckdkfckcki*

Now "kc" appears repeatedly, so we define:

D → kc

The sequence finally becomes:

*DkdkfcDi*

In this way, an 18-character sequence can be represented using fewer and more meaningful units. This is the core idea behind modern tokenization techniques such as Byte Pair Encoding (BPE), which many LLMs use internally.

Before jumping into Large Language Models (LLMs), it is important to understand how machines process language in general.

An NLP system typically follows a pipeline with steps like this:

1. Data Acquisition
2. Data Cleaning
3. Data Preprocessing
4. Feature Engineering
5. Model Building
6. Evaluation
7. Deployment

LLMs extend this pipeline further by using massive datasets, advanced tokenization strategies, and neural networks called transformers to learn patterns in language at scale.

We will learn more about the NLP pipeline and LLMs in the next article.