How LLM Works: Prompt Engineering

Large Language Models don’t read text the way humans do. They don’t understand meaning, intention, or context in the traditional sense. Instead, they operate as probability engines breaking language into tokens, converting them into numbers, and predicting what comes next.

When you type a sentence, an LLM first splits it into **tokens*, the smallest units it can process. These don’t always match words. A short word like “dog”* may be one token, while longer words are split into pieces. For example, “ButterFly” might become “Butter” + “Fly.”

Each token is then converted into a numeric ID. The model works entirely with these numbers, predicting which token should come next. Different models tokenize differently — GPT uses **Byte-Pair Encoding, while BERT uses [WordPiece](https://huggingface.co/learn/llm-course/chapter6/6)**, meaning the same sentence can be processed in different ways depending on the architecture.

This token-level processing is the foundation of how LLMs generate responses.

Why the Same Prompt Produces Different Answers

Unlike traditional programming, LLMs are non-deterministic. Ask the same question twice and you may get different responses. This happens because the model samples from probability distributions when selecting the next token rather than always choosing the single most likely option.

The Hidden Controls That Shape Every Response

LLMs expose parameters that influence how token selection works. These controls don’t change what the model knows; they change how it expresses knowledge.

1. Temperature — This is the Randomness Dial that controls how adventurous the model is when choosing tokens.

• 0.0–0.3: deterministic, factual, precise
• 0.7–1.0: creative and varied
• 1.2–1.5: unstable and unpredictable
• 1.5+: incoherent output

2. Max Tokens: This is the response length cap. Roughly, one token equals about 0.75 words. Setting this too low cuts responses mid-sentence; setting it too high allows unnecessary verbosity.

Typical usage:

• 50–150 tokens → short summaries
• 500–1000 tokens → detailed explanations
• 2000+ tokens → long articles

This parameter acts as a ceiling, not a target.

3. Top-P — Top-p (nucleus sampling) restricts the model to the most likely tokens that together form a probability threshold. For example, top-p = 0.9 means only tokens within the top 90% probability mass are considered.

Lower values produce focused responses. Higher values increase diversity.

It’s generally recommended to adjust temperature OR top-p, not both, to avoid unpredictable behaviour.

4. Context Window: Every LLM has a **context window**, which defines how much text it can remember at once. Modern LLMs range from 8k tokens (older GPT-3.5) to 1M+ tokens (Gemini 1.5 Pro). Claude 3.5 Sonnet handles 200k tokens, enough for 150,000 words or 500 pages of text. Once this limit is exceeded, earlier content is silently truncated.

Mastering these controls transforms prompting from guesswork into engineering.

Start prompting correctly from your next prompt; set Temperature OR top-p AND Max-tokens

Thank you.

Funmilola Fagbola (PhD)