LLM Part 8 — Token Sampling

1. Where We Left Off

The last article ended with a family of shapes. Same model, same prompt, same logits, one distribution per temperature. Sharper at low T, flatter at high T, always the same ranking. For “My favorite season is” the version at T = 1 kept spring on top at 23.24%, summer at 18.38%, winter and autumn behind and a long tail spreading the rest across the vocabulary.

Every article so far has stopped at that edge. The model lays out its candidates and sizes them. It never commits to one. Back in the softmax article we called the pick a separate step, later in the series. This is that step.

It gets an article of its own because it happens outside the model. Nothing in the 28 blocks of our model, the readout or the softmax selects anything. They hand over a distribution and stop. Turning that distribution into one written token is a rule applied on top and there is more than one rule to choose from. This article walks through the ones you meet in practice: taking the top token every time, drawing at random in proportion to the shares and trimming the field before the draw. At the end the token that comes out goes back in as input and the whole pipeline runs again. That move closes the loop the series opened seven articles ago.

2. Greedy Decoding

The simplest rule reads the whole distribution and keeps one fact from it: which token sits on top. Write that token down and move on. This is greedy decoding. The ranking alone decides, with no draw involved. For “My favorite season is” the ranking puts spring first at 23.24%, so greedy picks spring. The margin over summer plays no role. First place is all it checks.

We ran it five times, letting the model continue for twelve tokens after the prompt. Five runs, five identical continuations:

My favorite season is spring. It's the season of new life. It's

Same words, same punctuation, same cut at the twelfth token. Nothing in the procedure can vary. Once the prompt is fixed the distribution is fixed. Greedy reads only its top entry and the top entry never moves, so the first token is settled before anything runs. The same holds at every step after it. Each new token faces its own distribution and each time the leader wins. The whole continuation is decided in advance.

This ties off a thread from the last article. Pushing T toward zero drove the shares toward a spike, 100% on the token with the highest logit, a limit we could describe without ever reaching it. Greedy decoding is that limit put to work. It skips the shares and takes the leader directly, the same pick the spike was pointing at. This is also how T = 0 exists as a setting in practice. The division itself is undefined, yet APIs accept a temperature of zero and treat it as an instruction to switch to greedy decoding.

In principle that makes the output deterministic, the same text on every call.

In practice tiny implementation details can still nudge a token here and there, so the honest label is nearly deterministic.

The cost shows up over longer stretches. A rule that always takes the local leader tends to repeat itself on long generations, a documented weakness of the method. Our twelve tokens already hint at it. The continuation opens with It’s the season of new life and then a second It’s lines up, the same sentence shape starting over.

3. Sampling from the Distribution

The second rule uses everything the distribution says. It treats the shares as tickets in a lottery. Spring holds 23.24% of the tickets, summer 18.38%, winter 13.52% and so on down the whole vocabulary, every token with a nonzero share included. One ticket gets drawn and that token is the pick. The formal name is multinomial sampling and it sits at the opposite end from greedy: the ranking still matters, since more tickets mean better odds, yet no position is ever safe and none is ever hopeless.

The lottery is easy to watch: we drew the next token 1,000 times from the same distribution and counted the results. Here are the ten tokens drawn most often, next to the share each one held in the distribution

The counts track the shares. A share of 23.24% means spring should surface in roughly 23 draws out of 100 and over 1,000 draws it surfaced in 226. The match is close without being exact. Fall at 3.88% collected 49 draws while the underscore token at 4.11% collected 37. Chance keeps that much slack over 1,000 draws. The underscore tokens from the softmax article are in the list too, since every entry with a share gets its proportional turn, words and non-words alike.

A single draw picks a single token. To generate text the draw repeats at every step, each time on the distribution the new context produces. We ran five full continuations at T = 1, twelve tokens each, changing only the seed of the random draw.

seed 1:  autumn. Autumn is very sweet with warm sun and fresh water
seed 2:  winter. It's beautiful. You can enjoy the snow,
seed 3:  autumn.\nA. The season I love the most is autumn
seed 4:  ____.____\nwinter\nsummer\nautumn\n答案
seed 5:  summer. It is warm and colourful. I often stay at

Five runs, four different openings (the \n marks are line breaks). Autumn came out twice, winter once, summer once and in one run the draw landed on an underscore token, after which the continuation took the shape of a fill-in form, seasons listed on separate lines. Spring, the leader, appears in none of the five. Nothing went wrong there. A token holding less than a quarter of the tickets loses the draw more often than it wins it and five runs are few. Over many runs spring would lead the count, the way it did over the 1,000 single draws above.

This is also where the knob from the last article earns its keep. The temperature reshapes the shares before the softmax and sampling is the step that turns shares into picks, so turning the knob changes the lottery itself. At T = 0.3 the distribution concentrates on spring at 58.45%. Three sampled runs at that temperature:.

seed 100:  spring. It's a season of new beginnings and renewal.
seed 200:  spring. It is a season of new beginnings. It is
seed 300:  spring. It is the season of new beginnings. It is

All three open with spring and the sentences come out nearly alike, a word swapped here and there. A sharp lottery behaves almost like greedy while leaving every other ticket a small live chance.

4. Top-k and Top-p

Push the knob the other way and the lottery shows its weak side. The last article measured what a high temperature does to the shares. At T = 2 the five seasons keep 8.49% of the distribution and the rest spreads across the long tail, thousands of tokens holding a sliver each. Slivers add up. Draw from that lottery and the winning ticket usually comes from the tail. Three sampled runs at T = 2.

seed 100:  unquestionoptimized, combines palmיקה standard.widgets : Generate葺Marvel
seed 200:  chess developing acute terms just 😉estingMajor Politics_Result").']")
seed 300:  Wed−ervision하겠습니다\n\n\n\n\n\n\n\n\nIKNB 和Windows期间少许

Tokens pulled from every corner of the vocabulary, fragments, symbols and scripts side by side. Each of these tokens was a long shot on its own. Together the long shots owned the lottery.

The remedy is to trim the field before the draw. Two cuts are standard and they share the same moves: sort the tokens by share, keep the leaders, delete everyone else, rescale the survivors’ shares so they sum to one again and draw from what is left. The two differ only in where the cut lands.

Top-k keeps a fixed count. Set k = 5 and the five highest tokens survive, whatever they are. Top-p keeps a fixed amount of probability instead. Set p = 0.9 and the cut walks down the ranking, adding tokens until their combined share reaches 90%, then stops. The count of survivors comes out of the distribution itself.

Note: Top-p also goes by nucleus sampling, the name it carried in the paper that introduced it.

We ran both cuts, twice each, on our T = 1 distribution. Here is what survives and what each survivor holds after the rescale.

The rescale explains the climbing numbers. The five survivors of k = 5 held 70.00% of the original distribution between them. Scale that up to a full 100% and spring goes from 23.24% to 33.20%. The deleted mass gets handed to the survivors in proportion, so every cut makes the leaders stronger as well as safer.

The k = 5 column carries a lesson in its missing row. Fall is out and the fifth survivor is one of the underscore tokens, which at 4.11% outranked fall’s 3.88%. A fixed count keeps positions and knows nothing about meaning. Whatever sits in the top five gets a ticket, underscores included.

The two p columns show what adapting to the distribution means. At p = 0.5 the cut stops after three tokens, since spring, summer and winter already cover 55.14% of the mass. At p = 0.9 it takes fifteen. On a sharper distribution the same p = 0.9 might be covered by two tokens and on a flatter one it might take hundreds. Top-p reads the shape and sizes the field to match, while top-k keeps the same count whatever the shape.

In practice the controls stack. The temperature reshapes the shares, the cut trims the field and the draw runs on the survivors. The wild output at T = 2 above came from a lottery with no cut at all. Add one and the tail stays out of play at any temperature.

The four rules sit side by side below, all drawn on the distribution this article started from. Every panel shows the same bars with their original shares, before any rescale. What changes from panel to panel is which bars the rule leaves in play: one for greedy, all of them for sampling and the leaders for the two cuts.

5. Closing the Loop

The pick is one token. Say the draw landed on winter. That token joins the input just as it is, already in number form. The sequence becomes “My favorite season is winter” and the model needs a next token. It gets one the only way it knows: the whole pipeline runs again. Vectors, 28 blocks, readout, softmax, one more distribution, one more pick. Out comes another token. It joins the line too.

That’s the whole trick. One pass of the loop produces one token. A full paragraph is just the loop repeated, one word at a time. This way of generating text has a name, autoregressive: the model reads what it has written so far to decide what comes next.

The loop is also why runs drift apart. Our three runs at T = 0.3 all started with spring. Then one drew It’s, another It is and the sentences went separate ways from there. Every pick becomes part of the context, so one different word early on changes everything that follows.

The loop stops in one of two ways. The vocabulary includes a special token that means the text is over: when the pick lands on it, generation ends. The other brake is a plain length cap: our labs stopped every run at twelve new tokens, which is why so many continuations cut off mid-sentence.

With that the story is complete. Text goes in, gets chopped into tokens, becomes vectors, climbs the blocks and comes out as a distribution. A rule picks one token to write down. The token rejoins the input and everything runs again. Every word our model prints is one lap of that loop.

6. Key Takeaways

• The pick is a step of its own. The model hands over a probability distribution and a decoding rule turns it into one written token.
• Greedy decoding takes the top token every time, so the same prompt gives the same continuation. In practice APIs treat T = 0 as greedy.
• Sampling treats the shares as tickets in a lottery. Over many draws the frequencies settle around the probabilities.
• Top-k and top-p trim the field before the draw. Top-k keeps a fixed count while top-p keeps a fixed amount of probability and adapts to the shape.
• Generation is a loop: the chosen token joins the input and the pipeline runs again, one token per pass. Everything a language model writes comes out this way.