Temperature, top_p and top_k: Temperature zero does not always make an LLM deterministic

Temperature is an LLM setting that is often described vaguely as: “How creative the LLM should be”, and when you dig a little bit: “How likely the LLM selects the most probable word for it’s output”. Knowing that the network outputs a probability distribution, naively this sounds as if a higher temperature more often selects a less likely output and a lower temperature more often selects a more likely output. Temperature zero should then always select the most likely output, making it deterministic. Often, that is the advice you are given when you want repeatable output, i.e. set temperature to 0.

However, after some experimentation with LLMs, you will have noticed that temperature=0 is not always deterministic so something is fishy.

Temperature

First a little bit of back-story: Temperature was not initially introduced to “make an LLM more creative”. There are several reasons that LLMs have temperature, but the main reason is practical (AI is very much an experimental science, so practical solutions abound). Sometimes an LLM will predict a sequence of words and then predict a word within that sequence. At times, this locks the LLM into an infinite loop where the sequence of words keeps repeating. Temperature is a facile way to break such loops by randomly choosing something other than the very most probable next token.

Now back to determinism: It turns out, the temperature setting isn’t simply a linear increase in the probability of selecting a less predicted word. Instead, it is a parametre of the softmax function. To unpack that, I need to explain softmax and it’s role in an LLM:

The last weight matrix of an LLM is a linear transform (no activation function) from the output of the last transformer block into vectors, each vector of the same dimensionality as the token dictionary, and one such vector for each token in the context. The outputs from this layer can vary between (-inf, +inf) (note that the range is non-inclusive) and are interpreted as how likely the LLM considers any token in the dictionary to be for that position in the context. To make a probability distribution that sums to 1 out of the raw scores for one output position, the scores in a token vector are interpreted as the log-likelihood of the probability for every token in the dictionary (usually called logits).

The logits are converted into a probability distribution through the softmax function. Softmax is similar to the simple function of relative proportion:

Relative Proportion

however softmax exponentiates the values first. That makes softmax emphasize the difference between the more probable values and the less probable values, while still ensuring they all sum to 1. In a sense it exaggerates the max value while still retaining a trace of all the other values to allow differentiation for back-propagation.

The softmax function

A graph of the softmax function looks a bit like the sigmoid function, but it’s range is between the highest and lowest raw scores and the values are between [0,1]. When training, the loss function is usually directly applied to the result of softmax. This trains-in the interpretation of logits as log-likelihood probabilities, because softmax is the exponential of each logit distributed over all exponentials of the logits.

During inference, a token is finally selected for output according to the probability distribution from softmax.

Temperature is an additional parametre that can be added to the softmax function. It determines how “sharp” the sigmoid-like curve will be; it can emphasise or de-emphasise the relation between the top values and the other values. Temperature is not trained into an LLM, it is only used during inference. The softmax function including temperature T looks like this:

The softmax function with Temperature

When T is equal to one, this is just the normal softmax. However, if T is less than one, this will increase the exponents, making the slope of softmax more extreme. If T is more than one, the exponents are reduced, softening the slope of softmax. So, the closer T gets to zero, the more extreme the slope of softmax gets, however T can never actually reach zero.

So what does a temperature of zero mean then? That depends on the implementation. There are two common solutions: Most commonly, temperature is clamped to a minimum value ε when zero is requested, inherently making the LLM slightly random. Sometimes there is a separate code-path that directly selects the top logit when temperature is zero, though it makes less sense mathematically.

There are other inference parameters available that can be used to control the randomness of an LLM. Usually there are the top_k and top_p parameters, however they often require deep digging into the documentation to explain.

top_k and top_p

top_k is taking the top_k most probable results from softmax and sampling the output from them according to the strength of their predictions. top_k always samples from the same number of words, but it has the disadvantage that it is hard to calibrate, given that you don’t want the LLM to output words that it doesn’t believe are right at all. If only one or two words are deemed probable by the LLM, top_k will still sample top_k words and select from them. To be fully deterministic, it should be enough to set top_k to 1.

top_p is slightly more subtle: It selects the smallest set of words whose total probability exceeds top_p and samples from them. Commonly, top_p is implemented by sorting the potential words according to probability, then iteratively adding the most likely words until top_p has been exceeded. top_p avoids the need for calibration that top_k has. To be fully deterministic, it should be enough to set top_p to 0, since selection continues until the probability exceeds top_p.

Temperature is applied to softmax first, so there is an interaction between temperature, top_k and top_p. With temperature, you emphasise or deemphasise the differences between higher and lower probabilities for the tokens. After that level has been chosen, top_k selects the k most probable tokens and selects among them according to probability distribution among those k tokens. So if temperature has changed those ratios, the ratios of tokens that were selected by top_k will still be changed. For top_p, you select enough tokens to have higher probability than p. Then, like with top_k, you sample according the distribution of selected tokens. This means that top_p is impacted twice: if the original distribution has been emphasised by temperature then 1) fewer tokens need to be selected to get past p, and 2) The distribution among the selected tokens remains steeper when finally sampling a token. The contrary if temperature was used to emphasise differences.

Finally, other factors mean that most LLMs will never be perfectly deterministic. Floating point arithmetic is inherently non-deterministic unless you take special numerical stability precautions, like sorting arrays before adding, however such precautions can easily half the performance or worse for your LLM. Because there is a lot of parallelism in any LLM-implementation there is a lot of non-determinism from the parallel implementation as well. Finally, an added complexity is that two logits can be exactly equal, meaning that there isn’t always a single well-defined peak value. With quantization this may be unexpectedly common.