The model produces scores (logits). Softmax turns them into shares of the total, and a sampling rule picks one token. Temperature divides the scores before softmax. At temperature 0 the rule is greedy: always the top token (no division by zero is performed).
Toy: "Tomorrow the weather will be …"
Toy calculation· 6 made-up scoresScores (logits): fixed by the model
- sunny3.1
- cloudy2.4
- rainy1.9
- windy1.2
- snowy0.3
- purple-1.5
Shares at T = 1.00, and draws
- sunny49.5%29/50
- cloudy24.6%11/50
- rainy14.9%6/50
- windy7.4%0/50
- snowy3.0%4/50
- purple0.5%0/50
Entropy 1.302 nats. Shares sum to 100.0%. Same seed and settings give the same draws.
Moderate temperature: several tokens are drawn. The scores themselves never changed.
The formula and the greedy limit
p_i = exp(z_i / T) / Σ_j exp(z_j / T). As T → 0 the largest score takes all the probability, so T = 0 is implemented directly as “choose the argmax”, never by dividing by zero. Top-k and top-p further restrict which tokens can be drawn.