AI
How does temperature impact next token prediction in LLMs?
Ankur Manikandan · Follow Published in Towards Data Science · 4 min read · 4 hours ago — TLDR1. At a temperature of 1, the probability values are the same as those derived from the standard softmax function.2. Raising the temperature inflates the probabilities of the less likely tokens, thereby broadening the range of potential candidates (or diversity) for the model’s next token prediction.3. Lowering the temperature, on the other hand, makes the probability of
Ankur Manikandan
May 6, 2024