Hyperparameters in the prompting process
Temperature and Top P are often explained as “randomness”. Here is what they actually do to the words the model chooses between, and why Temperature 0 doesn’t always give the best answers.
Curious about how to balance creativity and accuracy in AI-generated content?
Even small adjustments to an AI model’s settings can have a big impact on the generated text. Finding the right balance between creativity and precision takes a deep understanding of how different settings affect the model’s behavior. This article gives insight into how these adjustments work, and explores how they can improve the model’s ability to solve problems and give precise answers to questions.
Understanding hyperparameters such as Temperature and Top P is essential to understanding the prompting process. These hyperparameters (that is, parameters that control how the prompt is processed by the “system”) govern how autocompletion works.
You will often hear Temperature and Top P described as a control for “randomness”.
There is a saying that “a simple explanation is better than a complicated truth”. But when it comes to understanding how language models work, the simple explanations are often so inaccurate that they do more harm than good. It’s a bit like explaining how a car works by saying “you turn the car on, and then it drives”. Technically true, but it doesn’t help you understand what is actually going on under the hood.
- Temperature usually ranges from 0 to around 2 (no hard limit, but practical constraints). The default is 1 in the API and playground, and around 0.65–0.8 in ChatGPT.
- Top P ranges from 0 to 1, with 1 as the default.
Let’s take a closer look. I’m going to simplify a little, but the goal is to give you an understanding that is both precise and useful, not just easy to remember.
Imagine the writer Nora writing sentences. She has a big bag full of words, and each word has a certain probability of fitting in based on what she has already written.
When Nora picks the next word, she adjusts how adventurous she feels by setting a “Temperature”. At a low Temperature she is careful and picks the words that fit best and are most likely. At a high Temperature she is bolder and also considers words that may not be as obvious, but that could give the sentence an unexpected twist.
She also decides how many words she wants to consider by setting a “Top P” value. She sorts the words by how well they fit, from most to least likely. Then she picks out the words that together make up a certain percentage of the total probability. This becomes her pool to choose from. (The number of words included when adjusting Top P is dynamic and depends directly on the probabilities of the available words.)
Let’s take a practical example
If Nora writes:
“No, Johnny, Steve didn’t jump off a bridge. That was just a figure of speech. I’m glad you’re ___”
With a Temperature of 1 and a Top P of 1, the possible next words and their probabilities look like this:
concerned ###################################### 46.317 %
interested ============================== 37.037 %
okay ===== 6.546 %
following == 3.000 %
excited == 2.000 %
listening = 1.500 %
here = 1.200 %
safe = 1.000 %
alright = 0.800 %
fine = 0.600 %With a higher Temperature and a higher Top P she opens up more possibilities and may end up with words like “excited” or “following”, giving the sentence a humorous or surprising ending.
For example: “No, Johnny, Steve didn’t jump off a bridge. That was just a figure of speech. I’m glad you’re excited.” Not quite fitting, but it could be funny.
So Temperature affects how conservative or creative Nora is in her choice of words, while Top P decides how many of the most likely words she considers. Together these settings help her find the perfect balance between predictability and originality in the sentences she creates.
Let’s see what happens if we lower the Temperature to 0.6 (Top P is still unchanged at 1). Then we get the following word probabilities:
concerned ###################################### 57.074 %
interested ========================== 39.318 %
okay = 2.189 %
following = 0.596 %
excited = 0.303 %
listening = 0.188 %
here = 0.129 %
safe = 0.096 %
alright = 0.066 %
fine 0.041 %- Likely words become even more likely
- Words that already have a high probability become even more likely when the Temperature is lowered.
- Less likely words become even less likely
- Words with a lower probability have their probabilities reduced further.
- No words are excluded here
- Even though the probabilities are adjusted, no word is removed from consideration entirely; only the probability of it being picked changes.
At an extremely low Temperature, such as 0.2, some words can end up with a probability of zero because of numerical underflow, which makes them unavailable for selection during sampling. Even though they are technically not removed from the “pool”.
So we have seen an example of the effect of lowering the Temperature, but what happens if we turn it back up to 1 and then lower Top P to 0.6?
Then it looks like this:
concerned ###################################### 46.317 % Σ 46.317 %
interested ============================== 37.037 % Σ 83.354 %
okay ===== 6.546 % Σ 89.900 %Now there are only three available words, not the long list we had earlier with a wide spread of probabilities.
In principle it adds up the probabilities from the top of the list, and keeps adding words until the cumulative probability reaches or exceeds the Top P threshold. With a Top P of 0.6 that means at least 60%.
If we look at the example above:
- After “concerned”: cumulative probability = 46.317%
- After “interested”: cumulative probability = 83.354% (above 60%)
On some occasions (depending on the language model), as seen above, an extra token can be included.
Possible reasons
- Implementation details in Top P sampling: The model includes the word that pushes the cumulative probability past the Top P threshold.
- Numerical rounding errors: Small numerical differences can cause extra words to be included.
- Minimum number of tokens: The model may be designed to always include a minimum number of words to ensure variation.
- Adjustments for better text flow: The model includes more words to make sure the text makes sense and flows naturally.
- The model’s internal probability distribution: The model adjusts the probabilities based on context, which affects which words are included.
Let’s turn Top P back to 1 and set the Temperature to 0
concerned ###################################### 100.000 %Okay, here you only see one word. Technically all the same words you saw earlier are here too, but with a probability of zero because of numerical underflow, which makes them unavailable for selection during sampling.
The writer Nora is completely restricted when she reaches into the bag for words. She can ONLY grab the very best word.
This is why we say the model is “deterministic” when the Temperature is set to zero. The output is fully determined by the content of the prompt you give it. This is super useful for testing prompts. You only need to run a prompt once at Temperature 0 to know how it will consistently behave under those conditions.
BUT. Many will say you should test primarily at 0. That the randomness at higher temperatures can make evaluating prompts more difficult. That you can’t judge a prompt’s effectiveness accurately with too much random variation. The “random” nature of the results can be challenging for those who prefer predictability.
Buut… they’re wrong.
Consider this example
“If five machines take five minutes to make five units, how long will it take a hundred machines to make a hundred units?”
“It will take a hundred minutes for a hundred machines to make a hundred units.”
“It will take five minutes for a hundred machines to make a hundred units.”
Why does the model give the wrong answer at a low Temperature?
This comes down to statistical bias:
- At a low Temperature the model focuses on the most likely answers based on patterns it has learned from the training data.
- Many people intuitively make the mistake of assuming that if five machines take five minutes for five units, a hundred machines will take a hundred minutes for a hundred units.
- The model reproduces this common fallacy because it is the most statistically likely answer.
Why does the model give the right answer at a higher Temperature?
This comes down to increased variation:
- At a higher Temperature the probability distribution for the next tokens is adjusted so that less likely alternatives have a greater chance of being picked.
- This lets the model generate answers that deviate from the most likely patterns it has learned from the training data, and so arrive at the correct answer.
Too many options can of course also have a downside and the opposite effect. That is why it’s important to understand how hyperparameters, especially Temperature and Top P, affect a language model’s output. This is particularly relevant in contexts where precision is critical, or when you want to balance creativity and fact-based answers.
Practical tips
Top P has limited effect at Temperatures below roughly 0.8–1.0. At extreme values (below about 0.1) Top P can affect the output even at lower Temperatures, but in general the effect is minimal. Temperature, on the other hand, always has a noticeable impact.
To get more predictable and likely results, lower the Temperature. A lower Temperature results in more conservative and fact-oriented answers. Values below about 0.35 give very conservative answers, while values below 0.1 produce extremely deterministic results. At Temperature 0 the model will consistently pick the most likely word.
- 0
- Consistent and predictable. Reliable, but can come across as boring. Very fact-oriented.
- 0.55–0.8
- Similar to ChatGPT’s balanced approach.
- 1
- Default setting in the Playground. Often preferred for a good balance between creativity and coherence.
- 1.2
- Extremely creative. It’s a good idea to start lowering Top P at this value.
- 1.6
- Can result in incoherent or overly fanciful text, unless Top P is reduced to around 0.18 or lower.
- Higher
- Risk of meaningless or incomprehensible output.
- 0
- Picks the most likely word almost exclusively.
- < 0.2
- Very selective. Suited to a high Temperature, to balance creativity with coherence.
- 1
- No restriction on word choice, giving maximum variation.
High Temperature + low Top P = A high degree of randomness in word choice, combined with a pull towards coherence and meaning. Ideal for lyrics, poetry, wordplay and creative writing.
There are other relevant parameters
- Top-K
- Beam search
- and more …
These parameters can affect text generation in different ways, often by balancing diversity and quality in the output. They aren’t always available in every interface, and their effectiveness can vary depending on the task.
But we’ll save those for another time!
First published . Also available in Norwegian.
← All articles