How tokens work in language models
Why can’t ChatGPT count the letters in “strawberry”? The answer lies in tokens: the pieces of text the model actually reads, never the letters themselves.
In this article you will get a basic understanding of tokens: what they are and why they are essential in the context of language models. Quite few people understand what tokens are or how they work. Having a basic grasp of how tokens work can be useful for anyone who works with or is interested in prompting.
Ever wondered why models as “smart” as ChatGPT can’t count how many R’s there are in the word “strawberry”?
Tokens directly affect how much information the model can process at once, known as the context window, and therefore the model’s ability to stay coherent in longer interactions. They also matter a great deal for how accurate the language model’s answers are.
Over the course of this article we will answer several key questions about tokens:
- What exactly is a token, and how do tokens work?
- Why are subword tokens more efficient than character-based or word-based alternatives?
- How do tokens affect a language model’s ability to process and generate language?
- What practical implications do tokens have for you as someone who writes prompts?
- What techniques can be used to optimize tokens?
- Which language should you prompt in?
What is a token?
For language models, tokens are the basic building blocks. They are small pieces of text (also known as chunks) that the models use to “understand” and generate language. Here are some key points about tokens:
- Tokens are small units that represent our language.
- Tokens let language models understand language efficiently.
- Tokens are the smallest unit of text that AI models process.
- Tokens have no fixed length. Some are just 1 character long, others are whole words.
- Tokens can be words, subwords, punctuation marks or special symbols.
- As a rule of thumb, one token equals 3/4 of a word. So 100 tokens is roughly 75 words (English words).
Why subwords (not characters or words) for tokens?
An example:
Show token IDs
[976, 4694, 55742, 2993, 8920, 316, 1072, 135596, 10789, 326, 1641, 46952, 289, 11273, 13, 6311, 92812, 5446, 1365, 1039, 7139, 11, 581, 322, 85679, 113770, 11, 38235, 76131, 11686, 13, 1328, 1729, 5283, 508, 2273, 11, 5495, 20282, 11, 25122, 32871, 306, 17855, 85, 59993, 780, 6456, 326, 18055, 33120, 13]Here you can see that the word “mismanaged” is split into two tokens, and the same goes for “overworked” (each color represents one token). “Readjustment” is split into three tokens.
Subwords are a way of splitting up words to improve the language understanding and efficiency of language models.
Why not characters, or whole words? Let me explain…
Character-based methods
Character-based methods have several drawbacks. First, each individual character carries very little meaning on its own. This means we need longer sequences to convey the same information, which in turn requires more computing power. Character tokens create needlessly long sequences, which makes them inefficient in practice.
Another challenge is that it becomes hard to grasp the meaning of words, their context and how they relate to each other when we only look at single characters. This makes training language models on character tokens more complicated and demanding. In short, character-based approaches are often poorly suited to efficient language processing and understanding.
This simple sentence would look like this with a character-based method: “Subword tokens are efficient.”
["S", "u", "b", "w", "o", "r", "d", " ", "t", "o", "k", "e", "n", "s", " ", "a", "r", "e", " ", "e", "f", "f", "i", "c", "i", "e", "n", "t", "."]A fully word-based approach
A word-based approach brings several challenges. First, it leads to an enormous token vocabulary, since every unique word becomes its own token. This is inefficient in terms of processing resources.
There are also problems with words that fall outside the vocabulary. The model simply has no way of handling rare or new words. On top of that come challenges with misspellings and word variation. It is practically impossible to include every conceivable variation or misspelled version of a word in the model’s vocabulary.
To illustrate, imagine we need separate tokens for words like “rewriting” and “postprocessing”. In reality, we can capture the meaning of these words perfectly using just two tokens for each. This shows how inefficient a purely word-based system can be compared with more flexible solutions.
# Full-word vocabulary:
["cat", "cats", "dog", "dogs", "running", "runner", "runners", ... thousands more variants]
# Subword vocabulary:
["cat", "dog", "run", "ing", "er", "s", ... thousands more variants](As you can see, you would need an extreme number of words in the vocabulary, and it would also be harder to handle misspelled words.)
So what makes subword tokens so good?
Subwords combine the best of both worlds while reducing the limitations.
- Smaller vocabulary. Subwords shrink the vocabulary by using smaller, reusable parts of words.
- Handling unknown words. When a model “sees” a word for the first time, it can break it down into smaller parts. Because the model knows the parts, it understands unknown words.
- Balance between length and information. Optimization methods find the best subword combinations for a language.
In summary
- Character tokens are too small. They carry too little information (they lack semantic meaning).
- Word tokens are too big and create enormous vocabularies.
- Subword tokens combine more efficient token vocabularies with semantic representation.
More examples of where subword tokens shine:
Show token IDs
[258, 639, 41989, 696, 25632, 537, 38245, 67862, 80987, 111126, 2367, 9446, 491, 2136, 5861, 124893, 536, 22725, 101998, 17140, 124868, 5095, 2367, 4694, 37352, 387, 10899, 6502, 56463, 3035, 9446, 54187, 7544, 2860, 1162, 313, 12279]Long words made up of many tokens. Using subwords, we can build long words from smaller tokens, and the language models capture their meaning. So subword tokens are the optimal approach.
Counting words, characters and tokens
Let’s use a real example: we ask ChatGPT to generate a short description of the Vikings. Then we count the number of words, characters and tokens.
Show token IDs
[53, 507, 963, 1504, 458, 1553, 4115, 144200, 95724, 11, 42315, 11, 326, 188224, 1218, 18755, 591, 290, 7844, 62979, 316, 6047, 4888, 81238, 39264, 13, 54336, 1365, 591, 3333, 14540, 51573, 11, 60104, 11, 326, 42009, 11, 1023, 553, 42374, 395, 1043, 12990, 1701, 23847, 11, 1118, 16770, 113716, 5251, 6267, 11, 16363, 11, 326, 290, 7180, 34906, 13, 90351, 25703, 306, 261, 7968, 328, 8173, 11, 3463, 10552, 11, 38070, 11, 34612, 11, 326, 10069, 4409, 7069, 11, 17000, 101483, 290, 15186, 326, 10480, 67057, 328, 290, 21043, 1023, 199698, 483, 13, 15273, 33304, 6664, 29298, 316, 84798, 6340, 11, 290, 11402, 328, 14854, 326, 7865, 11, 326, 290, 68777, 328, 55145, 11836, 63009, 13]The description has 117 tokens (or 89 words). Notice that most words use a single token (the words highlighted in a single color).
For our description we get about 76 words per 100 tokens. That is close to 75 (the rule of thumb).
What about tokens in non-English languages?
Let’s take a closer look! We translate the description into Norwegian and count the words, characters and tokens again.
Show token IDs
[53, 16768, 1900, 972, 120674, 19112, 2541, 1705, 86818, 19177, 7430, 158429, 11, 123126, 93775, 2085, 6323, 10726, 154714, 56976, 2541, 1796, 12114, 613, 6964, 59675, 1305, 1452, 1476, 7086, 1037, 2541, 3453, 2662, 24491, 50275, 1452, 1476, 650, 3944, 411, 17458, 59576, 264, 13, 3983, 991, 638, 3148, 2807, 6964, 155515, 59079, 11, 55719, 2085, 64714, 11, 1111, 334, 163436, 395, 44883, 87936, 599, 411, 6212, 38258, 11, 1796, 101740, 36784, 14735, 322, 7466, 1072, 13965, 19474, 11, 16363, 2085, 20341, 12, 2243, 8068, 58068, 13, 98293, 1900, 3115, 288, 8260, 411, 3055, 575, 469, 179715, 154961, 11, 178001, 805, 76807, 11, 22823, 41026, 1401, 11, 26134, 292, 1401, 2085, 49764, 145982, 85620, 301, 18173, 11, 49921, 1796, 150794, 12099, 334, 90660, 22088, 2085, 5086, 19177, 26518, 21100, 1900, 575, 7967, 1900, 334, 3535, 5172, 1347, 1475, 91392, 15570, 13, 1754, 1066, 48124, 169082, 259, 16023, 17764, 3453, 2674, 49138, 102300, 11, 1014, 1291, 1401, 1452, 117302, 2085, 2750, 259, 11, 2085, 1625, 5237, 1452, 50524, 5956, 409, 1272, 27603, 19177, 158502, 5770, 13]See the difference? Now most words use several tokens (the words highlighted in many colors).
For our description we get about 53 words per 100 tokens, far fewer words per 100 tokens than in English!
Here are the main reasons
- LLMs are mainly trained on English text. During training they haven’t “seen” enough Norwegian.
- Norwegian has special characters that other languages don’t use. Examples: æ, ø, å.
- Norwegian words take different endings. This means we write the same noun in many different ways (for example definite/indefinite form, singular/plural).
The implications of this are
- English tokens are more “efficient”.
- LLMs tend to make more grammatical mistakes in Norwegian.
- Using language models in Norwegian is more expensive, since we pay for token usage, not word usage.
In general I would recommend prompting in English but asking for the output in Norwegian. But it depends on the use case. There are some exceptions where you should prompt in Norwegian to “activate” as much of the Norwegian training data as possible, typically use cases like copywriting.
Counting the R’s in “strawberry”
Let’s move on to the viral example that many people have talked about and shared on social media: “How many R’s are there in the word strawberry?” or: “How many i are in antihistoricityness?”
Why do LLMs struggle with tasks for first-graders?
If they are so “smart”, why can’t they get it right? The answer may surprise most people.
Language models know nothing about human language.
Yes, you read that right. LANGUAGE models don’t understand our language. They don’t even know they are processing text.
When you talk to ChatGPT (or similar), the AI model never reads the text. It only reads the long list of numbers (tokens). When AI models answer, they never write text either. They write the list of numbers (tokens).
Let me describe the whole process, simplified, step by step:
- You send your request to the AI.
- Your request is encoded (turned into tokens).
- The LLM receives the list of tokens.
- The LLM generates the list of tokens for the answer.
- The answer is decoded into human language.
- You see the answer in human language.
Human language is just code
Human language can be seen as a kind of code – a system we have developed to convey thoughts and ideas efficiently.
Text is the code humans invented to communicate faster and better.
Humans gave names (words) to things, activities, behaviors and qualities. We use words to capture the meaning of all these things. But the “names” were invented. And that is why we have so many languages in the world.
When someone says “jordbær” (Norwegian for strawberry), most people picture a sweet, red fruit. But I have the same reaction when someone says “strawberry”. Because I also know English.
But the word itself is just a code that triggers this image in my mind. My imagination only really cares about what the strawberry represents, not how you spell it.
Language models work like the human understanding of concepts. They don’t care how we write it down. They only capture the meaning of a strawberry.
So how do language models capture meaning? This is where token embeddings come in.
Token embeddings
The language model receives a list of tokens. But the token numbers are just IDs. There is no meaning in IDs. To get the meaning, tokens are converted into vector embeddings. This conversion happens inside the large language models.
This is a bit simplified, but here is the most important information:
- Language models have a so-called “lookup table” for matching token IDs with token embeddings.
- The token embeddings are trained along with the model to find the best values.
Unfortunately we don’t have access to the parameters inside the GPT-4 models, so I can’t show you their token embeddings.
But we can take a look at open-source models. Let’s look at the BERT model, since it is the easiest one to show.
BERT Tokenizer
Let’s look at the “strawberry” example:
TokenIDs: [16876]
RawTokens: ["strawberry"]
Embeddings: tensor([[[ -3.3221e-01, -1.1379e-01, 6.3999e-01,
4.4558e-02, 8.5353e-01,
-1.0420e+00, 1.7256e-01, 1.3457e+00,
5.5995e-01, -1.2306e+00,
-5.1320e-01, 2.5210e-01, 4.0262e-01,
8.1291e-01, 5.0603e-01,
... a very long vector of 768 numbers ]]])This is how language models “understand” words – not as letters or sounds, but as complex mathematical representations of meaning.
That 768-number vector is how language models capture the meaning of a strawberry.
This is how language models know what a strawberry represents. But there is no information about the “human code” (also known as human language). There is nothing about which letters we use to represent a strawberry.
These vectors are not a direct representation of the word “strawberry” or “jordbær”. They are a representation of the meaning of the word.
What is the problem with the “How many letters …?” question?
When you say “strawberry”, I picture a strawberry in my imagination, but if you ask me “How many R’s are there in the word strawberry?” I will picture the actual word and not the fruit.
This illustrates the difference between how humans and machines “understand” the concept of a strawberry. Language models don’t have that ability: they always see the same thing, the vector embeddings assigned to the “strawberry” token.
An example of vector embeddings
In language models, the words in a sentence are converted into numeric tokens. The tokens are then converted into embeddings, which are high-dimensional vectors, as shown below.
INPUT: "Napoleon revolutionized milliatry organization"
↓ (Tokenizing)
TOKENS: [ [CLS], Napoleon, revolutionized, millitary, organisation ]
IDS: [ 101, 2024, 4046, 7890, 1042 ]
↓ (Embeddings Layer)
VECTOR: [ 0.0390, 0.0469, 0.0890, 0.0680, 0.0450 ]
[ 0.0370, 0.0479, 0.0879, 0.3725, 0.1220 ]Token usage for creatively written words
Those of you who have seen some of my earlier prompts have probably noticed “shortened” words, maybe even written in a creative way known as “leetspeak”, among other styles.
Most people assume at first that this “technique” is for squeezing a bigger prompt into the context window, but it actually has the opposite effect. Writing words in an unusual way gives you a much higher token count.
See the example below:
Show token IDs
[9866, 19, 12235, 4540, 499, 18, 45, 16, 3042, 391, 19, 4314, 16, 55, 198, 9866, 4769, 4540, 78255, 40, 3042, 195849, 47812, 47, 2366, 555, 77, 14, 1139, 2405, 1836, 1689, 7606, 20, 72, 20, 198, 4407, 1317, 14, 108118, 1500, 26536]As you can see, writing words this way uses considerably more tokens. “Prcptn” is three tokens, even though it is a shorter version of “Perception”, which is two tokens.
So what are the benefits of doing this?
When a specific word or phrase is mentioned in the input, the model will often treat that word as important to the context. This increases the likelihood that the model repeats the same word or similar phrases in the output.
If you write “1337” instead of “elite”, or “Prcptn” instead of “Perception”, the language model still understands what the word means (you can easily double-check this by asking the model), but you reduce the likelihood of the word being repeated in the output.
This doesn’t only apply to the word you write. Say that in the training data, the word “elite” is often followed by the word “player” later in the text. Then there is a relatively high likelihood that “player” also becomes one of the next words.
The next token is always generated based on the preceding tokens (the context).
See also my earlier article on hyperparameters in the prompting process for more insight into the words available in autocompletion.
Another example
Context: “I love to eat”
Examples of possible next tokens: “pizza”, “ice cream”, “sushi”
Explanation: Based on the preceding words, the model understands that the next word is probably a food.
So with this method you can reduce the likelihood that the model follows up with common associations tied to the original words.
This method is not token-efficient, so if you use methods like this, you should have a very clear picture of whether the trade-off is worth it.
Performance and costs
Focusing on efficient use of tokens improves the model’s performance in several ways:
- A good prompt with “efficient tokens” reduces the chance of misunderstandings and helps the model generate more accurate answers.
- By eliminating unnecessary tokens, you make sure the model’s attention is directed at the most important aspects of the prompt.
Costs
If you use OpenAI’s API, you pay for tokens; there is no fixed monthly cost. You simply pay for usage. The different models have different prices.
If a whole organization is going to use an AI product where you pay for tokens, or an AI application with many users, tokens and cost become a factor you need to think about.
| Model | Price | Price with Batch API* |
|---|---|---|
| gpt-4o | $2.50 / 1M input $1.25 / 1M cached** input $10.00 / 1M output | $1.25 / 1M input $5.00 / 1M output |
| gpt-4o-2024-08-06 | $2.50 / 1M input $1.25 / 1M cached** input $10.00 / 1M output | $1.25 / 1M input $5.00 / 1M output |
| gpt-4o-2024-05-13 | $5.00 / 1M input $15.00 / 1M output | $2.50 / 1M input $7.50 / 1M output |
Some key takeaways to finish
- A rule of thumb is that 100 tokens equals roughly 75 English words.
- Most of the time you will want to prompt in English but ask the model to answer in Norwegian (with exceptions in some cases, for example copywriting in Norwegian).
- Creative use of abbreviations or alternative spellings can affect how the model processes and repeats words in its answers.
- Language models don’t understand human language directly; they operate on numeric representations.
- Efficient tokens can directly affect the model’s performance.
- The number of tokens matters for cost when using the API.
I hope this gave you some insight into how tokens work, and why a seemingly “smart” technology can struggle to spell words like “strawberry”.
See you in the next one!
First published . Also available in Norwegian.
← All articles