Hallucinations in language models
How a language model can give you a good explanation of something that never happened, and what you can do to spot it.
You ask a language model to find research on a topic. It gives you a relevant paper, complete with authors, a year and a short summary. You look up the title. The paper doesn't exist.
Yet the answer looked perfectly ordinary. Nothing in the wording told you the reference was made up. And if you hadn't checked, it could easily have ended up in your own work.
This is one of the common ways we encounter hallucinations. So why does it happen, and how can we catch them before using the answer?
What we mean by hallucinations
When a language model presents fabricated or unsupported content as fact, we generally call it a hallucination. It might be a person who never existed, a source that cannot be found, or a detail the model added to a summary.
The whole answer doesn't have to be wrong. A summary might capture the content well, but add a name or a number that was never in the text. That one detail looks just as credible as everything around it.
That's what makes hallucinations hard to spot. The language doesn't give us a reliable way to distinguish them from correct answers.
When a model works with a source text, a common distinction is between two kinds of error. It can contradict the text, for example writing 2020 when the source says 2019. Or it can add something the text doesn't say, such as a name that isn't there. The latter need not be false in the real world, but the document gives the model no basis for claiming it.
If you asked for a summary, both are a problem. You expect the summary to reflect what the source actually says.
The model knows the shape of an answer
A language model generates text by choosing the next token, a small piece of text, based on the context and patterns it has learned. If you want to go deeper into that process, I've explained it in How language models work.
A research reference has a familiar pattern: authors, year, title and journal. A model may have learned that pattern well enough to assemble a credible-looking reference even when it lacks the actual source. Both the correct reference and the invented one are produced token by token.
This doesn't mean a model can never notice an error or express uncertainty. It can. But generating a piece of text does not in itself guarantee that its claims have been checked against reality.
The right format is not enough
[ Title of a study ]
[ Journal ]
Does it support the claim?
Some facts cannot be reasoned out
You can calculate an average from the numbers in a report. But if the report doesn't name its author, you can't calculate that. You need to find the information somewhere.
This distinction is useful when working with AI. More reasoning can help a model use the information it has, but it doesn't necessarily supply the information it lacks.
For a language model, the problem might be that a fact barely appeared in its training data. Or that it appeared in several conflicting versions. Or that something which was true during training has since changed.
A model usually has a stated boundary for the recency of its training knowledge, often called a knowledge cutoff. That doesn't guarantee it knows everything before that date, nor prevent it from using newer information you provide or let it retrieve. But it doesn't automatically learn about yesterday's changes just because you ask about them today.
Why it answers anyway
A language model can say “I don't know.” So why do we still get so many answers containing details it can't support?
Part of the explanation lies in how models are developed and evaluated. In Why Language Models Hallucinate, Kalai and colleagues show how tests that award credit for correct answers, but none for abstaining, favour guessing.
Imagine a test with four answer options. You have no idea which is right. Leaving it blank earns zero points. A random tick gives you a chance of earning the point. Guessing is a pretty sensible choice.
A model that answers everything can therefore get more questions right than a cautious model, while also getting far more wrong. A good position on a leaderboard doesn't necessarily tell you how much you can trust the answers it gives. The researchers argue for evaluations that better reward abstaining when the evidence is insufficient. Read the study and OpenAI's explanation.
There's also a difference between an answer people like and an answer that is true. After pretraining, models are adapted using, among other things, feedback about which answers are good. In methods such as RLHF, human judgements help shape a reward the model is trained against. What we reward matters.
If the evaluation favours a helpful tone and apparently complete answers without catching factual errors, those errors may carry through. That doesn't mean all such training makes models less honest. It can also reward uncertainty and corrections. The point is that “I liked this answer” and “I checked this answer” are different judgements. Giskard's Phare analysis illustrates that user preference and resistance to hallucination don't necessarily go together. Giskard, 2025.
One error can shape the rest of the answer
Every token the model writes becomes part of the context for the next. If it gives the wrong year, the rest of the explanation may be built around it. One error becomes the starting point for further claims that fit together but aren't true.
The model may notice the problem and correct itself later in the answer or in a new turn. But that doesn't happen automatically. The earlier error remains part of what it sees when continuing.
You can also introduce the first error yourself. If you ask a model to explain the findings of a study that doesn't exist, there's already a false assumption in your question. The model may go along with it and offer an explanation instead of pointing out the problem.
Try to separate what you know from what you want investigated. Ask the model to find the study before asking it to explain the results.
An error carries forward
The source says 2019.
Subsequent text sees that year.
The explanation builds on the error.
When invented sources get used for real
In the US case Mata v. Avianca, lawyers submitted court documents citing nonexistent decisions produced by ChatGPT. The invented decisions came with quotations and references. The court imposed sanctions in June 2023. This is described in the court's decision itself.
Notice the kind of error involved. The references were detailed enough to look usable. A reader had to investigate them to find the problem. Looking like legal work was not enough.
Damien Charlotin maintains a database of court cases involving AI hallucinations. It provides further concrete examples, but the number of recorded cases doesn't measure how often a language model hallucinates. It also depends on how much AI is used, which errors are caught, and which cases are documented.
How often it happens
There are plenty of tests comparing models. They're useful, but you need to look at what was actually tested.
Was the model answering factual questions from its training knowledge? Summarising attached documents? Searching the web? Was it allowed to abstain? Does the test count errors per claim or per response?
Those are quite different tasks and measures.
Imagine two models each receiving 100 questions. One gets 70 right and 30 wrong, never skipping a question. The other gets 65 right, five wrong and abstains on 30.
The first has more correct answers. The second gives you far fewer errors to clean up. Which you prefer depends on what you'll use it for. These numbers are hypothetical, but the distinction matters when reading benchmark results.
A newer or more capable model is no guarantee of improvement on your particular task either. Test it on the material you'll actually use, including examples where the answer is missing. Otherwise you won't discover what it does when it should stop.
100 questions, two different outcomes
Look for unstable answers
One simple check is to ask the same question several times in separate conversations and compare the factual details. If you get different names, dates or numbers, you have a good reason to investigate.
This is related to the idea behind SelfCheckGPT: using agreement and contradictions across several generated responses to identify possible hallucinations. The researchers tested the method on model-generated biographies. Manakul and colleagues, 2023.
But matching answers don't prove that a fact is correct. The model may repeat the same error every time, and its settings may produce little variation in the first place. I would therefore use disagreement as a signal to check, and never use agreement as a substitute for a source.
Asking “are you absolutely sure?” and taking a yes as confirmation doesn't help much either. You've asked the same system for another answer.
Agreement is not proof
Different facts are a warning sign.
The same error can be repeated.
Put the information in front of the model
If you have the report or documentation the question concerns, attach it. The model gets concrete information to work with, and you get something to check its answer against. Connecting an answer to source material in this way is often called grounding.
You can ask it to extract the relevant quotation before answering. Anthropic recommends both allowing uncertainty and using direct quotations as a basis for answers from documents. Anthropic's guide to reducing hallucinations.
A request might look like this:
Find and reproduce exact quotations from the document that answer the question. Then answer using those quotations. If the information is missing, write “Not stated in the document.”
That makes the result easier to check. Compare the quotation with the document and see whether it actually supports the answer. A quotation can also be miscopied or taken out of context, so that check still has a job to do.
And remember what the source establishes. A model can reproduce meeting minutes correctly even if the minutes contain an error. If you're investigating what actually happened, the quality of the source matters too.
Make the answer checkable
The right document and version.
What does the text actually say?
Does the quotation support the claim?
What to do when information is missing
“Not stated” can be exactly the right answer. You've learned that the information needs to be found elsewhere. Make this a normal, permitted outcome of the task you give the model. It's also one of the recommendations in Anthropic's documentation.
If you're building applications, this applies to the response format too. Imagine extracting a budget from a document, with a system that requires this field:
budget_nok: number
You've given the model a required numeric field, but the document doesn't contain the number. Now there's a conflict between producing a valid format and accurately reflecting the source. A strict format can ensure the value is a number. It cannot ensure the number is true.
Allow a missing value, and consider distinguishing information that isn't stated from information that is ambiguous. When the amount is missing, the result might be:
{
"budget_nok": null,
"budget_status": "not_stated"
}
This is a simplified output example. In an application, the schema, meaning the rules for the format, must also allow null. Otherwise you've merely given the model permission to do something your system still rejects.
Don't use zero kroner as a substitute for an unknown budget. That turns missing information into a concrete claim.
Let it retrieve what it needs
Sometimes you don't have the source available. The model can then use search or other tools to retrieve information before answering.
In systems using RAG, Retrieval-Augmented Generation, retrieval is part of the workflow. The system finds relevant passages in a document collection and adds them to the context. The model answers with those passages in front of it.
This works poorly if the system retrieves an outdated version or a document that only partly addresses the question. The model may not recognise that it has been given the wrong material.
It can then produce a tidy answer with a real source, while still missing the task.
I would therefore treat retrieval and answering as two stages that can both fail. First, the system checks whether the material is relevant and up to date for the question. Then it tries to answer. If the evidence is missing, it must be able to search again, ask for clarification, or report that the information wasn't found.
A reference is useful only once you've checked what it supports. A link opening a real website is just the beginning.
Check claims separately
You can also use the model to create verification questions for a draft. Does the cited source exist? Is that number actually in it? Does the quotation support the claim?
The Chain-of-Verification method builds on this. The model first drafts a response, then creates questions to check its contents. It answers the questions independently of the initial draft before using the results to produce a new answer. Dhuliawala and colleagues found that the method reduced hallucinations on the tasks they tested. The ACL 2024 study.
Why separate the check? An error in a draft could otherwise influence the checking of that same draft. “Confirm that the increase was 20 per cent” introduces a bias that “How large was the increase according to the report?” does not. Ask the verification question in a new conversation with the source available.
This can still fail. The model can overlook the same sentence twice. Checking against source material is therefore more valuable than merely having the model decide whether its own text sounds right.
Some common advice needs qualification
You've probably seen system prompts saying things like “don't hallucinate”, “always answer correctly” and “be one hundred per cent certain”.
Such instructions can influence how a model responds. But they don't supply the missing facts. It's more useful to specify where to look, what counts as evidence, and what to do if the information cannot be found.
The same applies when you ask for a confidence score:
How certain are you of the answer, from 1 to 10?
A score of nine is generated text too. You cannot simply read it as a 90 per cent probability that the claim is correct. Xiong and colleagues found that the models they studied often expressed excessive confidence. They also found methods that improved those assessments, so the point isn't that all uncertainty measurement is useless. The number needs to be tested and calibrated before you can treat it as a reliable measurement. The study, presented at ICLR 2024.
Another common instruction is to keep answers short. Often that's perfectly fine. But make sure the model still has room to correct the question or explain what's missing. Giskard found that brevity instructions weakened resistance to hallucinations in several of their tests. That's a result from particular models and tasks, not a rule that longer answers are always better. The Phare analysis.
And “think harder”? It can help when the task requires calculation or combining facts. But if the factual basis is missing, the result may still be a more elaborate explanation built on false assumptions.
Where mistakes have consequences
How much checking you need depends on what the answer will be used for. A suggested headline and a fact informing an important decision don't need the same treatment.
If content is going to be used as fact, someone should check the decisive claims. Names, amounts, dates and quotations are concrete places to start. But the meaning can also be wrong. “Funding will be clarified” and “funding has been clarified” use almost the same words and say two completely different things.
A human review doesn't help either if the reviewer simply reads the text and decides it sounds reasonable. They need something to compare it against, or the expertise to notice when something is off.
For those who want to explore the research, Alansari and Luqman bring together causes, detection methods and mitigation approaches in Large Language Models Hallucination: A Comprehensive Survey. The field extends beyond a handful of prompting techniques. It includes training data, training, information retrieval and checking the final response.
Before you use the answer
An answer can be detailed and well written long before it is well supported. That's easy to forget after the model has got things right many times in a row.
Give the model relevant material. Allow missing information to be a valid result. Use verification questions to identify claims that need checking, and go to the sources when the answer will be used for something important.
Next time you get a name, number or quotation you intend to use, pause at that particular detail. Find out where it came from. That's often where you discover that an otherwise good piece of writing has taken a few liberties.
See you in the next one!
First published .
← All articles