What is an AI hallucination?
An AI hallucination is output from a language model that is fluent and confident but false or unsupported by its sources, such as an invented fact, citation, file, API or tool argument.
Language models produce the most plausible continuation, and plausible is not the same as true. "Why Language Models Hallucinate" (Kalai and colleagues, OpenAI, September 2025) argues that hallucinations arise naturally from pretraining when a model cannot reliably tell true statements from false ones, and persist because training and evaluation reward guessing over admitting uncertainty. A model that guesses scores better on most benchmarks than one that says it does not know.
The paper's proposed fix is to change how widely used benchmarks score answers, so abstaining is not penalised. For a product team the lesson is the same at smaller scale: if the system and its evals reward an answer every time, they will get answers, including wrong ones.
How hallucination shows up in agents
Invented arguments
A tool call with an order number, file path or customer ID that does not exist.
Claims about unseen content
Describing code, a document or a record the agent never opened. Anthropic's prompting guidance includes an instruction to investigate before answering for exactly this case.
False completion
Reporting a task as done, or tests as passing, when they are not.
Unsupported citations
Pointing to a source that does not say what the answer claims.
What reduces it
Grounding
Retrieval and tools give the model real data to work from. The ReAct paper found that looking facts up through a simple Wikipedia API reduced the hallucination seen with chain-of-thought reasoning alone.
Permission to abstain
Instructions that make "not found" an acceptable answer, and evals that score an honest abstention above a wrong guess.
Verification
Tests, lookups against real records and a separate checker, so a claim is confirmed before it is acted on.
Short, relevant context
Accuracy degrades as context grows, so keeping the window lean also cuts errors.
Human gates
Irreversible actions wait for a person, so a hallucinated argument cannot do lasting damage.
Measuring it
Hallucination cannot be removed entirely, so it has to be measured. An eval set of real questions with known answers, including questions that should be declined, gives a hallucination rate that can be tracked across prompt and model changes. LLM-as-a-judge graders can check whether each claim is supported by the retrieved sources, once their own accuracy has been checked against human labels.
How Motiontic's engineers apply it
Motiontic's engineers ground answers in the client's own data, validate every tool argument against real records before execution, include unanswerable cases in eval sets, and report hallucination rate alongside task success on each release.
Hallucination types and controls
| Type | Example | Control |
|---|---|---|
| Factual | A wrong policy detail in a support answer | Retrieval with citations and a groundedness check |
| Tool argument | A refund issued against an order ID that does not exist | Strict schemas plus lookup before execution |
| Unseen content | A summary of a file the agent never read | Instructions to read before answering; traces |
| False completion | "All tests pass" when they do not | A check the harness runs, not the agent's word |
| Citation | A quote attributed to the wrong document | Verify quotes against the source text |
Common questions
What is an AI hallucination?
Output from a language model that sounds confident and fluent but is false or not supported by its sources, such as a made-up fact, citation or ID.
Why do language models hallucinate?
Research from OpenAI in 2025 argues that pretraining makes some errors statistically inevitable, and that training and evaluation reward guessing over saying "I don't know", so models learn to guess.
Does RAG stop hallucinations?
It reduces them by giving the model real sources, but does not stop them. The model can still misread a passage or answer when retrieval missed, so groundedness checks and evals are still needed.
Can hallucinations be eliminated?
Not today. They can be reduced through grounding, verification and permission to abstain, measured with evals, and contained with human approval on irreversible actions.
Related
Retrieval-augmented generation (RAG)
Fetch the right passages at query time, then answer from them with citations.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
LLM as a judge
A model grading model output against a rubric, checked against human labels.
AI guardrails
Checks on what goes into a model, what comes out, and what an agent is allowed to do.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.