85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is an AI hallucination?

An AI hallucination is output from a language model that is fluent and confident but false or unsupported by its sources, such as an invented fact, citation, file, API or tool argument.

Language models produce the most plausible continuation, and plausible is not the same as true. "Why Language Models Hallucinate" (Kalai and colleagues, OpenAI, September 2025) argues that hallucinations arise naturally from pretraining when a model cannot reliably tell true statements from false ones, and persist because training and evaluation reward guessing over admitting uncertainty. A model that guesses scores better on most benchmarks than one that says it does not know.

The paper's proposed fix is to change how widely used benchmarks score answers, so abstaining is not penalised. For a product team the lesson is the same at smaller scale: if the system and its evals reward an answer every time, they will get answers, including wrong ones.

How hallucination shows up in agents

  • Invented arguments

    A tool call with an order number, file path or customer ID that does not exist.

  • Claims about unseen content

    Describing code, a document or a record the agent never opened. Anthropic's prompting guidance includes an instruction to investigate before answering for exactly this case.

  • False completion

    Reporting a task as done, or tests as passing, when they are not.

  • Unsupported citations

    Pointing to a source that does not say what the answer claims.

What reduces it

  • Grounding

    Retrieval and tools give the model real data to work from. The ReAct paper found that looking facts up through a simple Wikipedia API reduced the hallucination seen with chain-of-thought reasoning alone.

  • Permission to abstain

    Instructions that make "not found" an acceptable answer, and evals that score an honest abstention above a wrong guess.

  • Verification

    Tests, lookups against real records and a separate checker, so a claim is confirmed before it is acted on.

  • Short, relevant context

    Accuracy degrades as context grows, so keeping the window lean also cuts errors.

  • Human gates

    Irreversible actions wait for a person, so a hallucinated argument cannot do lasting damage.

Measuring it

Hallucination cannot be removed entirely, so it has to be measured. An eval set of real questions with known answers, including questions that should be declined, gives a hallucination rate that can be tracked across prompt and model changes. LLM-as-a-judge graders can check whether each claim is supported by the retrieved sources, once their own accuracy has been checked against human labels.

How Motiontic's engineers apply it

Motiontic's engineers ground answers in the client's own data, validate every tool argument against real records before execution, include unanswerable cases in eval sets, and report hallucination rate alongside task success on each release.

Hallucination types and controls

Hallucination types and controls
TypeExampleControl
FactualA wrong policy detail in a support answerRetrieval with citations and a groundedness check
Tool argumentA refund issued against an order ID that does not existStrict schemas plus lookup before execution
Unseen contentA summary of a file the agent never readInstructions to read before answering; traces
False completion"All tests pass" when they do notA check the harness runs, not the agent's word
CitationA quote attributed to the wrong documentVerify quotes against the source text

Common questions

What is an AI hallucination?

Output from a language model that sounds confident and fluent but is false or not supported by its sources, such as a made-up fact, citation or ID.

Why do language models hallucinate?

Research from OpenAI in 2025 argues that pretraining makes some errors statistically inevitable, and that training and evaluation reward guessing over saying "I don't know", so models learn to guess.

Does RAG stop hallucinations?

It reduces them by giving the model real sources, but does not stop them. The model can still misread a passage or answer when retrieval missed, so groundedness checks and evals are still needed.

Can hallucinations be eliminated?

Not today. They can be reduced through grounding, verification and permission to abstain, measured with evals, and contained with human approval on irreversible actions.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.