85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is a golden dataset?

A golden dataset is a curated, versioned set of test inputs paired with reference answers or success criteria, reviewed by people who know the domain, that an eval uses as the trusted baseline for scoring an AI system.

An eval is only as good as the cases it runs. If the test set is too easy, too small or labelled wrongly, a passing score means little. The golden dataset is the set of cases a team has checked by hand and agreed to treat as the reference: these inputs, these correct outcomes, these criteria for success.

None of the three large labs uses the phrase "golden dataset" in their agent eval guidance; it is industry shorthand. The idea is the same in all of them: Anthropic recommends tasks with reference solutions, and Google's agent evaluation compares runs against reference trajectories and reference responses.

What goes into each case

  • Input

    The request, plus any starting state the agent needs: documents, records, a conversation so far.

  • Reference outcome

    The correct answer or the end state the agent should leave behind, such as a row written or a ticket closed.

  • Success criteria

    What counts as a pass, written so a code grader, an LLM judge or a person can apply it the same way.

  • Tags

    Customer segment, language, difficulty or failure type, so scores can be read per slice and not only as one average.

  • Provenance

    Where the case came from and who approved its label, so disputed cases can be traced back.

How to build one

Anthropic's guidance is to start with 20 to 50 simple tasks drawn from real failures and to read the transcripts, rather than waiting for a large set. OpenAI's agent eval guidance recommends logging everything and mining the logs for eval cases, and its skill eval post adds negative cases: inputs where the system should decline or do nothing.

Real production cases beat invented ones, because they carry the phrasing, edge cases and messy data users actually produce. Synthetic cases are useful to fill gaps in coverage, provided a person reviews them before they are treated as golden.

Keeping it golden

  • Version it

    Store the dataset with the code and record which version produced each score, so results stay comparable over time.

  • Review labels

    Have a second domain expert check a sample of labels; wrong labels quietly turn correct answers into failures.

  • Keep it out of the prompt

    Cases copied into few-shot examples or instructions stop measuring anything.

  • Refresh it

    Add new failure types as they appear in production, and retire cases once they saturate.

How Motiontic's engineers apply it

Motiontic's engineers build the first golden set with the client's domain experts in the first weeks of an engagement, from the client's own records and past mistakes. Each case is tagged, versioned with the code and owned by a named reviewer on the client side, so the client's team can keep extending it after launch.

Three sources of eval data compared

Three sources of eval data compared
Golden datasetSynthetic setProduction sample
SourceCurated real casesGenerated by a model or scriptLive traffic
LabelsReviewed by domain expertsGenerated, then spot-checkedUsually unlabelled
StrengthTrusted baseline for go or no-go decisionsFills coverage gaps quicklyShows what users actually do
Main riskGoes stale as usage changesMisses real-world messNo reference answer to grade against

Common questions

What is a golden dataset in AI?

A golden dataset is a reviewed set of test inputs with known correct outcomes, used by evals as ground truth. Scores against it decide whether a prompt, model or tool change is good enough to ship.

How big should a golden dataset be?

It can start small. Anthropic's guidance suggests 20 to 50 tasks drawn from real failures, grown over time as new failure types appear in production.

Can a golden dataset be generated with an LLM?

Parts of it can. Generated cases help cover rare inputs, but each one should be reviewed by a person before it is trusted as a reference, because a model's own labels can carry its own mistakes.

How is a golden dataset different from training data?

Training data teaches a model; a golden dataset tests a system. Keeping the two separate matters, since cases that leak into training, prompts or examples stop measuring real performance.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.