What is a golden dataset?
A golden dataset is a curated, versioned set of test inputs paired with reference answers or success criteria, reviewed by people who know the domain, that an eval uses as the trusted baseline for scoring an AI system.
An eval is only as good as the cases it runs. If the test set is too easy, too small or labelled wrongly, a passing score means little. The golden dataset is the set of cases a team has checked by hand and agreed to treat as the reference: these inputs, these correct outcomes, these criteria for success.
None of the three large labs uses the phrase "golden dataset" in their agent eval guidance; it is industry shorthand. The idea is the same in all of them: Anthropic recommends tasks with reference solutions, and Google's agent evaluation compares runs against reference trajectories and reference responses.
What goes into each case
Input
The request, plus any starting state the agent needs: documents, records, a conversation so far.
Reference outcome
The correct answer or the end state the agent should leave behind, such as a row written or a ticket closed.
Success criteria
What counts as a pass, written so a code grader, an LLM judge or a person can apply it the same way.
Tags
Customer segment, language, difficulty or failure type, so scores can be read per slice and not only as one average.
Provenance
Where the case came from and who approved its label, so disputed cases can be traced back.
How to build one
Anthropic's guidance is to start with 20 to 50 simple tasks drawn from real failures and to read the transcripts, rather than waiting for a large set. OpenAI's agent eval guidance recommends logging everything and mining the logs for eval cases, and its skill eval post adds negative cases: inputs where the system should decline or do nothing.
Real production cases beat invented ones, because they carry the phrasing, edge cases and messy data users actually produce. Synthetic cases are useful to fill gaps in coverage, provided a person reviews them before they are treated as golden.
Keeping it golden
Version it
Store the dataset with the code and record which version produced each score, so results stay comparable over time.
Review labels
Have a second domain expert check a sample of labels; wrong labels quietly turn correct answers into failures.
Keep it out of the prompt
Cases copied into few-shot examples or instructions stop measuring anything.
Refresh it
Add new failure types as they appear in production, and retire cases once they saturate.
How Motiontic's engineers apply it
Motiontic's engineers build the first golden set with the client's domain experts in the first weeks of an engagement, from the client's own records and past mistakes. Each case is tagged, versioned with the code and owned by a named reviewer on the client side, so the client's team can keep extending it after launch.
Three sources of eval data compared
| Golden dataset | Synthetic set | Production sample | |
|---|---|---|---|
| Source | Curated real cases | Generated by a model or script | Live traffic |
| Labels | Reviewed by domain experts | Generated, then spot-checked | Usually unlabelled |
| Strength | Trusted baseline for go or no-go decisions | Fills coverage gaps quickly | Shows what users actually do |
| Main risk | Goes stale as usage changes | Misses real-world mess | No reference answer to grade against |
Common questions
What is a golden dataset in AI?
A golden dataset is a reviewed set of test inputs with known correct outcomes, used by evals as ground truth. Scores against it decide whether a prompt, model or tool change is good enough to ship.
How big should a golden dataset be?
It can start small. Anthropic's guidance suggests 20 to 50 tasks drawn from real failures, grown over time as new failure types appear in production.
Can a golden dataset be generated with an LLM?
Parts of it can. Generated cases help cover rare inputs, but each one should be reviewed by a person before it is trusted as a reference, because a model's own labels can carry its own mistakes.
How is a golden dataset different from training data?
Training data teaches a model; a golden dataset tests a system. Keeping the two separate matters, since cases that leak into training, prompts or examples stop measuring real performance.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Regression eval
The suite that catches what a change broke before users do.
Eval saturation
When every score is near 100% and the eval stops telling anyone anything.
LLM as a judge
A model grading model output against a rubric, checked against human labels.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.