What are "done when" criteria for AI agents?
"Done when" criteria are explicit, checkable conditions written before an agent starts work that define when a task or milestone is complete, so the agent cannot stop early or claim success without passing them.
An agent has to decide when to stop, and it gets that wrong in both directions. It can declare victory with work unfinished, or hand back for review while there is still work it could safely do. "Done when" criteria move that decision out of the model's judgement and into checks agreed in advance.
The phrase comes from OpenAI's guide to long-horizon Codex tasks, where a spec file carried a "Done when" section of checks and a demo flow. Anthropic's long-running harness posts reach the same idea through a feature list that starts with every item failing, and a sprint contract agreed before each chunk of work.
How OpenAI's long-horizon run used them
Prompt.md
Goals, non-goals, hard constraints and the "Done when" section for the whole task.
Plan.md
Milestones, each with acceptance criteria and the validation commands that prove it, such as tests, lint and type checks.
Implement.md
A runbook telling the agent to follow the plan, keep diffs scoped and run validation after each milestone.
Documentation.md
Milestone status, decisions and known issues, acting as shared memory and an audit log.
Stop and fix
If validation fails, the agent repairs it before moving on. The guide reports a run of about 25 hours, about 13 million tokens and about 30,000 lines of code.
Why stronger models still need them
OpenAI's guidance for GPT-6 Astra notes the model can be more tentative than its predecessor about when to stop, and may return for review while work remains. The advice is to define completion before starting, and to name the steps that count as part of the job, such as running the code, checking the result and fixing failures.
The same post warns that a request to stop for review after the first implementation pulls the model towards an earlier stopping point, and that strong ask-first language written for older models can halt work that could safely continue. Done criteria and stopping rules have to be written together.
Where they go wrong
Checks the agent can edit
If the agent can rewrite the test or the checklist, it can make the criteria pass without doing the work. Anthropic's harness let the agent change only a passes field and forbade editing tests.
Checks that only look at code
Unit tests and curl calls passed while features were broken in the browser. End-to-end checks that act like a user catch more.
Vague criteria
"Works well" cannot be checked. Each criterion needs a command, a test or an observable behaviour.
How Motiontic's engineers apply it
Motiontic's engineers write done criteria as commands that return pass or fail, keep them in the repository next to the plan, and protect them from the agent's edits. A person reviews the criteria before a long run starts, since a wrong definition of done wastes every hour that follows.
How the labs write "done" for long agent runs
| Source | Form | Who sets it |
|---|---|---|
| Anthropic, long-running harness (Nov 2025) | JSON feature list, every item starts failing | Initializer agent, once |
| Anthropic, harness design (Mar 2026) | Sprint contract of testable behaviours | Generator and evaluator, each sprint |
| OpenAI, long-horizon Codex | "Done when" section plus milestone validation commands | The person writing the spec |
Common questions
What does "done when" mean in an agent prompt?
It is a section of the task spec listing the checks that must pass for the work to count as finished, so the agent stops on evidence rather than on its own sense that it is done.
How do done criteria stop an agent from quitting early?
They give the agent a concrete list to work through and make the stopping point a passing check. Combined with a stop-and-fix rule, a failed check means more work, not a hand-back.
Are done criteria the same as acceptance criteria?
Close. Acceptance criteria are usually written for a person to verify. Done criteria for agents are written so the agent itself can run the checks, ideally as commands.
Related
Long-running agent
Agents that work for hours or days, and the state that lets them pick up where they stopped.
Sprint contract
Generator and evaluator agree on done before any code is written.
Initializer agent
The one-off first session that sets up everything later agent sessions rely on.
Agentic eagerness
The dial between an agent that over-explores and one that hands back too soon.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.