85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is an eval harness?

An eval harness is the infrastructure that runs an evaluation end to end: it loads tasks, runs the agent or model on each one, records transcripts, applies graders and aggregates the scores.

Checked

Writing eval tasks is half the work. Something has to run them: start a clean environment for each trial, run the agent, capture everything it did, apply each grader and turn the results into a score a team can compare across versions. That machinery is the eval harness.

Anthropic's agent eval guide separates it from the agent harness, the system that lets a model act as an agent. One is the thing under test; the other is the test rig.

What an eval harness does

  • Loads tasks

    Reads the dataset, with inputs, references and success criteria, at a known version.

  • Isolates each trial

    Starts each attempt in a fresh environment, often a container, so one run cannot affect the next.

  • Runs trials

    Executes each task several times, in parallel where possible, with fixed resource limits.

  • Records transcripts

    Keeps the full log of every trial for reading and debugging.

  • Applies graders

    Runs code checks, LLM judges and human review queues on each trial.

  • Reports and gates

    Aggregates scores per task and slice, compares them with a baseline and fails the build when a threshold is missed.

Open options in 2026

Inspect, from the UK AI Security Institute and Meridian Labs, is an MIT-licensed framework with sandboxes including Docker and Kubernetes and more than 200 pre-built evals. Promptfoo is an MIT-licensed CLI for evals and red-teaming in CI; OpenAI announced it would acquire Promptfoo in March 2026 and names it as the migration path from OpenAI's Evals platform, which shuts down on 2026-11-30. Harbor runs containerised agent evals and is listed in Anthropic's guide. DeepEval, MLflow and Google's ADK eval tooling are further options.

Whatever the tool, Anthropic's research on infrastructure noise is a reason to record the harness configuration with every score: container limits alone moved agentic coding results by up to 6 points.

How Motiontic's engineers apply it

Motiontic's engineers keep the eval harness in the client's repository, next to the agent, and run it in the client's CI. The harness is chosen to fit the client's stack rather than a house default, and is documented so the client's team can add tasks and graders without outside help.

Eval harness compared with agent harness

Eval harness compared with agent harness
Eval harnessAgent harness
RoleRuns tests against the agentLets the model act as an agent
ContainsTasks, trial runner, graders, reportingAgent loop, tools, context, permissions, state
RunsIn development and CIIn production, for every user request
OutputScores and pass ratesActions and answers

Common questions

What is an eval harness?

An eval harness is the software that runs evals: it loads tasks, runs the system on each one in isolation, records transcripts, applies graders and reports scores.

What is the difference between an eval harness and an agent harness?

The agent harness is the loop, tools and state that let a model act as an agent. The eval harness is the separate infrastructure that runs tasks against that agent and scores the results.

What replaces the OpenAI Evals platform?

OpenAI's deprecations page names Promptfoo as the migration path. The hosted Evals platform went read-only on 2026-10-31 and shuts down on 2026-11-30.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.