85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is the agent quality flywheel?

The agent quality flywheel is a continuous loop in which an AI agent is evaluated, shipped, monitored in production and fixed, with each production failure turned into a test case for the next cycle.

One-off testing assumes an agent is finished when it ships. Agents are not: users ask new things, tools change and models are upgraded. The flywheel framing, which Google uses for its Gemini Enterprise Agent Platform evaluation tooling, treats quality as a loop rather than a gate at the end.

Google's June 2026 developer blog post describes three phases introduced at Cloud Next '26: Build and Test, Ship and Monitor, and Learn and Refine. Each turn of the loop makes the next one cheaper, because every failure found in production arrives as a ready test case.

The three phases

  • Build and Test

    Prepare eval data, run the agent, grade the results, analyse failures, apply a targeted fix and re-run against the previous baseline.

  • Ship and Monitor

    Online monitors grade live traffic and write quality scores into the monitoring system, so drift shows up without waiting for complaints.

  • Learn and Refine

    Failing production traces go back into Build and Test as new cases, and the cycle repeats.

Build and Test in five stages

Google's eval skill expands the first phase into five stages. Prepare data, from OpenTelemetry traces, hand-written cases or synthesised scenarios. Run inference to produce traces, a step skipped when production traces already exist. Grade, with model-based AutoRaters or custom metrics; this is the only stage that always runs. Analyse failures by reading rubric verdicts, clustering them automatically once there are ten or more. Then optimise and iterate: apply a targeted fix, rerun and compare.

In Google's travel-concierge example, one instruction change moved a revision-handling metric from 21% failures to 5%.

Where the flywheel stalls

  • Model scores treated as truth

    Google says AutoRater scores are directional signals and the deltas between runs matter more than absolute values.

  • Synthetic data only

    Generated scenarios bootstrap the loop; Google notes production data is what makes it useful.

  • Fixes with no human approval

    Google's skill proposes changes and a person approves them. An optimiser left alone can overfit the suite it is graded on.

  • No regression gate

    Without a regression suite, a fix for one failure can quietly break a case that was already solved.

Same idea, other names

Anthropic's January 2026 eval guide describes the same compounding effect without the flywheel label: failures become test cases, capability tasks graduate into regression suites, and automated evals are layered with production monitoring, A/B tests, user feedback and transcript review. It compares the layers to Swiss cheese: no single layer catches everything.

How Motiontic's engineers apply it

Motiontic's engineers set up the loop before launch: production traces in a format the eval harness can read, a triage step that turns confirmed failures into tasks, and a regression gate in CI. The client's team owns the loop after handover, so quality keeps compounding after the engagement ends.

Where each phase gets its signal

Where each phase gets its signal
PhaseInputOutput
Build and TestEval dataset or existing tracesGraded baseline and targeted fixes
Ship and MonitorLive trafficOnline quality scores and failing traces
Learn and RefineFailing production tracesNew eval cases for the next cycle

Common questions

What is an agent quality flywheel?

It is a continuous loop of testing, shipping, monitoring and fixing an AI agent, in which each production failure becomes a test case so the next cycle starts from a stronger suite.

How is the flywheel different from a regression suite?

The regression suite is one part of it: the set of solved cases that must keep passing. The flywheel is the loop that keeps feeding new cases from production into the suites and checks every fix against them.

Does the flywheel require Google's platform?

No. Google's tooling implements it, but the loop needs only traces, an eval harness, graders and a routine for turning failures into tasks, all of which open-source and commercial tools support.

Can the loop be fully automated?

Grading, clustering and re-running can be. Google's own skill still has a person approve each proposed change, and treats model-based scores as directional rather than final.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.