What is AI red teaming?
AI red teaming is adversarial testing in which people, models or both deliberately attack an AI model or agent, through prompts, documents, tools and infrastructure, to find harmful behaviour and security weaknesses before real attackers or users do.
Red teaming comes from security and military practice: a team plays the adversary to find weaknesses the defenders missed. Anthropic's June 2024 post on the challenges of red teaming AI systems describes it as adversarially testing a system to identify potential vulnerabilities, and notes that the field lacks standard practices, which makes results from different organisations hard to compare.
For a chat model, red teaming mostly means prompts that draw out harmful output. For an agent, the attack surface widens to everything it reads and every tool it can call, so red teaming also covers prompt injection through documents and tool outputs, misuse of permissions and plain infrastructure flaws.
Kinds of AI red teaming
Domain-expert
Policy experts probe usage-policy topics such as election integrity, and frontier-threat red teaming tests chemical, biological, radiological, nuclear, cyber and autonomy risks.
Multilingual and multicultural
Testing in other languages and cultural contexts, where safeguards trained mostly on English can be weaker.
Automated
One model generates attacks and another is trained to resist them. Open-source tools such as Microsoft's PyRIT and Promptfoo automate attack generation and scoring.
Multimodal
Attacks that combine images and text.
Open-ended
Crowdsourced testing and community events, such as the DEF CON AI Village challenges.
What Microsoft's AI red team found
Microsoft's AI red team, after more than 80 operations covering over 100 generative AI products by October 2024, published three lessons in January 2025. Generative AI amplifies existing security risks as well as adding new ones; one finding was an outdated FFmpeg component that enabled server-side request forgery, a conventional bug inside an AI product. People remain central, because automation helps with prompts, orchestration and scoring but cannot replace expertise and judgement. And defence in depth matters, through repeated break-fix cycles of red teaming, measurement and mitigation.
From red-team findings to evals
Anthropic describes a progression. Experts first probe a model by hand against a threat model and refine the inputs that work. A language model then generates hundreds or thousands of variations, turning a qualitative exercise into an automated, quantitative test that can run on every change, inside a loop of assessment, mitigation and retesting.
OWASP's GenAI Red Teaming Guide (January 2025) gives a structured, risk-based method that covers model-level issues such as toxicity and bias and system-level issues such as API misuse and data exposure, including agentic systems.
Limits to plan for
Never finished
Microsoft describes AI red teaming as a continuous process that has to keep up with new risks.
Finds, does not measure
A red team shows that an attack works, not how often it would succeed. Measurement comes from the evals built afterwards.
Only what is in scope
An agent's tools, data sources and infrastructure have to be in scope, or the most likely attack paths go untested.
How Motiontic's engineers apply it
Motiontic's engineers red-team agents before launch against the client's own threat model: injection through each untrusted source, misuse of each high-risk tool and attempts to reach outside the sandbox. Attacks that work become regression tasks, so the same hole is checked on every later change, and findings go to the client's security team in the format their own reviews use.
Red teaming compared with evals
| Red teaming | Evals | |
|---|---|---|
| Goal | Find failures no one has thought of | Measure known behaviour repeatably |
| Mindset | Adversarial | Specification |
| Output | Findings and working attacks | Scores against a baseline |
| Cadence | Before launch and periodically | On every change |
| Who | Security and domain experts, aided by attack tools | Engineers, with code, model and human graders |
Common questions
What is AI red teaming?
AI red teaming is deliberately attacking an AI model or agent, by hand and with automated tools, to find harmful behaviour and security weaknesses before real attackers or users find them.
How is red teaming different from an eval?
Red teaming searches for new failures with an adversarial mindset. Evals measure known behaviour repeatably. Successful red-team attacks are usually turned into eval cases so they are retested on every change.
What tools are used for AI red teaming?
Open-source options include Microsoft's PyRIT and Promptfoo, which OpenAI announced it would acquire in March 2026 and which says it will remain open source. OWASP's GenAI Red Teaming Guide provides a method to go with them.
Can AI red teaming be fully automated?
Not according to Microsoft's AI red team, which found that automation scales the work but cannot replace human expertise, cultural knowledge and judgement.
Related
Prompt injection
Untrusted text that hijacks a model's instructions, and how agents are built to survive it.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
AI guardrails
Checks on what goes into a model, what comes out, and what an agent is allowed to do.
Regression eval
The suite that catches what a change broke before users do.
Compliance-ready AI
Evals, traces, human gates and audit evidence built in, so the security review passes the first time.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.