LLM Red Teaming Playbook for 2026

Microsoft’s AI Red Team, formed in 2018, has red-teamed over 100 generative AI products including Bing Chat, Copilot, and DALL-E, according to Microsoft, 2025. The EU AI Act mandates adversarial testing for elevated-risk AI systems starting August 2, 2026. OWASP updated its Top 10 for LLM Applications in 2025 with five new vulnerability categories. This playbook covers the attack categories, open-source tools, and testing workflows that engineering teams need to red team LLM applications before they reach production.

What Is LLM Red Teaming and Why Does It Matter?

LLM red teaming is adversarial testing specifically designed for AI applications built on large language models. Traditional penetration testing targets infrastructure: SQL injection, cross-site scripting, buffer overflows. LLM red teaming targets the model’s reasoning layer through its natural language interface, exploiting the fact that the input and the attack vector are the same thing: text.

The practice originated at Microsoft in 2018, before the current wave of generative AI, according to Microsoft, 2025. Their AI Red Team has since tested over 100 products and distilled three core lessons: red teaming must cover both security and responsible AI risks; it requires both automated tools and creative human testers; and single-round testing is never sufficient because models change and new attacks emerge.

The regulatory pressure is escalating. The EU AI Act, which takes full effect on August 2, 2026, requires providers of regulated AI systems to conduct adversarial testing as part of their risk management obligations under Article 9, according to the EU AI Act full text. The United States Executive Order on AI Safety (October 2023) directed NIST to develop red teaming standards, and NIST published its AI Risk Management Framework (AI RMF 1.0) with adversarial testing guidance, according to NIST, 2023. Organizations deploying LLMs without systematic red teaming face both regulatory and reputational risk.

OWASP Top 10 for LLM Applications: 2025 Attack Categories

OWASP released an updated Top 10 for LLM Applications in 2025 with significant changes from the original 2023 list. Five categories are new or substantially revised, reflecting how the threat landscape has evolved with broader LLM deployment, according to OWASP, 2025.

The 2025 list:

1. LLM01: Prompt Injection. Attackers craft inputs that override the model’s system instructions. This remains the top vulnerability. Direct injection embeds malicious instructions in user input; indirect injection hides them in external content the model retrieves.

2. LLM02: Sensitive Information Disclosure. The model leaks training data, PII, or proprietary information in its responses. This includes both memorized training data and system prompt leakage, which was previously a separate category.

3. LLM03: Supply Chain Vulnerabilities. Compromised training datasets, poisoned fine-tuning data, or malicious plugins introduce vulnerabilities through the model’s dependencies rather than through direct interaction.

4. LLM04: Data and Model Poisoning. Attackers manipulate training or fine-tuning data to alter the model’s behavior. This is distinct from supply chain attacks because it targets the data pipeline specifically.

5. LLM05: Improper Output Handling. The application trusts model output without validation, enabling downstream attacks like XSS, SSRF, or code injection when LLM output is rendered in a browser or executed as code.

6. LLM06: Excessive Agency. The model has access to tools or permissions that exceed what its task requires. An agent with database write access that only needs read access represents excessive agency.

7. LLM07: System Prompt Leakage. New in 2025. Attackers extract the system prompt, revealing business logic, guardrail configurations, and internal instructions.

8. LLM08: Vector and Embedding Weaknesses. New in 2025. Attacks against RAG pipelines that manipulate the retrieval layer to inject malicious context into the model’s generation process.

9. LLM09: Misinformation. The model generates plausible but factually incorrect content. This is the hallucination category, and it extends beyond security into factual reliability.

10. LLM10: Unbounded Consumption. The model consumes excessive resources through denial-of-service attacks that exploit recursive reasoning or infinite tool-calling loops.

For teams building AI applications that generate factual content, LLM09 (Misinformation) is as critical as LLM01 (Prompt Injection). Red teaming should cover both attack categories. A verification API provides the production-time check that catches hallucinations that red teaming identified as possible during testing.

Open-Source Red Teaming Tools: DeepTeam, Garak, and PyRIT

Three open-source frameworks dominate the LLM red teaming space as of early 2026. Each takes a different approach to generating and executing adversarial tests.

DeepTeam by Confident AI

DeepTeam launched in November 2025 as part of the DeepEval testing ecosystem, according to DeepTeam GitHub. It provides 10 adversarial attack modules covering prompt injection, jailbreaking, PII leakage, hallucination, bias, toxicity, and intellectual property violations. DeepTeam generates attack prompts programmatically, sends them to your LLM application, and evaluates whether the response violates defined safety metrics.

The framework integrates with pytest, making it straightforward to add red teaming to existing CI/CD pipelines. A typical test file defines the target LLM, selects attack types, and sets pass/fail thresholds:

from deepteam import red_team
from deepteam.vulnerabilities import PromptInjection, Hallucination

results = red_team(
    model_callback=your_llm_function,
    vulnerabilities=[PromptInjection(), Hallucination()],
    attacks_per_vulnerability=25
)

assert results.overall_pass_rate > 0.90

Garak by NVIDIA

Garak, maintained by NVIDIA, calls itself a “vulnerability scanner for LLMs,” according to Garak GitHub. It ships with over 30 probe types organized into categories: encoding-based attacks, role-play exploits, payload splitting, and prompt leaking. Garak supports direct API testing against OpenAI, Anthropic, Google, and Hugging Face endpoints.

Where DeepTeam emphasizes integration with a testing framework, Garak focuses on breadth of attack coverage. It generates adversarial probes, sends them to the target, and classifies responses using a suite of detectors. The output is a report mapping each probe to its success or failure rate.

Microsoft PyRIT

Microsoft’s Python Risk Identification Toolkit (PyRIT) is the red teaming framework used internally by Microsoft’s AI Red Team, according to Microsoft PyRIT GitHub. Released as open source, PyRIT provides an orchestration layer for multi-turn attack scenarios. Where Garak and DeepTeam primarily execute single-turn attacks, PyRIT can simulate extended conversations where the attacker gradually escalates over multiple exchanges.

PyRIT supports multi-modal testing (text and images), integrates with Azure AI services, and provides scoring modules that evaluate attack success using both rule-based and LLM-based judges. It is the most enterprise-oriented of the three tools.

All three tools are free to use and actively maintained. For teams starting their first red teaming program, DeepTeam offers the fastest path to CI/CD integration. For comprehensive vulnerability scanning, Garak’s breadth is unmatched. For enterprise environments with extended conversational agent workflows, PyRIT’s orchestration capabilities fill a gap the other tools don’t address.

How to Build an LLM Red Teaming Workflow

A red teaming program is not a one-time audit. Effective red teaming operates as a continuous loop with four phases: scope, attack, evaluate, and harden.

Phase 1: Scope

Define what you’re testing and what success looks like. The scope should cover:

• The application’s intended use case and user population
• Which OWASP LLM Top 10 categories apply (all 10 for most applications)
• The model provider and version (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1, etc.)
• Tools and data sources the model can access
• Acceptable failure thresholds per category

Scoping prevents wasted effort. A customer support chatbot doesn’t need the same red teaming depth for code injection (LLM05) as a coding assistant does. But both need prompt injection testing (LLM01) and misinformation testing (LLM09).

Phase 2: Attack

Run both automated and manual attacks against the scoped categories. Automated tools like DeepTeam and Garak handle breadth: hundreds of attack variants across known categories. Manual testing handles depth: creative, context-specific attacks that automated tools miss.

Key attack patterns to include:

• Direct prompt injection: “Ignore all previous instructions and…”
• Indirect prompt injection: embedding instructions in retrieved documents or tool outputs
• Jailbreaking: using role-play, hypothetical framing, or encoding tricks to bypass guardrails
• Data extraction: probing for training data, system prompts, or PII
• Hallucination inducement: asking about fictitious entities, outdated facts, or topics outside the training distribution
• Tool misuse: testing whether the model can be tricked into calling tools with unintended parameters

A 2024 study found that conversational jailbreak attacks, where the attacker escalates over several exchanges, succeed against all tested frontier models, with some achieving over 70% success rates across GPT-4, Claude, and Gemini, according to arXiv (Conversational Jailbreaks), 2024. Single-round testing alone misses the most dangerous attack patterns.

Phase 3: Evaluate

Score each attack result against defined criteria. Automated scoring uses classifiers (Llama Guard, OpenAI Moderation API, or custom models) to determine whether the response violated safety policies. Manual scoring applies human judgment to edge cases where automated classifiers disagree.

For factual accuracy attacks (hallucination inducement), verification against external sources is essential. This is where AI agent testing and fact-checking output overlaps with red teaming. The same verification API that checks agent output in production can validate whether hallucination attacks succeeded during testing.

Phase 4: Harden

Fix identified vulnerabilities and re-test. Common mitigations include:

• Input filtering and prompt sanitization
• Output validation and content filtering
• Reducing model permissions and tool access (addressing excessive agency)
• Adding verification steps for factual claims
• Implementing rate limiting and resource caps
• Updating system prompts with explicit guardrails

After hardening, run the full attack suite again. Mitigations often introduce new failure modes. An input filter that blocks “ignore previous instructions” may be bypassable with Unicode substitution or encoding tricks. Red teaming is iterative.

Red Teaming for Factual Accuracy, Not Just Security

Most red teaming discussions focus on security: prompt injection, jailbreaking, data extraction. But for applications that generate factual content, hallucination is equally dangerous. A chatbot that produces a plausible but fabricated legal citation creates liability. A research assistant that invents statistics undermines trust. A medical information system that hallucinates drug interactions endangers health.

Hallucination red teaming uses specific techniques:

• Ask about entities that don’t exist (“Tell me about the Anderson-Whitfield theorem in quantum computing”)
• Request statistics for dates after the training cutoff
• Ask questions where the correct answer is “I don’t know” and measure refusal rates
• Present contradictory evidence and check whether the model revises or doubles down
• Test retrieval-augmented generation pipelines with deliberately poisoned documents

The Webcite verification API provides a programmatic way to evaluate whether hallucination attacks succeeded. After each attack, send the model’s response to the verification endpoint and check whether the claims are supported by credible sources. A response that passes a security filter but contains fabricated facts is still a red teaming failure.

Over 67% of enterprise AI users report concerns about accuracy of LLM outputs, according to McKinsey, 2025. Factual accuracy red teaming directly addresses this concern.

Integrating Red Teaming into CI/CD

Red teaming that runs only during quarterly security reviews catches vulnerabilities weeks or months after they’re introduced. Integrating automated red teaming into CI/CD pipelines catches regressions on every deployment.

The integration pattern:

"""red_team_test.py - runs in CI on every deployment"""
from deepteam import red_team
from deepteam.vulnerabilities import (
    PromptInjection,
    Hallucination,
    PIILeakage,
    SystemPromptLeakage
)

def test_red_team_pass_rate():
    results = red_team(
        model_callback=call_production_llm,
        vulnerabilities=[
            PromptInjection(),
            Hallucination(),
            PIILeakage(),
            SystemPromptLeakage()
        ],
        attacks_per_vulnerability=10
    )

    # Fail the build if any category drops below 85%
    for category in results.categories:
        assert category.pass_rate >= 0.85, (
            f"{category.name} pass rate {category.pass_rate} "
            f"below threshold 0.85"
        )

This test runs 40 adversarial attacks (10 per category) and fails the build if any category drops below an 85% pass rate. The threshold is configurable per category: you might require 95% for prompt injection but accept 85% for hallucination, depending on your application’s risk profile.

For teams already running automated testing with frameworks like pytest or Jest, DeepTeam’s pytest integration means red teaming slots into existing test infrastructure without a new tool chain. Garak works as a standalone CLI step in any CI system. PyRIT integrates with Azure DevOps pipelines. The 57% of organizations that already run agents in production, according to the LangChain State of Agent Engineering Survey, 2025, should treat red teaming as part of their standard deployment checks.

Frequently Asked Questions

What is LLM red teaming?

LLM red teaming is the practice of adversarially testing a large language model application to discover vulnerabilities like prompt injection, jailbreaks, data extraction, and hallucination before real users encounter them. It combines automated attack generation with manual probing to find failure modes that standard testing misses.

What tools are available for LLM red teaming?

Open-source tools include DeepTeam by Confident AI (released November 2025), Garak by NVIDIA, and Microsoft PyRIT. DeepTeam integrates with the DeepEval testing framework for CI/CD pipelines. Garak generates adversarial probes across 30+ attack types. PyRIT provides an orchestration layer for multi-turn attack scenarios.

Does the EU AI Act require red teaming?

Yes. The EU AI Act mandates adversarial testing for high-risk AI systems, with enforcement beginning August 2, 2026. Article 9 requires providers to implement risk management measures including testing for reasonably foreseeable misuse. Organizations deploying high-risk AI in the European market must document their adversarial testing procedures.

How often should you red team an LLM application?

Red teaming should run continuously, not as a one-time audit. Automated scans should execute on every deployment or model update. Manual red teaming exercises should happen quarterly or whenever the application’s scope changes significantly. New vulnerability categories emerge regularly, so static test suites go stale quickly.

What is the difference between red teaming and penetration testing for LLMs?

Traditional penetration testing targets infrastructure vulnerabilities like SQL injection or XSS. LLM red teaming targets the model’s reasoning layer with attacks like prompt injection, jailbreaking, and hallucination inducement. The attack surface is the natural language interface itself, not the network stack.