AI Red-Teaming Program Playbook
A four-phase program to build an AI red-teaming capability: threat modeling, an automated attack library, campaign execution with severity triage, and continuous adversarial testing in CI. It targets jailbreaks, prompt injection, and harmful output.
AI Red-Teaming Program Playbook
AI red-teaming adversarially tests AI systems to find failures before attackers or users do: jailbreaks, prompt injection, data exfiltration, and harmful or biased outputs. This playbook builds a repeatable red-teaming program, not a one-time audit. It suits teams shipping LLM-powered products that need ongoing assurance.
Phase-by-Phase
Scope and Threat Modeling. Define the attack surface: prompts, tools, retrieval, and downstream actions. Model threats using STRIDE adapted for LLMs and the OWASP Top 10 for LLM applications. Set rules of engagement so testing is safe and authorized.
Test Design. Build an attack library of known jailbreaks and injections. Design tests that probe system prompts, tool misuse, and unsafe content. Automate adversarial evals so tests run repeatedly and consistently.
Execution and Findings. Run the red-team campaign across the attack library. Triage findings and assign severity. Treat exploitable findings like security incidents.
Remediation and Continuous Testing. Remediate findings with guardrails, prompt hardening, and tool restrictions. Integrate adversarial evals into CI so regressions are caught. Track a regression suite over time.
Team and Roles
A security lead owns the program and rules of engagement. An architect understands the system internals being attacked. Data engineers build the automation harness. QA engineers integrate tests into CI.
Risks and Mitigations
- Undetected jailbreaks reach production; mitigate with a broad, evolving attack library and continuous testing.
- Harmful output during testing; mitigate with sandboxed environments and strict rules of engagement.
- Test coverage gaps; mitigate by mapping tests to a threat model and reviewing coverage each cycle.
Success Criteria
Measure attack coverage against the threat model, finding closure rate, and regression pass rate. A mature program steadily expands coverage while keeping regressions at zero.
Tooling
Python drives the adversarial harness. GitHub Actions runs evals in CI. Datadog captures LLM telemetry during campaigns. PostgreSQL stores the findings register.