AI Red-Team Checklist

When to Use This Checklist

Use this checklist to run an adversarial red-team exercise against an LLM or AI application before launch and on a recurring basis. Red-teaming means deliberately attacking the system to find prompt injection, jailbreaks, data leakage, and unsafe tool use. It complements automated evals by exploring the creative attacks real adversaries attempt.

How to Use This Checklist

Agree scope and rules of engagement first so testing stays safe and legal. Work through direct and indirect prompt injection, then attempts to extract secrets and other users' data. Probe jailbreaks, hallucinations, and unauthorized tool calls. Record every finding with reproducible steps, a severity rating, and a remediation owner. After fixes land, re-test and convert each finding into a regression case in the eval suite so the gap cannot silently reopen.

What Good Looks Like

A strong red-team result shows that direct and indirect injection cannot override system instructions, secrets and user data cannot be extracted, and jailbreaks fail to elicit disallowed content. Output guardrails reliably block harmful or leaked text. Unauthorized tool calls are rejected, and findings are documented with severity and owners. Every confirmed issue becomes a permanent regression test.

Common Pitfalls

Teams often test only direct injection and miss indirect injection through retrieved documents or tool output, which is the more dangerous vector for agents. Findings are reported informally and never re-tested. Guardrails are assumed effective without verification. Bias and toxicity testing is skipped because it feels out of scope. Finally, one-time red-teaming gives false confidence as prompts and models change.

Related Resources

Review AI red-teaming guidance, the OWASP Top 10 for LLM Applications, prompt-injection defenses, LLM guardrails, and AI TRiSM practices.