LLM/RAG Production-Readiness Checklist

When to Use This Checklist

Use this checklist when a retrieval-augmented generation (RAG) prototype is moving toward real users. RAG grounds a large language model (LLM) in documents fetched at query time, which reduces hallucination but adds a retrieval pipeline, an embedding store, and new failure modes. Production readiness means the system is accurate, observable, cost-bounded, and safe under load.

How to Use This Checklist

Work top to bottom, treating each required item as a launch gate. Start with discovery and compliance so you do not build on an unlicensed or stale corpus. Then validate retrieval quality with concrete metrics such as recall@k before tuning prompts. Pair every prompt or model change with a run of the offline eval set so regressions are caught before deployment. Assign owners for the guardrail, observability, and cost items, since these tend to be skipped under deadline pressure.

What Good Looks Like

A production-ready RAG service answers from retrieved, cited sources and degrades gracefully when retrieval fails. Every release passes a fixed eval set with tracked groundedness and accuracy scores. Prompt-injection and jailbreak attempts are blocked by input and output guardrails. Dashboards show latency, token spend, cache hit rate, and a groundedness signal per request, and alerts fire on cost or quality drift. Token budgets are enforced per tenant, and a documented rollback returns the system to a known-good prompt and model version within minutes.

Common Pitfalls

The most common mistake is shipping with no offline evaluation, so quality is measured only by anecdote. Teams often ignore prompt injection until an incident forces the issue. Unbounded context windows and verbose prompts quietly inflate cost. Stale indexes serve outdated answers because no re-indexing SLA exists. Finally, responses without citations cannot be audited, which blocks adoption in regulated settings.

Related Resources

Review RAG and evaluation best practices, guardrail patterns, vector database guidance, and hallucination mitigation techniques before sign-off.