LLM/RAG Production-Readiness Checklist
A launch checklist for retrieval-augmented generation apps covering retrieval quality, evaluation, guardrails, cost control, and observability. It turns a RAG prototype into a reliable, auditable production service.
When to Use This Checklist
Use this checklist when a retrieval-augmented generation (RAG) prototype is moving toward real users. RAG grounds a large language model (LLM) in documents fetched at query time, which reduces hallucination but adds a retrieval pipeline, an embedding store, and new failure modes. Production readiness means the system is accurate, observable, cost-bounded, and safe under load.
How to Use This Checklist
Work top to bottom, treating each required item as a launch gate. Start with discovery and compliance so you do not build on an unlicensed or stale corpus. Then validate retrieval quality with concrete metrics such as recall@k before tuning prompts. Pair every prompt or model change with a run of the offline eval set so regressions are caught before deployment. Assign owners for the guardrail, observability, and cost items, since these tend to be skipped under deadline pressure.
What Good Looks Like
A production-ready RAG service answers from retrieved, cited sources and degrades gracefully when retrieval fails. Every release passes a fixed eval set with tracked groundedness and accuracy scores. Prompt-injection and jailbreak attempts are blocked by input and output guardrails. Dashboards show latency, token spend, cache hit rate, and a groundedness signal per request, and alerts fire on cost or quality drift. Token budgets are enforced per tenant, and a documented rollback returns the system to a known-good prompt and model version within minutes.
Common Pitfalls
The most common mistake is shipping with no offline evaluation, so quality is measured only by anecdote. Teams often ignore prompt injection until an incident forces the issue. Unbounded context windows and verbose prompts quietly inflate cost. Stale indexes serve outdated answers because no re-indexing SLA exists. Finally, responses without citations cannot be audited, which blocks adoption in regulated settings.
Related Resources
Review RAG and evaluation best practices, guardrail patterns, vector database guidance, and hallucination mitigation techniques before sign-off.