What should you monitor in a production RAG system?

Monitor retrieval quality via offline evaluation metrics re-run on a fixed evaluation set after every change, online signals like user feedback and follow-up-question patterns, latency at each pipeline stage, token usage per query for cost tracking, and error rates for each component. Track retrieval-specific metrics like average similarity score distributions, which shift when content drifts from query distributions. Log queries, retrieved documents, and generated answers with unique identifiers to enable post-hoc analysis when users report bad answers.