How do you A/B test changes to a RAG system?

Use offline evaluation on a fixed labeled set for rapid iteration on retrieval and prompt changes, then roll out winning variants to a small percentage of production traffic for live A/B testing. Measure online signals like user ratings, follow-up patterns, session length, and task completion. Ensure both variants use identical user segments to avoid confounding. Changes to embedding models require shadow indexing before rollout since embeddings from different models are not comparable. Track retrieval quality and generation quality separately to attribute changes correctly. ---