How do you A/B test prompts and models?

A/B testing for LLMs splits traffic between variants and compares outcomes on metrics like user satisfaction, task completion, latency, and cost. It is essential because offline evaluations rarely predict production behavior. Use a feature-flagging system, log everything, and run tests long enough to reach statistical significance on your KPIs.