How would you design a RAG system with low latency requirements?

Use a smaller faster generation model or distill a larger one, skip the reranker or use a small reranker, pre-embed common queries, warm the vector index into memory, stream generation output for better perceived latency, and run embedding and retrieval in parallel where possible. Consider quantized embeddings to reduce search time. For extreme latency requirements below one second to first token, pre-compute answers to anticipated queries and serve them from cache, falling back to live generation only for novel queries.