How do you choose embedding dimensions for a RAG system?

Higher dimensions improve retrieval quality but cost more in storage, memory, and search latency. A 1536-dimensional vector takes six kilobytes per item in float32, so a million items consumes six gigabytes before index overhead. Most teams start with a strong small model at 768 or 1024 dimensions, measure recall on a domain evaluation set, and only move to 3072 dimensions if the smaller model misses queries. Matryoshka embeddings let you truncate a longer vector to fewer dimensions with minimal quality loss, giving flexibility without re-embedding.