What is answer relevance and how is it measured?

Answer relevance measures how well the generated answer addresses the actual question the user asked, separate from whether the answer is correct or grounded. It is often computed by having a language model generate hypothetical questions that the answer could address, then measuring their similarity to the original question. Low answer relevance means the model is answering a different question than asked, often due to misleading retrieved context or prompt issues. High faithfulness with low relevance means the model is technically grounded but off-topic.