LLM Engineer Interview Questions: LLM Evaluation, Hallucinations, Guardrails, Production Monitoring

LLM output evaluation combines automated metrics, LLM-as-judge, and human review. Automated metrics like BLEU and ROUGE work for tasks with reference outputs. LLM-as-judge uses a strong model to score outputs against criteria. Human evaluation remains the gold standard for nuanced quality. Production systems typically use a mix, with LLM-as-judge for fast iteration and human spot-checks for ground truth.

LLM-as-judge is an evaluation approach where you use a strong language model to score the outputs of another model against criteria like helpfulness, accuracy, or relevance. It scales much better than human evaluation but inherits the judge's biases and blind spots. Common practice is to use a stronger model than the one being evaluated, and to validate the judge against human ratings on a sample.

Faithfulness measures whether the generated answer is grounded in the retrieved context, not in the model's internal knowledge. A faithful answer only uses facts from the retrieved chunks. Faithfulness catches hallucinations where the model asserts something the context does not support. It is one of the most important RAG metrics, often computed by an LLM judge comparing answer claims to context.

Answer relevance measures whether the generated answer actually addresses the user's question, regardless of whether it is grounded. A faithful answer can still be irrelevant if it answers a different question. Both faithfulness and relevance are needed; high faithfulness with low relevance means correct but useless answers, and high relevance with low faithfulness means useful but possibly fabricated answers.

Hallucinations are outputs where the model states something false with apparent confidence. They occur because LLMs generate plausible-sounding text without grounding in truth. Hallucinations are especially common for facts the model never learned, niche topics, recent events, and numerical details. They are the central reliability problem for LLM applications.

Reduce hallucinations with RAG to ground answers in real sources, with prompts that explicitly ask the model to abstain when uncertain, with structured outputs that constrain format, with citation requirements that force the model to point to evidence, with fact-checking against trusted sources, and with calibration prompts that ask the model to rate its own confidence. No single method eliminates hallucinations.

Guardrails are mechanisms that constrain LLM behavior to prevent unsafe, off-topic, or policy-violating outputs. They include input filters that block malicious prompts, output filters that catch harmful or off-topic responses, classifiers that detect PII or toxic content, and topic restrictions that keep the model on its intended use case. Guardrails are usually layered for defense in depth.

A content filter is a classifier that examines text for unsafe categories like violence, hate, sexual content, or self-harm. It can run on inputs before they reach the LLM and on outputs before they reach users. Major LLM providers ship built-in content filters, but production systems often add custom filters for domain-specific concerns.

Monitor latency including TTFT and inter-token latency, throughput in tokens per second, cost per request and per user, error rates, retrieval quality metrics for RAG systems, user feedback scores, hallucination rates from automated checks, and content filter trigger rates. Set up alerts for regressions on any of these. Most teams use observability tools like LangSmith, Arize, or LangFuse.

LLM observability is the practice of capturing detailed traces of LLM calls in production, including prompts, responses, latencies, costs, retrieved context, and tool calls. It enables debugging individual failures, identifying patterns across requests, and optimizing prompts and models over time. Tools include LangSmith, LangFuse, Arize Phoenix, and Helicone.

A/B testing for LLMs splits traffic between variants and compares outcomes on metrics like user satisfaction, task completion, latency, and cost. It is essential because offline evaluations rarely predict production behavior. Use a feature-flagging system, log everything, and run tests long enough to reach statistical significance on your KPIs.

Canary deployment routes a small percentage of traffic to a new prompt, model, or RAG configuration before rolling it out fully. This catches regressions early without risking the whole user base. Combined with monitoring and automatic rollback, canaries are essential for safely shipping changes to production LLM systems where outputs are non-deterministic. ---