Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Evaluate retrieval and generation separately, because they fail for different reasons and get fixed by different people. Measure retrieval with recall@k on labelled question-to-article pairs, generation with groundedness and answer correctness, then confirm in production with re-contact and escalation rates.
How to approach it
Ask what a bad answer costs here. A help center for a note-taking app and one for a payroll product deserve different bars. Then ask whether the articles are current and whether every user can see every article, since stale or permissioned content changes the retrieval design before it changes the eval. Only then lay out the plan.
A strong answer
A RAG answer can go wrong in two places. The retriever can fetch the wrong articles, or the generator can misuse the right ones. If you score only the final answer, a 70% pass rate tells you nothing about where to spend next week. So I would build two evals and one production loop.
Retrieval. Take a few hundred real questions from search logs and support tickets, and have a support lead mark which article (or articles) actually answers each one. That gives labelled pairs. The metric is recall@k: for what share of questions does a correct article appear in the top k results the generator sees? If the generator reads five chunks, recall@5 is the number that matters. While it is low, prompt work cannot help, because the model cannot answer from an article it never received; fix retrieval first and set its bar from the misses you read.
The retrieval fixes are measurable, and the published numbers show how much room there usually is. Anthropic's contextual retrieval write-up (September 2024) scored its pipelines on 1 minus recall@20 and cut the top-20 retrieval failure rate by 67%, from 5.7% to 1.9%, by combining contextual embeddings, keyword (BM25) search and a reranker. Its example of why keyword search matters is a user typing an exact error code, the kind of query embeddings blur.
Generation. Hold retrieval fixed by feeding the correct article, then check three things per answer, each pass or fail:
| Check | Question it answers | How to score |
|---|---|---|
| Groundedness | Is every claim supported by the retrieved text? | LLM judge validated against human labels |
| Answer correctness | Does it actually resolve the question? | Reference answer for a golden subset, judge for the rest |
| Citation precision | Do the cited links support the sentence they sit on? | Code check that the link exists, judge for support |
Groundedness and correctness are different failures. An answer can be perfectly grounded in an outdated article and still be wrong, which is why the content team belongs in this eval.
Production. Offline sets drift away from what users ask. Online, I would watch the share of conversations followed by a ticket within 24 hours, thumbs-down rate, query reformulation (the user asks the same thing again in other words) and escalation to a human. Every week, sample 50 low-rated conversations, read them, and add the new failure types to the golden set.
The launch bar I would propose, set with the support lead and adjusted to what a wrong answer costs: recall@5 above 90% on the top 50 intents, groundedness failures under 2% on a set large enough that the interval is meaningful, and a ticket-creation rate no worse than today's search-only help center. That last one keeps the comparison honest. The baseline is the search box, not a perfect agent.
If the product is payroll or tax, I would raise the groundedness bar, require a citation on every factual sentence, and route anything involving a figure the user owes to a human-reviewed answer.
What interviewers probe next
"Recall@5 is 72%. What do you look at first?" The misses. Usually it is vocabulary mismatch (users say "invoice", articles say "billing statement"), so hybrid keyword plus semantic search and a reranker beat re-chunking.
"How do you get labelled pairs without a data team?" Mine tickets where an agent pasted an article link in the reply. Each one is a free label.
"The judge says 96% grounded. Do you believe it?" Only after checking it against 100 to 200 human-labelled answers and looking at how often it catches the ungrounded ones specifically. Methods such as ARES (2023) go further and use a small human-labelled set to correct the judge's estimate statistically.
"Why not use a framework that scores RAG without any human labels?" Reference-free scoring such as Ragas is useful for fast iteration. For a launch decision I still want a few hundred human labels, because a judge that is wrong about your domain is wrong silently.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Structure | One end-to-end accuracy score | Separates retrieval from generation | Names recall@k, groundedness and correctness as distinct, fixable failures |
| Data | "Create a test set" | Uses real user questions | Mines tickets and search logs for labels and grows the set from production failures |
| Bar | No threshold | A threshold per metric | Thresholds tied to the cost of a wrong answer and compared to the current help center |
Common mistakes
Grading only the final answer, so a retrieval bug gets "fixed" with three weeks of prompt edits.
Writing the test questions yourself. Teams write questions in the vocabulary of their own docs, which inflates recall.
Treating groundedness as correctness. A faithful summary of a stale article is a wrong answer.
Skipping the baseline. Leadership will ask whether the assistant beats the search box, and "it scores 88%" does not answer that.
See evaluating RAG and error analysis first for the full method.