Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: RAG looks up the relevant documents at question time and gives them to the model to answer from; fine-tuning changes the model's behaviour by training it on examples. For a changing, permissioned document collection, start by testing retrieval: it can supply current sources, while the application enforces access and checks whether citations support the answer. Fine-tuning is for changing how the model behaves (format, tone, a narrow skill), and is often combined with RAG later.
How to approach it
Lead with an analogy the CEO will remember, then the decision, then the two or three reasons that matter to the business. Keep jargon out until asked. The interviewer will likely follow up with "when would you fine-tune?", so have a clear answer ready.
A strong answer
The analogy. RAG is an open-book exam: the model gets the relevant pages in front of it each time it answers. Fine-tuning is studying for a closed-book exam: the model practises on examples until it behaves differently, but it cannot look anything up during the test.
What that means for us. We want an assistant that answers questions from our help docs, policies and product information. Four business requirements decide the choice:
| Requirement | RAG | Fine-tuning |
|---|---|---|
| Information changes weekly | Re-ingest updates and invalidate affected caches; measure lag | Weight-based recall does not update automatically; retrieval can still supply fresh facts |
| Answers must show where they came from | Each answer can cite the document it used | The model cannot reliably tell you where it learned something |
| Each customer sees only their own documents | Enforce user scope before content reaches the model | Shared weights provide no per-record access-control boundary |
| Cost and time to start | Ingestion, access controls, retrieval and evaluation | Training data, training runs and evaluation; schedule from the actual scope |
So for knowledge, my recommendation is RAG. We would start with a prototype in a couple of weeks and measure it on real customer questions.
When fine-tuning is the right tool. When the problem is behaviour rather than knowledge: the model must improve a measured task behavior, such as a house style or a narrow classification task. For a strict parseable format, try structured output and validation first. Another candidate is lowering the cost of a narrow task, such as classifying ticket categories, with a smaller model. A fine-tuned small model can sometimes replace a large general model on one high-volume task at a fraction of the cost per request.
What the research says, for when the CEO's team pushes back. Ovadia et al. (Fine-Tuning or Retrieval?, 2023) compared the two for injecting knowledge and found that retrieval consistently outperformed unsupervised fine-tuning, both for facts the model had seen in pre-training and for entirely new ones. Gekhman et al. (EMNLP 2024) found that fine-tuning examples containing new facts are learned significantly more slowly than examples consistent with what the model already knows, and that as the model does learn them, its tendency to hallucinate rises linearly. "Train it on our documents" is therefore slow at teaching facts and can make the model less reliable. There is also a third option worth naming: Anthropic's contextual retrieval guidance (September 2024) says a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt with caching and skip retrieval entirely. Our help centre is past that size, which is why retrieval wins here.
Using both. Retrieval can supply current facts to a fine-tuned model. Whether tuning improves how it uses those facts must be measured. The order matters. Start with RAG and good prompting; fine-tune only when evaluation shows a specific behaviour gap that prompting cannot close.
What I would say in one sentence to the CEO: "We'll retrieve the current pages each customer may access, test whether the answer uses them correctly, and consider tuning if a measured behavior problem remains."
What interviewers probe next
"Doesn't fine-tuning make the model 'know' our business better?" It can change behavior and factual associations, but recall lacks automatic provenance and the facts can go stale. For facts, retrieval is more reliable and auditable.
"What could go wrong with RAG?" Retrieval can miss the right document, documents can be outdated, and the model can misread what it retrieved. That is why we evaluate retrieval and answers separately.
"Couldn't we just put all our documents into a huge context window?" For a small, stable set, sometimes. For a large or changing corpus with per-customer permissions, compare retrieval with scoped long context on cost and evidence quality. Both need access controls before generation.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Explanation | Jargon-heavy or wrong | Correct distinction | Clear analogy plus the business consequences of each approach |
| Recommendation | Fine-tune because "it learns our data" | RAG for knowledge | RAG for knowledge with freshness, citation and permission reasons, backed by the knowledge-injection and hallucination evidence; fine-tuning for behaviour; full context for small corpora |
| Communication | Lectures | Adapts to the audience | One-sentence summary for the CEO, decision tree for the team |
Common mistakes
Recommending fine-tuning to teach a model company facts.
Explaining embeddings and vector databases to a CEO who asked for a decision.
Presenting RAG and fine-tuning as mutually exclusive.
See RAG vs fine-tuning and try the RAG vs fine-tuning decision tool.