TL;DR: Use RAG when the problem is knowledge: facts that change, answers that must cite sources, or data that differs by customer or user permission. Use fine-tuning when the problem is behaviour: a format, a style or a narrow skill that prompting cannot hold, or a high-volume task where a small tuned model can replace a large one. Start with prompting and RAG; fine-tune when a measured gap remains. Mature systems often use both.
The one-line distinction
RAG changes what the model sees at answer time. Fine-tuning changes how the model behaves by default.
An open-book exam versus studying for a closed-book one: RAG hands the model the relevant pages every time; fine-tuning has the model practise until its habits change, with nothing to look up during the exam.
Decision table
| Requirement | Favours | Why |
|---|---|---|
| Information changes often | RAG | Update documents, not models |
| Answers must cite sources | RAG | Retrieved passages are the citations |
| Different users see different data | RAG | Filter by permission before retrieval |
| Very large knowledge base | RAG | Too much to train in reliably |
| Strict output format or schema | Structured output plus validation first | Fine-tuning may improve adherence but cannot enforce authorization or truth |
| Specific brand voice | Prompt examples first; tuning if a measured gap remains | Compare against a written style rubric |
| Narrow task at very high volume | Fine-tuning a small model | Cost and latency |
| Domain jargon understanding | Either, test both | Retrieval of glossaries or tuning on domain text |
| Small supplied document set | Prompt with permitted context | Avoid an index until selection is useful |
Why fine-tuning is a poor knowledge store
Teams often reach for fine-tuning with the intuition that training a model "on our documents" will make it know the business. In practice:
- Facts learned through fine-tuning are recalled unreliably, mixed with general knowledge, and the model cannot say where it learned them.
- A knowledge change will not automatically alter the weights. You can supply the updated fact through retrieval or a tool; retrain only when the intended parameter-level behavior warrants it.
- Learned content has no built-in per-record access-control boundary. Do not rely on a shared model's prompt to isolate confidential training examples between customers.
- Hallucination does not go away; the model may confidently blend trained facts with invented ones.
Why RAG alone is sometimes not enough
RAG supplies facts but does not change behaviour much beyond what prompting achieves. If the model consistently formats answers wrongly, ignores a required style, or is too slow and expensive at volume for a narrow task, fine-tuning addresses that. Structured-output modes have also reduced the need to fine-tune purely for format.
Combining them
A common mature pattern: retrieval supplies current, permissioned facts; a fine-tuned model (often smaller and cheaper) is better at using retrieved passages in the required format and voice, citing correctly and abstaining when evidence is missing. The order of work matters:
The CEO explanation
"We'll give the model the right pages to read every time it answers, so it stays current, cites its sources and respects who can see what. We'll only retrain the model itself if we find a behaviour we cannot fix any other way, or if a smaller trained model can do a high-volume job much more cheaply."
Cost comparison, roughly
RAG costs: ingestion pipeline, a vector and keyword index, and more input tokens per request (the retrieved passages). Fine-tuning costs: data preparation, training, evaluation, and ongoing retraining as content or base models change; but potentially lower per-request cost if a small tuned model replaces a large one. Compare complete alternatives for the workload. A small supplied corpus may need no retrieval service; at high volume, a tuned small model can reduce serving cost if it preserves the required behavior.
In the interview
For a RAG-or-fine-tuning prompt, a strong answer leads with knowledge versus behaviour, gives the business reasons (freshness, citations, permissions), names when fine-tuning wins, recommends starting with prompting and RAG, and notes that both are often combined. Delivering it as a clear analogy plus a one-sentence recommendation shows communication skill as well as understanding. The RAG vs fine-tuning decision tool walks through the same logic.
Next: RAG explained · fine-tuning and LoRA