RAG vs fine-tuning: retrieval supplies knowledge, tuning shapes behaviour
Retrieval-augmented generation (RAG) gives a model the facts it needs at query time by fetching them from a company's own documents and data, while fine-tuning retrains a model's weights on examples so it adopts a format, style or narrow skill; RAG suits knowledge that changes, and fine-tuning suits behaviour that does not.
Checked
Both approaches answer the same complaint: the model does not know the business. They fix different parts of it. RAG changes what the model sees for each request. Fine-tuning changes the model itself.
Most teams reach for fine-tuning too early. In practice the order is: put the right context in front of a capable model, add retrieval when the knowledge is too large or changes too often, and tune only when evals show a gap that context cannot close.
What each one changes
RAG
Documents and records are indexed. At query time, the most relevant passages are retrieved and placed in the prompt, so the answer can cite them. Updating knowledge means updating the index.
Fine-tuning
A model is trained further on example inputs and outputs. It learns a format, tone or narrow task. Updating it means preparing data and running another training job.
When retrieval fits
Retrieval fits knowledge that changes, knowledge that must be cited, and knowledge that different users are allowed to see differently. Permissions can be applied at retrieval time, and a deleted document stops appearing as soon as the index updates.
Retrieval quality is a tuning problem of its own. In Anthropic's Contextual Retrieval study, contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 keyword search cut them by 49%, and adding a reranker cut them by 67%. The same study notes that a knowledge base under about 200,000 tokens can go straight into the prompt, with prompt caching, and skip retrieval altogether.
When fine-tuning earns its place
A fixed output format at volume
Extraction or classification that must match a strict schema thousands of times a day.
A smaller, cheaper model
OpenAI's guide advises prototyping on the most capable model, setting an eval baseline, then trying smaller models. Tuning is one way to bring a smaller model up to that baseline.
Style that prompts cannot hold
A tone or domain convention that examples in the prompt do not reproduce reliably.
How Motiontic decides
Motiontic's engineers build an eval set from the client's own questions and documents first, then try context, retrieval and tuning in that order, keeping whichever closes the gap at the lower running cost. Long prompts are not a free fallback: Chroma's Context Rot study found that reliability drops as input length grows, across 18 models.
RAG compared with fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| What changes | What the model sees per request | The model's weights |
| Fresh knowledge | Update the index | Retrain |
| Citations | Answers can point to the source passage | No built-in link to a source |
| Access control | Applied per user at retrieval time | Whatever was trained in is available to everyone |
| Upfront work | Indexing, chunking, search tuning | Labelled examples and training runs |
| Running cost | Longer prompts per request | Shorter prompts, possibly a smaller model |
| Typical failure | The right passage is not retrieved | Confident answers from outdated training data |
Common questions
What is the difference between RAG and fine-tuning?
RAG retrieves relevant documents at query time and puts them in the prompt. Fine-tuning retrains the model on examples. RAG changes what the model knows for one request; fine-tuning changes how the model behaves on every request.
Can fine-tuning teach a model new facts?
It can embed some, but it is a weak way to manage knowledge that changes. Every update needs another training run, access cannot be limited per user, and the model cannot point to where an answer came from.
Can RAG and fine-tuning be combined?
Yes. A tuned model can handle format and tone while retrieval supplies current facts. Most systems should prove retrieval works first, then tune only if evals show a remaining gap.
When is the context window enough without RAG?
Anthropic's Contextual Retrieval study notes that a knowledge base under about 200,000 tokens can go straight into the prompt, with prompt caching to cut cost. Above that, or when content changes often, retrieval is needed.
Is agentic search replacing RAG?
For some corpora. Coding agents such as Claude Code search files directly instead of using a pre-built index. For large, conceptual or multi-tenant document sets, indexed hybrid search with a reranker remains the stronger option, often called by the agent as a tool.
Related
RAG vs Agentic Search
One retrieval before the answer, compared with an agent that searches as it works.
Context Engineering vs Prompt Engineering
Wording the instructions, compared with curating every token the model sees.
Prompt caching
Paying less for the part of the prompt that does not change between calls.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.