85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Compare

RAG vs fine-tuning: retrieval supplies knowledge, tuning shapes behaviour

Retrieval-augmented generation (RAG) gives a model the facts it needs at query time by fetching them from a company's own documents and data, while fine-tuning retrains a model's weights on examples so it adopts a format, style or narrow skill; RAG suits knowledge that changes, and fine-tuning suits behaviour that does not.

Checked

Both approaches answer the same complaint: the model does not know the business. They fix different parts of it. RAG changes what the model sees for each request. Fine-tuning changes the model itself.

Most teams reach for fine-tuning too early. In practice the order is: put the right context in front of a capable model, add retrieval when the knowledge is too large or changes too often, and tune only when evals show a gap that context cannot close.

What each one changes

  • RAG

    Documents and records are indexed. At query time, the most relevant passages are retrieved and placed in the prompt, so the answer can cite them. Updating knowledge means updating the index.

  • Fine-tuning

    A model is trained further on example inputs and outputs. It learns a format, tone or narrow task. Updating it means preparing data and running another training job.

When retrieval fits

Retrieval fits knowledge that changes, knowledge that must be cited, and knowledge that different users are allowed to see differently. Permissions can be applied at retrieval time, and a deleted document stops appearing as soon as the index updates.

Retrieval quality is a tuning problem of its own. In Anthropic's Contextual Retrieval study, contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 keyword search cut them by 49%, and adding a reranker cut them by 67%. The same study notes that a knowledge base under about 200,000 tokens can go straight into the prompt, with prompt caching, and skip retrieval altogether.

When fine-tuning earns its place

  • A fixed output format at volume

    Extraction or classification that must match a strict schema thousands of times a day.

  • A smaller, cheaper model

    OpenAI's guide advises prototyping on the most capable model, setting an eval baseline, then trying smaller models. Tuning is one way to bring a smaller model up to that baseline.

  • Style that prompts cannot hold

    A tone or domain convention that examples in the prompt do not reproduce reliably.

How Motiontic decides

Motiontic's engineers build an eval set from the client's own questions and documents first, then try context, retrieval and tuning in that order, keeping whichever closes the gap at the lower running cost. Long prompts are not a free fallback: Chroma's Context Rot study found that reliability drops as input length grows, across 18 models.

RAG compared with fine-tuning

RAG compared with fine-tuning
RAGFine-tuning
What changesWhat the model sees per requestThe model's weights
Fresh knowledgeUpdate the indexRetrain
CitationsAnswers can point to the source passageNo built-in link to a source
Access controlApplied per user at retrieval timeWhatever was trained in is available to everyone
Upfront workIndexing, chunking, search tuningLabelled examples and training runs
Running costLonger prompts per requestShorter prompts, possibly a smaller model
Typical failureThe right passage is not retrievedConfident answers from outdated training data

Common questions

What is the difference between RAG and fine-tuning?

RAG retrieves relevant documents at query time and puts them in the prompt. Fine-tuning retrains the model on examples. RAG changes what the model knows for one request; fine-tuning changes how the model behaves on every request.

Can fine-tuning teach a model new facts?

It can embed some, but it is a weak way to manage knowledge that changes. Every update needs another training run, access cannot be limited per user, and the model cannot point to where an answer came from.

Can RAG and fine-tuning be combined?

Yes. A tuned model can handle format and tone while retrieval supplies current facts. Most systems should prove retrieval works first, then tune only if evals show a remaining gap.

When is the context window enough without RAG?

Anthropic's Contextual Retrieval study notes that a knowledge base under about 200,000 tokens can go straight into the prompt, with prompt caching to cut cost. Above that, or when content changes often, retrieval is needed.

Is agentic search replacing RAG?

For some corpora. Coding agents such as Claude Code search files directly instead of using a pre-built index. For large, conceptual or multi-tenant document sets, indexed hybrid search with a reranker remains the stronger option, often called by the agent as a tool.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.