Engineering depth

RAG or Fine-Tuning: How to Decide

Use retrieval-augmented generation (RAG) when the model needs facts that change or belong to you. Use fine-tuning when you need a consistent style or behaviour that prompts alone will not produce. Most products should start with RAG.

By Cloud Technopix · Published 3 October 2026 · 11 min read

Two different problems

Retrieval-augmented generation and fine-tuning are often presented as rivals, but they solve different problems. RAG changes what the model can see at the moment it answers: relevant documents are fetched and placed in the prompt. Fine-tuning changes the model itself, by training it further on examples so that it behaves differently. Choosing well starts with asking which problem you actually have.

What RAG does well

RAG is the right tool when the answer depends on information the base model does not have or that changes over time: your product documentation, contracts, policies, tickets, prices, or a customer's own records. Because the facts are retrieved at question time, updating them means updating the index, not retraining anything.

  • Fresh data: new documents are available as soon as they are indexed
  • Traceability: each answer can cite the passages it used, so a person can check
  • Access control: retrieval can respect who is allowed to see what
  • Lower cost to start: no training run, and easy to swap the underlying model

What fine-tuning does well

Fine-tuning shines when you need consistent behaviour rather than new facts: a particular tone, a strict output format, a classification scheme, a specialised task that prompts struggle to teach, or lower latency and cost from a smaller model trained to do one job. It is a poor way to teach a model facts, because knowledge baked into weights is hard to update, hard to cite and easy to get subtly wrong.

A first-pass decision table. Real systems often use both.
QuestionPoints to RAGPoints to fine-tuning
Does the answer depend on your private or changing data?YesNo
Do you need citations or an audit trail?YesHard to provide
Do you need a fixed style or output format?Prompting may be enoughOften yes
Is the task narrow, repeated and high volume?PossibleOften yes
Must knowledge update daily?YesNo

A diagram of a typical RAG pipeline

RAG and agent pipelineDocuments are ingested, embedded and indexed in Postgres with pgvector, relevant passages are retrieved with access rules applied, an agent reasons with tools, and a person approves irreversible actions.Ingestaccess kept01Embed, indexpgvector02Retrievecited03Reasontools, steps04Approvehuman gate05
Documents are ingested, embedded and indexed, retrieved with access rules applied, and used by an agent that a person approves for irreversible steps.

Why most products should start with RAG

Almost every business application has some private context: the customer, the account, the document, the policy. RAG gives the model that context cheaply and keeps the facts inspectable. It also fails more visibly. When retrieval returns the wrong passage, you can see it in the logs and fix the chunking, the index or the query, which is much easier than diagnosing a changed model.

Start with a strong base model, a clear prompt and good retrieval. Only when you have measured a shortfall that retrieval and prompting cannot fix should you consider training. That order protects you from spending weeks on a fine-tune that a better prompt would have matched.

Making RAG work in practice

Chunk with the question in mind

Split documents along their natural structure, such as sections, clauses or table rows, rather than at arbitrary lengths. Keep titles and headings with their text so a retrieved passage still makes sense out of context.

Combine search methods

Vector similarity finds passages with similar meaning, and keyword search finds exact terms such as product codes and names. Using both, and re-ranking the results, usually beats either alone. Storing embeddings next to your relational data, for example with Postgres and pgvector, also makes permission filters straightforward.

Enforce access at retrieval time

Filter by the user's permissions before passages reach the model. Telling the model "do not reveal confidential content" is not a control. If a passage is retrieved, assume it can appear in the answer.

Cite, and say when you do not know

Show the source passages with each answer and design the prompt so the model says when the retrieved material does not contain the answer. An honest "not found" builds more trust than a confident guess.

How to test before you commit

  1. Collect 30 to 100 real questions from the people who will use the system, with the answers you would accept.
  2. Run them against a base model with a good prompt and record the failures.
  3. Add retrieval, run the same set, and compare. Look at which failures are about missing facts and which are about behaviour.
  4. Only if the remaining failures are about style, format or a narrow task, test a fine-tune on a held-out set before adopting it.

Keep that test set. It becomes your regression suite when you change models, prompts or the index, which you will.

Using both together

The approaches combine well. A small fine-tuned model can handle a narrow step such as routing or extraction cheaply, while RAG supplies the facts for the answer. The point is to match each tool to the problem it solves rather than treating the choice as a contest.

We build retrieval and agent systems as part of our AI agents and RAG development work, and we use AI features in our own products, such as the itinerary and receipt features in SplitMitra. If you are deciding between approaches for your product, contact us and we will help you design the test before you spend on the build.

Can RAG remove hallucinations completely?

No. It reduces them by grounding answers in retrieved text, but the model can still misread a passage or answer beyond it. Citations, a not-found behaviour and a person approving consequential actions are the practical safeguards.

Is a bigger context window a replacement for RAG?

Not usually. Long contexts help, but cost and latency grow with input size, and you still need to decide what to put in. Retrieval is how you choose it.