---
title: 'Fine-tuning vs RAG: how to choose, and when a better prompt wins'
description: >-
  RAG fixes missing facts, fine-tuning fixes behaviour. We tested retrieval on
  our own blog and found where it really fails: the right page, the wrong
  paragraph.
author: elroot_labs
date: '2026-11-13'
categories:
  - artificial-intelligence
---
Use retrieval-augmented generation (RAG) when the model gets answers wrong because it is missing facts. Use fine-tuning when it has the facts and still answers in the wrong shape. Try a better prompt before either. That is the standard advice, and the research behind it is solid. What the standard advice skips is where RAG itself breaks, so we tested it on the one set of documents we know best, our own blog. Retrieval found the right post almost every time and the right paragraph only about half the time. If your assistant gives wrong answers, check that before you spend money on fine-tuning.

## The rule, briefly

Three sources agree on the basic split.

A comparison of knowledge-injection methods found that RAG outperformed unsupervised fine-tuning "both for existing knowledge encountered during training and entirely new knowledge" ([Ovadia et al., 2023](https://arxiv.org/abs/2312.05934)). A second study found that models "struggle to acquire new factual knowledge through fine-tuning," and that as they eventually learn new facts that way, "they linearly increase the model's tendency to hallucinate" ([Gekhman et al., 2024](https://arxiv.org/abs/2405.05904)). So facts belong in retrieval.

OpenAI's [accuracy guide](https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) frames the same split as two levers: context for missing knowledge, and fine-tuning for behaviour, which it says "maximizes consistency of behavior." It recommends starting with a good prompt and an evaluation set before either.

All of that assumes retrieval puts the right text in front of the model. We wanted to see how often it does.

## We tested retrieval on our own blog

The setup was deliberately simple. We took the 14 posts live on this site, split them into 451 paragraphs, and wrote 24 questions a reader might ask, each with the section of a post that answers it. We wrote the questions before running anything, half using the posts' own vocabulary and half phrased the way a buyer would actually ask. Then we ran BM25, the classic keyword-ranking method behind many search boxes, in about a hundred lines of standard-library Python.

| Method                                                               | Right section first | Right section in top 3 | Right post in top 3 |
| -------------------------------------------------------------------- | ------------------- | ---------------------- | ------------------- |
| Plain keyword search                                                 | 10 of 24            | 13 of 24               | 21 of 24            |
| Same search, each paragraph labelled with its post title and heading | 14 of 24            | 14 of 24               | 23 of 24            |

![Bar chart of a retrieval test for RAG: labelling each paragraph with its post title and heading raised right-section-first hits from 10 to 14 of 24 questions and right-post-in-top-3 hits from 21 to 23.](/images/blog/elroot/figures/keyword-retrieval-plain-vs-labelled-paragraphs.png)

It is a small test on one corpus with one method, and the numbers will differ for your documents. The pattern in the last two columns is the useful part. Finding the right document was easy. Finding the paragraph that actually answers the question was not, and the model only ever sees the paragraph.

## The misses that matter point the wrong way

Most misses landed in the right post, one or two sections away from the answer. A few were worse, and they are worth reading closely.

"Should I fine-tune to teach the model our internal policies?" The right answer is in the section that says to use RAG for missing facts. Search returned the section titled "Fine-tune when the model knows enough but behaves inconsistently", because the question and that heading share the word that matters most to a keyword ranker. A model given that paragraph would likely recommend fine-tuning, the opposite of the post's advice, and it would sound well sourced while doing it.

"How do I stop the vendor from charging more when the work grows?" The answer is in our contracts post, under time and materials, where it says to agree a ceiling. The question never uses "ceiling", "scope" or "time and materials", so plain search returned a paragraph about AI context platforms, and the labelled version returned one about cloud outages.

These are the failures an evaluation set has to catch. An assistant built on our blog would answer both questions confidently and wrongly, and no amount of fine-tuning would fix it, because the problem happened before the model saw anything.

## Labelling chunks helped the first result, not the top three

Adding each paragraph's post title and section heading to the text it is indexed under is a crude, no-model version of what Anthropic calls contextual retrieval. Their version uses a model to write a short, document-aware description for each chunk, and they report it cut the top-20 retrieval failure rate by 49% when combined with keyword search, and by 67% with reranking ([contextual retrieval](https://www.anthropic.com/news/contextual-retrieval)).

Our cheap version moved the right section into first place for four more questions and barely changed the top three. That fits the idea: labels help when the heading carries the meaning, and do nothing for a question phrased in words the document never uses. For that, you need either better retrieval, such as embeddings or reranking, or documents written with the reader's words in them.

## What this changes about the fine-tuning decision

Sort every wrong answer from your evaluation set into one of three groups before choosing a fix:

1. The answer was never available to the model. Add it to the prompt if it is small, or to retrieval if it is large or changes often.
1. The right passage was in front of the model, and it still answered in the wrong format, tone or order. This is the case for examples in the prompt first, then fine-tuning.
1. The answer existed in your documents, and retrieval did not deliver it. Fix retrieval. Changing the model will not help, and fine-tuning on facts risks the hallucination effect the research describes.

The third group is the one teams skip, because spotting it requires logging what retrieval returned for each question alongside the final answer. Log both. In our small test, it was the largest group.

OpenAI's guide adds a warning that applies in reverse: in one of its case studies, adding retrieval to a behaviour problem "actually confounded the model by giving it extra noise." More passages do not fix the wrong shape.

## Run the same test on your own documents

You do not need a model or a vendor to start:

1. Write 20 to 30 questions real users ask, and for each, note the exact passage that answers it.
1. Split your documents the way your system will, and index them with the retrieval you plan to use.
1. For each question, record whether the right passage appears first and in the top three.
1. Read every miss. Group them: wrong post, right post but wrong section, or a section that points to the wrong answer.
1. Only then decide between better retrieval, a better prompt, and fine-tuning.

Our guide to [building an evaluation set from your own failure cases](/blog/agent-evaluation-test-set) covers how to collect the questions.

## Frequently asked questions

### Should I fine-tune a model to teach it our company's information?

Usually not. A [2024 study](https://arxiv.org/abs/2405.05904) found that models learn new facts slowly through fine-tuning and become more prone to hallucinate as they do. Put the information in the prompt or in retrieval, and fine-tune only for consistent format or behaviour.

### Why does RAG still give wrong answers?

Often because retrieval returns the right document and the wrong passage. In our test on our own blog, keyword search found the right post for 21 of 24 questions and the right section for only 13.

### When is fine-tuning worth it?

When the model has the right information in front of it and still answers in the wrong format, tone or order across many cases, and examples in the prompt have not fixed it.

If you are deciding how to adapt a model to your own documents, we start with an evaluation set like the one above; you can [tell us about your use case](/contact).
