To measure an LLM's hallucination rate on your own documents, ask it questions your documents do not answer, as well as ones they do. In our test on passages from this blog, two current models answered every answerable question correctly. Given a plain prompt and a question the passage did not cover, they still answered six times out of nine, stating facts that were not in the text. We checked those six: three were true, one was wrong, and two we could not confirm, and nothing in the wording of the answers separated one kind from another. Adding one instruction line to the prompt turned all ten unanswerable questions in that condition into a clean "not in passage", with no loss on the answerable ones.
Most hallucination tests ask only questions the documents can answer
A team building an assistant over its own documents usually tests it the obvious way: write questions, check the answers. If the answers are right, the hallucination rate looks like zero. That test misses the case that causes the damage, which is the question your documents cannot answer. It happens whenever retrieval fetches the wrong passage, whenever a user asks about something you never wrote down, and whenever the honest answer is "we don't say".
A model that has the answer in front of it rarely invents one. The useful measurement is what it does when it does not. So for every passage in our test we wrote a pair of questions before running anything: one the passage answers, with the expected answer noted, and one on the same subject that it does not answer. The second question is deliberately plausible. "What registration fee does the scheme charge?" sits naturally next to a paragraph about that registration scheme, and the paragraph says nothing about fees.
What we ran, and what we could not finish
The passages are paragraphs from posts already published on this site: an Indian electronics registration scheme, crystal placement on a circuit board, an AI agent's refund incident, a research paper on context compaction, and a measurement of free memory on our own laptop. We used each one in two prompts. The plain prompt gave the passage and the question and asked for a one-sentence answer. The grounded prompt added "using only the passage. If the passage does not contain the answer, reply exactly: NOT IN PASSAGE".
The models were gpt-5.6-luna and gpt-5.6-terra, called through OpenAI's Codex command-line tool with low reasoning effort. Codex wraps each call in its own system prompt and can search the web, which turned out to matter.
We planned 104 calls across eleven passages, plus a set of questions with no passage at all. The account reached its usage limit part-way through, so this post reports the 42 calls that completed: every planned call for five passages except one, and a few for a sixth. That is a small sample, and we say so wherever it matters. The complete inputs, prompts and outputs are in our evidence file.
Asked something the passage did not say, the models usually answered anyway
All 23 answerable questions came back right across both models and both prompts. The unanswerable ones are where the two prompts parted company.
| Prompt | Answerable questions right | Unanswerable questions declined | Unanswerable questions answered from elsewhere |
|---|---|---|---|
| Plain | 12/12 | 3/9 | 6/9 |
| Grounded | 11/11 | 10/10 | 0/10 |

With the plain prompt, the models declined to give a refund amount the passage never mentioned, and one declined to invent a speed figure for a test that measured memory. On the other three questions they answered confidently. We checked each answer against a primary source.
Both models said the context-compaction research was evaluated on Qwen3-8B and Qwen3-32B. That is true: the paper's abstract says exactly that. The passage they were given does not mention models at all.
Asked what crystal load capacitance Espressif recommends, terra said to take it from the crystal's datasheet and account for stray capacitance on the board, which matches Espressif's schematic checklist. Luna said Espressif recommends a crystal with a 10 pF load capacitance. The checklist gives no fixed value and points to the crystal's datasheet, so that answer is unsupported, and someone ordering parts from it could pick the wrong crystal.
Asked about the registration fee, both models gave the same answer: an annual fee of 1,000 rupees per product category. Terra attached a link to a fee schedule on the regulator's registration portal that it had found by searching. We opened that schedule, a PDF on the Bureau of Indian Standards' CRS portal, and it lists separate processing, per-test-report, inclusion and renewal fees over a two-year validity. A single annual fee per category is at best a simplification of it, and we could not confirm the figure as the models stated it.
So the plain-prompt answers from outside the passage split three ways: true, wrong and unconfirmed. All of them were written in the same confident one-sentence style. A reader, or a downstream system, has no way to tell which is which from the answer.
An answer from outside your documents is a defect even when it is true
It is tempting to score the true answers as wins. In a product built on your own documents they are still defects, for three reasons. You cannot show a source for them, so a user who asks "where does it say that?" gets nothing. You cannot audit them, because the next answer from the same habit may be the wrong one, as the crystal question showed. And in a regulated setting, a correct figure from an unknown source is still a figure your organisation has now stated.
The web search in Codex makes this sharper than it would be in many products, since the model could go and find things. A model without search draws on what it absorbed in training, which is older and harder to trace. Either way, the behaviour you are measuring is the same: does the model stay inside the material you gave it?
One instruction line moved six outside answers to zero
The grounded prompt differs from the plain one by a single sentence. With it, all ten unanswerable questions came back as NOT IN PASSAGE, and all eleven answerable questions in that condition were still answered correctly. On this sample, the instruction cost nothing in false refusals, which is the usual worry about prompts that encourage a model to decline.
Treat that as a starting point, not a guarantee. Ten questions is a small number, the passages were short and clean, and longer or messier context makes staying inside it harder. An instruction changes the odds and leaves the rest to measurement, which is why the test below is worth keeping and rerunning.
How to measure hallucination rate on your own documents
- Take 20 or more passages your system will actually retrieve. Prefer ones with specific facts: numbers, names, dates, conditions.
- For each, write one question it answers, with the expected answer, and one plausible question on the same subject that it does not answer. Write them all before the first run.
- Run both through the exact prompt and model you ship. If you have a grounding instruction, also run without it once, so you know what it is buying you.
- Grade answerable questions on whether the expected fact appears. Grade unanswerable ones on whether the model declines. Count those as two separate numbers: accuracy, and abstention when the answer is absent.
- Read every answer that came from outside the passage and check it against a primary source. Note which were true. The point is to know how often the system goes outside your material, whatever the outcome.
- Rerun after every change to the prompt, the model or the retrieval, and add the failures you see in production to the set.
Once this works passage by passage, put retrieval back in the loop and ask the same questions end to end, because a retrieval miss turns an answerable question into an unanswerable one. Our test of RAG chunking strategies uses the same kind of question set to measure that half, and building an evaluation set from your own failure cases covers keeping it alive after launch.
