Skip to content

LLM structured output: no broken JSON in 80 calls, eight wrong answers

18 Nov 2026

·

8 minute read

Getting an LLM to return valid JSON is no longer the hard part. In our test of 80 extraction calls on two current OpenAI models, every reply was bare, parseable JSON that matched the schema, whether or not the schema was enforced. The eight mistakes we found, out of 480 extracted values, were all values that broke the rules we had written: a duration converted wrongly, a budget kept when the rule said to drop it, an ambiguous request put in the wrong category. LLM structured output features guarantee the shape. Your effort belongs on the values, and the most reliable way to check those is in code.

The test: 20 messy enquiries, two models, with and without a schema

We wrote 20 short project enquiries of the kind a software studio receives through its contact form, and the correct answer for each, before running anything. They are invented, and several carry traps we have seen in real messages: budgets in lakh and crore, one in dollars, one with no currency at all, a day rate where a total belongs, a calendar date where a duration belongs, "two quarters", "not urgent" and "bit of a rush".

The model had to extract six fields. Name and company are strings or null. Service is one of five values: ai_agent, web_app, mobile_app, hardware_iot or other. The budget is an integer number of rupees or null, timeline_weeks is an integer or null, and urgent is true or false. The prompt spelled out the rules for each field, for example that a range uses its lower bound, a budget with no currency is null, and a month counts as four weeks.

We ran each enquiry through two models, gpt-5.6-luna and gpt-5.6-terra, in two ways. The first used the prompt alone, ending with an instruction to reply with only a JSON object. The second used the identical prompt and also enforced a strict JSON Schema, with every key required, no extra keys allowed and the service field limited to its five values. The calls went through the Codex command-line tool with reasoning effort set to low, so they carry Codex's own system prompt and are not raw API calls. That makes 80 calls, all of which completed, at a median of about nine seconds each.

A script graded every reply in three steps: does it parse as JSON, does it match the schema, and is each of the six values right?

Every reply parsed, with or without enforcement

ModelMethodValid JSONMatched the schemaAll six fields right
gpt-5.6-lunaprompt only20/2020/2016/20
gpt-5.6-lunaschema enforced20/2020/2017/20
gpt-5.6-terraprompt only20/2020/2020/20
gpt-5.6-terraschema enforced20/2020/2019/20

Bar chart of an LLM structured output test: all six fields correct in 16 of 20 runs for gpt-5.6-luna with prompt only and 17 with the schema enforced, and 20 and 19 for gpt-5.6-terra.

The failure most guides warn about, a reply wrapped in a Markdown code fence or preceded by "Here is the JSON:", did not happen once in the 40 prompt-only calls. Neither did a missing key, an extra key or a number returned as a string.

That result is specific to these models, this prompt and this kind of task. Older models, smaller open models and other providers can behave differently, and that is the real case for enforcement: when you do not control which model your code will meet next year, a guarantee about shape is worth having. But if you have been adding retry loops and regular expressions to strip code fences, measure how often you need them before you build more of them.

The mistakes were all in the values

Here is every error the script found, with the enquiry text that caused it.

  1. "Need someone to fix it this week." Expected a timeline of 1 week. Luna returned null both times, reading "this week" as a date rather than a duration.
  2. "Timeline two quarters." Expected 24 weeks. Luna, prompt only, returned 12, converting two quarters into weeks wrongly.
  3. "6 months" on a request for two contract developers. Expected 24 weeks. Luna returned null both times, reading the contract length as something other than a project timeline.
  4. "50k budget." with no currency. The rule says null. Luna, schema enforced, returned 50000.
  5. "Maybe with cameras or some AI?" Expected other, since the sender does not know what they need. Luna, prompt only, and terra, schema enforced, both chose ai_agent.

Three of these are arguable. "This week" as one week and a six-month contract as a 24-week timeline were our own calls when we wrote the answers, and a reasonable person could have written null. That is a finding in itself: the hardest values to extract were the ones where the rules needed judgement, and no schema can express judgement.

OpenAI's documentation says as much. Its structured outputs guide states plainly that structured outputs "can still contain mistakes", and that an application still needs to handle refusals, incomplete responses and errors, and validate any business rules that the schema does not express.

Enforcement cost nothing here, but it can elsewhere

Enforcing the schema made no measurable difference to accuracy in our test: luna went from 16 fully correct records to 17 and terra from 20 to 19, a difference of one record each way on 20 inputs, which is noise.

That is worth knowing because the opposite has been reported. A study titled Let Me Speak Freely? found a significant decline in reasoning ability when models were restricted to structured formats, and found that stricter constraints generally hurt more. Extraction is light on reasoning, which may be why we saw no effect. If your task needs the model to work something out before it answers, test enforced and free-form output side by side, or let the model reason in free text first and produce the structured record in a second step.

Move the hard rules out of the prompt and into code

Look at the errors again. Converting quarters to weeks is arithmetic. "No currency means null" is a rule with no ambiguity. Deciding whether "this week" is a duration is a policy choice. A model can do all three, and in our test it usually did, but code does the first two without fail and states the third where anyone can review it.

So the design we now use for extraction asks the model for what it is good at, finding the relevant words, and leaves the decisions to code:

  • Ask for the raw text of each value as well as the value: the budget phrase, its currency word and the timeline phrase, exactly as the sender wrote them.
  • Convert units in code. Lakh, crore, quarters and months are lookup tables.
  • Apply the business rules in code: a budget without a recognised rupee marker becomes null, and a timeline phrase that names a date becomes null.
  • Keep an explicit "unclear" option for categories, and route those records to a person instead of forcing a guess.

In this test, that design would have taken four of the five error types out of the model's hands: the quarters arithmetic, the missing currency, and both timeline readings, which become a rule in code that someone chose on purpose. The ambiguous service request would still need a human, and would now reach one.

How to test extraction on your own inputs

Twenty inputs and an afternoon are enough to find out where your extraction fails. This is the method we used, and the full prompt, schema, inputs and results are in our evidence file.

  1. Collect 20 to 50 real inputs, and add the awkward cases you know about: other currencies, missing values, dates where you expect durations, people who contradict themselves.
  2. Write the expected value for every field before the first run.
  3. Run each input with and without schema enforcement, on the model you will ship with.
  4. Grade format and values separately: parse failures, schema failures, then each field.
  5. Read every wrong value and ask whether code could have decided it.

The same discipline applies to agents that act on what they extract, where a wrong value turns into a wrong action. We cover building that kind of test set in how to build an evaluation set from your own failure cases.

AI tools

Ask ChatGPTAsk Claude

Start with the problem that matters