An agent crawls a partner directory and comes back with twelve records. Every one of them is a clean JSON object: company name, tier, country, website, a satisfaction percentage, a list of technologies. No trailing prose to strip, no half-closed brace, no "Here's the data you asked for!" preamble. The schema was passed to the model as a decoding constraint, so the output could not have come back malformed.
Eleven of those records are real. The twelfth has a company name that is half-invented, a tier that was never written anywhere on the page, and a satisfaction percentage that is entirely plausible and entirely made up. It syncs to Airtable next to the other eleven, formatted identically, and a salesperson works it on Thursday.
Constrained decoding did exactly what it promised. It guaranteed the shape. Nobody ever claimed it guaranteed the contents.
What structured output actually buys you
When you hand a JSON Schema to a model as a format constraint, you are restricting the token space at each decoding step so that only tokens consistent with the grammar can be emitted. That is genuinely useful. It removes an entire class of production failure — the parse error, the fenced code block, the model that decided this time it would return a markdown table.
Here is roughly the schema our partner extractor constrains against:
{
"type": "object",
"properties": {
"partners": {
"type": "array",
"items": {
"type": "object",
"properties": {
"companyName": { "type": "string" },
"websiteUrl": { "type": ["string", "null"] },
"partnerTier": { "type": ["string", "null"] },
"technologies": { "type": "array", "items": { "type": "string" } },
"evidence": {
"type": "array",
"items": {
"type": "object",
"properties": {
"field": { "type": "string" },
"quote": { "type": "string" }
},
"required": ["field", "quote"]
}
}
},
"required": ["companyName", "evidence"]
}
}
},
"required": ["partners"]
}Read that carefully and notice what it can and cannot enforce. It can enforce that partnerTier is a string or null. It cannot enforce that the string is a tier that appeared on the page. Schema validation is a statement about types, and hallucination is not a type error. A fabricated tier is a perfectly valid string.
So the cheap class of failure is gone and the expensive one is untouched — and worse, the output now looks more trustworthy than it did when it was messy. Well-formed data is data people stop reading.
The gate: every value carries a quote
The fix is to make the model pay for each claim. Alongside the extracted fields, it must emit an evidence array: one row per claim, each naming the field it supports and quoting the span of page text that supports it. Then the extractor — not the model — checks the quotes.
Two conditions have to hold before a value is allowed through:
- The quote is real. After whitespace and case normalization, the quote must appear as a literal substring of the source text we fetched. A quote the model composed rather than copied fails here.
- The quote actually contains the claim. The normalized quote must contain the complete normalized value being claimed, and the evidence row must name the same field. A real quote about something else supports nothing.
Condition one alone is the version most teams ship, and it is not enough. A model that quotes a genuine sentence from the page and then writes an unrelated value beside it passes a naive grounding check every time. Grounding has to be per-field, not per-record.
/**
* True when a grounded quote for `field` supports `value`.
* The quote must contain the complete normalized claimed value.
*/
export function valueSupportedByEvidence(value, field, evidence, sourceText) {
const v = canonicalizeEvidenceValue(value, field);
if (!v || v.length < 2) return false;
const fieldNorm = normalizeWhitespace(field);
return (evidence || []).some((e) => {
// The evidence row must be about THIS field.
if (normalizeWhitespace(e.field) !== fieldNorm) return false;
// The quote must literally exist in the page we fetched.
if (!quoteFoundInSource(e.quote, sourceText)) return false;
// The quote must contain the full claimed value — never the reverse.
return normalizeWhitespace(e.quote).includes(v);
});
}The boundary bug that made it pass anyway
Our first version of that last line was more generous, and the generosity is instructive. It read:
// Quote must mention the claimed value (or vice-versa for short quotes).
return q.includes(v) || (v.length >= 4 && v.includes(q));The second clause exists because of a reasonable-sounding worry: what if the model quotes tightly — just Gold Partner — while the value it claims is longer? Rather than lose a true positive, we accepted the reverse containment too.
Look at what that admits. Page text: IROKOO Gold partner in France. The model emits a quote of IROKOO Gold — genuinely present, condition one satisfied — and claims a company name of IROKOO Gold International. The forward check fails, because the quote does not contain the full claim. The reverse check passes, because the claim contains the quote. A company that does not exist inherits the credibility of a real substring of a real page.
That is the shape of nearly every grounding-check bug we have hit: the escape hatch added to avoid false rejections is exactly the hatch fabrication walks through. Half a grounded name plus an invented suffix is more dangerous than a wholly invented name, because it survives a human spot-check. Somebody skimming the sync sees IROKOO, recognizes it, and moves on.
We deleted the reverse clause and wrote the regression as a boundary test — partial prefix quote, longer claim, must be rejected. If your validator has a symmetric comparison anywhere in it, that is where to look first. Evidence is directional: the source contains the claim, never the other way around.
Normalization is where correctness leaks
Every normalization you apply before comparing makes the check more forgiving, and forgiveness is surface area. Collapsing non-breaking spaces and case-folding is nearly free — it recovers true positives lost to HTML noise without meaningfully widening what counts as a match.
URL comparison is where it gets tempting to go further. A page says https://irokoo.example/ and the model claims https://irokoo.example. Same resource, different string, and a strict substring check rejects a correct extraction over a trailing slash. So we canonicalize URLs — drop the fragment, drop default ports, lowercase the host, strip a bare trailing slash — but only for fields we know are URLs:
export function canonicalizeEvidenceValue(value, field) {
const raw = String(value || "").trim();
if (!raw) return "";
if (field === "websiteUrl" || field === "profileUrl") {
try {
const u = new URL(raw);
u.hash = "";
const path = u.pathname === "/" ? "" : u.pathname.replace(/\/$/, "");
return normalizeWhitespace(`${u.protocol}//${u.host}${path}${u.search}`);
} catch {
return normalizeWhitespace(raw.replace(/\/$/, ""));
}
}
return normalizeWhitespace(raw);
}The restriction to two named fields is the whole point. It would have been one line shorter to canonicalize everything, and it would have meant punctuation-stripping applied to company names — which is precisely how Acme, Inc. starts matching Acme Inc starts matching things that are not the same company. Each field type gets the loosest comparison that is still correct for that type, and no looser.
Reject the field, keep the record
An all-or-nothing gate does not survive contact with a real directory. If one unsupported technologies entry destroys an otherwise well-grounded partner record, the extractor's yield collapses and somebody turns the validation off. That is the actual failure mode — not bad data getting through, but a good check getting disabled.
So the gate degrades per field. Each scalar is tested independently and dropped if unsupported; each array is filtered down to its grounded members; ungrounded evidence rows are stripped from the record entirely so nothing downstream can re-derive a claim from them. Only one failure is fatal: if companyName itself is not grounded, there is no record to speak of and the whole item is rejected with a reason.
The errors do not vanish either — they accumulate alongside the results, per index, with the reason. A directory page that suddenly produces forty grounding rejections is telling you the site's markup changed, and that signal is worth as much as the extractions.
The page's own furniture is not data
One more failure that structured output makes worse rather than better. Directory pages are full of chrome that reads like content: filter labels, category pickers, sector dropdowns. A model asked for company names on a page whose sidebar lists Agriculture, Finance/Assurance, Construction et rénovation will occasionally hand you those as partners.
And they pass the evidence gate cleanly, because they are on the page. The quote is real. The value matches the quote. Grounding is satisfied — the extraction is faithful to the source and still wrong, because the source text at that location was navigation, not a record.
Grounding is a necessary condition, not a sufficient one. We keep a small explicit denylist of known chrome labels for the directories we crawl, checked before the evidence gate runs. It is not elegant and it does not generalize, and both of those are acceptable for a check that costs a regex and catches a category of nonsense no amount of prompt engineering reliably prevents.
This is a different question from provenance
It is worth separating two things that sound similar. A provenance trail answers how was this produced — which query ran, against which tables, at which freshness. It is what you need when a number is right and nobody can explain why.
Evidence grounding answers a blunter question: did the source ever say this at all. It applies where there is no query to trace, because the input was unstructured text that a model read. You need both, and neither substitutes for the other. A perfect provenance trail around an extraction step tells you which page was fetched and which model was called — and tells you nothing about whether the tier on that record was in the HTML.
The test to run on your own extractor
Take any record your pipeline produced this week and go field by field. For each value, can you point at the exact substring of the source it came from? Not the page it came from — the substring.
Where you can't, you do not have an extraction. You have a guess with good grammar and a valid schema, and the schema is what made it look like the others.