Structured extraction: the validator is the product

Week 6 of the fundamentals series: structured extraction, the workhorse job of production AI. Read a messy document, produce typed records. My research pipeline has done it 28,812 times, and most of what I know about it comes down to one design stance: the model proposes a record, and something that isn't the model decides whether the record is real.

Why validation can't live inside the model

The temptation is to put the rules in the prompt. "Always return valid JSON. The year must be a four-digit number. Only include statistics that appear in the text." And the model will mostly comply, which is the problem, because mostly is a property of the model and the property you need is always, and no instruction makes a probabilistic system deterministic. That was week 1's lesson and it's this week's foundation.

There's also a subtler reason. A prompt rule and a code rule fail differently. When a prompt rule fails, a malformed record flows downstream and corrupts whatever aggregates it. When a code rule fails, you get a rejected row with a reason attached, sitting in a queue where a person can look at it. Validation outside the model converts silent corruption into visible workload, and that conversion is most of the value.

The tooling for the model side has gotten genuinely good, structured output modes and schema-constrained decoding that guarantee the output parses. Use them, they remove a whole class of retries. But parsing is the shallow half of valid. A schema-conformant record can still quote a sentence that isn't in the document, or cite a page that doesn't exist. Conformance is the model's job. Truth conditions are the validator's.

The schema is a contract with teeth

Here's the shape of the extraction schema my pipeline runs against every paper, trimmed to the load-bearing fields:

{
  "findings": [{
    "claim": "one sentence, the author's assertion",
    "statistics": "effect sizes, sample sizes, p-values, as reported",
    "population": "who was studied",
    "evidence_quote": "verbatim sentence from the paper",
    "page_or_section": "where"
  }]
}

Every field is there to be checkable. The evidence_quote field is the one I'd defend in a fight: requiring a verbatim quote per finding means the validator can do the cheapest grounding check that exists, a substring test against the source text. A model can hallucinate a claim, and it can hallucinate a quote, but hallucinating a quote that happens to appear character-for-character in the document is a much taller order. One string comparison buys you that.

And the rule that shapes everything: fields the model can't fill stay empty rather than guessed. An empty statistics field is a fact, "this extraction found no statistics." A guessed one is a landmine with a timestamp. The prompt says all of this, and then the validator enforces the parts of it that can be enforced.

The tiny build: fifty messy PDFs

The build for this week is the pipeline in miniature, and the mess is the point. Feed it fifty real-world PDFs and you get the full zoo: clean born-digital text, two-column layouts that extract in scrambled order, scanned pages OCR'd with errors, a forty-year-old typewriter page. PDFs are the swamp. The build's flow:

1

Extract text, record the quality

Clean parse, layout-scrambled, or OCR. The quality grade rides along with the document.

2

Model extracts against the schema

Typed fields, one record per finding, empty fields allowed and encouraged over guesses.

3

Validate every rowgate

Parse check, then truth conditions: the evidence quote must appear in the source text, page references must exist, numbers in the claim must appear in the quote.

4

Reject with a reason, retry once

A rejected row goes back with its specific violation. A row that fails twice goes to quarantine, not to the database.

5

Downgrade, don't launder

Records from bad text carry the low-quality flag forward. Bad OCR in means low confidence out, visibly.

Step 1 matters more than it looks. The pipeline records extraction quality per document and downgrades confidence accordingly, because the most dangerous input isn't the scan that fails to parse, it's the scan that parses into fluent garbage. A validator can't tell you the source text was wrong, so the quality grade has to travel with the record for a human to weigh later.

Step 4's cap matters too. Retrying a rejected row once with the violation named is cheap and often works. Retrying until it passes is how you launder a bad extraction into a compliant-looking one, which is the same lesson the citation gate taught me: a gate you retry into submission isn't a gate.

Three systems, one stance

In the research pipeline, this is the machinery behind every published claim: schema-enforced extraction, rejected rows quarantined with reasons, provenance stamped on what passes.

In my fitness app, the same stance guards writes instead of reads. Every model-proposed mutation passes through a Zod schema before it can touch the database, and here's the detail I like: a validation failure isn't an exception, it's round-tripped to the model as a structured error, so the model gets one informed chance to correct exactly what was wrong. The validator is both the wall and the error message.

In Enacted, the validated field is the prose itself: a deterministic check scans every generated summary for citations and dates that weren't in its input, and a rejected summary publishes as a diff with no summary at all rather than being coaxed into compliance.

Same pattern three ways: the model's output is a proposal, and code decides whether the proposal is real.

If you're building an extraction pipeline this week, write the validator before you tune the prompt. Decide what a checkable record looks like, which fields carry their own proof, what happens to a row that fails, and how many retries you'll allow before quarantine. The prompt work goes faster afterward anyway, because now you have a definition of done that a script can check.