Making the model speak JSON

Prompt a model for JSON and it mostly obeys. Raise the temperature, hand it a messier input, and every so often it emits a trailing comma, an unquoted key, a stray sentence before the opening brace. One malformed record and the parser downstream throws, and now you are writing retry loops and regex patches around a model that was supposed to save you work. The HuggingFace cookbook shows this cleanly: the same task that produces valid JSON at low temperature starts producing broken JSON as the temperature climbs.

Constrained decoding fixes this at the root, and the fix is smaller than you would guess. It is not a better prompt or a special model. It is one intervention on the numbers the model was already producing.

The trick lives at the logits

You met logits back in modules 1 and 6. At each step, the model does not emit a token, it emits a score for every token in its vocabulary, tens of thousands of them, and sampling turns those scores into one choice. Constrained decoding steps in between: before sampling, it sets the score of every token that would break your format to negative infinity, so that token has zero probability of being chosen. The model picks only from what is left.

That is the entire idea. Everything else in this module is about doing that masking cheaply and correctly.

A schema is a state machine over tokens

The question is which tokens to mask at each step, and it changes as the output grows. After an opening brace, a quote is legal and a digit is not. Halfway through a number, a digit is legal and a quote is not. Something has to track where you are in the structure and hand back the legal set.

That something is a state machine. The insight, from Willard and Louf's paper, is that a regex or a JSON schema can be compiled into a finite-state machine, and generation becomes transitions between its states. Each state knows exactly which tokens are allowed to come next. Nested structures like recursive JSON need a slightly more powerful machine, a pushdown automaton that can remember how deep you are, but the principle holds: the current state determines the legal tokens.

The naive version is too slow to ship. Checking all hundred-thousand-plus tokens against the grammar at every single step would swamp the model's own compute. Willard and Louf's move is to precompute an index once, mapping each state of the machine to its set of allowed tokens, so the per-step work becomes a lookup instead of a scan. They report it adds little overhead to generation. Later engines like XGrammar and llguidance push the same idea further with adaptive caches and overlapping the mask computation with GPU work, to the point where it is nearly free.

Watch it happen

Tokens do not align with characters, and that is the detail that makes naive approaches fail. A single token can span a quote, a colon, and a brace, so you cannot enforce structure character by character. The machine has to reason about which whole tokens keep the output valid. Step through a tiny example and watch the mask move:

Two things in that walk are worth naming. First, most steps still leave the model a real choice, several digits survive, and the model's own preferences decide among them, so constraining the format does not mean dictating the content. Second, one step had exactly one legal continuation, and the engine emitted it without calling the model at all.

That second thing is the coalescence trick, and the dottxt writeup is the best walkthrough of it. When the state machine has only one legal next token, or a whole deterministic run of them like a fixed key or the punctuation between fields, there is nothing for the model to decide, so you skip the forward pass and fast-forward the text directly. Because a real schema is full of fixed structure, this makes constrained generation faster than unconstrained generation, the writeup reports at least a 5x speedup on some workloads. The counterintuitive result is that adding constraints can speed the model up, because the constraints let you stop calling it.

Function calling, demystified

Function calling looks like magic and is not. OpenAI's own docs spell out the mechanics. Your tool definitions get injected into the prompt as text the model was post-trained to recognize, and yes, they count as input tokens you pay for. The model is trained to emit special tokens that delimit a call. And strict mode, the part that guarantees the arguments match your schema, is just constrained decoding applied to the arguments JSON. There is no separate function-calling engine. It is a prompt convention plus the logit masking you just watched.

Valid is not correct

Now the honest limit. The state machine guarantees the output parses against your schema. It cannot guarantee the values are true. A model constrained to emit {"age": <integer>} will always give you an integer, and that integer can still be wrong. Syntax is enforceable; truth is not.

There is also a live debate about whether constraining hurts quality. The Let Me Speak Freely paper argued that format restrictions degrade reasoning, since masking removes tokens the model wanted and can push it down a less likely path. The dottxt team replicated it and pushed back: most of the reported degradation came from mismatched prompts, and once the prompt is held constant and the schema leaves the model room to reason, for example a free-text field before the constrained answer field, structured generation matches or beats the unconstrained version. Both halves are worth carrying. There is a real failure mode, and schema design mitigates it, so give the model somewhere to think inside the structure.

All of which is why validation still belongs outside the model, the same stance as the validator is the product. Constrained decoding is the model's job, making the output parse. Whether the parsed values are real is a separate check that code has to do, because no format constraint makes a number true.

In production

Enacted validates the prose itself. A deterministic check scans every generated summary for citations and dates that were not in the input, and a summary that fails publishes as a diff with no summary rather than getting coaxed into passing. That gate does not care whether the JSON parsed, it cares whether the claims are grounded, which is the layer constrained decoding cannot reach. Verify-kit takes the same line for citation verification: the model proposes, and a precision-first check either grounds the claim or abstains. The constrained-decoding engine and the validator are two different walls, and shipping usually needs both.

Where to go from here

That closes the course. You started by watching a model pick the next token from a probability distribution, and you can now trace the whole arc: tokens and embeddings, attention, the arithmetic of a forward pass, sampling and temperature, extraction and grounding, agents and their compounding math, test-time compute, and now the logit mask that makes a model speak JSON. The tools are yours to break now. Paste a long document into a model and watch recall degrade. Turn the temperature up until the JSON cracks, then reach for a schema. Wire a two-step agent and count how often it fails on the second step. The depth essays over in the writing go further on the pieces that grabbed you. The best way to keep learning this is to build something small enough to see all the way through, and then push on it until it breaks.