2026-06-14 · fundamentals
Why models fabricate, mechanically
Week 7 of the fundamentals series: hallucination mechanics. The distribution has to put its mass somewhere, and 'I don't know' is rarely the most probable continuation. A temperature demo you can drag, what the research says about incentives, and the autopsy of my own four fabricated citations.
Week 7 of the fundamentals series, and it's the one the whole series orbits: why do language models fabricate? I've built gates, abstention layers, and a verification tool against this failure. This week I wanted the mechanism itself, because every defense I've shipped was built on an intuition I'd never sat down and verified against the sources.
The distribution has to go somewhere
A language model computes a probability distribution over the next token and the distribution sums to one, always, whether or not the model has anything worth saying. Ask it to complete "The definitive study on this was published in ____" and the probability mass lands somewhere. If the training data contains no strong answer, the mass spreads across the plausible answers, and plausible years, plausible author names, and plausible journal titles are precisely what a model trained on millions of papers has in unlimited supply.
Here's that situation as a toy you can drag. The logits are illustrative, the softmax is real:
Prompt: “The definitive study on this was published in ____”
Every year here is wrong: in this toy setup the model has no idea, and the honest continuation (green) was barely trained into it. Slide the temperature to 0.1 and the model doesn’t get more honest, it just picks its favourite wrong year every time. Slide it to 2 and the wrongness diversifies. No temperature turns “I don’t know” into the winner, because the distribution itself is the problem.
The demo kills the most common folk theory on contact: that fabrication is a sampling problem you can cool away with temperature. Slide it to 0.1. The model doesn't get more honest, it gets more consistent, committing to its single favorite wrong year every time. Slide to 2.0 and the wrongness diversifies. Temperature reshapes the distribution. It cannot add knowledge the distribution doesn't contain, and it cannot promote "I'm not sure" from a low-probability continuation to a winning one. The problem is upstream of sampling.
Why is the honest answer so improbable in the first place? Partly the data: text on the internet that poses a question usually proceeds to answer it, so "I don't know" is rare in exactly the positions where a specific answer is expected. And partly, the research argues, the incentives. OpenAI's 2025 paper "Why Language Models Hallucinate" (Kalai and coauthors) makes the case that our own benchmarks teach the behavior: most evals score a confident wrong answer the same as an abstention, zero, which means guessing strictly dominates honesty in expectation. Train and select models against enough of those tests and you've bred good test-takers, in the worst sense.
The genuinely hopeful finding sits next to it: Anthropic's "Language Models (Mostly) Know What They Know" (Kadavath et al., 2022) showed that models' self-assessments carry real signal, larger models are reasonably calibrated about whether they can answer correctly. The information for abstention is substantially in there. The generation process just isn't built to act on it, which is why abstention has to be designed in from outside, a thing I'd already learned in production before I knew there was a paper explaining it.
The autopsy of my own four
The best dataset I own on this is small and personal: the spot-check where four of my pipeline's five citations turned out to be wrong. Line them up against the mechanism and each failure is the same move at a different scale.
One paper didn't exist at all: the model composed a plausible title from the topic's vocabulary, attached plausible authors, and formatted it perfectly, pure generation from the "what citations look like" distribution. One paper existed but said something different from what was attributed to it: the retrieval-ish failure, right neighborhood, wrong assertion, which is week 3's relatedness-versus-assertion problem wearing a citation. And two had real authors attached to titles those authors never wrote: recombination, real fragments from the training data stitched along the seams of plausibility.
That last category is the one that changed how I think. The model wasn't inventing from nothing, it was interpolating between true things, and interpolation is the whole job we trained it for. Fabrication is next-token prediction operating exactly as designed in a region where the data runs thin, which is why I've stopped expecting model upgrades to fix it, and why the incentives framing in the OpenAI paper rings true.
What raises the rate, what lowers it
From the mechanism you can read off the risk factors directly, and they match what I've seen across my own systems.
Specificity raises it. A citation is the worst case I know: title, authors, year, venue, identifier, each one an exact token sequence, each one an independent opportunity to interpolate. The more exact the ask, the thinner the true distribution and the fatter the plausible one.
Sparse topics raise it. My pipeline works in a niche research literature, thin training data by construction. The four-of-five disaster wasn't bad luck, it was the expected value of the terrain.
Format pressure raises it. Demand a filled field and you've made abstention structurally impossible, the week 6 lesson: an extraction schema that doesn't allow empty fields is a fabrication generator with types.
And the lowering levers are the same list inverted. Ground the model in retrieved source text and you've fattened the true distribution where it matters. Make "not verifiable" a legal output and the calibration Kadavath measured has somewhere to go. Then gate what comes out anyway, because lowering a rate is not the same as an invariant.
The cleanest evidence I have for that last sentence is Enacted's citation gate. That system is built to make fabrication nearly impossible: the model is handed the diff and the metadata, everything it needs, no recall required. Across the 128 seeded change events, the gate still caught one summary introducing a citation that wasn't in its input. One in 128, in the most grounded setup I know how to build. The rate went down enormously, and it didn't go to zero, and nothing about the mechanism says it ever will. That's the entire argument for the gate.
Week 7's summary, then: the model is a plausibility engine, and the gap between plausible and true doesn't close with scale or temperature. It closes with sources in the context and abstention as a legal output, and then a gate outside the model for whatever still gets through. Which is weeks 1 through 6 of this series.