Why the same prompt gives different answers

You send the same prompt to an API twice and get two different answers. You ask for a citation and it hands you a paper that does not exist, formatted perfectly, authors and all. Both of these look like the model misbehaving. Neither is. They're the same mechanism seen from two angles, and once you can see the mechanism, a lot of "the model is broken" tickets turn into "the model is doing exactly what it does."

That mechanism is one loop. The model reads what it has so far, produces a probability distribution over what comes next, and something picks one option from that distribution. Then it does it again. Everything else, chat, code, agents, the whole product surface, is that loop running fast. This module is about the loop.

What the model actually reads

Start with a smaller surprise than you'd expect. The model never sees your text. It sees integers.

Before anything runs, your string gets split into tokens, chunks from a fixed vocabulary of around 100,000 pieces, and each token is just an ID number. Common words are usually one token. Rare words, code, and numbers get shredded into several. "strawberry" might be three tokens, and "1234567" might split in a place that has nothing to do with digits. This is why models miscount the letters in a word and fumble arithmetic on long numbers: they're not reasoning over characters, they're reasoning over IDs that don't line up with the characters you see.

The fastest way to feel this is to paste text into a tokenizer and watch the boundaries. Try a normal sentence, then a long number, then a line of code, then a non-English word. The Tiktokenizer playground shows the exact tokens and IDs per model. It takes about ten seconds to stop trusting that "the model reads words."

The one thing the model does

Once the text is tokens, the model runs and produces one thing: a score for every token in the vocabulary, saying how likely each is to come next. Run those scores through softmax and you get a probability distribution, 100,000 numbers that sum to one. The token "sat" after "the cat" gets a big number. The token "helicopter" gets a tiny one.

That distribution is the entire output. The model doesn't decide, it doesn't choose, it reports odds. "Autocomplete on steroids" is the honest metaphor here, and it does real work: like your phone keyboard, the model is ranking next-tokens by how well they fit what came before, and it has no plan beyond the next one.

Here's where that metaphor lies to you, and it's worth marking the exact spot. Your phone keyboard ranks by simple frequency and stops at the word. This model's ranking was shaped by training on a huge slice of human text, so the "fit" it scores can encode grammar, facts, style, and enough structure to write a working function. The metaphor gets the shape right, one-token-at-a-time prediction, and gets the depth wrong. Don't let "just autocomplete" talk you out of respecting what's in the distribution.

Where the different answers come from

So the model hands you a distribution. Now something has to pick one token from it. That picker is the sampler, and it's where your two-different-answers question gets answered.

If you always take the single highest-probability token, that's greedy decoding, and it's deterministic, same prompt, same output, every time. It also tends to produce flat, repetitive text, because the safest next word over and over is a rut. So most deployed sampling adds randomness: draw a token in proportion to its probability. High-probability tokens usually win, low ones occasionally sneak through, and two runs diverge the moment they pick differently once. That's your different answers. It's not the model being moody, it's a dice roll you didn't know was there.

Temperature is the knob on that dice roll. Low temperature sharpens the distribution toward its favorite, so sampling behaves almost like greedy. High temperature flattens it, so long-shot tokens get a real chance. Top-p and top-k are the other common controls, and they're truncation filters: before sampling, throw away the unlikely tail (top-k keeps the k best, top-p keeps the smallest set whose probability adds up to p) so the dice can only land on plausible options.

Play with it directly. Same prompt, one knob, watch the distribution reshape and then watch what sampling does to it.

Prompt: “The definitive study on this was published in ____”

Temperature: 1.0Sample 50 completions

2019

27.4%

2020

22.4%

2018

18.4%

2021

13.6%

2017

10.1%

2015

6.1%

…I'm not sure

2.0%

Every year here is wrong: in this toy setup the model has no idea, and the honest continuation (green) was barely trained into it. Slide the temperature to 0.1 and the model doesn’t get more honest, it just picks its favourite wrong year every time. Slide it to 2 and the wrongness diversifies. No temperature turns “I don’t know” into the winner, because the distribution itself is the problem.

The thing to try to break: find a temperature that turns "I'm not sure" into the winning answer. You can't. Temperature reshapes probabilities that already exist, it never invents mass for an option the model didn't already rank. Slide it to 0.1 and the model isn't more honest, it's more committed to its favorite. Slide it to 2.0 and it's more varied, not more truthful. Hold that thought, because it's the whole of the next section.

Fabrication is the same mechanism, not a lie

Now the citation that doesn't exist. Ask "the definitive study on this was published in ____" about a topic where the model has no real answer. The sampler still has to pick a token, the distribution still sums to one, and the mass lands on whatever looks like a plausible continuation. A believable year. A believable author. A believable title assembled from the vocabulary of the field. Nothing in the loop can output "there is no such study" unless that was a high-probability continuation, and in most training text a question is followed by an answer, not by an admission of ignorance.

That's fabrication, and calling it a lie gives the model too much credit. A lie needs a known truth to depart from. This is a plausibility engine with no separate fact-store to consult, producing the most probable-looking string on an out-of-distribution question. It's the system working as designed in a region where the data ran thin. The 2025 paper "Why Language Models Hallucinate" makes the sharper version of this argument: our benchmarks reward a confident guess the same as they'd reward abstaining, so guessing wins in expectation and we train it in. I wrote up the mechanism and my own four fabricated citations in more detail in why models fabricate.

There's one instrument that sees into this: logprobs. Ask the API for them and it returns the probability the model assigned to each token it emitted. That's confidence made inspectable, and it lives in the numbers, not in the prose. Fluent text tells you nothing about certainty. A low logprob on the year in a citation tells you plenty.

In production

The NoeticMap pipeline extracts structured findings from about 13,000 near-death-experience research papers, and it lives or dies on this mechanism. Ask a model to fill a citation field for a claim in a thin corner of the literature and it will produce a perfectly formatted reference that may be interpolated from true fragments. The defense is structural. Hand the model the real source text so the true continuation has probability mass, let "not verifiable" be a legal output so abstention has somewhere to go, and then verify what comes out anyway. That last step is its own project, verify-kit, a citation checker built precision-first so it abstains rather than guess. The whole design assumes fabrication is mechanical and builds around it, because nothing about the sampling loop promises to stop on its own.