Thinking costs tokens

Ask a model a five-step arithmetic word problem and let it answer in one shot, and it often blurts a wrong number with total confidence. Add "work through it step by step" and the same model gets it right, except now the reply is ten times longer and the bill is ten times bigger. That tradeoff is the whole subject of this module.

Reasoning models like o1 and DeepSeek-R1 turned that trick into a product. They are slower and pricier than their non-reasoning siblings, and on the hard problems they are much better. This module is about why the extra tokens buy accuracy at all, and then about a newer idea that spends inference compute in a stranger way to read inputs no context window could hold.

One forward pass, one fixed budget

Back in module 6 you traced where the arithmetic lives: a token comes out of one pass through a fixed stack of layers, a fixed amount of matrix multiplication, the same for an easy token and a hard one. Nothing in that pass lets the model linger on a difficult step. The depth is set by the architecture, not by the problem.

So if a problem needs more computation than one forward pass provides, the only place to put that computation is in more tokens. This is what chain-of-thought actually is. "Let's think step by step" is not a magic phrase that unlocks a smarter mode. It gives the model room to write down intermediate results, and those written tokens become inputs the later tokens can attend to. The scratchpad is real. The model reads its own working the same way it reads the prompt, and the intermediate steps carry the computation that a single pass could not.

Serial is the word that matters. More tokens in a chain of reasoning are more sequential steps of computation, each able to build on the last. A four-digit multiplication a model fumbles in one shot becomes reliable when it writes out the partial products, because now each partial product is a token it can lean on.

Trained to think, not told to

Prompting a base model to reason gets you some of this. Reasoning models get much more, and the difference is training, not prompting.

The clean account is in Sebastian Raschka's writeup. DeepSeek-R1-Zero was trained with reinforcement learning and almost nothing else: give the model a problem with a checkable answer, let it generate a long chain of reasoning, and reward it when the final answer is correct. Math answers verified against a key, code checked by running tests. No human wrote the reasoning traces. The reward was just "did you get it right," and over many rounds the model learned that longer, more careful chains got rewarded more often.

What comes out of that is the surprising part. Behaviors nobody hardcoded show up on their own: the model starts double-checking its own steps, catching a mistake mid-chain and backing up to try again. Raschka calls it the point where the model learns to verify itself without being told to. And it is not gated behind a giant lab budget. TinyZero, a 3-billion-parameter model, showed the same self-checking after training that reportedly cost under thirty dollars, and Sky-T1 reached o1-comparable reasoning on about 17,000 examples for roughly $450. The receipts are small enough to be worth stating plainly.

Reasoning training has a few recipes, not one. Pure RL is the DeepSeek-R1-Zero path. Most production models mix supervised fine-tuning on good reasoning traces with an RL stage on top, and you can distill a big reasoning model's traces into a smaller one and keep much of the skill. The through-line is the same: the model is being taught to spend inference tokens well on problems where correctness can be checked.

The second scaling axis

For years, better meant a bigger model trained on more data. Test-time compute is a second dial. Hold the model fixed and spend more at inference, through longer chains, or sampling several answers and picking the best, or searching over reasoning steps, and accuracy climbs.

The counterintuitive result is that a small model given a smart search at inference can beat a much larger model answering in one shot. HuggingFace showed a 3B Llama model closing the gap to a 70B model on the MATH benchmark by spending test-time compute wisely instead of parameters. Same weights, more thinking, better answers. It is a real knob you turn per query, and the harder the query, the more turning it is worth.

Recursive language models: don't swallow the input

Module 3 gave this failure a name: context rot. Stuff a long document into the window and the model's attention spreads thin, recall degrades, and the relevant sentence gets lost well before the window is technically full. Bigger windows do not fix it, because the problem is attention quality, not capacity.

Recursive language models take a different route, and it is worth understanding because it inverts the usual assumption. The idea, from Alex Zhang's writeup and worked through end to end in Avishek Biswas's deep dive, is to never put the long input in the model's window at all.

Instead, the context lives as a variable in a Python session. A root model gets the question and a REPL where that variable is already loaded, and it interacts with the context by writing code. It can print the first chunk to see the shape of things, grep for a keyword to narrow down, slice out a section, and, when it needs a model to read a slice, call one recursively as if it were a function.

1

Root model gets the question, not the context

The long input sits in a Python variable. The root model's own window stays nearly empty, so its attention never has to span millions of tokens.

2

It writes code to explore

print a slice to see the structure, grep with a regex to find relevant sections, chunk the rest. The context is data it queries, not text it reads.

3

It calls sub-models on slicesgate

When a piece needs reading, the root launches a recursive model call on just that slice. Each sub-call sees a small, clean context and returns an answer.

4

It composes the results

Sub-answers come back as variables the root assembles directly, so nothing has to be reproduced token by token from a bloated window.

Because the root model never attends over the whole thing, its window grows slowly and context rot never gets a chance to set in. Zhang reports that on the OOLONG benchmark at 132,000 tokens, a recursive setup built on GPT-5-mini scored over 34 points higher than GPT-5 answering directly, roughly a 114% improvement, at about the same total API cost per query. On a retrieval task spanning up to 1,000 documents and more than five million tokens, the recursive version was the only one that held perfect performance at the largest scale. The December 2025 paper formalizes it and reports handling inputs up to about 100 times beyond the base model's window. These are early results from one research group, so hold them as promising rather than settled, but the mechanism is clean: this is inference-time orchestration, not a new architecture. Any capable model can be driven this way.

The cost knob is recursion depth and fan-out. Every sub-call spends tokens, so a root model that recurses on everything gets expensive fast. The economical strategy is to filter first and recurse only where the answer might actually be, which is exactly the decomposition instinct that makes a good engineer good.

In production

The NoeticMap pipeline reads roughly 13,000 near-death-experience papers, and it never tries to hold them in one context. Each paper is extracted on its own, one clean context at a time, and the results are assembled downstream in code. I did not build that as a recursive language model, and I would not overclaim the parallel, but the instinct is the same one the RLM formalizes: when a corpus is far too big to attend over, you do not widen the window, you chunk, filter, and process the pieces, then compose. Context is a budget, and the cheapest way to stay inside it is often to never load the whole thing at once.