Why long contexts cost so much and work so badly

You move to the big-context model, paste in the whole manual, and ask your question. The answer is worse than it was on the 5,000-token version, and the bill is ten times higher. You paid more to get less. This is one of the most common surprises in production, and it's baked into how attention and context work.

"Just put everything in the context" feels free once the window is a million tokens wide. It isn't free on cost and it isn't free on quality, and the two problems come from the same place. This module is why long contexts are expensive, why they degrade, and what to do instead.

The context window is working memory, not storage

The context window is everything the model can see right now: your system prompt, the conversation, the documents, all of it, as one sequence of tokens. Here's the part people skip. The model has no memory outside this window. Nothing from a previous request carries over unless you resend it. Every token the model uses to answer, it re-reads on this request, from scratch.

So the window behaves like working memory. It doesn't file your document away for later, it holds the whole thing in mind at once, on every generation step. That reframe is the key to both the cost and the quality problem, so it's worth saying plainly: the model re-processes the entire context to produce each new token.

Why length costs so much: attention

The reason re-processing is expensive is attention, the mechanism that lets the model relate tokens to each other. On each step, every token looks at every earlier token to decide what matters. Ten tokens, ten each look back. A thousand tokens, a thousand each look back. The comparisons grow with roughly the square of the length, which is why the initial processing of a long prompt (prefill) gets expensive fast and why context windows are a hard engineering wall, not a number a product manager picked.

The KV cache, and why generation is fast anyway

If every step re-attended to everything from scratch, generating a long answer would crawl. It doesn't, because of the KV cache. When the model processes a token, it computes that token's keys and values (the things later tokens attend to) and saves them. The next step reuses those saved keys and values instead of recomputing them, so generation after the first pass is fast.

The cache is a physical object sitting in GPU memory, and it grows with every token in the context, linearly. That's why a long conversation eats memory as it goes, and why memory, not compute, is often the real ceiling on how long a context can get. It's also the exact thing "prompt caching" caches, which is where the pricing comes from.

Why prompt caching is priced the way it is

Prompt caching lets you reuse that saved KV state across requests instead of rebuilding it every time. If a thousand requests share the same big system prompt and tool definitions, you can process that prefix once and reuse it. The economics reflect the work: at one major provider, writing to the cache costs about 25% more than normal input tokens, and reading from it costs about 10% of the normal price, a 90% discount. Another provider gives roughly a 50% discount on cached reads, applied automatically past about 1,024 tokens.

The catch is what makes the pricing actionable: caching works on exact prefix matches. The cache is reusable only up to the first token that differs. Change one token near the front and everything after it has to be reprocessed. That single fact dictates prompt structure. Put the stable stuff first, system prompt, tool definitions, the big reference document, and put the variable stuff, the user's actual question, last. Same content, reordered, and the difference is whether a thousand requests hit a 90%-off cache or pay full price every time. I priced this out on a real pipeline in context is a budget, not a backpack.

Advertised length is not usable length

Now the quality half, and it's the one that catches people who solved the cost half. A model advertised at 200,000 tokens does not reliably use 200,000 tokens. Performance degrades as the input grows, non-uniformly, even on tasks the model aces when the input is short. Chroma's context-rot report tested 18 models and found this across all of them: as input length climbs, reliability drops, and it drops faster when the target information is only loosely related to the question. A single distracting passage measurably hurt accuracy, and adding more compounded it. Irrelevant text is not neutral padding, it actively pulls the answer down.

Position matters too, and in a specific shape. The Lost in the Middle paper found a U-curve: models reliably use information at the beginning and end of the context and are worst at using information stuck in the middle. Same fact, same length, move it from the edge to the middle and the answer can flip.

Feel the two effects together.

Where the fact sits

StartMiddleEnd

Model

Sturdier modelMore fragile model

Context length: 64k tokenspredicted recall: 58%

Try to break it: put the fact at the Start, drag the length all the way right, then switch the same fact to Middle. Same fact, same length, the recall drops, that is the lost-in-the-middle shape. Now flip to the more fragile model and watch the whole curve sag.

Try to break it: put the fact at the start and drag the length to the far right, then switch the same fact to the middle without changing anything else. Recall drops. That gap is lost-in-the-middle. Then flip to the more fragile model and watch the whole curve sag, because how much a model rots is model-dependent. The curve is illustrative, hand-shaped from the two reports above, not measured data, but the shape it's teaching is real: degradation grows with length, and the middle is where it bites hardest.

Why "stuff everything in" fails, and the fix

Put the cost and quality halves together and the maximal-context strategy collapses on both. You pay for every token whether it changed the answer or not, and the tokens that didn't help aren't harmless, they're distractors dragging accuracy down while the useful fact sits in the neglected middle. More context is not more signal. Past a point it's more noise at higher cost.

The fix is the previous module. Instead of sending everything and hoping the model finds the needle, retrieve the few high-signal chunks that actually bear on the question and send those. Minimal, relevant, well-placed context beats a maximal dump on cost and on accuracy at the same time. Retrieval isn't a workaround for small windows, it's the right move even when the window is huge, because the window being huge is exactly what makes the dump tempting and wrong.

In production

The NoeticMap pipeline extracts findings from about 13,000 papers, and the naive design, one giant prompt with a stack of papers, loses on every axis I just described: it costs the most, and the paper that matters lands in the rotting middle. The build that works retrieves the handful of relevant passages per question, orders the prompt stable-prefix-first so the shared instructions and schema hit the cache while the variable query stays last, and keeps the working context small on purpose. Permit-check does the same for permit questions, pulling the specific bylaw sections rather than pasting an entire code and praying. The discipline both share is treating context as a budget you spend, not a backpack you fill, and spending it on the tokens that change the answer.