Context is a budget, not a backpack

Week 9 of the fundamentals series: context engineering, the name that stuck for deciding what the model gets to read. Context windows have grown to the point where you can dump a whole corpus into one request, and that option being available has quietly replaced the question of whether it's good. This week was about pricing that question properly.

What the sources say about big windows

The paper I kept coming back to is "Lost in the Middle" (Liu et al., 2023). The authors placed a needed fact at controlled positions inside a long context and measured whether models used it. The result is a U-shape: information at the very beginning and very end of the context gets used well, information in the middle gets used worst, and performance can degrade substantially as the context grows even for models rated for the full length.

Newer models do better on this than the 2023 crop, but the direction of the finding has kept mattering in my work: a fact being in the context is not the same as the model attending to it. Which reframes context stuffing. The strategy that feels safe and thorough buries the load-bearing sentence in the middle of 200,000 tokens of accompaniment, and charges you for the burial.

Because the second half of the argument is just arithmetic. Input tokens cost money linearly, so every token you send that couldn't change the answer is pure spend. Send a 90,000-token corpus to answer a question grounded in one 400-token section and you paid roughly 200 times the necessary input cost, for a worse read of the section.

One task, three strategies

The build for this week is a comparison I ran in miniature and have lived at scale: answer questions against a corpus of research papers. Three shapes.

Stuff. Concatenate the papers into the context, ask the question. Zero infrastructure, best case for one-off exploration of a few documents. Costs scale with corpus size per question, and the middle-of-context problem grows with every paper added.

Retrieve. Weeks 3 through 5 of this series: embed chunks, pull the handful relevant to the question, send only those. Per-question cost is small and flat regardless of corpus size. New failure mode: the answer is only as good as the retrieval, and a missed chunk is invisible in the output, which is why the retrieval layer needs its own eval.

Extract once, query forever. The shape my pipeline actually converged on. Each paper is read once, whole, by a model extracting against the week 6 schema, and every question after that hits structured records in Postgres, often with no model in the loop at all. The expensive full-document read happens one time per document instead of one time per question, and it happens under conditions I control.

That third shape is worth dwelling on because it's underrepresented in tutorials. "Retrieval versus long context" is a false binary when the real choice includes "neither, at query time." My pipeline sends each paper whole to the extractor, deliberately, because extraction wants the full document, the methods section informs how to read the results section. Chunking there would destroy exactly the cross-references the schema captures. But that's a one-time read. The 28,812 extracted papers get queried thousands of times without any of them re-entering a context window. The context engineering happened upstream, once.

1

Read each document whole, once

Extraction wants the full paper, so it gets the full paper. This is the only step that pays full-document token costs.

2

Run it as a batch job

Nobody is waiting, so it prices at half the interactive rate.

3

Skip anything already processedgate

Content-hash caching: a paper seen before never re-enters a context window on reruns.

4

Answer questions from the records

Queries hit structured rows in Postgres. Most never involve a model at all.

The two cost levers that made it viable

Two pricing mechanics turn the extract-once shape from expensive to cheap, and both reward the same discipline: knowing which parts of your workload repeat.

Batch APIs price non-interactive work at half the interactive rate. Extraction is the perfect batch workload, thousands of independent documents and nobody waiting, and the discount is why triaging and extracting a 25,037-paper corpus cost me about twenty five dollars instead of fifty. The rule I gave in the pipeline essay stands: if no human is waiting on the response, you should almost never pay interactive prices.

Prompt caching prices repetition. Providers charge cached input tokens at roughly a tenth of the normal input price, against a write premium of about 1.25x when the prefix is first cached, so a reused prefix pays for itself on the second hit. The catch is that it's a prefix cache: the request has to be byte-identical up to the cached point, so the stable content (system prompt, schema, reference material) has to come first and the varying content (the document, the question) last. A timestamp interpolated at the top of a system prompt silently zeroes the whole benefit. Structuring prompts stable-first isn't a style preference, it's a price.

And one layer up, the cheapest token is the request you never send: my pipeline caches by content hash, so a paper already processed is skipped entirely on reruns. Reruns went from days to hours when I added it. Boring, decisive.

| Strategy | Model reads per question | Where it wins | | --- | --- | --- | | Stuff the context | The whole corpus, every time | A few documents, one-off questions | | Retrieve chunks | A handful of chunks | Big corpus, varied questions, eval'd retrieval | | Extract once, query the records | Zero (the read happened upstream, batched and cached) | Big corpus, repeated structured questions |

The smallest sufficient context

The counterpoint to all the scale talk is my favorite context decision in anything I run, and it's tiny. When a law changes, Enacted's summarizer receives the computed diff and the official metadata. Nothing else. Not the full statute, not the version history, not retrieval over related regulations.

That choice pins the spend per summary to the size of the actual change, which is why the whole system runs for under ten dollars a month, and it keeps the window free of anything irrelevant, so there's no middle to get lost in. It's also a security decision, though that story belongs to week 12: a model can't leak or misuse context it never received, and every claim in its output can be checked against a small, known input, which is exactly what the citation gate does.

That's the week 9 principle in one system: the question isn't how much context you can send, it's the smallest context that determines the answer. Everything past that point is spend and attention dilution, purchased together.