Token prices fell by roughly two thirds in a year while enterprise AI budgets grew about sixfold. The gap is volume, and the cause is structural: a language model has no index into a document, so every question pays to read everything.
Two numbers from the last two years sit oddly together. The blended price of a million tokens fell by roughly two thirds. In the same period, the average enterprise AI budget grew from about $1.2 million to about $7 million, and 73 percent of enterprises exceeded their AI cost projections. 2026 is the first year the industry spends more running models than training them.
Prices went down. Bills went up. If that looks like a contradiction, it is worth being precise about why it is not.
The gap between price and bill is volume
A unit price falling does not lower a bill if the number of units rises faster. That is what has happened. Agents fire ten to twenty model calls per task and re-send the conversation at every step. Retrieval pipelines inflate the context around each question three to five fold. And every question pays, again, to read material the organisation already owns.
Cheaper tokens made it affordable to ask far more questions, which is exactly what everyone did. The saving was real and it was consumed immediately by volume.
Why the volume is structural, not wasteful
It is tempting to treat this as a discipline problem: better prompts, tighter pipelines, a stricter budget. Those help at the margin. They do not touch the cause.
A language model has no index into a document. To answer any question about a document, it reads the document. Every question therefore pays for every token, and the same document is paid for again on the next question, and the one after that.
Retrieval helps by sending less of the document, but a retrieved chunk is still prose: an arbitrary window of text that happens to sit near the answer. Prompt compression shaves the prose. Neither changes what a unit of knowledge is. Both are ways of sending less of the same thing.
This matters at scale because between 80 and 90 percent of enterprise data is unstructured, and around 90 percent of it is never analysed at all. The industry's standard answer to that problem, a retrieval stack of chunks, embeddings and indexes, more than doubles the size of the estate it was meant to tame. The architecture built to make unstructured data usable makes there be more of it.
Changing the unit of knowledge
There is a different move available: change what gets stored and retrieved in the first place.
Read a document once and decompose it, one way, into knowledge fragments. A fragment is a small structured statement of something the document actually asserts, tied back to the sentence it came from. Once the knowledge is addressable, the whole document is no longer needed to answer a question about it. A question retrieves the dozen fragments that hold its answer, not the eight thousand tokens surrounding them.
On our own benchmark, a frontier model answering from fragments matched its own full-document accuracy on 8 of 9 questions while using 77 percent fewer tokens. A clinical guideline that cost 8,000 tokens to read whole answered a specific factual question from 355 tokens of fragments. The fragment tables for a representative document came to roughly 300 kilobytes against a 3.5 megabyte source, about 91 percent smaller, for the same answerable knowledge.
Two limits worth stating plainly
Decomposition does not make a document smaller. At whole-document level, the fragment set is larger than the source. The saving comes from never needing the whole document to answer a question, not from shrinking it.
And if the workload is to summarise entire documents every time, fragments cost more, not less. The value is in targeted, repeated questioning of a stable corpus, which happens to describe most enterprise AI work: policy libraries, product documents, guidelines, precedent, regulatory correspondence, asked the same classes of question at volume.
Any claim in this area that does not name its limits is not a measurement. It is a marketing line.
What to do about it
The practical question for a finance or technology leader is not whether the arithmetic works in principle. It is whether it works on your documents, with your questions, against the models you already use.
That is answerable. Take a representative slice of the estate, decompose it, ask the same questions of the documents and of the fragments, mark the answers and total the tokens each route consumed. The result is a number, and it is allowed to come out flat.
Key takeaways
- Falling token prices do not lower bills when volume rises faster
- The cause is structural: a model with no index reads everything, every time
- Retrieval and compression send less of the same unit; they do not change the unit
- Changing the unit of knowledge cuts tokens and storage together, with limits worth stating
- The only claim that counts is the one measured on your own documents
The full argument, sourced throughout and with the measured evidence and the honest limits, is set out in our whitepaper, The Cost of Asking. You can read it at /analect/whitepaper.