The free upgrade that raised your bill

Somewhere in your company there's a line on last month's invoice that is higher than the month before, and nobody can quite say why. The product didn't change, the number of users didn't change, and the only thing anyone did was move to the new version of the model, because it was better and its list price was a third lower. The bill went up anyway.

This spring a group of researchers at Stanford, Berkeley, Carnegie Mellon and Microsoft put a number on a related surprise. Across the model pairs they tested, the model with the lower list price cost more to run 32% of the time, and in the worst case by a factor of 28.

That finding was widely shared, and it deserved to be, but it's a story about choosing the wrong model. What we wanted to understand is the less visible case, the one that happens when you choose nothing at all: same feature, same prompt, same vendor, and the next version of the model you were already paying for.

Opinions were easy to find, so we decided to measure instead. For this post we put together a workload that imitates a feature many products have, a reading assistant that takes a paragraph of a technical document and does ordinary things with it: a one-sentence summary, a two-sentence explanation for someone who hasn't read the paper, a question with a numeric answer, an extraction into JSON, a translation into Spanish.

The document is "Attention Is All You Need", partly for the symbolism and mostly because its source is public, so anyone can rebuild our 191 requests and repeat what we did.

We sent those requests to five versions of two vendors' models, between three and five times each, and priced every billed token at the vendor's own list price. What follows is what we found.

You pay per token, not per word

A model doesn't charge you per word. It charges you for the pieces it chops your text into, which the industry calls tokens.

The chopping isn't done word by word. Before training, each model is given a fixed vocabulary of tokens, somewhere in the tens of thousands of them, chosen so that common words like "the" or "model" are a single token while a rarer word like "convolutions" is assembled from two or three. The part of the model that does this chopping is the tokenizer, and a new version of a model often comes with a new vocabulary, so the same text breaks into a different number of tokens.

Take the sentence that started all of this, the one in which the paper introduces its invention: "We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Sonnet 4.6 cuts it into 28 tokens. Sonnet 5, the version that replaced it, cuts the very same sentence into 48.

Nothing about the sentence changed, and nothing about its meaning. The tokenizer changed, and since you pay per token, the same sentence now costs 71% more to read.

How much more depends on what you send, as the bars on the right show: prose and Spanish suffer most, code and JSON least, and across our whole workload it came to 1.37x. It isn't a Sonnet quirk either; Opus 4.6 and Opus 4.7, two versions that list at an identical price, differ by 1.40x on the same text.

So there is an exchange rate between versions, and it isn't printed anywhere on the price sheet.

The price cut that never reached the bill

In June, Anthropic cut Sonnet's price by a third. Anthropic listed Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens, and Sonnet 5 arrived at $2 and $10. If you assumed your bill fell by a third after the switch, you probably haven't checked it.

On our 191 requests, with thinking switched off on both versions, the list price fell 33% and the bill rose 4%. With both models left on their defaults, the setting you get unless you change it, it rose 6%.

Part of this is the exchange rate from the previous section, since the input grew 1.37x. The larger part is what came back. The output grew 1.66x, partly because the finer chopping applies to what the model writes as well as to what it reads, and partly because Sonnet 5 says more.

Given the same paragraph and the same instruction, a one-sentence summary went from 48 billed tokens to 90, and a two-sentence explanation from 82 to 160. Vendors bill a token written at five times the rate of a token read, $10 against $2, so the longer answer is where most of the money went: output was about two thirds of the bill before the upgrade and over 70% after it.

Sonnet 5 is the better model. It answered every one of our fact questions, though so did its predecessor, so we can't show you a quality gain here, only that nothing was lost. What we can show is that a 33% price cut, applied to a feature nobody touched, produced a bigger invoice.

Where the extra dollar went

Then we looked at the harder case, the same price twice. Three pairs qualify. Sonnet 5 to Sonnet 5.5 is $2 and $10 both times, and so is GPT-6 Sol to GPT-6.1 Sol; we sent the 191 requests to all four, at their defaults and at their cheapest settings.

Opus 4.6 to Opus 4.7 is $5 and $25 both times, an older upgrade that we ran with thinking off, so that only the chopping and the length of the answer could move. Then we separated each bill into its causes, one at a time.

The chopping. This is the leak people have heard of, and in these two pairs it happens to be zero, since Sonnet 5 and 5.5 share a tokenizer and so do the two GPT-6 Sols. It has happened at an unchanged price, though: Opus 4.6 and Opus 4.7 both list at $5 and $25, and the same 191 requests are 40% more input tokens on the newer one. The other three leaks are the ones that surprise people.

The longer answer. A new version tends to write more for the same question, though not always. Opus 4.7, with thinking off and at the same price as Opus 4.6, wrote 51% more output tokens for the same 191 requests, and between the two leaks its bill came out 47% higher. On their defaults, where neither version thinks, the gap narrows to 35%, because Opus 4.6 writes more on its defaults.

GPT-6.1 Sol at low effort wrote 12% more than GPT-6 Sol at low effort, and cost 8% more for it. Sonnet 5.5 at its cheapest setting, in contrast, produced exactly the same bill as Sonnet 5 at its cheapest, by writing slightly shorter answers on some tasks and slightly longer ones on others.

The default that thinks. Left on its defaults, Sonnet 5.5 decided to reason before answering on 12% of our requests, including questions whose entire answer is a single number. The vendor bills reasoning as output, so the bill rose 7% overall, and on the fact questions, where thinking is pure overhead, it rose 50%. Nothing in the request asked for any of this; the model's idea of a sensible default moved, and the invoice moved with it.

The setting that vanished. GPT-6 Sol offered a reasoning setting called none, and teams tuned their cheapest requests around it. GPT-6.1 Sol doesn't have it. The request a team had carefully tuned yesterday comes back today as an error, and its nearest replacement, low, costs 14% more and takes more than three times as long to answer.

At least this one fails visibly, with an error. The others don't fail at all; they arrive, weeks later, as a number on an invoice.

We measured each of these on its own, and they don't add up to one tidy number, which is exactly the point. A bill can leak through one of them or through all four at once, and the price sheet, honest as it is, shows none of them.

It goes both ways

None of this means upgrades always cost more. On these requests GPT-6.1 Sol on its defaults was 6% cheaper than GPT-6 Sol on its defaults, because the newer version reasons less often. And the same Sonnet upgrade that cost 7% more overall was 12% cheaper on JSON extraction and 50% more expensive on fact questions.

Switch reasoning off and the ranking changes, as the lower bars show: the explanation task, 8% more expensive on defaults, is 12% cheaper with reasoning off. There is no such thing as what Sonnet 5.5 costs. There is what it costs on your task, at your setting, and that number can go from negative to positive between two tasks of the same feature.

Because the same 191 requests went to both vendors, the runs also allow a comparison we had not set out to make: the two newest models against each other, at the same $2 and $10. Sonnet 5.5 cost 53% to 61% more than GPT-6.1 Sol, and most of that gap is the exchange rate again.

GPT-6.1 Sol's lower bill came with higher latency. Compared with GPT-6 Sol, its median response time was 1.5 times longer on defaults and 3.4 times longer at the lowest reasoning setting, which for a feature with a latency budget is a cost of its own. Both facts appear again in the replay table later in the post, measured through three.dev rather than our scripts.

Nobody can give you the number

Not the vendor, whose price sheet is perfectly honest and tells you none of this. Not a benchmark, which measures its own questions at a setting you may never use. We saw how far the leaderboard favorite can be from the right choice when we replayed one real task through seven models, and that was before looking at the bill. Not even this article. Every figure above comes from our 191 requests, and the same upgrade ranged from 12% cheaper to 50% more expensive across five tasks that all belong to one small workload, so our numbers describe our workload, not yours.

How you would have known

The way to find out is the one the researchers suggest. Before you flip the version, send yesterday's traffic through tomorrow's model and read the bill. We had recorded our requests through our own proxy as we ran them, so for the last experiment we replayed them from there: the production configuration against the two candidates at the same price, with quality, latency and cost next to each other, and nothing in the product touched.

The replay says what the raw runs said, with the trade-offs attached. The upgrade at the same price costs about the same here and gains a point on the judge's score. The other vendor's model at the same price costs a third less and takes twice as long, which for a feature with a latency budget decides the question. It is not a decision you can make from the two price sheets, which are identical.

Some teams choose a different solution and run a model of their own. None of the four leaks can happen to a model nobody upgrades for you: the tokenizer stays the same, the defaults stay the same, and no setting disappears, because the next version only ships when you decide it does. We wrote about when that is the right call.

Conclusions

The price sheet is not the bill. Between the two sits an exchange rate that moves with every version: how the text is chopped, how much the model writes back, whether it thinks before answering, and which settings still exist. On our requests a third off the list price came back as a 4% bigger invoice, and an upgrade at an identical price ran anywhere from 12% cheaper to 50% more expensive depending on the task.

Our test was small on purpose: one public text, short single-turn requests, no tools, no agents. Long conversations and tool calls would show the same four leaks in different proportions, and probably larger ones, since the study we opened with found its biggest gaps in reasoning tokens and agent turns. And because every model answered our fact questions correctly, this post says nothing about which version is better, only about what each one costs.

Knowing what happens to a feature when its model changes is one of the problems we are working on at three.dev. Mostly, though, it is a habit. The teams whose AI bill holds no surprises are not the ones on the newest version. They're the ones who ran yesterday's requests through it first.

The next time an upgrade is free, you'll know what to ask for first: the bill.

Appendix: How We Measured

You are a reading assistant for a technical paper. Answer the task at the end of the message using only the text provided. Be precise, keep the author's terminology, and do not add commentary beyond what the task asks for.