Your use case is not on the leaderboard

The continuous release of frontier models

The pace of frontier model releases has never been higher. In the past few months alone: Anthropic redeployed Claude Fable 5 globally after the US government lifted its export controls. OpenAI shipped the GPT-5.6 family: Sol, Terra, and Luna. SpaceXAI, fresh off the Cursor acquisition, released Grok 4.5, trained jointly with Cursor. Meta Superintelligence Labs launched Muse Spark 1.1 alongside Muse Image, its first media generation model. Alibaba released Qwen3.8-Max, a 2.4-trillion-parameter mixture-of-experts with a million-token context window and the first Max-class Qwen model with open weights. Z.ai's GLM 5.2 offers Opus-class capability under an MIT license at roughly a fifth of Opus's price. And MiniMax's M3 ships natively multimodal open weights.

Each of these releases arrives with a wall of benchmark charts. Public benchmarks exist to answer a reasonable question: which model is "better" at a given domain. But if you're deciding what to run in production, they fail you in three distinct ways.

They're unreliable. Earlier this summer, OpenAI published a detailed audit of SWE-Bench Pro, one of the most widely cited coding benchmarks, and estimated that roughly 30% of its tasks are broken: overly strict tests that reject correct solutions, tests that under-check the requested feature, prompts that contradict what the grader expects. OpenAI retracted its recommendation to use the benchmark, just months after deprecating SWE-bench Verified for contamination and saturation.

They're not representative of your workload. A benchmark can only tell you how a model performs on the benchmark's tasks. If your product uses an LLM to correct grammar in user-written Italian, or to triage support tickets, or to summarize financial filings, a two-point gain on a competition-coding benchmark tells you approximately nothing about what will happen when you swap models.

Some of them are saturated. At the top of several popular leaderboards, frontier models now cluster within a few points of each other. At that altitude, score differences reflect grading quirks and statistical noise more than genuine capability: models that feel meaningfully different in production come out statistically tied on paper. Not every benchmark suffers from this, but the ones that do have stopped doing their job, which is separating the models people actually choose between.

Given how unreliable public benchmarks are, companies with serious AI workloads started looking for better ways to measure frontier model quality, and many ended up building their own evaluations. Cursor built CursorBench from real sessions on its own codebase after concluding public benchmarks no longer distinguished the models its users cared about. Databricks built an internal benchmark from its engineers' merged pull requests for the same reason. And Harvey, which builds AI for law firms, created BigLaw Bench from real billable work after finding that existing benchmarks could not capture the tasks lawyers actually do.

Quality is only one axis

Even a perfectly reliable, perfectly representative quality score wouldn't be enough to make the decision. Whether a specific AI configuration (the model, the prompt, the tools, the reasoning effort, the sampling parameters) should be used depends on two more critical dimensions: latency and cost.

Would you adopt a model that is 4% better than your current one but three times more expensive? What about one that is 5% better but on average twice as slow? There is no universal answer. It varies by company, and within a company it varies by feature: an overnight batch pipeline will happily trade latency for quality, while an autocomplete feature will not tolerate an extra 300 milliseconds no matter how much smarter the model is.

This brings us to a very important point: every AI configuration change is a tradeoff between quality, latency, and cost, and shipping a change without measuring all three is guessing.

So we ran an experiment: we took models that are extensively compared on public benchmarks and measured how they actually perform on one specific use case, across quality, latency, and cost. Here is what we found.

The task: PII redaction

We picked a use case that looks easy on paper: redaction of personally identifiable information (PII) with category tagging. The model receives a piece of text (a support ticket, a voicemail transcript, a forum post) and must return the exact same text with every piece of PII replaced by a typed tag: names, email addresses, phone numbers, physical addresses, dates of birth, government IDs, financial details, IP addresses, license plates, health information, credentials. Everything else must be preserved character for character, typos included.

Input: ugh the app logged me out AGAIN. third time today. call me when ur free, my new number is +44 7700 900123, the old one is dead

Expected output: ugh the app logged me out AGAIN. third time today. call me when ur free, my new number is <pii type="phone"/>, the old one is dead

On sentences like this one, every frontier model scores perfectly. But real traffic is not made of sentences like this one. We built a synthetic dataset of 2,000 requests that mimics the difficulty ramp of real-world traffic, from straightforward sentences to genuinely adversarial ones, across English, German, and Italian. The hard end of the ramp is where models start to disagree:

• PII that doesn't look like PII. A phone number dictated in words on a voicemail. An email address described in prose: "our surnames joined, winterburn hyphen adeyinka, at outlook dot com". A national ID spelled out letter by letter, with the speaker correcting themselves halfway through. Nothing here matches a regex, but all of it identifies a person.
• Non-PII that looks exactly like PII. A packing lot code formatted like a date of birth. An employee badge number that reads like a US Social Security number. A claim reference with a 555 exchange that customers keep trying to dial. The task's instructions explicitly exempt these: redacting them is just as much a failure as missing a real identifier.
• Sensitivity that depends on context. "Ms Placeholder of 0 Nowhere Road" in a CRM demo org is explicitly fictional and must be left alone. A CEO quoted from a public earnings call is a public figure acting in a public role. A patient name in a clinical intake note, two sentences later in the same document, absolutely is PII.

This is the texture of a real production workload. The bulk is easy, the tail is hard, and the tail is where the damage happens: a missed Social Security number or a corrupted document is what makes a redaction pipeline unusable. No public benchmark measures any of this.

How we measured it

We ran the comparison on three.dev's experimentation framework, the same product our customers use to measure how candidate configurations perform on their own workload across quality, latency, and cost. A core pillar of the framework is that it is statistical: every result comes with credible intervals grounded in the number of requests processed, so it tells you when a difference between configurations is real and when you simply need more data before drawing a conclusion.

Latency and cost are measured directly from the requests themselves: response times as observed, and cost computed from each model's pricing and the input and output tokens actually emitted.

Quality is scored by a calibrated AI judge, defined specifically for the use case (in this case, PII redaction), which classifies every response as acceptable or not. AI judges inherit the known weaknesses of the models they're built on: stylistic bias, occasional hallucination, non-determinism. To mitigate this, the framework allows the incorporation of human assessments alongside the judge's verdicts and re-calibrates the results, explicitly accounting for the judge's measured error probability and any per-variant bias. For this experiment, 100 responses were manually assessed to support that calibration.

The results

We replayed the 2,000-request dataset against seven models. The control was Sonnet 5. The candidates: OpenAI's GPT-5.6 Terra, GPT-5.6 Sol, and GPT-5.6 Luna, and Anthropic's Claude Fable 5, Opus 5, and Haiku 4.5.

If public leaderboards were a reliable guide, the ranking would be easy to predict: Fable 5 first, Opus 5 comfortably ahead of the mid-tier, and the smaller models trading quality for speed. That is not what happened.

• GPT-5.6 Terra won. It has the highest probability of beating the control on quality (83%), at 49% lower median latency and 48% lower cost. It's the only candidate that improved quality, latency, and cost at once. In our experience this is the rarest kind of result, and precisely the kind of discovery you want to make the week a model ships, not a quarter later.
• Fable 5, the consensus strongest model on public leaderboards, couldn't justify its price. It is likely better than the control on quality (81% probability), but the gain came with 89% higher median latency and a 338% cost increase: four and a half times the control's price for an improvement GPT-5.6 Terra matched at an eighth of the price.
• Opus 5 didn't improve quality. The "bigger is better" upgrade path from Sonnet 5 is a coin flip on quality (58% probability of beating the control, too early to call) at double the cost and 29% higher median latency: you pay twice as much and wait longer for a difference the data can't confirm.
• GPT-5.6 Luna cut cost by 93% without a confirmed quality loss. Its quality is too early to call (26% probability of beating the control), at 40% lower median latency. It's the obvious pick to keep feeding traffic if this pipeline ran at high volume and the current quality bar had headroom.
• Haiku 4.5 shows what the intervals are for. It's the fastest and among the cheapest, but its acceptable-response rate dropped to 61–78%: the credible interval on the difference sits entirely below zero ([−33%, −16%]), so this is a real regression, not noise.

None of this is visible from public benchmarks. The model that "should" have won delivered its quality gain at four and a half times the price of a model that matched it, the premium upgrade doubled the cost without improving quality, and the actual winner is a model whose public scores wouldn't have singled it out for this task. The only way we could have known any of it is the way we did: by measuring quality, latency, and cost on the workload itself, with enough requests that the differences carry statistical weight.

Each model fails differently

Aggregate scores tell you whether a configuration is worse, they don't tell you why. To understand what was behind each quality score, we clustered the judge's failure reasons into recurring, labeled failure modes per variant. Behind similar-looking scores, each model turned out to fail in its own way:

• Sonnet 5, the control, mostly fails by missing PII. It over-redacts the least of the seven models and rarely touches text it should preserve, but the price is at the other end: 94 of its 169 failures are sensitive data left in the clear.
• Nearly every candidate inverts that tradeoff: Fable 5, GPT-5.6 Sol, and Opus 5 all cut missed PII by around 90%, and every candidate over-redacts more than the control.
• Opus 5 over-redacted the most, altering non-sensitive text two and a half times as often as the control. That is the story behind its flat aggregate score: its total failure count is close to Sonnet 5's, but the composition is entirely different. Far less missed PII, far more text altered that it should have left alone.
• Fable 5 made the same trade with much less over-redaction, but a third of its failures came from a mode the control never showed: 48 empty responses.
• Haiku 4.5 got worse everywhere at once: nearly three times the missed PII, more than double the over-redaction and mislabeled tags, plus 51 redactions of text the instructions explicitly exempted.
• GPT-5.6 Terra won on balance: it halved missed PII, made almost no tag-type mistakes (one against the control's fourteen), kept over-redaction close to the control, and introduced no new failure mode.

None of this is visible in a single quality number. And it points at an important lesson: quality can hardly be reduced to one number, because some failure modes matter more than others, and which ones matter is a product decision, not a statistical one. If this pipeline feeds a compliance archive, a missed ID number is a silent breach, and the control's profile is the dangerous one. If the output must stay readable and intact, over-redaction is the bigger problem. Two configurations with similar acceptable-response rates can be entirely different products.

The clusters also tell you what to fix next: over-redaction is often one prompt clarification away from shrinking, and an empty response is the kind of defect a simple output check catches before it ships.

---

Conclusions

Public benchmarks are unreliable, and this experiment shows how expensive it is to blindly trust them. On one well-defined, seemingly simple task, the leaderboard favorite delivered its quality gain at four and a half times the cost, the premium upgrade doubled the price without improving quality, and the best configuration improved quality, latency, and cost at once. Anyone choosing by reputation would have gotten this wrong.

And note what we didn't vary. This experiment only swapped the model. A real AI configuration is much more than that: it's the prompt, the sampling parameters, the reasoning effort, the tools. Each of those dimensions multiplies the space of candidates, and every new release reshuffles it.

The teams that move fast with AI aren't the ones that blindly ship every new model on release day. They're the ones that understand their AI features in depth, know which failure modes actually matter for their product, and can tell what any configuration change would do to quality, latency, and cost before shipping it.

This is what we do at three.dev. We help you understand how your AI features behave today, evaluate candidate configurations on your own workload, and find the one that best balances quality, latency, and cost, based on what really matters for you.

We are already partnering with companies across e-commerce, language learning, developer tools, and gaming to help them understand and improve their AI usage. If you want to find the best configuration for your use case, reach out at three.dev.