Turn vibes into evidence
Test any AI change against the scenarios you're worried about, confirm the winner live, and distill what works into a specialized model you own.
one loop · benchmark → live → yours
Benchmark
92%
24 scenarios · 3 trials
Simulated users push agents through multi-turn edge cases before anyone real does.
Live experiment
+7.2%
Hotel rooms booked · 18,400 sessions
Real traffic confirms the winner on the metric your business tracks.
Specialized model
−55%
cost · your-slm-v1 vs Sonnet 5
What survives the loop becomes a model you own and keep tuning.
The new prompt is live. And no one can say if it's actually better.
The demo looked better. Then a screenshot lands in Slack. Your agent promised a customer a refund policy that doesn't exist. Now you're the team that ships on vibes.
three.dev is one loop: test the change, prove it live, own what wins.
Run every candidate through the scenarios that scare you before real users hit them.
Live traffic settles it, on the metrics you already report: bookings, conversions, resolutions.
The winner becomes a specialized model you own and keep tuning.
Every change gets an answer. Every answer sharpens the next.
Test agents before users do
Run every candidate through the scenarios you're worried about. Multi-turn conversations, simulated users, real tool calls, isolated state, and not a single production user touched.
01 · The scenario
Cancel inside the refund window
Goal · hidden from the agent
cancel the trip, keep the deposit
Environment
1 active booking
within the refund window
Assertions
02 · The simulation
simulated user · turn 4 of 6
03 · The score
3 trials · mixed outcomes
Multi-turn scenarios, not golden prompts
Each scenario seeds your feature's state, gives a simulated user a hidden goal, and lets the agent call real tools across a full conversation.
Agents under real operating conditions
Test tool calls, memory, state transitions, retries, and handoffs instead of judging a single input-output pair.
Assertions and AI judges
Grade each run with deterministic assertions, AI judges, or both. Run any number of trials to expose flaky behavior, not one lucky pass.
“You changed the system. Did it actually get better?”For most teams, the honest answer is still: we don't know.
three.dev exists to answer it
Confirm on live traffic
The benchmark identifies the contenders. A live experiment splits real traffic across control and the strongest candidates, shifting allocation toward whatever wins on the business metric you actually care about.
90% CI · hotel rooms booked vs control
Business metrics, not proxies
Measure the thing that matters: hotel rooms booked, conversion, or resolution rate, not only an offline score.
Adaptive traffic allocation
Control plus two challengers. Allocation shifts toward winners as evidence accumulates, so losing variants see less of your traffic.
Quality, latency, and cost
A variant can win on quality while losing on cost or speed. See the full tradeoff before you ship.
Graduate to a model you own
Your production traffic and expert assessments distill into a specialized language model tuned to your use case — better, faster, and cheaper than the frontier model it replaces. No ML team required.
Production traffic
286K requests · 90 days
business outcomes — price quoted, booking made
Human assessments
150 expert labels · collected as you go
AI-derived signals
AI Judge pass/fail · failure clustering
Trained on your traffic
The same production requests that power your experiments become the training set. No dataset curation project.
Aligned with your experts
The human assessments you collect along the way steer the model toward what your domain experts call good.
Proven before it ships
Your SLM enters the same loop as any candidate: benchmarked offline, then confirmed live against the model it replaces.
Integration
One base URL away
Keep the OpenAI or Anthropic SDK you already use. Swap the base URL, add a couple of headers, and your traffic starts flowing — every experiment and distillation above runs on top of it.
No new client library · Benchmarks never touch your users
client = OpenAI(- base_url="https://api.openai.com/v1",+ base_url="https://gate.three.dev/v1",+ api_key=os.environ["THREE_DEV_API_KEY"],+ default_headers={"x-three-use-case": "pr-review",+ "x-three-ai-provider": "openai"}, )Request access
Ship on evidence, not vibes
Your next agent change deserves better than “LGTM, ship it.” Test it in simulation, then confirm it on live traffic.