Launch offerSix months of the Starter plan, free for teams who sign up by November 22Claim it →

Compare

three.dev vs Comet Opik (2026): which one fits your AI feature?

Opik gives you tracing, evals, test suites, and Diagnostics scans run by its Ollie assistant on credits. three.dev scores live traffic and names the failure modes with nothing to turn on. Here is where each one fits.

Updated

The short answer

Pick three.dev if your AI feature is live, nobody has set up evals for it yet, and your team fixes code with Claude Code, Codex, or Cursor. You change one base URL. three.dev scores a sample of live requests, sorts the issues into named failure modes with the most severe on top, and passes each one to your coding agent. There is no assistant to turn on, no credit balance, and no daily switch in each project.

Pick Opik if you want one platform for tracing, online evals, test suites, datasets, experiments, prompts, and alerts, either open source on your own servers or on a $19 cloud plan. Its Diagnostics feature also finds repeated failures for you, once Ollie is on and you have credits to run it.

At a glance

Best for
three.dev

A live AI feature with no evaluators yet, and a team that fixes with a coding agent

Comet Opik

Teams that want tracing, online evals, test suites, and prompt tools on a $19 cloud plan or open source, with failure scans run by an AI assistant

What it is
three.dev

A proxy that scores live LLM traffic, names its failure modes, and hands the fix to your coding agent

Comet Opik

An AI observability and evals platform: tracing, online evaluation, test suites, datasets, experiments, prompt management, alerts, Diagnostics, and the Ollie assistant

How you connect
three.dev

Base URL gate.three.dev and three headers: a three.dev API key, a use case, and your provider. Your AI SDK and provider key stay the same. Works through LiteLLM and the other documented providers

Comet Opik

Python or TypeScript SDK, 40+ framework, provider, and gateway integrations (LiteLLM among them), or OpenTelemetry over HTTP. The Opik skill can add the tracing for you

Setup before you see failures
three.dev

None. No assistant to turn on, no credits to buy, and no daily switch in each project

Comet Opik

For Diagnostics: Ollie turned on for the workspace and Ollie credits. Click Run diagnostic, or switch on daily scans in each project. Some Cloud projects get a free first scan after 100 traces in 7 days

For online evaluation: an AI provider key, then a judge prompt, a decision model, or Python code, with a scope, filters, and a sampling rate

Finding failures
three.dev

A sample of requests is scored after each response returns. Each issue gives the reason and quotes the lines behind it

Issues group into named failure modes, each with a severity, most severe first

Comet Opik

Diagnostics, run by Ollie, scans the last 24 hours on demand, or the previous day once daily scans are on. It groups repeated failures into one issue with a severity, a root cause, and a suggested fix

Each scan covers only its own window. Days missed while daily scans were off are not scanned later

Shaping the list
three.dev

three.dev sets each severity, and your team can change it. Your team can also dismiss a failure mode that is not worth acting on, and restore it later

Comet Opik

Close an issue as Resolved or Not useful, and write project guidance that later scans read. On self-hosted Opik, guidance and Not useful are off by default

Asking about your traffic
three.dev

Chat answers plain questions about your traffic and links each answer to the requests behind it

Comet Opik

Ollie answers questions about traces, threads, datasets, experiments, and prompts. It runs on tokens: a free trial on Free and Pro, then you buy more

After you find a failure
three.dev

A Fix with your agent button copies a prompt that starts your coding agent on one failure mode. The MCP server gives it the failing requests and each issue's reasoning

Comet Opik

Continue with Ollie hands an issue to the chat. With opik connect running on your machine, Ollie reads your code, proposes edits you approve, reruns your agent, and adds the trace to a test suite

Coding-agent access
three.dev

An MCP server with failure modes, failing requests, and docs. It reads your data, and changes only a failure mode's severity or whether it is dismissed, when you ask. Plugins for Claude Code and Codex install it; Cursor and other clients paste its URL

Comet Opik

An MCP server with read and write tools (no deletes) and a skill pack for Claude Code, Cursor, Codex, VS Code Copilot, and opencode. Your agent can read Diagnostics issues and start a scan

Checking a fix
three.dev

Tag requests with a release, then compare two releases or two date ranges on quality, each failure mode, p50 and p95 latency, and cost

Comet Opik

Test suites with plain-text assertions, experiments on datasets, and Ollie reruns of a failing trace's inputs through opik connect

What gets recorded
three.dev

LLM calls and the tool calls the model asks for, grouped into sessions. Tool runs and retrieval are not separate spans

Comet Opik

Every step of an agent run: LLM calls, tool calls, retrieval, and threads

Effect on your requests
three.dev

In the path, as a proxy. Scoring waits until the response has returned, so it adds no latency

Comet Opik

Off the path. The SDKs log traces next to your LLM calls, and the Python SDK sends them to Opik in batches

Evals, prompts, datasets
three.dev

Not offered today

Comet Opik

Yes, on every plan, the open-source build included

Multi-turn evals
three.dev

Not offered. Sessions keep whole conversations for review

Comet Opik

Thread evaluators score a whole conversation once it has been quiet for 15 minutes (the default)

CI gates
three.dev

Not offered today

Comet Opik

No CI gate or GitHub Action in the docs. Test suites and experiments run from the SDKs

Alerts
three.dev

Not offered today

Comet Opik

Alerts on trace errors, cost, latency, feedback scores, and prompt changes go to Slack, PagerDuty, or any webhook, on every plan

What it costs to find failures
three.dev

Scoring is included in the plan price. No credits to buy

Comet Opik

Diagnostics and Ollie use Ollie credits: a free trial on Free and Pro, then tokens you buy, at a price Comet does not publish. Online-evaluation judge calls bill to your own AI provider

Seats
three.dev

Unlimited on every plan

Comet Opik

Up to 10 team members on Free and 50 on Pro, unlimited on Enterprise. A FAQ on the same pricing page says every plan has unlimited team members

History kept
three.dev

30 days on Free, 90 on Starter, 180 on Pro

Comet Opik

60 days on Free and Pro. Pro can extend to 400 days for $29 per 100k spans. Custom on Enterprise

Open source and self-hosting
three.dev

Not open source. No self-hosting today

Comet Opik

Apache 2.0, free to self-host, without user management. The pricing table leaves Ollie out of the open-source build, but the Diagnostics docs say it works on self-host when Ollie is enabled

Data use
three.dev

three.dev does not train models on your traffic for anyone other than you

Comet Opik

Comet's AI assistant sends your questions, which can hold trace data, to OpenAI. Comet keeps those logs for quality assurance and says they do not train third-party models. Free and Pro keep data in the US

Regulated data
three.dev

Not for health data (PHI) or government workloads yet

Comet Opik

SOC 2, ISO 27001, ISO 9001, HIPAA, and GDPR compliance, SSO, and RBAC on Enterprise only

Pricing
three.dev

Free, Starter $79, or Pro $249 a month, each with scoring and unlimited seats

Comet Opik

Free Cloud plan; Pro $19 a month; Enterprise custom; Ollie tokens extra; self-hosting free

What each product is

Opik is Comet's AI observability and evals platform, "for the Agentic Era" in Comet's words. You trace your app with the Python or TypeScript SDK, an integration, or OpenTelemetry, and Opik records every step: LLM calls, tool calls, retrieval, and threads. On top of the traces you get online evaluation (judge prompts, decision models, or Python code that you set up), test suites with plain-text assertions, datasets, experiments, a prompt library and optimizer, annotation queues, dashboards, and alerts. Ollie is Opik's built-in assistant. It answers questions about your traces, runs Diagnostics, and, with opik connect running on your machine, reads your code, proposes edits you approve, and reruns your agent. Opik is Apache 2.0 and comes as a free open-source build, Opik Cloud (Free, Pro, Enterprise), and Enterprise deployments in the cloud, on-premises, or fully managed by Comet.

three.dev works on the LLM calls your feature already makes, with nothing for you to write or turn on. Your requests go through its proxy at gate.three.dev:

  1. Connect. Point the base URL at gate.three.dev and add three headers: your three.dev API key, a use case, and your provider. Your AI SDK and provider key stay the same. A setup skill lets your coding agent make the change for you.
  2. Find. After each response returns, three.dev scores a sample of requests in the background. Each defect is an issue with its reason and the quoted lines behind it. As issues collect, three.dev names failure modes, describes them, gives each a severity of high, medium, or low, and lists the most severe first. Issues that fit no failure mode wait in Unmatched.
  3. Manage. Your team can change a severity, or dismiss a failure mode that does not matter and restore it later.
  4. Fix. Claude Code, Codex, or Cursor reads a failure mode and its failing requests over the MCP server at https://mcp.three.dev/mcp.
  5. Compare. Tag requests with a release, then set two releases or two date ranges side by side on quality (the share of scored requests without issues), latency (p50 and p95), and cost.

Chat answers plain questions about your traffic and links each answer to the requests behind it.

Where Opik is the better choice

  • You want open source or self-hosting. Opik is Apache 2.0, and Comet says the open-source build runs the same codebase as the hosted versions. Self-hosting is free, with unlimited spans and retention. three.dev is not open source and has no self-hosting today.
  • You want the lowest paid price. Opik Pro Cloud is $19 a month for 100k spans, then $5 per 100k. three.dev's first paid plan, Starter, is $79 a month.
  • You want more history on the free plan. Opik Cloud Free keeps 60 days of data. three.dev Free keeps 30 days.
  • You want test suites, datasets, and experiments. Opik has test suites with plain-text assertions and execution policies, evaluation datasets, experiments, 30+ built-in metrics, a prompt library, and a prompt optimizer, on every plan. three.dev offers none of these today.
  • You want evaluators you control. Opik's online evaluation runs your own judge prompts, decision models, or Python code on traces, threads, or spans, with filters and a sampling rate. Thread evaluators score a whole conversation. three.dev does not offer multi-turn evals; sessions keep whole conversations for review.
  • You need alerts. Opik alerts on trace errors, cost, latency, feedback scores, and prompt changes, and sends them to Slack, PagerDuty, or any webhook. three.dev has no alerts today.
  • You want the fix written inside the platform. Both products hand a failure to an outside coding agent over MCP. Opik also has its own: with opik connect, Ollie reads your code, proposes edits you approve, reruns the agent on the failing inputs, and adds the trace to a test suite.
  • You need to trace tool runs and retrieval steps. Opik records every step of an agent run. three.dev records each LLM call, including the tool calls the model asks for, and does not trace tool runs or retrieval steps as their own spans.
  • You want nothing in your request path. Opik's SDKs send trace data to Opik on their own. three.dev is a proxy, so your LLM requests pass through it.
  • You handle regulated data. Opik Enterprise lists SOC 2, ISO 27001, ISO 9001, HIPAA, and GDPR compliance, and Comet's Trust Center shows SOC 2 Type 2 and ISO 27001:2022. three.dev does not take health data (PHI) or government workloads yet.

Where three.dev is the better choice

  • You want failures found with nothing to turn on. Opik Diagnostics needs Ollie on for the workspace and Ollie credits. Someone clicks Run diagnostic, or switches on daily scans in each project. three.dev starts scoring with your first requests.
  • You want every day covered. A Diagnostics scan covers only its own window: the last 24 hours on demand, or the previous day when daily scans are on. Days missed while daily scans were off are never scanned. three.dev scores a sample of requests as they come in.
  • You want to know what finding failures costs. Diagnostics and Ollie chat draw on Ollie credits: a free trial, then tokens at a price Comet does not publish. When credits run out, the scan does not run and daily scans switch off for that project. Opik's online evaluation judges bill to your own AI provider. Every three.dev plan includes scoring.
  • You want the whole team in. Opik Cloud Free allows up to 10 team members and Pro up to 50, per its plan cards. three.dev has unlimited seats on every plan.
  • You want to shape the list directly. In Opik you steer later scans with project guidance and by closing issues as Not useful. In three.dev your team can change a severity, or dismiss a failure mode and restore it later.
  • You would rather change a setting than add a tracing SDK. Your AI SDK stays as it is, and one base URL change covers OpenAI, Anthropic, Gemini, Azure OpenAI, OpenRouter, Concentrate, Amazon Bedrock, LiteLLM, the Vercel AI SDK, and LangChain.
  • You want to check each failure before you act. Every three.dev issue gives the reason the request failed and quotes the lines behind it.

Pricing

Opik pricing (the Opik tab on Comet's pricing page):

Open Source
Price$0
IncludedSelf-hosted: unlimited spans and retention, community support, AI guardrails. No user management, and no Ollie in the pricing table
Extra usageNone: you run the servers
Free Cloud
Price$0 a month
Included25k spans a month, 60 days of data, up to 10 team members, US data region, Ollie trial
Extra usageNo extra spans listed; Ollie tokens after the trial
Pro Cloud
Price$19 a month
Included100k spans a month, 60 days of data, up to 50 team members, US data region, email support, Ollie trial
Extra usage$5 per 100k spans; $29 per 100k spans for 400-day retention; Ollie tokens after the trial
Enterprise
PriceCustom
IncludedUnlimited team members, custom usage and data region, cloud, on-premises, or fully managed deployment, SSO, RBAC, SOC 2, ISO 27001, ISO 9001, HIPAA, and GDPR compliance, 2-hour response SLA
Extra usageCustom, Ollie included at a custom price

A span is one tracked step, such as an LLM call, a tool call, or any traced function. Ollie (chat, trace analysis, Diagnostics, and the opik connect coding harness) is "Free trial then purchase tokens" on Free and Pro, and Comet does not publish the token price. Online evaluation judges call the AI provider you connect, so your provider bills you for them. The plan cards cap team members at 10 and 50, while a FAQ on the same page says every plan includes unlimited team members.

three.dev has three plans: Free, Starter at $79 a month, and Pro at $249 a month. Each includes scoring and unlimited seats. three.dev is generally available: compare its plans, or start free and point it at your own traffic.

Our recommendation

Choose Opik if you want open source you can run yourself, the cheapest paid cloud plan here, or one platform for tracing, evals, test suites, prompts, and alerts. If your team already uses Opik, turn on Ollie and Diagnostics before you look elsewhere: it is the closest thing to what three.dev does.

Choose three.dev if nobody on your team has time to set up evals or watch a credit balance, your users are the ones reporting failures, and a coding agent already writes most of your fixes. One base URL change gets you failure modes with a severity on each, ready for the agent.

Key facts

  • three.dev: a proxy at gate.three.dev that scores a sample of live LLM requests, explains each issue with quoted lines, and groups the issues into failure modes ranked by severity. Your team can change a severity, and dismiss or restore a failure mode. Generally available since November 2026.
  • Opik: Comet's Apache 2.0 platform for tracing, online evaluation, test suites, datasets, experiments, prompts, and alerts, with the Ollie assistant and Diagnostics.
  • Failure discovery: Opik Diagnostics needs Ollie and credits, and scans on demand or daily once switched on. three.dev's runs with no switch, and scoring is in the plan price.
  • Prices: Opik Cloud Free is $0, Pro $19 a month, Enterprise custom, self-hosting free, Ollie tokens extra. three.dev Free is $0, Starter $79, and Pro $249 a month.
  • Missing from three.dev today: alerts, CI gates, prompt management, datasets, experiments, multi-turn evals, and self-hosting. It does not take PHI or government workloads yet.

Sources

FAQ

Is three.dev an Opik alternative?

For a team with a live AI feature and no eval setup, yes. You change one base URL, and three.dev scores a sample of live traffic, sorts the issues into failure modes by severity, and gives them to your coding agent, with no assistant to turn on and no credits to buy. For a team that wants open source, self-hosting, test suites, datasets, experiments, prompt management, or alerts, Opik covers more today.

Is Opik free?

The open-source build is free under the Apache 2.0 license, with unlimited spans and retention, and you run the servers. Opik Cloud has a Free plan with 25k spans a month, 60 days of data, and up to 10 team members. Pro Cloud is $19 a month. Ollie, the assistant that runs Diagnostics, comes with a free trial on Free and Pro, and after that you buy tokens.

Does Opik find production failures automatically?

Yes, with Diagnostics, as of November 18, 2026. Ollie reads your traces, groups repeated failures into one issue each, and gives each issue a severity, a root cause, and a suggested fix. It needs Ollie turned on for the workspace and Ollie credits. A scan runs when you click Run diagnostic, or every day once someone switches on daily scans in a project. Some Opik Cloud projects get a free first scan after 100 traces in 7 days. three.dev scores a sample of requests after each response returns, with no switch to turn on, and groups the issues into failure modes.

What is the difference between Opik Diagnostics and three.dev failure modes?

Both group repeated failures and rank them by severity. Opik Diagnostics is a scan: it reads the last 24 hours when you start it, or the previous day when daily scans are on, and a day missed while daily scans were off is not scanned later. It runs on Ollie credits. three.dev scores a sample of live requests as they come in and names failure modes once enough issues collect, with scoring in the plan price. In Opik you steer later scans with project guidance and by closing issues as Not useful. In three.dev your team can change a severity, or dismiss a failure mode and restore it later.

How much do Ollie credits cost?

Comet does not publish a price, as of November 18, 2026. The pricing page says "Free trial then purchase tokens" for Ollie on Free and Pro, and "Custom" on Enterprise. The Diagnostics docs say scans use your organization's Ollie credits, shared with Ollie's other features, and that a scan does not run when credits are out. three.dev has no credits: scoring is in the plan price.

How much does Opik cost compared to three.dev?

Opik Cloud Free is $0 for 25k spans a month. Pro is $19 a month for 100k spans, then $5 per 100k spans, with 60 days of data and up to 50 team members. Enterprise is custom, and self-hosting is free. Ollie tokens are extra. three.dev costs $0 on Free, $79 a month on Starter, and $249 a month on Pro, with scoring and unlimited seats on each. Opik's paid entry price is lower. The two products meter different things (spans against data and scored requests), so price your own traffic on each pricing page.

Is Opik open source? Can I self-host it?

Yes. Opik is Apache 2.0, and Comet says the open-source build runs the same codebase as the hosted versions. Self-hosted Opik has no user management, and SSO and RBAC are Enterprise only. Comet recommends Kubernetes for production, and the stack includes ClickHouse, MySQL, Redis, and MinIO. The pricing table leaves Ollie out of the open-source build, while the Diagnostics docs say Diagnostics works on self-host when Ollie is enabled. three.dev is not open source and has no self-hosting today.

Does Opik work with coding agents like Claude Code or Cursor?

Yes. Opik's MCP server and skill pack set up with one command for Claude Code, Cursor, Codex, VS Code Copilot, and opencode. The server can read and write traces, scores, prompt versions, test suites, and experiments, read Diagnostics issues, and start a scan. Ollie can also edit your code itself through opik connect, after you approve each edit. three.dev's MCP server reads failure modes and their failing requests in Claude Code, Codex, Cursor, and other clients, and changes only a failure mode's severity or whether it is dismissed, when you ask, and a Fix with your agent button copies a prompt that starts the agent on one failure mode.

Opik stays out of the request path. Does three.dev?

No. three.dev is a proxy, so your LLM requests pass through gate.three.dev. Scoring waits until the response has returned, so it adds no latency to the request. Opik's SDKs log traces next to your calls and send them to Opik on their own. three.dev does not train models on your traffic for anyone other than you.

More comparisons

three.dev vs Braintrust

three.dev vs Braintrust (2026): which one fits your AI feature?

Braintrust is built around datasets, scorers, and CI, with failure discovery you turn on per project. three.dev scores live traffic and names the failure modes with no setup. Here is where each one fits.

As of November 18, 2026Read

three.dev vs Langfuse

three.dev vs Langfuse (2026): which one fits your AI feature?

Langfuse gives you open-source tracing and evals that you set up and run. three.dev names your failure modes from live traffic with nothing to configure. Here is where each one fits.

As of November 18, 2026Read

three.dev vs LangSmith

three.dev vs LangSmith (2026): which one fits your AI feature?

LangSmith runs the whole agent lifecycle, and its Engine finds failures on paid plans. three.dev finds them without moving your stack onto a platform. Here is where each one fits.

As of November 18, 2026Read

three.dev vs Raindrop

three.dev vs Raindrop (2026): which one fits your AI feature?

Raindrop and three.dev both find failures in production that nobody wrote a test for. Raindrop does it on its Pro plan, with alerts, full agent traces, and simulations around it. three.dev does it on every plan, from one base URL change. Here is where each one fits.

As of November 18, 2026Read

three.dev vs Arize AX

three.dev vs Arize AX (2026): which one fits your AI feature?

Arize AX traces, evaluates, and monitors your agents, and its Signal finds recurring failures on a schedule. three.dev names your failure modes from live traffic with no SDK and nothing to turn on. Here is where each one fits.

As of November 18, 2026Read

three.dev vs Galileo

three.dev vs Galileo (2026): which one fits your AI feature?

Galileo is now Splunk Agent Observability: evals, Luna models, Signals, and runtime guardrails, sold through Splunk. three.dev names your failure modes from live traffic with nothing to configure. Here is where each one fits.

As of November 18, 2026Read

See how your AI feature fails.

Point the AI SDK you already use at one base URL, and get your failure modes named and ranked by severity.