Benchmark Sources
Atomic’s model-selection docs are keyed to two live external benchmark sources rather than a hand-maintained table of scores. This page lists each benchmark, what it measures, and when to reference it for a given workflow role — so the docs stay useful as new models ship without a manual rewrite every time.
No single benchmark is the source of truth. Use these as inputs and validate against Atomic’s own workflow evals — public suites test different task distributions than real engineering loops. When Atomic’s numbers disagree with a public index, Atomic’s evals win. Last reviewed: 2026-07-17.
The two sources at a glance
DeepSWE is the closest public proxy for what Atomic actually does. Tasks are written from scratch (not scraped from PRs), so no model has seen the solutions; solutions require substantially more code than SWE-bench-style suites; and verifiers test behavior rather than implementation.
- Metric:
pass@1, plus average cost per task, output tokens, and agent steps.
- When to reference: default weighting for debugger, worker, and any code-writing role. This is the table that drives Model Selection and Pareto Efficiency.
- Watch: cost and step count, not just score — a model that passes but takes 268 steps (e.g. sonnet-5) is a poor worker even at a good pass rate.
Artificial Analysis — intelligence and capability breakdown
Artificial Analysis separates performance by benchmark, which lets a workflow pick the model that is strong at the specific thing a role needs. Reference the individual evaluations, not just the composite index.
Composite indices
- Intelligence Index (v4.1) — composite of the nine evaluations below. Use as a first-pass filter when a new model appears.
- Coding Index — coding-weighted sub-index. Cross-check against DeepSWE.
- Agentic Index — tool use, planning, autonomy, complex problem solving. The best AA signal for orchestrator and reviewer roles.
Individual evaluations — what each measures and when to reference
Capability indices
Artificial Analysis also publishes per-domain capability indices — Agentic, Coding, Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, Economics. When a workflow is domain-specific, pick by the matching capability index rather than the general Intelligence Index.
Role → benchmark map
A quick lookup for which benchmark to weight per role:
Keeping the docs fresh
flora131’s guidance on this issue: point the model at the live benchmark URLs and describe what each measures and when to reference it, rather than hardcoding scores that go stale on every release.
- Treat the model-selection pages as timestamped snapshots that read from the live sources above.
- When a new model appears on DeepSWE or Artificial Analysis, add it by pulling its numbers from the source — the frontier may move (as the gpt-5.6 family did).
- Mark a model unmeasured only if it is absent from both sources; unmeasured models may still be operational defaults.
- Prefer generating the docs and any future routing policy from the same underlying data, so documentation and routing cannot drift apart.
