Benchmark Sources
Atomic’s model-selection docs are keyed to two live external benchmark sources rather than a hand-maintained table of scores. This page lists each benchmark, what it measures, and when to reference it for a given workflow role — so the docs stay useful as new models ship without a manual rewrite every time.
No single benchmark is the source of truth. Use these as inputs and validate against Atomic’s own workflow evals — public suites test different task distributions than real engineering loops. When Atomic’s numbers disagree with a public index, Atomic’s evals win. The DeepSWE snapshot used by the linked model-selection pages was updated September 3, 2026 and was re-read from the live source on 2026-09-03. The Artificial Analysis sections below were last reviewed 2026-09-01 and were not re-fetched in that pass.
The two sources at a glance
DeepSWE is the closest public proxy for what Atomic actually does. Tasks are written from scratch (not scraped from PRs), so no model has seen the solutions; solutions require substantially more code than SWE-bench-style suites; and verifiers test behavior rather than implementation.
- Current snapshot: DeepSWE v1.1, 113 tasks across 91 repositories and 5 languages, updated September 3, 2026. The site reports 28 measured models and displays 21 leaderboard rows by default, out of 70 published model/effort configurations.
- Metric:
pass@1, plus average cost per task, output tokens, and agent steps.
- When to reference: default weighting for debugger, worker, and any code-writing role. This is the table that drives Model Selection and Pareto Efficiency.
- Watch: cost and step count, not just score — a model that passes but takes 268 steps (e.g. sonnet-5) is a poor worker even at a good pass rate, and the two accuracy leaders sit at opposite ends of that axis: Gemini 3.8 Flash leads the highest-published-effort reading the linked pages use at 166 average steps, while the live default Best view’s leader, GPT-6 Astra [xhigh], averages 29.
Artificial Analysis — intelligence and capability breakdown
Artificial Analysis separates performance by benchmark, which lets a workflow pick the model that is strong at the specific thing a role needs. Reference the individual evaluations, not just the composite index.
Composite indices
- Intelligence Index (v4.1.1) — composite of the nine evaluations below, weighted across four categories: Agents 34%, Coding 24%, Scientific Reasoning 24%, General 18%. Use as a first-pass filter when a new model appears.
- Coding Index — coding-weighted sub-index. Cross-check against DeepSWE.
- Agentic Index — tool use, planning, autonomy, complex problem solving. The best AA signal for orchestrator and reviewer roles.
Individual evaluations — what each measures and when to reference
Capability indices
Artificial Analysis also publishes per-domain capability indices — Agentic, Coding, Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, Economics. When a workflow is domain-specific, pick by the matching capability index rather than the general Intelligence Index.
Role → benchmark map
A quick lookup for which benchmark to weight per role:
Keeping the docs fresh
flora131’s guidance on this issue: point the model at the live benchmark URLs and describe what each measures and when to reference it, rather than hardcoding scores that go stale on every release.
- Treat the model-selection pages as timestamped snapshots that read from the live sources above.
- When a new model appears on DeepSWE or Artificial Analysis, add it by pulling its numbers from the source — the frontier may move (as the gpt-5.6 family did).
- Mark a model unmeasured only if it is absent from both sources; unmeasured models may still be operational defaults.
- Prefer generating the docs and any future routing policy from the same underlying data, so documentation and routing cannot drift apart.
