Smithers Benchmarks
Frontier coding benchmarks run through a Claude-only delegation fleet.
delegation-chain (Opus plans/reviews, Opus->Opus, Opus->Sonnet implement)
Rows marked pending await the first full cloud pass. Public rows are each benchmark's own leaderboard. Small subsets are never comparable to full-benchmark numbers.
SWE-Bench Pro
731 multi-file tasks across 11 professional repos (Go/Python/JS/TS), hidden tests in canonical per-instance Docker.
| Model | Scaffold | Pass@1 (%) | n | Subset / caveat |
|---|---|---|---|---|
| smithers (Opus + Sonnet) pending | delegation-chain | — | — | full-731 pendingClaude-only fleet run pending |
RoadmapBench
Long-horizon multi-target version upgrades (median ~3,700 LOC / ~51 files); the best single agent on the official board resolves under 40%.
| Model | Scaffold | Resolved Rate (%) | n | Subset / caveat |
|---|---|---|---|---|
| public SOTA (best single agent) public | official | 39 | 115 | full-115approximate published best |
| smithers (Opus + Sonnet) pending | delegation-chain | — | — | full-115 pendingClaude-only fleet run pending |
Claw-Eval-Live
Agentic enterprise workflows scored on observable execution evidence; no public model clears the 70% pass bar.
| Model | Scaffold | Score (pass bar 75) | n | Subset / caveat |
|---|---|---|---|---|
| Claude Opus 4.6 public | official | 83.6 | 105 | full-105public leaderboard |
| GPT-5.4 public | official | 81.7 | 105 | full-105public leaderboard |
| smithers (mixture) pending | mixture-gateway | — | — | pendingcannot run faithfully Claude-only (native OpenAI gather leg + neutral Gemini judge); asterisked variant only |
SWE-EVO
Carry a Python project across a whole release transition; frontier agents resolve only ~21-25% (full-48).
| Model | Scaffold | Resolved Rate (%) | n | Subset / caveat |
|---|---|---|---|---|
| gpt-5.4 public | official (OpenHands) | 25 | 48 | full-48Wilson 95% CI [14.9, 38.8] |
| gpt-5.2 public | official (OpenHands) | 18.75 | 48 | full-48Wilson 95% CI [10.2, 31.9] |
| smithers (Opus + Sonnet) pending | delegation-chain | — | — | dvc subset pendingClaude-only fleet run pending; dvc subset is easier than full-48, not directly comparable |