# Smithers Benchmarks

Frontier coding benchmarks run through a Claude-only delegation fleet.
`delegation-chain (Opus plans/reviews, Opus->Opus, Opus->Sonnet implement)`

Rows marked pending await the first full cloud pass. Public rows are each benchmark's own leaderboard. Small subsets are never comparable to full-benchmark numbers.

## SWE-Bench Pro

ScaleAI/SWE-bench_Pro (test, 731)

731 multi-file tasks across 11 professional repos (Go/Python/JS/TS), hidden tests in canonical per-instance Docker.

Model | Scaffold | Pass@1 (%) | n | Subset / caveat
--- | --- | --- | --- | ---
smithers (Opus + Sonnet) pending | delegation-chain | — | — | full-731 pendingClaude-only fleet run pending

## RoadmapBench

UniPat-AI/RoadmapBench (115)

Long-horizon multi-target version upgrades (median ~3,700 LOC / ~51 files); the best single agent on the official board resolves under 40%.

Model | Scaffold | Resolved Rate (%) | n | Subset / caveat
--- | --- | --- | --- | ---
public SOTA (best single agent) public | official | 39 | 115 | full-115approximate published best
smithers (Opus + Sonnet) pending | delegation-chain | — | — | full-115 pendingClaude-only fleet run pending

## Claw-Eval-Live

Claw-Eval-Live (105 tasks / 17 families)

Agentic enterprise workflows scored on observable execution evidence; no public model clears the 70% pass bar.

Model | Scaffold | Score (pass bar 75) | n | Subset / caveat
--- | --- | --- | --- | ---
Claude Opus 4.6 public | official | 83.6 | 105 | full-105public leaderboard
GPT-5.4 public | official | 81.7 | 105 | full-105public leaderboard
smithers (mixture) pending | mixture-gateway | — | — | pendingcannot run faithfully Claude-only (native OpenAI gather leg + neutral Gemini judge); asterisked variant only

## SWE-EVO

SWE-EVO (48 release-sized tasks / 7 repos)

Carry a Python project across a whole release transition; frontier agents resolve only ~21-25% (full-48).

Model | Scaffold | Resolved Rate (%) | n | Subset / caveat
--- | --- | --- | --- | ---
gpt-5.4 public | official (OpenHands) | 25 | 48 | full-48Wilson 95% CI [14.9, 38.8]
gpt-5.2 public | official (OpenHands) | 18.75 | 48 | full-48Wilson 95% CI [10.2, 31.9]
smithers (Opus + Sonnet) pending | delegation-chain | — | — | dvc subset pendingClaude-only fleet run pending; dvc subset is easier than full-48, not directly comparable