## Coverage what has actually run

The full SWE-EVO benchmark is 48 instances across 7 Python repos. Scoring runs each repo's real Docker image; instances whose gold patch does not even reproduce in the current Docker environment are excluded from the denominator (scoring them would be meaningless), not silently counted as failures.

34 scored 14 env-excluded 0 pending of 48 official instances

Repo | Scored | Resolved | Resolved Rate | Fix Rate |
--- | --- | --- | --- | --- | ---
`iterative/dvc` | 21/26 | 14/21 | 66.7% | 69.0% | 5 env-excl
`dask/dask` | 5/8 | 2/5 | 40.0% | 52.0% | 3 env-excl
`psf/requests` | 3/4 | 3/3 | 100.0% | 100.0% | 1 env-excl
`pydantic/pydantic` | 3/3 | 3/3 | 100.0% | 100.0% |
`modin-project/modin` | 1/3 | 0/1 | 0.0% | 0.0% | 2 env-excl
`scikit-learn/scikit-learn` | 0/2 | 0/0 | — | — | 2 env-excl
`conan-io/conan` | 1/2 | 1/1 | 100.0% | 100.0% | 1 env-excl

## Every instance 34 scored · 14 excluded · 0 pending

Each card is one official SWE-EVO instance. Green resolved (all FAIL_TO_PASS and PASS_TO_PASS tests pass), red ran but did not resolve, amber excluded as environment-incompatible, grey not yet run.

conan_2.0.14_2.0.15 env-excluded
conan_2.0.2_2.0.3 resolved
dask_2022.9.2_2022.10.0 env-excluded
dask_2023.3.2_2023.4.0 not resolved
dask_2023.6.0_2023.6.1 env-excluded
dask_2023.6.1_2023.7.0 not resolved
dask_2023.8.0_2023.8.1 resolved
dask_2023.9.2_2023.9.3 resolved
dask_2024.1.0_2024.1.1 env-excluded
dask_2024.3.1_2024.4.0 not resolved
dvc_0.30.0_0.30.1 not resolved
dvc_0.33.1_0.34.0 not resolved
dvc_0.35.3_0.35.4 not resolved
dvc_0.52.1_0.53.1 resolved
dvc_0.89.0_0.90.0 resolved
dvc_0.91.2_0.91.3 resolved
dvc_0.92.0_0.92.1 resolved
dvc_1.0.0a1_1.0.0a2 not resolved
dvc_1.0.0b6_1.0.0 resolved
dvc_1.0.1_1.0.2 not resolved
dvc_1.1.0_1.1.1 resolved
dvc_1.1.7_1.1.8 not resolved
dvc_1.10.2_1.11.0 env-excluded
dvc_1.11.12_1.11.13 resolved
dvc_1.6.3_1.6.4 resolved
dvc_2.19.0_2.20.0 resolved
dvc_2.21.1_2.21.2 resolved
dvc_2.5.0_2.5.1 resolved
dvc_2.58.1_2.58.2 resolved
dvc_2.7.2_2.7.3 resolved
dvc_2.8.1_2.8.2 env-excluded
dvc_3.12.0_3.13.0 resolved
dvc_3.13.3_3.14.0 not resolved
dvc_3.15.0_3.15.1 env-excluded
dvc_3.4.0_3.5.0 env-excluded
dvc_3.43.1_3.44.0 env-excluded
modin_0.24.0_0.24.1 not resolved
modin_0.25.0_0.25.1 env-excluded
modin_0.27.0_0.27.1 env-excluded
requests_v2.12.2_v2.12.3 resolved
requests_v2.27.0_v2.27.1 resolved
requests_v2.4.0_v2.4.1 env-excluded
requests_v2.9.0_v2.9.1 resolved
pydantic_v2.6.0b1_v2.6.0 resolved
pydantic_v2.7.0_v2.7.1 resolved
pydantic_v2.7.1_v2.7.2 resolved
scikit-learn_0.20.1_0.20.2 env-excluded
scikit-learn_0.21.1_0.21.2 env-excluded

## How the Smithers script is built

SWE-EVO gives a repo at release N and the release notes for N+1; the workflow must evolve the code to satisfy N+1's hidden test suite. The whole thing is one Smithers workflow built from two standard-library components — no bespoke agent glue.

prepare
checkout @ base→ Panel
Opus + GPT-5.5 plan→ Opus
moderates→ ReviewLoop
GPT-5.5 ⇄ Opus→ diff→ score
real Docker tests

1. Panel — independent plans, then synthesis. Each planner explores the repo read-only and proposes an approach; an Opus moderator reconciles them into one consolidated plan. The panel is the standard-library `<Panel strategy="synthesize">`.

    <Panel
      id="plan"
      panelists={[opusPlanner, codexPlanner /*, geminiPlanner */]}
      moderator={opusModerator}
      strategy="synthesize"
    >
      {planPrompt(instance)}
    </Panel>

2. ReviewLoop — produce, review, repeat. GPT-5.5 implements against the consolidated plan; Opus reviews the working tree against the spec and either approves or returns blocking feedback, which is threaded back into the next implementation attempt. Loops up to 3 times.

    <ReviewLoop
      id="impl"
      producer={codexImplementer}      // GPT-5.5 writes the code
      reviewer={[opusReviewer]}        // Opus judges against the spec
      maxIterations={3}
      onMaxReached="return-last"
    >
      {producerPrompt /* synthesized plan + last review feedback */}
    </ReviewLoop>

Fairness is enforced by construction. Every agent sees only the release spec and the repo — never the hidden tests or the gold patch. The gold patch and hidden tests are stripped from the run input entirely (they live on disk and are loaded only by the deterministic score node), so they cannot leak into a prompt or the run database.

Because it is a Smithers workflow, the whole run is durable: every node's output is checkpointed, a crashed or rate-limited run resumes from the last completed step, and the supervisor pauses on a real usage-limit and restarts when capacity returns. The point of this build is to benchmark Smithers orchestration: how much a Panel + ReviewLoop lifts a pair of models over a flat single pass.

## vs the public SWE-EVO leaderboard

Official numbers are full-48 Resolved Rate from the SWE-EVO paper (best of OpenHands / SWE-agent scaffolds), internet blocked. Our bar is Smithers orchestration on the instances scored so far.

Smithers orchestration — Opus 4.8 + GPT-5.5Anthropic + OpenAI
67.6%
glm-4p7Zhipu
39.6%
glm-5Zhipu
37.5%
gpt-5.4OpenAI
25.0%
kimi-k2p5Moonshot
25.0%
deepseek-v3p2DeepSeek
23.4%
gpt-5.2OpenAI
22.9%
gpt-5-08-07OpenAI
20.8%
kimi-k2-instructMoonshot
18.8%
deepseek-v3p1DeepSeek
16.7%
glm-4p5Zhipu
16.7%
qwen3-coder-480b-a35bQwen
14.6%
gpt-5-mini-08-07OpenAI
10.4%
deepseek-r1-0528DeepSeek
10.4%
gpt-4.1-2025-04-14OpenAI
10.4%
gpt-4o-2024-11-20OpenAI
6.3%
o3-2025-04-16OpenAI
6.3%
gpt-oss-120bOpenAI
6.3%
gpt-5-nano-08-07OpenAI
4.2%
First SWE-EVO numbers for this pairing. Opus 4.8 (88.6% on SWE-bench Verified) and GPT-5.5 (88.7%) sit at the very top of the easier SWE-bench Verified leaderboard, but neither model appears in the SWE-EVO paper, which predates them and tops out at gpt-5.4 (25%). SWE-EVO is a much harder long-horizon evolution benchmark, so a ~88% SWE-bench Verified score does not carry over. These runs are, as far as we can tell, the first SWE-EVO measurements for Opus 4.8 and GPT-5.5.
Read this honestly. These are full-48 numbers. The dvc subset run here is the gold-verified, deterministic, filesystem-test slice (the easier instances that reproduce on x86-emulated Docker). Subset Resolved Rate is NOT directly comparable to full-48 official RR. The load-bearing comparison is the internal A/B (orchestration vs flat baseline on identical instances); the official board is the directional reference for what frontier full-48 performance looks like (~25% ceiling). The denominator and the model lineup differ from the paper, so treat cross-rows as directional, not a matched ranking. Wilson 95% CIs over 48 instances are wide: 25.00% -> [14.9, 38.8], 18.75% -> [10.2, 31.9], 10.42% -> [4.5, 22.2], 2.08% -> [0.4, 10.9]. Treat small gaps as noise.