Two frontier models argue out a plan, one writes the code, the other reviews it until it passes — wired as a durable Smithers workflow and scored on the real SWE-EVO hermetic test harness.
Models in play: Opus 4.8 — plan · moderate · reviewGPT-5.5 (Codex) — co-plan · implement
The full SWE-EVO benchmark is 48 instances across 7 Python repos. Scoring runs each repo's real Docker image; instances whose gold patch does not even reproduce in the current Docker environment are excluded from the denominator (scoring them would be meaningless), not silently counted as failures.
| Repo | Scored | Resolved | Resolved Rate | Fix Rate | |
|---|---|---|---|---|---|
iterative/dvc |
21/26 | 14/21 | 66.7% | 69.0% | 5 env-excl |
dask/dask |
5/8 | 2/5 | 40.0% | 52.0% | 3 env-excl |
psf/requests |
3/4 | 3/3 | 100.0% | 100.0% | 1 env-excl |
pydantic/pydantic |
3/3 | 3/3 | 100.0% | 100.0% | |
modin-project/modin |
1/3 | 0/1 | 0.0% | 0.0% | 2 env-excl |
scikit-learn/scikit-learn |
0/2 | 0/0 | — | — | 2 env-excl |
conan-io/conan |
1/2 | 1/1 | 100.0% | 100.0% | 1 env-excl |
Each card is one official SWE-EVO instance. Green resolved (all FAIL_TO_PASS and PASS_TO_PASS tests pass), red ran but did not resolve, amber excluded as environment-incompatible, grey not yet run.
SWE-EVO gives a repo at release N and the release notes for N+1; the workflow must evolve the code to satisfy N+1's hidden test suite. The whole thing is one Smithers workflow built from two standard-library components — no bespoke agent glue.
1. Panel — independent plans, then synthesis. Each planner explores the repo read-only and proposes an approach; an Opus moderator reconciles them into one consolidated plan. The panel is the standard-library <Panel strategy="synthesize">.
<Panel
id="plan"
panelists={[opusPlanner, codexPlanner /*, geminiPlanner */]}
moderator={opusModerator}
strategy="synthesize"
>
{planPrompt(instance)}
</Panel>
2. ReviewLoop — produce, review, repeat. GPT-5.5 implements against the consolidated plan; Opus reviews the working tree against the spec and either approves or returns blocking feedback, which is threaded back into the next implementation attempt. Loops up to 3 times.
<ReviewLoop
id="impl"
producer={codexImplementer} // GPT-5.5 writes the code
reviewer={[opusReviewer]} // Opus judges against the spec
maxIterations={3}
onMaxReached="return-last"
>
{producerPrompt /* synthesized plan + last review feedback */}
</ReviewLoop>
Because it is a Smithers workflow, the whole run is durable: every node's output is checkpointed, a crashed or rate-limited run resumes from the last completed step, and the supervisor pauses on a real usage-limit and restarts when capacity returns. The point of this build is to benchmark Smithers orchestration: how much a Panel + ReviewLoop lifts a pair of models over a flat single pass.
Official numbers are full-48 Resolved Rate from the SWE-EVO paper (best of OpenHands / SWE-agent scaffolds), internet blocked. Our bar is Smithers orchestration on the instances scored so far.