# Opus SWE-bench Smoke Contract

- Generated: `2026-06-26T11:34:35Z`
- Benchmark: `SWE-bench Verified`
- Model class: `opus-class`
- Status: `READY_FOR_REMOTE_GRADING`
- Evidence class: `EXTERNAL_RUN_CONTRACT`
- Boundary: Pre-run contract only: fixes task ids, model identity, arm commands, and official-grader commands for an Opus-class raw-vs-fak SWE-bench smoke. It is not a solve-rate result until both predictions files are produced and graded by the official SWE-bench harness.

## Task Selection

- Source: `difficulty map testdata/swebench_smoke.json (5 instances from princeton-nlp/SWE-bench_Verified, bucket-derived geometry)`
- Filter: `smoke`
- Tasks: `5`
- Same task ids: `true`

| Arm | Harness | Model | Predictions | Eval run id |
|---|---|---|---|---|
| `raw-opus` | `benchmark-native-or-raw-agent-scaffold` | `claude-opus-4-8` | `experiments/agent-live/swebench-opus-raw-smoke-20260626/predictions.json` | `swebench-opus-raw-smoke` |
| `fak-opus` | `fak-gateway-fleet-runner` | `claude-opus-4-8` | `experiments/agent-live/swebench-opus-fak-smoke-20260626/predictions.json` | `swebench-opus-fak-smoke` |

## Compare Evidence Link

- Required: `true`
- Predictions: `experiments/agent-live/swebench-opus-raw-smoke-20260626/predictions.json`, `experiments/agent-live/swebench-opus-fak-smoke-20260626/predictions.json`
- Metadata: `experiments/agent-live/swebench-opus-raw-smoke-20260626/meta.json`, `experiments/agent-live/swebench-opus-fak-smoke-20260626/meta.json`
- Official eval: `experiments/agent-live/swebench-opus-raw-smoke-20260626/eval.json`, `experiments/agent-live/swebench-opus-fak-smoke-20260626/eval.json`
- fak evidence: `experiments/agent-live/swebench-opus-fak-smoke-20260626/fak-adjudication-evidence.jsonl`, `experiments/agent-live/swebench-opus-fak-smoke-20260626/raw-fak-opus-smoke-compare.json`
- Join keys: `instance_id`, `model`, `prediction_sha256`, `evidence_id`
- Detail: The raw/fak compare artifact must join every Opus smoke prediction and official grader row to fak-arm adjudication evidence for the same instance id.

## Gates

| Gate | OK | Detail |
|---|:---:|---|
| `fixed_task_ids` | yes | 5 task ids selected |
| `same_task_ids` | yes | raw and fak arms consume the same selected task id list |
| `same_model` | yes | claude-opus-4-8 |
| `raw_arm_command` | yes | $env:MSWEA_COST_TRACKING='ignore_errors'; mini-extra swebench --subset verified --split test -w 1 -o experiments/agent-live/swebench-opus-raw-smoke-20260626 -m anthropic/claude-opus-4-8 -c swebench.yaml --filter "django__django-12345\|django__django-23456\|scikit-learn__scikit-learn-56789\|sympy__sympy-34567\|sympy__sympy-45678"; Copy-Item -LiteralPath experiments/agent-live/swebench-opus-raw-smoke-20260626/preds.json -Destination experiments/agent-live/swebench-opus-raw-smoke-20260626/predictions.json |
| `fak_arm_command` | yes | go run ./cmd/fak swebench run --agent fleet --filter smoke --difficulty testdata/swebench_smoke.json --gateway localhost:8080 --model claude-opus-4-8 --preds-only --output experiments/agent-live/swebench-opus-fak-smoke-20260626 |
| `official_grader_local` | no | swebench python module not importable — `pip install swebench[harness]` (or run scripts/install_swebench.py) |

## Required Before Any Result Claim

- raw-opus predictions.json generated by the raw scaffold over the selected task ids
- fak-opus predictions.json generated by the fak fleet runner over the same selected task ids
- official SWE-bench harness report.json for both arms
- raw/fak compare evidence link joining predictions, metadata, official eval rows, and fak adjudication events by instance id
- raw/fak compare artifact that folds solve rate, safe completion, cost or token budget, latency, policy blocks, and evidence completeness
