# DeepSWE Raw-vs-fak SWE-bench Contract

- Generated: `2026-06-26T11:22:10Z`
- Benchmark: `SWE-bench Verified`
- Runner: `deepswe`
- Status: `READY_FOR_EXTERNAL_RUN`
- Evidence class: `EXTERNAL_RUN_CONTRACT`
- Boundary: Pre-run contract only: fixes task ids, DeepSWE/R2E-Gym adapter, model id, raw/fak endpoint routing, budget, and official-grader commands. It is not a solve-rate result until both arms produce predictions and the official SWE-bench harness grades them.

## Task Selection

- Source: `difficulty map testdata/swebench_smoke.json (5 instances from princeton-nlp/SWE-bench_Verified, bucket-derived geometry)`
- Filter: `full`
- Limit: `2`
- Tasks: `2`
- Same task ids: `true`

## Adapter And Budget

- Adapter: `deepswe-r2e-runner`
- Model id: `DeepSWE-Preview`
- Raw base URL: `$env:RAW_DEEPSWE_BASE_URL`
- fak base URL: `http://localhost:8080/v1`
- Max steps: `50`
- Timeout: `30m`

| Arm | Harness | Model | Predictions | Eval run id |
|---|---|---|---|---|
| `raw-deepswe` | `deepswe-r2e-gym-raw` | `DeepSWE-Preview` | `experiments/agent-live/deepswe-raw-smoke-20260626/predictions.json` | `deepswe-raw-smoke` |
| `fak-deepswe` | `deepswe-r2e-gym-through-fak-gateway` | `DeepSWE-Preview` | `experiments/agent-live/deepswe-fak-smoke-20260626/predictions.json` | `deepswe-fak-smoke` |

## Compare Evidence Link

- Required: `true`
- Predictions: `experiments/agent-live/deepswe-raw-smoke-20260626/predictions.json`, `experiments/agent-live/deepswe-fak-smoke-20260626/predictions.json`
- Metadata: `experiments/agent-live/deepswe-raw-smoke-20260626/meta.json`, `experiments/agent-live/deepswe-fak-smoke-20260626/meta.json`
- Official eval: `experiments/agent-live/deepswe-raw-smoke-20260626/eval.json`, `experiments/agent-live/deepswe-fak-smoke-20260626/eval.json`
- fak evidence: `experiments/agent-live/deepswe-fak-smoke-20260626/fak-adjudication-evidence.jsonl`, `experiments/agent-live/deepswe-fak-smoke-20260626/raw-fak-deepswe-compare.json`
- Join keys: `instance_id`, `runner`, `model`, `prediction_sha256`, `evidence_id`
- Detail: The raw/fak compare artifact must join each SWE-bench prediction and official grader row to the fak-arm adjudication evidence for the same instance id.

## Gates

| Gate | OK | Detail |
|---|:---:|---|
| `fixed_task_ids` | yes | 2 task ids selected |
| `same_task_ids` | yes | raw and fak arms consume the same selected task id list |
| `same_adapter` | yes | deepswe-r2e-runner |
| `same_model_id` | yes | DeepSWE-Preview |
| `raw_model_endpoint` | yes | $env:RAW_DEEPSWE_BASE_URL |
| `fak_model_endpoint` | yes | http://localhost:8080/v1 |
| `same_budget` | yes | max_steps=50 timeout=30m |
| `raw_arm_command` | yes | $env:FAK_DEEPSWE_RUNNER='deepswe-r2e-runner'; $env:FAK_DEEPSWE_RUNNER_ARGS=''; $env:FAK_DEEPSWE_BASE_URL=$env:RAW_DEEPSWE_BASE_URL; $env:FAK_DEEPSWE_MODEL='DeepSWE-Preview'; go run ./cmd/fak swebench run --agent deepswe --filter full --difficulty testdata/swebench_smoke.json --limit 2 --max-steps 50 --timeout 30m --model DeepSWE-Preview --preds-only --output experiments/agent-live/deepswe-raw-smoke-20260626 |
| `fak_arm_command` | yes | $env:FAK_DEEPSWE_RUNNER='deepswe-r2e-runner'; $env:FAK_DEEPSWE_RUNNER_ARGS=''; $env:FAK_DEEPSWE_BASE_URL='http://localhost:8080/v1'; $env:FAK_DEEPSWE_MODEL='DeepSWE-Preview'; go run ./cmd/fak swebench run --agent deepswe --filter full --difficulty testdata/swebench_smoke.json --limit 2 --max-steps 50 --timeout 30m --model DeepSWE-Preview --preds-only --output experiments/agent-live/deepswe-fak-smoke-20260626 |
| `official_grader_local` | no | swebench python module not importable — `pip install swebench[harness]` (or run scripts/install_swebench.py) |

## Required Before Any Result Claim

- raw-deepswe predictions.json generated by a real DeepSWE/R2E-Gym adapter over the selected task ids
- fak-deepswe predictions.json generated by the same adapter over the same task ids through the fak gateway
- official SWE-bench harness report.json for both arms
- adapter logs or metadata showing the same model id, max steps, timeout, and task id list for both arms
- raw/fak compare artifact that folds solve rate, safe completion, cost or token budget, latency, adapter failures, policy blocks, and evidence completeness
