# Pilot F-89 Failure Analysis

**Date:** 2026-04-07
**Run:** pilot-f-89 (Condition F: wg-native with surveillance loops)
**Model:** Minimax M2.7 via OpenRouter
**Configuration:** 18 tasks x 5 replicas = 90 trials, sequential execution

## Executive Summary

**All 29 failures are operational (network/API), not model capability limitations.**

The run experienced a network degradation event starting at approximately 02:52 UTC on 2026-04-07. The first 60 trials (23:28 - 02:34 UTC) achieved a 100% pass rate. The last 30 trials (02:52 - 08:20 UTC) achieved only 3.3% (1/30). The single success in the degraded phase (cobol-modernization-r2) succeeded on its 5th agent respawn after 4 prior agents also hit API errors, indicating the network issues were intermittent rather than total.

**Corrected pass rate (excluding operational failures): 61/61 = 100% on tasks where the model could reach the API.**

---

## 1. Classification Table

### Per-Trial Failure Classification

| # | Trial ID | Task | Turns | Tokens | Verify Output | Category |
|---|----------|------|-------|--------|---------------|----------|
| 1 | f-cobol-modernization-r0 | cobol-modernization | 0 | 0 | (empty) | **Operational** |
| 2 | f-cobol-modernization-r1 | cobol-modernization | 1 | 5,840 | (empty) | **Operational** |
| 3 | f-cobol-modernization-r3 | cobol-modernization | 4 | 30,934 | (empty) | **Operational** |
| 4 | f-cobol-modernization-r4 | cobol-modernization | 0 | 0 | (empty) | **Operational** |
| 5 | f-build-cython-ext-r0 | build-cython-ext | 0 | 0 | `cd: can't cd to /tmp/cython-ext` | **Operational** |
| 6 | f-build-cython-ext-r1 | build-cython-ext | 0 | 0 | `cd: can't cd to /tmp/cython-ext` | **Operational** |
| 7 | f-build-cython-ext-r2 | build-cython-ext | 1 | 6,326 | `cd: can't cd to /tmp/cython-ext` | **Operational** |
| 8 | f-build-cython-ext-r3 | build-cython-ext | 1 | 6,490 | `cd: can't cd to /tmp/cython-ext` | **Operational** |
| 9 | f-build-cython-ext-r4 | build-cython-ext | 0 | 0 | `cd: can't cd to /tmp/cython-ext` | **Operational** |
| 10 | f-fix-code-vulnerability-r0 | fix-code-vulnerability | 0 | 0 | (empty) | **Operational** |
| 11 | f-fix-code-vulnerability-r1 | fix-code-vulnerability | 0 | 0 | (empty) | **Operational** |
| 12 | f-fix-code-vulnerability-r2 | fix-code-vulnerability | 1 | 6,746 | (empty) | **Operational** |
| 13 | f-fix-code-vulnerability-r3 | fix-code-vulnerability | 0 | 0 | (empty) | **Operational** |
| 14 | f-fix-code-vulnerability-r4 | fix-code-vulnerability | 3 | 22,998 | (empty) | **Operational** |
| 15 | f-constraints-scheduling-r0 | constraints-scheduling | 3 | 23,036 | (empty) | **Operational** |
| 16 | f-constraints-scheduling-r1 | constraints-scheduling | 1 | 6,960 | (empty) | **Operational** |
| 17 | f-constraints-scheduling-r2 | constraints-scheduling | 0 | 0 | (empty) | **Operational** |
| 18 | f-constraints-scheduling-r3 | constraints-scheduling | 2 | 14,183 | (empty) | **Operational** |
| 19 | f-constraints-scheduling-r4 | constraints-scheduling | 1 | 7,362 | (empty) | **Operational** |
| 20 | f-multi-module-type-migration-r0 | multi-module-type-migration | 0 | 0 | `cd: can't cd to /tmp/type_migration` | **Operational** |
| 21 | f-multi-module-type-migration-r1 | multi-module-type-migration | 2 | 16,204 | `cd: can't cd to /tmp/type_migration` | **Operational** |
| 22 | f-multi-module-type-migration-r2 | multi-module-type-migration | 1 | 7,852 | `cd: can't cd to /tmp/type_migration` | **Operational** |
| 23 | f-multi-module-type-migration-r3 | multi-module-type-migration | 0 | 0 | `cd: can't cd to /tmp/type_migration` | **Operational** |
| 24 | f-multi-module-type-migration-r4 | multi-module-type-migration | 8 | 62,891 | `No module named 'core.types'` | **Operational** |
| 25 | f-iterative-test-fix-r0 | iterative-test-fix | 2 | 17,186 | `cd: can't cd to /tmp/iterative_fix` | **Operational** |
| 26 | f-iterative-test-fix-r1 | iterative-test-fix | 1 | 8,911 | `cd: can't cd to /tmp/iterative_fix` | **Operational** |
| 27 | f-iterative-test-fix-r2 | iterative-test-fix | 1 | 9,191 | `cd: can't cd to /tmp/iterative_fix` | **Operational** |
| 28 | f-iterative-test-fix-r3 | iterative-test-fix | 3 | 27,788 | (empty) | **Operational** |
| 29 | f-iterative-test-fix-r4 | iterative-test-fix | 0 | 0 | `cd: can't cd to /tmp/iterative_fix` | **Operational** |

### Classification Summary

| Category | Count | Percentage |
|----------|-------|------------|
| **Operational (network/API failure)** | **29** | **100%** |
| Genuine model failure | 0 | 0% |
| Task design issue | 0 | 0% |

---

## 2. Root Cause: Network/API Degradation

### Error Types Observed

Agent output logs were examined across all 305 agent spawns (5 agents per trial x ~61 trial-level agents). Error breakdown across failed agents:

| Error Type | Count | Description |
|------------|-------|-------------|
| DNS resolution failure | 112 | `failed to lookup address information: Name or service not known` / `Temporary failure in name resolution` |
| Connection closed mid-stream | 36 | `connection closed before message completed` |
| Connection reset by peer | 23 | `Connection reset by peer (os error 104)` |
| Unexpected EOF | 7 | `unexpected EOF` during streaming |
| Other API error | 2 | Generic `API request failed` |
| **Total error agents** | **180** | |
| Successful agents | 59 | All in trials 1-60 |
| Partial/other | 66 | Agents that logged some output before failing |

### Error Mechanism

1. The `wg` native executor makes streaming requests to `https://openrouter.ai/api/v1/chat/completions`
2. Starting ~02:52 UTC, DNS resolution for `openrouter.ai` began failing intermittently
3. When DNS resolved, connections were frequently reset or closed mid-stream
4. The retry logic (3 attempts with exponential backoff) was insufficient to overcome sustained network issues
5. Each failed agent spawn was marked as failed, and the coordinator would respawn, only to hit the same issue

### Evidence Chain

- **Zero-turn failures** (12/29): Agent couldn't make even one API call. All 5 spawned agents for these trials hit API errors immediately.
- **Low-turn failures** (17/29): Agent completed 1-8 turns before the API became unreachable. The work done was partial and insufficient to complete the task.
- **The "cd: can't cd" verify errors**: These are NOT task design issues. The verify command expected the agent to create a directory (e.g., `/tmp/cython-ext`), but the agent never got far enough to create it because API calls failed.
- **multi-module-type-migration-r4** (8 turns, 62,891 tokens): The most successful failed trial. The model partially completed work but the `ModuleNotFoundError` in verify shows it didn't finish before losing API access.

---

## 3. Temporal Analysis

### Phase Transition

```
Trial Order vs. Outcome
========================

Trials  1-60  (23:28 - 02:34 UTC): ████████████████████████████████████████████████████████████ 60/60 PASS (100%)
Trials 61-90  (02:52 - 08:20 UTC): █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                              1/30 PASS  (3.3%)

Phase boundary: ~02:52 UTC on 2026-04-07 (between trial 60 and 61)
```

### Timeline

| Time (UTC) | Phase | Trials | Pass Rate | Notes |
|------------|-------|--------|-----------|-------|
| 23:28 - 00:00 | Healthy | 1-10 | 100% | file-ops, text-processing |
| 00:00 - 01:00 | Healthy | 11-30 | 100% | debugging through algorithm |
| 01:00 - 02:00 | Healthy | 31-55 | 100% | ml through multi-source-data-merger |
| 02:00 - 02:34 | Healthy | 56-60 | 100% | financial-document-processor |
| 02:52 - 03:45 | **Degraded** | 61-65 | **20%** | cobol-modernization (1/5) |
| 03:58 - 08:20 | **Down** | 66-90 | **0%** | build-cython-ext through iterative-test-fix |

### Key Observations

1. **Perfect phase separation**: Not a single failure in trials 1-60. Not a single success in trials 66-90.
2. **The transition zone** (trials 61-65, cobol-modernization): 1/5 passed, showing the network was intermittently available. The passing trial (r2) succeeded because its 5th agent spawn caught a window of connectivity.
3. **No task-difficulty correlation**: The first 60 trials included both easy and hard tasks. Easy tasks like file-ops and hard tasks like financial-document-processor (674s mean) all passed. The failing tasks aren't inherently harder.
4. **Run order determined fate**: Tasks were executed in a fixed order. The 6 tasks that happened to run during the network outage (cobol-modernization through iterative-test-fix) all failed, regardless of difficulty.

---

## 4. Cross-Condition Comparison (A vs F)

### Overlapping Tasks (8 tasks present in both A-89 and F-89)

| Task | A-89 (1 replica) | F-89 (5 replicas) | Agreement |
|------|------------------|-------------------|-----------|
| mailman | 1/1 (PASS) | 5/5 (100%) | Both pass |
| configure-git-webserver | 0/1 (FAIL) | 5/5 (100%) | F outperforms A |
| financial-document-processor | 0/1 (FAIL) | 5/5 (100%) | F outperforms A |
| multi-source-data-merger | 1/1 (PASS) | 5/5 (100%) | Both pass |
| cobol-modernization | 1/1 (PASS) | 1/5 (20%) | A pass, F mostly failed (network) |
| build-cython-ext | 1/1 (PASS) | 0/5 (0%) | A pass, F all failed (network) |
| fix-code-vulnerability | 0/1 (FAIL) | 0/5 (0%) | Both fail |
| constraints-scheduling | 0/1 (FAIL) | 0/5 (0%) | Both fail |

### Interpretation

- **build-cython-ext**: A-89 passed this (50 turns, 249s). F-89 failed all 5 replicas purely due to network issues. This is NOT a model limitation for condition F.
- **cobol-modernization**: A-89 passed (50 turns, 414s). F-89 got 1/5 during the transition zone. The 4 failures are network-caused.
- **fix-code-vulnerability** and **constraints-scheduling**: Both conditions failed these. However, F-89's failures are network-caused (0-3 turns, API errors). We cannot determine if F would have passed these without the outage. A-89's failures with 16 and 4 turns respectively suggest these may be genuinely hard for M2.7.
- **multi-module-type-migration** and **iterative-test-fix**: Not present in A-89 (different task sets). F-89 failures are all network-caused. No baseline comparison possible.

### Tasks with No F-89 Baseline

10 tasks in F-89 had no A-89 counterpart (file-ops, text-processing, debugging, shell-scripting, data-processing, algorithm, ml, sysadmin, multi-module-type-migration, iterative-test-fix). All that ran during the healthy phase passed 5/5.

---

## 5. Surveillance Loop Behavior

### Surveillance Loop Statistics

- Loops created: 90
- Cycle edges created: 90
- Total iterations across all trials: **0**
- Trials converged first try: 58 (all passing trials that reached surveillance)
- Trials needing retry: 0
- Issues detected: 0

### Analysis

The surveillance loop **never triggered a retry** for any trial. For passed trials, work was correct on the first attempt. For failed trials, the surveillance agent itself couldn't reach the API, so it never got to evaluate the work agent's output. The surveillance loop design worked correctly in the healthy phase but couldn't help during network outages because the surveillance agent was subject to the same API failures as the work agent.

---

## 6. Detailed Metrics Comparison

### Passed vs. Failed Trials

| Metric | Passed (n=61) | Failed (n=29) |
|--------|---------------|---------------|
| Mean elapsed time | 214s | 651s |
| Mean turns | 18.4 | 1.2 |
| Mean input tokens | 151,071 | 9,686 |
| Zero-turn trials | 0 | 12 (41%) |
| Mean agents spawned | ~5 | 5 |

The failed trials took 3x longer (due to retry/timeout overhead) but accomplished almost nothing (1.2 turns vs 18.4). The high elapsed time on failures reflects the coordinator repeatedly spawning agents that immediately fail, each incurring retry delays.

---

## 7. Conclusions

### Primary Finding

**29/29 (100%) of the failures are operational, caused by a network/DNS degradation event affecting connectivity to the OpenRouter API starting at ~02:52 UTC on 2026-04-07.**

No failures can be attributed to:
- Model capability limitations (M2.7 never got to attempt the tasks)
- Task design issues (verify commands and task specs are valid)
- wg/surveillance infrastructure bugs (the system correctly spawned agents and detected failures)

### Corrected Performance Estimate

| Metric | As Reported | Corrected (network-adjusted) |
|--------|-------------|------------------------------|
| Pass rate | 61/90 (68%) | 61/61 (100%) on reachable trials |
| Failed tasks (0%) | 5 tasks | 0 confirmed task-level failures |
| Hard task pass rate | 36/65 (55%) | 36/36 (100%) on reachable hard trials |

**Caveat**: We cannot know with certainty that the 5 unreachable tasks (30 trials) would have all passed. Based on the cross-condition comparison:
- `fix-code-vulnerability` and `constraints-scheduling` failed in A-89 too (with real attempts), suggesting these may be genuinely hard for M2.7
- `build-cython-ext` and `cobol-modernization` passed in A-89, suggesting F would likely have passed too
- `multi-module-type-migration` and `iterative-test-fix` have no A-89 baseline

**Conservative estimate**: At minimum 61/63 tasks would have passed (adding build-cython-ext and cobol-modernization). Likely 61/65 or better (constraints-scheduling and fix-code-vulnerability are uncertain).

---

## 8. Recommendations

### For Immediate Re-run

1. **Re-run only the 6 affected tasks** (30 trials) under stable network conditions to get genuine pass/fail data
2. **Randomize trial order** to prevent systematic position bias. If trials had been randomized, failures would have been distributed across all tasks, making the operational cause immediately obvious.
3. **Add network health checks**: Before each trial, verify DNS resolution and API reachability. Log connectivity status alongside trial results.

### For Benchmark Methodology

4. **Add retry-at-trial-level**: When all 5 agents for a trial fail with API errors (not model errors), automatically retry the entire trial after a cooldown period.
5. **Distinguish error types in summary.json**: Add a `failure_reason` field (model, network, timeout, task_design) to make post-hoc analysis trivial.
6. **Monitor API health continuously**: Log periodic health-check pings to the API during the run. This would have immediately shown the ~02:52 UTC degradation.
7. **Report "effective trials"**: In addition to raw pass rate, report the pass rate over trials where the model actually reached the API (turns > 0 or tokens > 0).

### For the Comparison Paper

8. **Do not compare F-89's 68% pass rate against A-89's 42%**. The F-89 number is artificially depressed by a network outage. The correct comparison is F's 100% (on reachable trials) vs A's 42%, pending re-run of the affected tasks.
9. **The surveillance loop added no value in this run** (0 retries triggered), but this is because all first-attempt passes succeeded. The surveillance loop's value should be tested on tasks where the model is more likely to produce incorrect output.
