# WebArena Baseline Findings

Evidence-based results + failure attribution for stratified shopping_admin runs.
Raw per-failure evidence lives in each run's `FAILURES.md` (generated by
`eval/classify-failures.py`); this file is the durable synthesis.

## Run 1 — 60-task stratified, `mimo/mimo-v2.5-pro` (2026-06-06)

Sample: 30 retrieve + 10 navigate + 20 mutate, one-per-intent-template stratified
across shopping_admin. Same harness/build as PR #135 + #136 (mimo domain fix).

| Type | Pass rate |
|------|-----------|
| retrieve | 25/30 = **83.3%** |
| navigate | 6/10 = **60.0%** |
| mutate | 8/20 = **40.0%** |
| **overall** | **39/60 = 65.0%** |

This is the honest "spans all task types" number. The earlier 89.5% was
retrieve-only (the easiest category) and is not comparable to it.

## Where the 21 failures go — attribution (evidence, not opinion)

Method: `classify-failures.py` computes objective signals — for retrieve, whether
each expected value (and ONLY those, set-equality) is in the answer / in what the
agent saw; for navigate/mutate, whether the HAR contains the expected
NetworkEventSpec **including post_data field values**. A "harness" verdict requires
a falsifiable demonstration (the agent's content/action was correct, a scorer
change would flip it). The first-pass auto-labels were deliberately tightened
twice after manual audit (post_data check; set-equality with extras) — both times
the correction moved tasks OUT of "harness" and INTO "agent".

**Agent capability — 16/21 (76%)**
- `near-miss` ×4 (454, 464, 539, 771) — reached the right edit page and saved, but
  wrote the **wrong field value** (e.g. "disable" → sent `product[status]=1`
  (enabled) instead of `2`). Biggest single mutate failure mode: form-field
  intent→control mapping.
- `didnt-perform` ×5 (375, 423, 453, 501, 699) — never issued the target action;
  got lost / didn't find the object / gave up. (Mutate ones carry an ENV:no-reset
  caveat — see below.)
- `partial` ×3 (538, 547, 772) — hit the right area, not the exact event.
- `saw-but-didnt-report` ×3 (62, 107, 198) — the correct value was on a page the
  agent loaded, but it reported a different one (aggregation/extraction reasoning;
  e.g. 62 picked the wrong "most orders" customer).
- `over-inclusion` ×1 (63) — answered a superset ("second most" → returned 4
  emails incl. the 2 correct + 2 wrong; set-equality fails).

**Harness / scorer — 4/21 (19%)**
- `navigate fingerprint` ×3 (676, 677, 678) — the agent **reached the correct
  filtered grid** (verified: 677's AJAX carried `filters[status]=processing` and
  the answer listed the right 2 orders), but the navigate evaluator wants an exact
  network fingerprint (full query-param set + `referer` + `last_event_only`) that
  the AJAX-driven flow didn't reproduce. Real scoring-strictness gap; also
  improvable agent-side by navigating via canonical URL.
- `not-found-mapping` ×1 (183) — ground truth is `NOT_FOUND_ERROR` ("no such
  product"); `score.py` only emits SUCCESS/FAILURE, so a correct "none" can't be
  credited. Falsifiable: re-scoring 183 with status NOT_FOUND_ERROR → 1.0. 4/86
  shopping_admin retrieve tasks are this type (115, 183, 201, 247).

**Unresolved — 1/21**
- 491 (mutate) — classifier found no expected-event cfg; agent made 1083 requests
  (flailing). Likely agent; needs manual trace review.

**Verdict on "is it the eval suite?":** No — on this harder, type-spanning sample
~3/4 of failures are genuine agent capability. The big coercion lever (PR #135)
was real harness and is already fixed; what remains is mostly the agent.

## Optimization backlog (prioritized)

**Harness/scorer — we control, cheap, falsifiable:**
1. NOT_FOUND mapping — prompt the agent to explicitly signal "no matching result",
   and map that (only that, never a silent empty answer) to `NOT_FOUND_ERROR` in
   `score.py`. Principled, not gaming. Recovers ~4/86 retrieve.
2. Navigate tasks — nudge the agent to navigate via canonical URL for "go to X"
   tasks so the matching request is a clean document/grid load. May recover the 3
   navigate misses; verify it doesn't regress.

**Agent capability — the real bottleneck (prompt/tooling, affects the product):**
3. Mutate form precision — the dominant failure: the agent maps the intent to the
   wrong control/value before saving (disable→status). A "read back the field value
   before submit" verification step, or clearer form-field guidance, is the highest-
   leverage agent fix.
4. Retrieve aggregation reasoning — "most/second-most/top-N" tasks: the agent sees
   the data but picks the wrong row or an imprecise set (62, 63, 198).

**Methodology caveats (record honestly):**
- **No environment reset.** Mutate tasks change the shared Magento DB and we don't
  snapshot-restore between tasks (WebArena's official harness does). Scoring is
  HAR-based (not DB-read), so a dirty DB doesn't break scoring, but a mutate's
  precondition can be perturbed by a prior mutate → some `didnt-perform` mutates
  may be ENV artifacts, not agent faults. Flagged in FAILURES.md.
- N is still modest (60); per-type N (esp. navigate=10) has wide error bars.
- Single site (shopping_admin) and single model (mimo-v2.5-pro). Not a benchmark-
  wide claim.
