# Model Routing Measurement Plan

Last reviewed: 2026-05-06

This note defines the measurement layer needed to decide whether prompt-language
is useful or just extra ceremony when compared with direct local or frontier model
use.

## Measurement Question

For a fixed bounded task with a locked oracle, does PL improve verified success
enough to justify its extra wall time, tokens, routing complexity, and maintenance
cost?

The answer must be measured per arm, not inferred from narrative quality.

## Required Metrics

| Metric                     | Why it matters                                          | Source                                               |
| -------------------------- | ------------------------------------------------------- | ---------------------------------------------------- |
| Oracle score / pass        | Decides whether the result actually works               | Locked public and hidden verifier                    |
| Wall time                  | Measures operator wait and runtime overhead             | Harness start/end timestamps                         |
| Prompt/input tokens        | Main frontier cost driver                               | Codex JSONL usage, Claude JSON result, provider logs |
| Output/reasoning tokens    | Secondary frontier cost driver and reasoning intensity  | Codex JSONL usage, Claude JSON result, provider logs |
| Estimated USD              | Makes cloud/frontier use comparable                     | Provider pricing table captured at run time          |
| Frontier calls             | Measures hybrid escalation cost                         | Harness route manifest                               |
| Local GPU active time      | Makes local inference cost visible                      | `ollama ps` snapshots and process sampling           |
| Retry count                | Shows whether PL is doing useful repair or looping      | PL audit trace and route manifest                    |
| Human intervention minutes | Captures real babysitting cost                          | Operator log                                         |
| Failure class              | Separates model failure from runtime/harness failure    | Manifest classification                              |
| Scope/oracle leak status   | Prevents invalid evidence                               | Fixture leak audit and protected-file checks         |
| Maintenance/review defects | Measures whether the output is actually easier to trust | Blinded review or deterministic defect checklist     |

## First Probe Results

These are smoke probes only. They are useful for telemetry shape, not model
quality claims.

| Arm                    | Model               | Result       | Wall time   | Usage captured                                                             | Direct cost signal               |
| ---------------------- | ------------------- | ------------ | ----------- | -------------------------------------------------------------------------- | -------------------------------- |
| Raw Ollama             | `qwen3:8b`          | Exit 0       | 13.0s       | No token usage from raw CLI; output included thinking/control text         | $0 API cost                      |
| Codex frontier         | `gpt-5.4-mini`      | Exit 0       | 9.2s        | 19,597 input, 2,432 cached input, 238 output, 153 reasoning tokens         | API-equivalent estimate required |
| Claude bare print      | `haiku`             | Auth failure | 2.2s        | JSON result reported zero tokens and `Not logged in`                       | $0                               |
| Codex local via Ollama | `qwen3:8b`          | Exit 0       | 26.6s       | 4,096 input, 369 output tokens in Codex JSONL; `ollama ps` showed 100% GPU | $0 API cost                      |
| HA-HR1 fake-live       | deterministic shell | Exit 0       | ~1.1s total | Per-step wall time, timeout, exit code, estimated USD, GPU seconds         | Harness only                     |
| PL Ollama smoke `A`    | `qwen3:8b`          | Failed `0/1` | 21.0s       | 496 input, 716 output, 1212 total tokens; `ollama ps` showed 100% GPU      | $0 API cost                      |
| PL Codex smoke `A`     | `gpt-5.2`           | Passed `1/1` | 26.9s       | 112,399 input, 502 output, 110,336 cached input, 251 reasoning output      | Cost null until pricing basis    |
| PL Claude smoke `A`    | `claude-opus-4-7`   | Passed `1/1` | 31.9s       | 23 input, 796 output, 308,988 cache read, 62,602 cache creation            | Provider-reported $0.5657715     |

## Repeated Provider Telemetry Matrix

The first bounded repetition used smoke id `A`, because it exercises real prompt
turns and therefore produces provider telemetry. The local artifacts are under
`scripts/eval/results/`; regenerate the aggregate view with:

```sh
npm run eval:smoke:summary -- --test A --harness ollama,codex,claude --last 12
```

Observed on 2026-05-06 in this bounded prompt-backed smoke slice:

| Provider | Harness | Model label                    | Runs | Result | Median wall time | Token/cost signal                                                                                    | Cost basis        |
| -------- | ------- | ------------------------------ | ---: | ------ | ---------------: | ---------------------------------------------------------------------------------------------------- | ----------------- |
| Ollama   | ollama  | `ollama/qwen3:8b`              |    3 | `0/3`  |            21.1s | 1,488 input, 2,148 output, 3,636 total; `ollama ps` showed GPU residency                             | zero API cost     |
| Codex    | codex   | `gpt-5.2`                      |    5 | `5/5`  |            27.1s | 314,408 input, 1,382 output, 308,736 cached input, 772 reasoning output                              | unknown           |
| Claude   | claude  | CLI default / partial metadata |    3 | `3/3`  |            44.4s | 69 input, 2,487 output, 967,945 cache read, 146,753 cache creation; provider-reported cost $1.463699 | provider-reported |

Claim boundary: this is telemetry-comparable, not outcome-comparable proof.
`k=3` is a stability probe, not statistical proof. The Ollama result is a
behavioral failure with useful local telemetry; the Codex result is a pass-rate
signal with cost intentionally left null; the Claude result is pass-rate plus
provider-reported cost for this smoke scope.

Using OpenAI's published GPT-5.4 mini API price as a rough equivalent
($0.75/1M input tokens and $4.50/1M output tokens, published 2026-03-17), the
Codex frontier probe is roughly $0.014-$0.016 if billed like API usage. Codex
CLI may account by quota rather than raw API billing, so store the raw token
usage and compute cost from the run's declared pricing table.

## Decision Criteria

PL is useful when at least one of these is true:

- It raises oracle pass rate enough to offset added wall time and setup.
- It prevents false completion or invalid output that direct prompting misses.
- It reduces human intervention minutes through gates, retries, and clearer
  failure states.
- It enables local models to complete bounded work that direct local prompting
  cannot complete.
- It reduces frontier calls per successful task without lowering oracle score.
- It produces better traceability when auditability is an explicit requirement.

PL is probably wasteful when:

- Direct prompting matches oracle score with materially lower wall time.
- PL adds retries but does not improve pass rate, defect count, or intervention
  time.
- The task is trivial, one-shot, and low risk.
- The harness/runtime failure rate is high enough that model quality cannot be
  interpreted.
- Hybrid routing uses frontier calls close to frontier-only volume without a
  quality gain.

## Next Implementation Slice

Continue live measurement support before running larger pilots:

1. Add `--live-probe` or `--live` support for one read-only prompt task.
2. Capture stdout, stderr, timeout, exit code, wall time, and raw provider events.
3. Reuse the prompt-turn provider telemetry artifact for PL-routed Codex,
   Claude, and Ollama runs.
4. Attach a versioned pricing basis before converting Codex/Claude token counts
   into estimated cost claims unless the provider reports exact cost.
5. Use the harness `--local-resource-snapshot-command` option for local live
   runs, and add `--local-resource-snapshot-interval-ms` when residency can be
   missed by before/after snapshots; `ollama ps` proves model residency, not
   sustained utilization. Read `resourceSnapshotSummary` first to confirm sample
   count, probe failures, and whether sampled stdout/stderr carried any output.
6. Keep `local-only`, `frontier-only`, `advisor-only`, and `hybrid-router` claims
   separate.

## Minimum Pilot

Run one task across four arms:

| Arm             | Purpose                                       |
| --------------- | --------------------------------------------- |
| `local-only`    | Measures cheap local capability               |
| `frontier-only` | Measures quality/time/cost of direct frontier |
| `advisor-only`  | Measures whether advice without routing helps |
| `hybrid-router` | Measures local bulk plus frontier escalation  |

Hard caps for the first pilot:

- Wall time: 15 minutes per arm.
- Frontier spend: $1 total for the smoke pilot.
- Frontier calls: 2 maximum for hybrid on one task.
- Local repair loops: 2 maximum before classifying failure.
- Repeat count: `k=1` only until telemetry is clean, then `k=3`.
