# Pipeline status & ways forward — 2026-05-12

This document consolidates the two independent assessments produced
today (`20260512_codex.md` and `20260512_claude.md`), then proposes
concrete ways forward — including an honest evaluation of splitting
`extract-parameters-from-full` into a candidate-identification step and a
structuring step.

## TL;DR

- The two AI assessments **agree on the dominant failure mode**:
  ids can fall into a gap between extraction, bounds, and
  calculations, producing zero finite results for the pipeline's
  payoff outputs. Codex frames this as a *schema contract* problem;
  Claude frames it as the *bounds-skip-but-no-calc black hole*. Same
  bug, different framing levels.
- Both agree the pipeline is genuinely useful when it works, and
  that the Monte Carlo runner is not the bottleneck — model
  construction is.
- **Highest-leverage next move: tighten the validation contract**
  along the lines Codex proposes. Add the rule "every important
  model variable must be explicit, bounded, or calculated — no
  fourth state."
- **Splitting `extract-parameters-from-full` into a 2-step
  candidate-then-structure flow is worth trying**, but it does not
  directly fix the dominant failure mode. Best framed as a
  provenance/debuggability win, not a model-correctness fix. Do it
  *after* the validation contract is tightened, not before.

## Comparison: codex vs claude assessments

### Headline: same diagnosis, different framing levels

Two independent AI reviews of the same pipeline reaching the same
diagnosis is reasonably strong evidence the diagnosis is right.

### Where they agree

| Claim | Both agree |
|---|---|
| Monte Carlo runner is fundamentally solid | yes |
| Pipeline is genuinely improving v20 → v23 | yes |
| The dependency black hole is the #1 issue | yes |
| Dead-end inputs persist across versions | yes |
| Duplicate calcs in v23 (`combined_viability_surplus_dkk` vs `combined_shock_surplus_dkk`) | yes |
| Off-peak coverage gate (Nuuk's stated KPI) is structurally absent | yes |
| Validation is too permissive | yes |

### Where they differ

| Dimension | Codex | Claude |
|---|---|---|
| **Evidence base** | Nuuk only (v20–v23) | All 3 domains (v11–v23) |
| **Diagnosis level** | Schema contract — "no fourth state should exist" | Symptom-level — "bounds-skip-but-no-calc" |
| **Prescriptiveness** | High — 7 concrete validation rules + v24 goal | Low — describes failure modes, no rule text |
| **Numbers cited** | Few specific | Many: `P(gate_pass)=0/10000`, `r=−1.00`, etc. |
| **Coverage of operational wins** | Doesn't mention them | Faraday's P=0, threshold pass-rate as most useful output |
| **Schema redesign suggestion** | `missing_inputs_to_bound` vs `derived_calculations_needed` split | Not proposed |
| **Pipeline-as-product framing** | "Not yet reliable enough to trust unattended" | "Genuinely useful at this point" |

### Where Codex is sharper

- **Schema-contract framing** is more useful than symptom framing — it exposes that the fix belongs at the schema level, not as a per-stage patch.
- **The `missing_inputs_to_bound` vs `derived_calculations_needed` split** directly maps to the bug. Claude noticed the bug 3 times but didn't propose a structural fix.
- **The 7 proposed validation rules** are drop-in actionable.
- **Closing invariant** is the cleanest single sentence either produced: *"Every important model variable must be either explicit, bounded, or calculated. No fourth state should exist."*

### Where Claude is sharper

- **Three-domain coverage** prevents overfitting to small-business plans. Codex's Nuuk-only evidence might miss patterns visible in Faraday and Heatwave.
- **Concrete operational wins** (Faraday's hard zero, Nuuk's single-shock dominance, Heatwave's 25 median deaths avoided) — context for "is the pipeline already useful?".
- **Function-name discovery bug at v19a** — evidence Monte Carlo had at least one real wiring bug. Codex grades MC unconditionally "good".
- **Threshold pass-rate framing** — Codex doesn't mention this is the operationally most useful output.

### Bottom line of the comparison

If acting on one document, **act on Codex** — its prescriptions are at the right level. The schema-contract framing and the 7 validation rules are directly implementable. Use Claude for context on what to keep working while fixing.

## Highest-leverage ways forward

### 1. Tighten the validation contract (highest priority)

Add Codex's invariant as a hard validator rule:

```text
For every RHS variable in every formula:
  it must resolve to one of:
    - explicit numeric key_value
    - bounded missing primitive (in missing_values_to_estimate AND in bounds.json)
    - output of a recommended_first_calculation
    - output of a derived_question

If not, validate-parameters must ERROR (not WARN).
```

This single rule catches the `rental_revenue_shortfall_dkk` failure
in v23 *before* scenarios and Monte Carlo waste 10,000 runs producing
null. It also catches `required_members_for_payroll` in v20 and
`registered_vulnerable_population_denominator` in v15.

### 2. Add primitive-vs-derived discipline to `missing_values_to_estimate`

Either change the schema (Codex's preferred fix) — split into
`missing_inputs_to_bound` for primitives and
`derived_calculations_needed` for missing derivations — or enforce
the distinction in the validator. The naming heuristic (Codex
suggests warning on ids containing `gap`, `shortfall`, `surplus`,
`deficit`, `ratio`, `required`, `expected`, `remaining`, `_after_`)
is a reasonable first cut.

### 3. Mandate combined-gate preservation

When multiple shocks draw on the same reserve / runway / margin, the
pipeline must produce the combined-surplus output. Nuuk has shown
twice (v22 lost it after v21 added it; v23 attempted it but black-holed).

### 4. Validate end-to-end coverage of stated funding gates

Every plan KPI named as a "gate" or "pass/fail criterion" in the
extracted `comment` or `source_text` should appear as either a
calculated output OR an explicit `monte-carlo` threshold target.
Nuuk's off-peak coverage gate has been absent across v20–v23 despite
being the plan's named funding gate.

### 5. Add explicit-value parsing rule

If `value_type: "explicit"` and `value: null` but the literal value
appears in `source_text` or `comment` (parseable number or date),
WARN. This catches v23's `operational_readiness_date` issue
("2026-09-15" lived in the comment but `value` was null).

### 6. Detect and dedupe equivalent calculations

When two functions have the same RHS and same `depends_on` but
different LHS ids (v23 had `combined_viability_surplus_dkk` and
`combined_shock_surplus_dkk` doing the same arithmetic), warn or
auto-dedup. Cosmetic but it confuses interpretation.

## On splitting `extract-parameters-from-full` into two steps

### The proposal

Currently `extract-parameters-from-full` does everything in one shot: read the
report → identify what matters → triage to ≤8 → categorise → assign
units → write formulas → produce JSON. The proposal is to split into:

- **Step A — candidate identification**: scan the report, return a
  list of `{line_number, snippet, why_relevant}` items pointing at
  every potentially-important parameter
- **Step B — structuring**: read the much smaller candidate list and
  produce the schema-conformant JSON (key_values,
  missing_values_to_estimate, derived_questions,
  recommended_first_calculations)

### What this would buy

| Benefit | Strength |
|---|---|
| **Provenance** — every key_value traceable to a specific line, not paraphrased into `source_text` | high |
| **Debuggability** — when the pipeline misses an important parameter (Nuuk off-peak gate), you can inspect Step A's output to see whether it was identified at all | high |
| **Cheaper iteration on structuring rules** — changing the schema/categorisation prompt only re-runs Step B on the candidate list, not the whole 350 KB markdown | medium [†] |
| **Independent test surfaces** — Step A measures recall (did we find the candidate?), Step B measures precision (did we structure it correctly?). Today they're entangled | medium |
| **Smaller second-pass context** — Step B operates on a candidate list that's maybe 5–10 KB, not the full report | medium [†] |

[†] These two benefits assume Step A returns extracted snippets. They
weaken substantially if Step A uses **in-place XML tagging** (preferred
— see "Step A output format" below). With in-place tagging, Step B
still receives the full report, just marked up.

### What this would NOT buy

- **It does not fix the dominant failure mode.** The black hole sits
  between extraction (any version), bounds, and calculations. Splitting
  extraction in two doesn't change the fact that a derived id can land
  in `missing_values_to_estimate` and get stranded.
- **It does not improve modelling quality of the off-peak coverage
  gate** unless Step A is *better* at noticing the gate than the
  current single-shot extractor. There's no obvious reason it would be
  — both have the same input.
- **Triage to ≤8 gets harder when split**, not easier. Step B
  re-evaluating "is this candidate more important than that one?"
  needs the full prose context, which Step A's
  `{line_number, snippet}` may not preserve enough of.

### Risks

| Risk | Severity |
|---|---|
| **Step A recall failures poison everything downstream** — if Step A misses an important parameter, Step B has no way to recover it. Today the single-shot at least sees the whole report at structuring time. | high |
| **Calibrating "candidate" is hard** — too narrow and you re-introduce today's blind spots; too wide and Step B is overwhelmed | medium |
| **More moving parts** — two skills, two prompts, two sets of iteration overhead | medium |
| **Loss of holistic context** — sometimes the right key_value isn't directly stated; it's inferred from a constellation of mentions. Step A focused on explicit hits may miss the constellation. | medium |

### Step A output format: in-place XML tagging

The natural output for Step A is **the report verbatim, with
`<planexe-candidate>` tags wrapped around the relevant lines**.
Concretely:

```markdown
The Year 1 budget is fixed at <planexe-candidate reason="budget">2,000,000 DKK</planexe-candidate>
under the Builder's Balance strategy.

The contingency is <planexe-candidate reason="risk-buffer">15% of budget (300,000 DKK)</planexe-candidate>,
ring-fenced for unexpected shocks.
```

Step B then receives this marked-up document instead of the original.

#### Why in-place XML tagging beats a JSON `{line, snippet}` list

| Property | Verbatim XML tagging | JSON snippet list |
|---|---|---|
| Preserves surrounding context for Step B | yes — natural | no — lost |
| Verbatim integrity checkable by diffing | yes | no — Step A may rephrase the snippet |
| Step B sees structural cues (section headers, tables) | yes — for free | no — must encode separately |
| Provenance | yes — verbatim source text via tag position | depends on snippet quality |
| Second-pass token cost | same as today (full report) | smaller |

The verbatim form trades token cost for context — and at ~350 KB for
the Nuuk markdown, context matters more than tokens.

#### Specific design choices

| Decision | Recommendation | Why |
|---|---|---|
| Tag name | `<planexe-candidate>` (or another project-namespaced tag) | Avoids collisions with markdown that may already contain HTML/XML |
| Granularity | Semantic unit (sentence, list item, table row) — not literal "line" | Markdown lines are inconsistent: `# header` is one line, a wrapped paragraph is one line, a budget table row is one line. Sentences match the unit of meaning |
| Attributes | One short `reason="..."` attribute | Lets Step B triage when overwhelmed; keeps prompt complexity low |
| Step A output | Entire report with tags inserted, nothing else | Preserves global structure; no sidecar to keep in sync |
| JSON sidecar | Defer — re-derive line numbers later from tag positions if needed | Reduces moving parts in v1 |
| Tag-density budget | Cap at ~50 tags per report; treat overflow as a prompt-quality signal | Without a budget Step A will tag too liberally and Step B has no signal |

#### What this changes about the tradeoff

In-place XML tagging means **Step B still receives the full ~350 KB
report** with tags. The "smaller second-pass context" benefit listed
in the table earlier mostly evaporates, and "cheaper iteration on
structuring rules" weakens too. The case for the split now rests on
**provenance, debuggability, and independent test surfaces** — not on
token cost.

If small-context is also wanted, Step A would need to **prune** —
return only the sections containing tags, plus a context window (e.g.
±10 lines). That's an additional design choice, and pruning loses
cross-section relationships (a budget figure in section 2 referenced
by a risk in section 8 needs both visible to Step B). I'd defer the
pruning question — start with full-document in-place tagging, since
context matters more for correctness, and add pruning later if Step
B's input actually becomes a problem.

### My recommendation

**Worth trying, but it is the second-priority fix, not the first.**
The contract-tightening work above closes the failure mode both
documents identified. The split is a provenance/debuggability win that
would *help diagnose* why the off-peak coverage gate keeps getting
missed, but doesn't directly fix it.

Concrete sequencing:

1. **First**, do the contract-tightening pass (validation rules above
   + the missing-values discipline). This is roughly v24 in Codex's
   numbering. Re-run all three domains (Heatwave v17, Faraday v20,
   Nuuk v24) and confirm the black hole is closed.
2. **Then** evaluate whether the persistent extraction misses (Nuuk
   off-peak gate; instructor capacity dead-end across v20–v23) are
   recall problems — i.e. the extractor never noticed — or
   structuring problems — it noticed but discarded. Today we can't
   tell the difference.
3. **If it's a recall problem**, build the 2-step split as a
   diagnostic experiment: run Step A on Nuuk and inspect whether the
   off-peak gate appears in the candidate list. If it does, then the
   structuring step is the issue (and the split helps). If it
   doesn't, then a different recall-improvement intervention is
   needed (chunking, different scanning prompt, or
   domain-specific extractor variants).
4. **If the contract tightening alone fixes the per-run failures**,
   the split becomes lower priority — provenance is nice but not
   load-bearing.

### Alternative: keep the 1-step extract but add a `candidates_inspected`
side-channel

A lighter-weight version of the proposal: keep `extract-parameters-from-full`
as one step, but require it to also emit a `candidates_inspected`
list — every candidate it considered, with a brief
`accepted | rejected: <reason>` annotation. This gets most of the
debuggability benefit (you can see what was rejected and why) without
splitting the skill in two. Worth weighing against the full split.

## Recommended v24 goal

Tightly scoped:

1. Add validator rule: every formula RHS variable must resolve to
   one of {explicit key_value, bounded missing primitive, recommended
   calc output, derived question output}. ERROR severity.
2. Add validator rule: ids in `missing_values_to_estimate` whose name
   matches `gap | shortfall | surplus | deficit | ratio | required_ |
   expected_ | remaining_ | _after_` produce a WARN unless explicitly
   justified.
3. Add validator rule: explicit value_type with null value triggers a
   WARN if `source_text` or `comment` contains a parseable number or
   date.
4. Add validator rule: high or critical priority key_value that feeds
   no calculation triggers a WARN.
5. Add validator rule: two calculations with equivalent RHS and
   equivalent dependencies trigger a WARN.
6. Re-run Nuuk v24. Success criterion: the combined-survival output
   has finite values in 100% of Monte Carlo runs.

## Recommended v25 (or v24.5) goal

Decide whether to do the extract-parameters-from-full split based on what v24
shows:

- If v24 fixes the dominant failures and the model now produces the
  off-peak coverage gate correctly: defer the split indefinitely.
- If v24 fixes the contract bug but recall problems persist (the
  off-peak gate still doesn't show up; instructor capacity still
  unconnected): build the split as a diagnostic. Start with the
  candidate list as a side-channel (Alternative above) before
  committing to a full skill split.

## Closing thought

The pipeline is at the stage where the single-paragraph answers it
produces (Faraday: `P(gate_pass) = 0`; Nuuk: labor-shock dominance at
`r = −1.00`) are genuinely useful — useful enough that the marginal
return on more domain-coverage testing is now small. The marginal
return on **closing the structural bug both documents identified** is
much higher. Once the black hole is closed, the cycle of "v22 added
the gate, v23 lost it to a dependency, v24 re-adds it" stops, and
each subsequent iteration moves the model forward instead of
sideways.
