# Pipeline status — Claude's view, 2026-05-12

This document captures my (Claude's) first-hand perception of the
PlanExe parameter modelling pipeline after running it across three
unrelated plan domains and 23 versioned iterations of the system
prompts. It is intentionally honest: things that work, things that
don't, and where the recurring structural gaps are.

## TL;DR

Six skills are built and round-trip end-to-end on three unrelated
plans: a public-health resilience plan (Leipzig Heat Response), a
commercial product launch (Estonian Faraday enclosure), and a
community small-business plan (Nuuk Clay Workshop). The pipeline
produces materially useful answers — `P(gate_pass) ≈ 0` for Faraday,
single-shock dominance with r=−1.00 for Nuuk's labor-law variable,
~25 median deaths avoided for the Leipzig pilot. The same prompts
handle all three domains without per-domain customisation.

The recurring structural gap is a **bounds-skip-but-no-calc black
hole** that has bitten three times across versions (v15, v20, v23).
When the extractor places a derivable id in
`missing_values_to_estimate`, the bounds skill skips it expecting
`generate-calculations` to handle it, but `generate-calculations`
only emits functions for entries in `recommended_first_calculations`
and `derived_questions`. The id ends up neither bounded nor computed,
and any calculation that depends on it produces 0 finite results.
Closing this gap is the single highest-leverage improvement available.

## Skill-by-skill status

### 1. extract-parameters-from-full — solid

Reliably produces a parameter JSON that other stages can consume. Hard
caps (8/5/5/5) are respected. Schema discipline (depends_on uniqueness,
formula RHS declaration, `_eur`/`_dkk` currency suffixes,
fractions-as-fractions) is consistent across runs. Markdown ingestion
(introduced at v20) is materially cheaper than HTML — same plan as
markdown is roughly 22% the file size and parses cleanly without HTML
artifact stripping in `source_text`.

The dead-end-prevention rule is partly working: the extractor proactively
drops variables that no calculation would consume (the Nuuk extractor
explicitly dropped `staffing_fte` once for this reason). But it still
misses about 2-3 dead-ends per run on average across the runs I have
inspected.

### 2. validate-parameters — solid

Catches real schema drift cleanly. The Faraday v16 caught a S003 error
(extractor included `suggested_low/base/high` on a key_value despite the
schema disallowing it) — that's the kind of catch this skill is for.
Stable rule IDs (S/C/E/V/I/F/D/Q/T) make violations machine-parseable.
Across 11 full pipeline runs, validation has produced exactly one ERROR
(Faraday v16) and a handful of WARNs.

### 3. repair-parameters — unbuilt and not blocking

Still optional and not implemented. The extractor has produced clean
output in 10 of 11 runs without it. Worth deferring until a real
recurring extraction failure appears that local schema fixes can resolve.

### 4. generate-bounds — solid for sampling, has the dependency black hole

For variables it bounds, the spread heuristics (low uncertainty
±10–20%, medium ±25–50%, high 2–5×, fractions clamped to [0,1],
counts as integers, binary-gate Bernoulli for monetary `low=0,
base=high=X` patterns) produce defensible ranges. Source labelling
(`data` vs `assumption`) is honest.

The black hole: when an `missing_values_to_estimate` entry looks like
a derived output (id contains `_shortfall`, `_gap`, etc.), bounds
skips it under the rule "skip if calculable from declared inputs". But
nothing else picks it up — `generate-calculations` only emits
functions for `recommended_first_calculations` and `derived_questions`
entries. Net: the id is unresolvable, and calculations depending on
it produce 0 finite results. This pattern bit Nuuk v20
(`required_members_for_payroll`), Heatwave v15
(`registered_vulnerable_population_denominator`), and most recently
Nuuk v23 (`rental_revenue_shortfall_dkk` — both `combined_*_surplus`
outputs were null in 100% of MC runs).

### 5. generate-calculations — solid

Always produces an AST-parsable Python module. Function names use the
`formula_hint` LHS; arguments match `depends_on` exactly in declared
order; division guards are placed correctly; `P(...)` notation
becomes a `NotImplementedError` stub. Argument-name shadowing of
sibling function names is handled correctly per the explicit rule.

LHS collisions between `recommended_first_calculations` and
`derived_questions` (when two entries name the same output) produce a
fallback to the entry id, which can leave duplicate functions in the
module computing identical values (e.g. v23's `combined_shock_surplus_dkk`
and `combined_viability_surplus_dkk` are identical bodies under
different names). Cosmetic but worth dedup'ing upstream.

### 6. run-scenarios — solid

Deterministic low/base/high evaluation produces clean tables.
`spread_ratio = high/low` correctly inverts (`<1`) for outputs where
higher inputs make the result worse (runway, contingency-after-shock).
Skips and warnings are scenario-agnostic when the cause is structural
(null `formula_hint`).

### 7. monte-carlo — solid after one wiring fix

Function-name discovery from `formula_hint` LHS (rather than from the
`q_*`-prefixed entry id) was a real bug that shipped at v19 and was
fixed at the prompt level — a re-run on the same artifacts then
discovered all generated functions. The Bernoulli-on-binary-gate rule
correctly bimodalises `total_available_funding_eur`. Sensitivity
rankings via Pearson correlation produce the right top driver in every
case I've inspected (e.g. Nuuk's `labor_law_cost_uplift_fraction` at
r=−0.94 against contingency-after-shock).

Threshold pass-rate computation against explicit `settings.thresholds`
is the most operationally useful output the pipeline produces.
Faraday: `P(gate_surplus_eur ≥ 0) = 0.0%` across 10k runs is a
concrete answer to a concrete question.

## How well does the pipeline model the most important parameters?

### Leipzig Heat Response (public health) — well

By v16: the model computes a coverage cascade (vulnerable population
estimate → contacted → cooling-centre served → protected → avoided
deaths) and a unit-economics line (`cost_per_protected_person`).
Monte Carlo produces median 25 avoided deaths with p05–p95 of 7–64.
Sensitivity correctly identifies `baseline_heat_mortality_rate_per_person`
as the dominant unknown, then `intervention_effectiveness`, then
`registered_vulnerable_population`. Zero dead-end inputs in v16.
Genuinely useful for tuning the plan's data-gathering priorities.

### Faraday Enclosure (commercial) — very well, with a clear "no" answer

By v19: the model goes all the way through Year-1 P&L
(revenue → gross profit → operations net cash → gate surplus →
runway → MIL-STD funding gap → required market penetration). With
explicit thresholds in v19c/d, the pipeline returns
`P(gate_surplus_eur ≥ 0) = 0/10000 = 0%` and
`P(required_market_penetration ≤ 1) = 28.8%`. The plan needs
146% of the addressable market to break even at base case. This is the
clearest single answer the pipeline has produced — the model says
unambiguously "this product cannot pass its own gate at any plausible
parameter combination." That answer would be hard to derive from the
prose plan alone.

### Nuuk Clay Workshop (community business) — partially

By v22: the model decomposes the labor-law shock into
`baseline_fixed_labor_cost × labor_law_cost_uplift_fraction`,
mechanistically grounded in the report's "20–35%" uplift quote.
Sensitivity correctly identifies the labor-uplift fraction as the
dominant driver (r=−0.94 against contingency surplus).

What the model doesn't capture across v20–v23: the **off-peak
hourly-rental coverage gate** that the report names as the actual
funding-gate KPI. The hourly rental rate, off-peak hours, and utility
overhead are extracted and bounded but no calculation tests "does
off-peak revenue cover utility overhead?". Three iterations in, that
gate is structurally absent. This is a real model-quality gap.

By v23: the model attempts to combine the survival metric
(`combined_viability_surplus_dkk`) but trips on the dependency black
hole described above — every Monte Carlo run produces null. So the
intent has caught up to the plan's actual structure but the wiring
between bounds and calculations hasn't.

## Recurring failure modes

In rough order of impact:

| # | Issue | Domains hit | Severity |
|---|---|---|---|
| 1 | Bounds-skip-but-no-calc dependency black hole | Heatwave v15, Nuuk v20, Nuuk v23 | high — produces null outputs in 100% of MC runs for affected calcs |
| 2 | Dead-end bounded inputs persist across iterations | All three domains | medium — wastes sampling budget, distorts no answer |
| 3 | Unmodelled stated funding gates (Nuuk off-peak coverage) | Nuuk v20–v23 | medium — pipeline answers a different question than the plan asks |
| 4 | Unit-inference substring matching false positives (`operations` matches `ratio`) | Faraday v18 | low — fixed at prompt level by v19 with whole-token matching |
| 5 | Function-name discovery via entry id rather than formula LHS | Faraday v19 | low — fixed at prompt level after one missed run |
| 6 | LHS collisions producing semantically duplicate functions/outputs | Nuuk v21, v23 | low — cosmetic, no incorrect numbers |
| 7 | Currency support beyond EUR done ad-hoc by agents | Nuuk v20–v23 (DKK) | low — agents extended `_eur` symmetrically and it worked, but not codified |

## What the pipeline answers well today

- **"Can this plan pass its own funding gate?"** Faraday makes this a
  hard zero. Threshold settings are the right interface.
- **"Which single input most drives output uncertainty?"** Sensitivity
  rankings are reliable across all three domains.
- **"What does the downside / base / upside case look like?"**
  `run-scenarios` is fast, deterministic, and easy to read.
- **"Where does the model collapse?"** Heavy-tail / `inf` warnings on
  outputs whose denominator can go negative (Faraday's contribution
  margin in p05) surface real model fragility.

## What the pipeline doesn't yet answer well

- **"Does the off-peak coverage gate pass?"** Stated KPI, structurally
  unmodelled in Nuuk v20–v23.
- **"What if all key levers move adversely together?"** Correlation
  groups exist in the monte-carlo settings spec but I haven't seen them
  actually used. Default is full independence.
- **"Why is `breakeven_units` `inf` in 26% of runs?"** The model reports
  the collapse rate but doesn't explain which input combination
  triggered it.
- **"What's the smallest set of input revisions that flips
  `P(gate_pass)` above 50%?"** The pipeline can answer the gate-pass
  question but not the inverse design question.

## Closing thought

The pipeline is genuinely useful at this point — it produces the kind
of single-paragraph "this plan can/can't pass its own gate" answer
that a careful human reading the prose plan would take much longer to
arrive at. The biggest concrete improvement available is closing the
bounds-vs-calculations dependency black hole, because every
recurrence of it removes a calculated output from the model's payoff
column. Beyond that, the model-quality gap to close domain-by-domain
is making sure every funding-gate KPI named in the prose plan ends up
as a calculated output the pipeline can test against.
