# Monte Carlo Pipeline Status Assessment

## Scope

This document summarizes the current state of the plan-to-Monte-Carlo simulation pipeline, based primarily on the Nuuk Clay Workshop test case across versions v20, v21, v22, and v23.

The focus is not whether the clay workshop plan itself is viable. The focus is whether the pipeline is successfully extracting the most important modelling parameters from plans, turning them into executable calculations, assigning bounds, running scenarios, and producing Monte Carlo outputs that answer the plan's core viability questions.

## Executive Summary

The Monte Carlo simulation code appears directionally solid once it receives a coherent model. It can run deterministic scenarios, sample bounded inputs, produce distributions, report percentiles, and calculate sensitivity correlations. The main problems are upstream: parameter extraction, schema semantics, bounds generation, calculation generation, and validation contracts.

The pipeline is improving. v20 failed because critical derived questions had null formulas. v21 fixed executable formulas and produced a meaningful combined viability gate. v22 improved mechanistic modelling of labor-law shock, but lost the composite gate. v23 restored the combined survival-gate intent, but exposed a deeper wiring bug: a derived intermediate was placed in `missing_values_to_estimate`, skipped by bounds, and not generated as a calculation, causing the most important Monte Carlo output to be missing in every run.

The current state is best described as:

> The Monte Carlo engine is probably usable, but the model-building pipeline is not yet reliable enough to trust unattended.

The pipeline can now often identify the right risks, but it does not consistently preserve them as executable, bounded, end-to-end calculations.

## Current Assessment

| Area | Status | Assessment |
|---|---:|---|
| Monte Carlo runner | Good | Works when given bounded inputs and executable calculations. |
| Scenario runner | Mostly good | Correctly exposes skipped/unresolvable calculations. |
| Sensitivity analysis | Good enough | Correlations are useful for identifying dominant drivers. |
| Parameter extraction | Improving | Now better at extracting executable formulas, but still misclassifies derived intermediates. |
| Bounds generation | Fragile | Skips some `missing_values_to_estimate` entries when their names look derivable. |
| Calculation generation | Narrow contract | Only emits functions from recommended calculations and derived questions. This is fine, but other stages must respect it. |
| Validation | Too permissive | Allows unresolved dependencies, duplicate formula logic, dead-end important variables, and missing explicit values. |
| End-to-end reliability | Not yet good | The most important output can disappear while validation still reports valid. |

## What the Monte Carlo Code Seems to Do Well

The Monte Carlo code itself is not where I see the earliest or most serious problem.

It appears to do these things correctly:

- Runs 10,000 sampled simulations.
- Produces finite-output summaries when dependencies resolve.
- Reports mean, standard deviation, min, percentiles, and max.
- Tracks missing outputs when calculations cannot resolve.
- Produces sensitivity correlations between outputs and input drivers.
- Surfaces warnings when calculations are skipped or unresolvable.

The Monte Carlo output is useful when the model is wired correctly. For example, in earlier versions it clearly showed that the plan's contingency logic was fragile under labor-law shock and speculative revenue shortfall. That is exactly the kind of insight the simulation should produce.

The main issue is that Monte Carlo can only evaluate the model it is handed. When upstream stages produce a structurally incomplete model, Monte Carlo accurately reports missing outputs, but it cannot repair the model.

## Main Pipeline Pattern Across Versions

### v20

v20 had the earliest obvious failure:

- Important `derived_questions` existed.
- Their `formula_hint` values were null.
- The scenario and Monte Carlo stages skipped those questions.
- The pipeline did not answer the core plan-viability gates.

The key problem was that parameter extraction identified the right questions but did not turn them into executable formulas.

### v21

v21 was a major improvement:

- Derived questions received formulas.
- Skipped calculations dropped to zero.
- Scenario warnings disappeared.
- A meaningful combined gate appeared: whether the contingency cushion covered the speculative revenue gap.

This was the first version that felt like the pipeline was modelling the plan's core risk structure rather than just extracting decorative numbers.

Remaining issues in v21:

- Some inputs were still dead-ended.
- Duplicate output/collision behavior appeared.
- The off-peak utility coverage gate was still missing.
- Instructor capacity was computed but not connected to revenue capacity or staffing cost.

### v22

v22 improved the mechanism of the labor shock model:

- Labor shock became a function of baseline fixed labor cost multiplied by a cost uplift fraction.
- This is better than sampling a free-floating scalar shock.

However, v22 regressed on decision-gate quality:

- The combined viability gate disappeared.
- The user had to mentally combine labor-shock survival with revenue-gap risk.
- One derived question regressed to a null formula.
- Dead-end inputs remained.

The lesson from v22:

> Better mechanistic decomposition is not enough. The pipeline must preserve the highest-level viability question after decomposition.

### v23

v23 restored the combined-survival-gate intent:

- `combined_viability_surplus_dkk` appeared.
- `combined_shock_surplus_dkk` appeared.
- Null derived questions were eliminated.
- The pipeline attempted to model the actual Builder's Balance survival test.

But v23 exposed a deeper schema-contract bug:

- `rental_revenue_shortfall_dkk` was referenced by the combined-surplus calculations.
- It was placed in `missing_values_to_estimate`.
- Bounds generation skipped it, apparently because the name looked like a derivable output.
- Calculation generation did not generate it, because it only emits functions from recommended calculations and derived questions.
- Result: the most important combined-surplus outputs had zero finite Monte Carlo results.

This is the most important current diagnosis:

> The pipeline lacks a strict contract for whether a quantity is a primitive unknown, a bounded assumption, or a derived calculated output.

## The Most Important Current Failure Mode

The recurring failure is not just one bad prompt. It is a schema-contract problem.

Currently, an identifier can fall into a gap between pipeline stages:

1. Parameter extraction references an id in a formula.
2. The id is declared as `missing_values_to_estimate`.
3. Bounds generation decides not to bound it because it looks derived.
4. Calculation generation does not create a function for it because it is not in `recommended_first_calculations` or `derived_questions`.
5. Scenario and Monte Carlo cannot resolve it.
6. Validation still passes or only weakly warns.

This creates a black hole for ids such as:

- `*_gap`
- `*_shortfall`
- `*_surplus`
- `*_deficit`
- `*_ratio`
- `required_*`
- `expected_*`
- `remaining_*`
- `*_after_*`

Many of these names are usually derived outputs, not primitive missing assumptions. They should usually be calculations, not bounded inputs.

## How Well the Pipeline Models Important Plan Parameters

The pipeline is getting better at identifying important plan parameters. It repeatedly surfaced the correct high-leverage dimensions in the Nuuk case:

- Contingency reserve.
- Labor-law reclassification or labor-cost uplift risk.
- Speculative/off-peak revenue realization.
- Taster-session conversion.
- Hourly rental revenue assumptions.
- Material buffer / working capital.
- Staffing capacity.
- Operational readiness deadline.

That is good. The extractor is not merely pulling random numbers.

The problem is that the pipeline is inconsistent at turning those important parameters into complete causal calculations.

Examples:

- It identifies rental rate, hours, and utilization-like quantities, but does not always turn them into bottom-up revenue or coverage calculations.
- It identifies instructor capacity, but does not reliably connect it to revenue capacity, service reliability, staffing cost, or session coverage.
- It identifies material buffer costs, but may leave them detached from liquidity or runway calculations.
- It identifies plan deadlines, but may leave them as decorative dates rather than schedule-risk gates.
- It identifies combined viability pressure, but may wire it through an unbounded missing intermediate.

So the current quality is:

> Good at finding important concepts. Still unreliable at preserving them as executable model structure.

## Specific Evaluation of the Clay Workshop Test Case

The clay workshop is a good test case because it exposes several classes of modelling difficulty:

1. Narrative confidence vs numerical fragility.
   The plan sounds robust, but the simulation can reveal whether the contingency actually survives shocks.

2. Multiple interacting gates.
   Labor-law risk, speculative revenue, utility overhead, membership conversion, and staffing reliability interact.

3. Mixed explicit and implied quantities.
   Some values are stated directly, while others must be inferred from relationships such as rate × hours × utilization.

4. Physical operations.
   The plan involves inventory, utilities, staffing, equipment downtime, local demand, and seasonality.

5. Real schema stress.
   It repeatedly surfaces boundary bugs between extraction, bounds, calculations, validation, scenarios, and Monte Carlo.

The case is not ideal as the only benchmark because it may overfit the prompt toward small-business/community-workshop patterns. But it is a strong regression test because it keeps revealing whether the pipeline can preserve real viability logic.

## What Is Working

### 1. The pipeline can now improve iteratively

The v20 to v23 progression shows meaningful learning:

- Null formulas were reduced.
- Derived questions became executable.
- Mechanistic decomposition improved.
- Combined gate logic was introduced.
- Warning surfaces became more informative.

This suggests the architecture is amenable to improvement.

### 2. Monte Carlo output is interpretable

When outputs are finite, the simulation summaries are useful. The mean/p05/p50/p95 summaries and sensitivity drivers clearly identify which assumptions dominate model behavior.

### 3. The pipeline exposes model incompleteness

Even when it fails, the later stages expose useful symptoms:

- skipped functions
- missing outputs
- finite vs missing counts
- dead-end inputs
- sensitivity gaps

These are valuable diagnostics.

## What Is Not Working Yet

### 1. Validation is too weak

Validation should not report clean or near-clean status when:

- A formula RHS dependency is neither bounded, explicit, nor calculated.
- A critical output has zero finite Monte Carlo runs.
- A high-priority uncertain key value feeds no calculation.
- A derived question duplicates a recommended calculation.
- An explicit value has `value: null` while the value appears in comment/source text.
- A missing value looks like a derived output and is referenced by calculations.

The validator should become the main guardrail. Prompt text alone is not enough.

### 2. `missing_values_to_estimate` is semantically overloaded

It currently mixes two categories:

- primitive external assumptions, which should receive bounds
- derived-but-missing intermediates, which should become calculations

These need to be separated by schema or enforced by validation.

Suggested distinction:

- `missing_inputs_to_bound`: primitive uncertain inputs
- `derived_calculations_needed`: calculated quantities that require formulas

If the schema cannot change, then validation must enforce the distinction using naming and dependency rules.

### 3. Highest-level viability gates are not reliably preserved

The pipeline may decompose a model into parts but lose the top-level question.

For planning evaluation, the top-level question is often the most important output:

- Does the reserve survive combined shocks?
- Does capacity cover demand?
- Does revenue cover fixed and variable costs?
- Does staffing cover required service levels?
- Does the deadline survive realistic task duration uncertainty?

The pipeline must preserve these as final aggregate surplus, deficit, ratio, or boolean gates.

### 4. Rate × volume × utilization is still not automatic enough

When a plan contains a rate-like value and a volume/capacity-like value, the pipeline should strongly prefer bottom-up formulas:

- revenue = rate × volume × utilization
- cost = unit_cost × units
- capacity = hours × throughput × utilization
- staffing cost = FTE × hours × wage rate

Leaving rate-like values dead-ended is a sign that the model is not really testing the plan's economics or capacity.

### 5. Duplicate calculations are still possible

v23 had both `combined_viability_surplus_dkk` and `combined_shock_surplus_dkk`, which appear to be overlapping or duplicative.

Duplicate calculations are not fatal, but they confuse interpretation and may hide the fact that the model has fewer real outputs than it appears.

## Recommended Fixes

### Highest priority: validation contract

Add a validation rule:

```text
For every RHS variable in every formula:
  it must resolve to one of:
    - explicit numeric key_value
    - bounded missing primitive
    - output of a recommended_first_calculation
    - output of a derived_question

If not, validation must fail or warn loudly.
```

This single rule would catch the v23 `rental_revenue_shortfall_dkk` failure before scenarios and Monte Carlo.

### Add primitive-vs-derived missing-value rule

```text
missing_values_to_estimate must contain primitive external assumptions, not calculated outputs.

Ids containing gap, shortfall, surplus, deficit, ratio, required, expected, total, remaining, or after should trigger a warning unless explicitly justified as primitive external assumptions.
```

### Add zero-finite-output validation

```text
If a calculated output has zero finite values in scenarios or Monte Carlo, treat it as an error for critical outputs and a warning otherwise.
```

### Add composite-gate preservation rule

```text
If multiple pressures draw on the same reserve, budget, capacity, runway, or margin, compute a combined surplus or coverage ratio unless the plan treats them as independent.
```

### Add dead-end high-priority input rule

```text
Any critical or high-priority uncertain input should feed at least one calculation. If it does not, warn.
```

### Add duplicate formula detection

```text
If two calculations have equivalent RHS formulas and equivalent dependencies but different LHS ids, warn or deduplicate.
```

### Add explicit-value parsing warning

```text
If value_type is explicit and value is null, but source_text or comment contains a parseable number/date, warn.
```

## Recommended v24 Goal

The next target should not be a broad prompt rewrite. It should be a contract tightening pass.

v24 should aim for:

1. Keep v23's combined-survival gate.
2. Ensure every dependency in that gate is explicit, bounded, or calculated.
3. Prevent derived intermediates from entering `missing_values_to_estimate` without bounds or formulas.
4. Preserve rate × volume × utilization calculations.
5. Deduplicate equivalent outputs.
6. Make validation fail or warn before scenario/Monte Carlo if the payoff variable cannot resolve.

A successful v24 Nuuk run should have:

- zero skipped calculations
- zero unresolvable scenario outputs
- zero zero-finite Monte Carlo outputs for critical gates
- no duplicate combined gates
- no high-priority dead-end rental/staffing/capacity values
- a finite distribution for the final combined viability surplus

## Overall Verdict

The Monte Carlo simulation code is probably not the bottleneck. The bottleneck is the model construction pipeline around it.

The pipeline has made real progress:

- It now often identifies the right risks.
- It can produce executable formulas.
- It can build mechanistic relationships.
- It can attempt combined viability gates.

But it is not yet reliable because the schema contract is loose. Important quantities can become stranded between extraction, bounds, and calculation generation. Validation is currently too permissive to catch this early.

The right next move is not another large prompt rewrite. The right next move is stricter validation and clearer semantics:

> Every important model variable must be either explicit, bounded, or calculated. No fourth state should exist.

Once that invariant holds, the Monte Carlo outputs will become much more trustworthy.
