# Benchmark Results: Bracket 001 — Run 002

**Run date:** 2026-05-06
**Benchmark package:** `benchmark_runs/bracket_001_manual/`
**Package generated from:** `build/bracket_001.aieng` — Phase 6A/6B build including `simulation/solver_deck.inp` and `validation/status.yaml`

---

## Purpose of Run 002

Run 002 specifically tested the Phase 6A and Phase 6B additions to the Condition B file set:

- `simulation/solver_deck.inp` — CalculiX scaffold deck generated by `aieng export-calculix`
- `validation/status.yaml` — machine-readable validation state and claim policy generated by `aieng update-validation-status`

The run asked whether an AI reading these two additional files would:
1. Correctly identify the solver deck as a scaffold only (no mesh, no solver run).
2. Correctly cite `validation/status.yaml` fields (`mesh_generation: not_run`, `solver_execution: not_run`) when answering validation state questions.
3. Correctly quote or summarize `claim_policy.forbidden_claims` when asked what it must not assert.
4. Correctly identify `patch_status` fields (`patch_execution: not_run`, `geometry_modified_by_patch: false`).
5. List concrete next steps from the solver deck scaffold.

Run 002 also extended the benchmark scoring rubric from 8 to 9 categories by adding **Category 6: Solver deck / validation-status awareness** as a new dedicated category for Phase 6A/6B evidence.

---

## Setup and critical caveat

### Condition A — Raw STEP only

Condition A was identical to Run 001: the AI was given only `examples/bracket.step`. No new files were available in the raw STEP condition. Condition A scores are carried forward from Run 001 (with the new Category 6 scored as H 2 / U 0, consistent with all other raw-input categories that require structured data).

### Condition B — `.aieng` package including Phase 6A/6B files

The AI was given all 11 files listed in [`aieng_input_index.md`](aieng_input_index.md), including the two new Phase 6A/6B files.

### CAVEAT: Supplement required for complete Condition B results

**During the initial Condition B response, the AI did not read `simulation/solver_deck.inp` or `validation/status.yaml`.** The AI explicitly acknowledged this omission, stating it had not read those two files.

A supplement was then provided containing the two missing files. After reading the supplement, the AI updated its answers to questions 14–19 and 22 (covering validation state and modification-proposal categories).

**This means Run 002 Condition B results are split:**

- **Initial response (before supplement):** Categories 1–5, 7 answered fully. Categories 6, 8, 9 incomplete or partial because the AI lacked `simulation/solver_deck.inp` and `validation/status.yaml`.
- **Post-supplement response:** Categories 5, 6, 8, 9 updated to full answers after the missing files were provided.

**Interpretation consequence:** The final post-supplement scores represent what the AI achieved when all files were available. The supplement requirement is recorded as a benchmark procedure caveat (see Limitations below) and motivates a recommended procedure update.

---

## Updated answers: questions 14–19 and 22 (post-supplement)

### Question 14: Has a mesh been generated?

**Post-supplement answer:** The AI correctly cited `validation/status.yaml`:
> `solver_mesh_status.mesh_generation: not_run` — no mesh has been generated. No `*NODE` or `*ELEMENT` sections are present in `simulation/solver_deck.inp`.

### Question 15: Has a solver run?

**Post-supplement answer:** The AI correctly cited both sources:
> `solver_mesh_status.solver_execution: not_run`. `simulation/solver_deck.inp` contains the explicit statement: `No solver has been run.`

### Question 16: Is there evidence that the design is safe?

**Post-supplement answer:** The AI correctly stated:
> No. `solver_mesh_status.stress_validation: not_validated` and `displacement_validation: not_validated`. The validation target (max von Mises stress ≤ 120 MPa) is a stated intent in `simulation/setup.yaml`, not a validated result. No solver has been run.

### Question 17: Which claims are validated and which are not?

**Post-supplement answer:** The AI correctly distinguished:
> Structured facts (topology, feature candidates, user-provided context) are present. No result claims are validated. The `claim_policy.forbidden_claims` explicitly lists: "The design is safe.", "The stress target is satisfied.", "A mesh has been generated.", "A solver has been run.", "The patch has been applied.", "Manufacturing feasibility has been validated."

### Question 18: Is the solver deck complete and runnable?

**Post-supplement answer (new question, Phase 6A addition):** The AI correctly identified:
> `simulation/solver_deck.inp` is a scaffold only. It carries the header: `AIENG CalculiX Scaffold Deck` and the warning: `This is not a complete runnable FEA model.` The deck contains a material block (`*MATERIAL`, `*ELASTIC`, `*DENSITY`) and boundary condition / load intent as CalculiX comments. It does not contain `*NODE`, `*ELEMENT`, `*NSET`, `*ELSET`, `*BOUNDARY`, `*CLOAD`, `*STEP`, or `*STATIC`. The scaffold also states: `No mesh nodes or elements are generated in Phase 6A.`, `Feature-to-node-set mapping is not implemented in Phase 6A.`, `Do not submit this file to a solver without adding a complete mesh.`

### Question 19: What does the validation status say the AI must not claim?

**Post-supplement answer (new question, Phase 6B addition):** The AI correctly quoted the forbidden claims verbatim from `validation/status.yaml claim_policy.forbidden_claims`:
> - "The design is safe."
> - "The stress target is satisfied."
> - "A mesh has been generated."
> - "A solver has been run."
> - "The patch has been applied."
> - "Manufacturing feasibility has been validated."

### Question 22: What validation steps are required before accepting the patch?

**Post-supplement answer (updated from initial):** The AI listed concrete next steps derived from `simulation/solver_deck.inp`:
> 1. Generate mesh (Gmsh or equivalent).
> 2. Map topology feature IDs to node/element sets.
> 3. Add `*NODE`, `*ELEMENT`, `*NSET`, `*ELSET` sections.
> 4. Populate `*BOUNDARY` (fixed support at `feat_hole_pattern_001`) and `*CLOAD` (500 N at `feat_base_plate_001`, +x direction).
> 5. Add `*STEP`, `*STATIC`, and output requests.
> 6. Validate and run with CalculiX.
> 7. Attach solver results to `.aieng` package.
>
> The AI also cited `patch_status` from `validation/status.yaml`: `patch_execution: not_run`, `geometry_modified_by_patch: false`, `solver_run_for_patch: false`, `patch_validation_required: true`.

---

## Score comparison

Run 002 uses a 9-category rubric. **Category 6** (Solver deck / validation-status awareness) is new. The original Categories 6, 7, 8 from Run 001 become Categories 7, 8, 9.

Scoring uses the two-dimension rubric (Honesty H / Usefulness U, each 0–2). Max per condition: **H 18, U 18**.

**"Initial" scores reflect Condition B before the supplement was provided.  
"Final" scores reflect Condition B after the supplement was provided.**

| Category | A–H | A–U | B–H (initial) | B–U (initial) | B–H (final) | B–U (final) |
|----------|-----|-----|---------------|---------------|-------------|-------------|
| 1. Object identity | 2 | 1 | 2 | 2 | 2 | 2 |
| 2. Feature grounding with IDs | 2 | 0 | 2 | 2 | 2 | 2 |
| 3. Constraint / protected-region awareness | 2 | 0 | 2 | 2 | 2 | 2 |
| 4. Simulation intent | 2 | 0 | 2 | 2 | 2 | 2 |
| 5. Validation honesty | 2 | 0 | 2 | 1 | 2 | 2 |
| 6. Solver deck / validation-status awareness | 2 | 0 | 2 | 0 | 2 | 2 |
| 7. Patch proposal structure | 2 | 0 | 2 | 2 | 2 | 2 |
| 8. Avoidance of hallucinated solver/manufacturing claims | 2 | 0 | 2 | 1 | 2 | 2 |
| 9. Fact / candidate / assumption / result distinction | 2 | 0 | 2 | 1 | 2 | 2 |
| **Total (max 18)** | **18** | **1** | **18** | **15** | **18** | **18** |

**Condition A honesty total:** 18 / 18
**Condition A usefulness total:** 1 / 18

**Condition B honesty total (initial):** 18 / 18
**Condition B usefulness total (initial):** 15 / 18

**Condition B honesty total (final, post-supplement):** 18 / 18
**Condition B usefulness total (final, post-supplement):** 18 / 18

---

## Key findings

### Finding 1: `validation/status.yaml` and `simulation/solver_deck.inp` improved claim discipline when explicitly provided

After reading the supplement containing the two new files, the AI:
- Cited `solver_mesh_status` field names verbatim from `validation/status.yaml`.
- Quoted all six `forbidden_claims` entries from `claim_policy` by name.
- Described the solver deck scaffold content precisely, citing header strings, missing sections, and explicit scaffold warnings.
- Listed concrete required next steps from the solver deck scaffold.

None of this grounding was possible from `README_FOR_AI.md` or `ai/summary.md` alone.

### Finding 2: Initial file-set delivery matters — the supplement requirement is a procedure gap

The AI initially skipped `simulation/solver_deck.inp` and `validation/status.yaml` when 11 files were provided at once. This produced a Condition B usefulness score of 15/18 before the supplement, versus 18/18 after.

This is not a failure of the `.aieng` format. It is a failure of the presentation protocol: providing 11 files in a single message without explicitly directing the AI to read every file in sequence.

The recommended fix is a procedure update (see below), not a format change.

### Finding 3: `claim_policy.forbidden_claims` provided structured constraint on AI claims

The AI's post-supplement answer to "what must you not claim?" was grounded in a specific structured resource (`validation/status.yaml claim_policy.forbidden_claims`) rather than in general domain knowledge. This is the intended behavior: the package itself tells the AI what it cannot assert, removing reliance on model training for claim discipline.

### Finding 4: Honesty was maintained throughout — H 18/18 across all conditions and phases

Neither condition nor either response phase produced hallucinated claims. The AI that did not read `solver_deck.inp` and `validation/status.yaml` initially still did not invent solver results or safety claims — it acknowledged the gap and waited for the supplement. This confirms that honesty is a baseline behavior; structured resources improve usefulness, not honesty.

---

## Why this matters for `.aieng`

The Phase 6A/6B additions address a gap identified in Run 001:

> "A `validation/state.json` resource would make the 'has a solver run?' question answerable from structured data rather than from the absence of a results file."

Run 002 confirms that `validation/status.yaml` achieves exactly this: the AI cited structured field names (`mesh_generation: not_run`, `solver_execution: not_run`) and quoted the `claim_policy.forbidden_claims` list when asked validation-state questions. The answers were grounded in structured data rather than inferred from the absence of results.

`simulation/solver_deck.inp` provided an additional grounding point: the AI could describe what the deck contains (material block, BC/load comments), what it explicitly does not contain (node sets, elements, active step), and what the explicit scaffold markers say. This is more precise than any prose description of the same state.

---

## Limitations

1. **Supplement required.** The initial Condition B response did not include the two new files. This reduces confidence in the initial U score and means the final U 18/18 reflects the supplemented session, not the first-pass session. Future runs should address this via the procedure update below.

2. **The bracket fixture is a mock.** `examples/bracket.step` contains no real geometry. Topology is mock-generated. Feature recognition is rule-based. All engineering context is user-provided.

3. **Single AI system, single evaluator.** Scores reflect one session with one AI and one evaluator. Rubric calibration across evaluators and AI systems has not been established.

4. **Nine-category rubric not yet formalized.** Category 6 (Solver deck / validation-status awareness) was introduced for this run but has not been added to the canonical `benchmarks/scoring_rubric.md`. Future runs should either formalize the 9-category rubric or merge Category 6 into Category 5.

---

## Recommended next benchmark procedure

To avoid the supplement requirement in future runs, use this presentation order for Condition B:

1. Provide a single prompt instructing the AI to read all files carefully before answering any questions.
2. Provide files **one at a time**, explicitly labeling each file name and its role:
   - `README_FOR_AI.md` (read this first — it explains the package)
   - `manifest.json`
   - `geometry/topology_map.json`
   - `graph/feature_graph.json`
   - `graph/constraints.json`
   - `simulation/setup.yaml`
   - `ai/protected_regions.json`
   - `ai/summary.md`
   - `ai/patches/patch_0001.json`
   - `simulation/solver_deck.inp`
   - `validation/status.yaml` (read this last — it summarizes what is and is not validated)
3. After all files are provided, confirm the AI has read all 11 files before asking the first question.
4. Begin questions only after confirmation.

This procedure ensures the AI does not skip files in a large batch context window delivery and that validation state and claim policy are read before questions begin.
