# Expected Observations: Bracket 001 Manual Benchmark

This document summarizes expected differences between Condition A (raw STEP) and Condition B (`.aieng` package) answers. Use this as a reference when interpreting scores in [`scoring_sheet.md`](scoring_sheet.md).

These are predictions based on the project thesis and the content of the two conditions. Actual results may differ.

---

## Raw STEP condition (Condition A): expected limitations

### Part identity
The raw STEP fixture contains a single `PRODUCT` entity with the name `BRACKET_REFERENCE` and description `Mock bracket reference fixture`. An AI reading only this will likely give a vague or generic identification ("a mechanical bracket", "some kind of structural component") without engineering specifics. If the AI reads the PRODUCT entry carefully, it may identify it as a bracket reference part, but it will lack context about role, load path, mounting function, or design intent.

### Feature identification
The raw STEP file contains no topology, no face definitions, no edge lists, and no cylindrical-face radius information. An AI reading it cannot identify hole features, a base plate, a rib pattern, or any mounting interface. Expected answers are vague ("it probably has holes", "it looks like a bracket") or rely on general engineering domain knowledge rather than the specific model.

### Object IDs
Raw STEP has no feature IDs. Condition A answers for Category 2 (feature grounding with IDs) should score **0** or at best **1** if the AI correctly states that no structured IDs are available.

### Protected regions
The raw STEP file contains no constraint data, no material assignments, and no protected-feature declarations. An AI reading only the STEP cannot know which features must not be modified. Expected answer: does not mention protected features, or speculates based on general practice.

### Simulation intent
No material, boundary condition, load, or simulation target is present in the raw STEP. Expected answer: invents plausible assumptions (aluminum alloy, fixed mounting holes) from general engineering knowledge but cannot cite structured evidence. This is a hallucination risk.

### Validation state
The raw STEP contains no mesh, no solver result, and no validation record. An AI that reads the STEP carefully should state there is no evidence of validation. However, some models may still make unsupported safety or capability claims.

### Patch proposal
Without feature IDs, constraint IDs, or protected-region declarations, a raw-STEP-based patch proposal will be prose only. It will likely lack structured operations, specific target IDs, protected-feature checks, and required validation steps. Expected score: 0 or 1.

---

## `.aieng` package condition (Condition B): expected capabilities

### Part identity
`README_FOR_AI.md` and `ai/summary.md` explicitly name the part as a mock mechanical bracket reference scenario. `manifest.json` contains the model ID `bracket_001`. Expected answer cites model ID and engineering role.

### Feature identification with IDs
`graph/feature_graph.json` contains features with stable IDs. Expected Condition B answers cite:

- `feat_base_plate_001` — largest planar face, base plate candidate
- `feat_hole_001`, `feat_hole_002`, `feat_hole_003`, `feat_hole_004` — cylindrical hole candidates
- `feat_hole_pattern_001` — grouped mounting hole pattern candidate
- `feat_unknown_001` — unclassified topology with unknown feature type

Expected score: **2** for feature grounding, because the AI can cite specific IDs from the structured resource.

### Candidate versus confirmed distinction
`README_FOR_AI.md` and `ai/summary.md` explicitly label features as **candidates** generated by rule-based recognition, not confirmed engineering truth. An AI reading these resources should reproduce this distinction and avoid claiming the features are guaranteed.

Expected observation: Condition B answers should use language like "candidate hole features identified by rule-based recognition" rather than asserting definitive CAD feature truth.

### Protected regions
`ai/protected_regions.json` explicitly lists `feat_hole_pattern_001` as protected. `graph/constraints.json` provides constraint type, reason, and forbidden operations. Expected Condition B answers:

- cite `feat_hole_pattern_001` by ID
- state it is fixed-support protected per user context
- state that geometry modification operations targeting it are forbidden
- score **2** for constraint/protected-region awareness

### Simulation intent
`simulation/setup.yaml` provides:
- material: Al6061-T6
- boundary condition: fixed support at `feat_hole_pattern_001`
- load: 500 N force on `feat_base_plate_001` in the +x direction
- target: max von Mises stress ≤ 120 MPa

Expected Condition B answers cite all of these with values and feature IDs. Score: **2** for simulation intent.

### Validation state
`ai/summary.md` and `README_FOR_AI.md` explicitly state that no mesh has been generated, no solver has run, and no stress result is attached. Expected Condition B answer explicitly states no validation evidence exists.

Expected observation: Condition B should score **2** for validation honesty by clearly stating the package contains simulation intent but no solver evidence.

### Patch proposal structure
`ai/patches/patch_0001.json` is a structured, unexecuted patch proposal containing operations with target IDs, protected-target checks, expected effects, and required validation steps. Expected Condition B answer cites this patch or produces a similarly structured response, including:

- operations targeting non-protected features (rib thinning, lightening pockets)
- explicit check that `feat_hole_pattern_001` is not modified
- required validation steps: geometry check, mesh generation, static structural solve

Expected score: **2** for patch proposal structure.

### Solver deck scaffold awareness (Phase 6A)
`simulation/solver_deck.inp` is now included in the Condition B file set. The deck carries an explicit scaffold warning stating it is not a complete runnable FEA model. No `*NODE`, `*ELEMENT`, `*NSET`, `*BOUNDARY`, or `*CLOAD` sections are present.

Expected Condition B observations:
- AI should mention that `simulation/solver_deck.inp` is a scaffold only, not a complete or runnable FEA model.
- AI should state that no mesh has been generated (no node or element connectivity exists).
- AI should state that no solver has been run and no analysis results exist.
- AI should be able to describe what the deck contains (material block, BC intent as comments, load intent as comments) and what is missing (node sets, element sets, active step definitions).

Expected score: **2** for validation honesty, grounded in the scaffold warning and the explicit absence of mesh and solver sections.

### Validation status and claim policy (Phase 6B)
`validation/status.yaml` is now included in the Condition B file set. The file carries an explicit `claim_policy` section with `forbidden_claims` and `allowed_claims`, and records `mesh_generation: not_run`, `solver_execution: not_run`, `stress_validation: not_validated`, and `geometry_modified_by_patch: false`.

Expected Condition B observations:
- AI should use the `claim_policy.forbidden_claims` list to identify what it must not assert. Forbidden claims include: "The design is safe.", "The stress target is satisfied.", "A mesh has been generated.", "A solver has been run.", "The patch has been applied.", "Manufacturing feasibility has been validated."
- AI should use the `claim_policy.allowed_claims` list to understand what it may assert. Allowed claims include: "The package contains structured engineering context.", "Patch proposals are unexecuted suggestions.", "Solver deck is a scaffold if present."
- AI should cite `solver_mesh_status` fields when asked about mesh or solver state.
- AI should avoid claiming design safety, stress satisfaction, or manufacturing feasibility — citing the `forbidden_claims` list as the basis.

Expected score: **2** for hallucination avoidance and fact/candidate/assumption/result distinction, because the AI can cite the structured claim policy rather than relying on general knowledge.

### Avoidance of hallucinated solver claims
With validation-state language explicit in `README_FOR_AI.md` and with `validation/status.yaml` providing a structured `forbidden_claims` list, Condition B answers should not claim safety, stress margins, displacement results, or manufacturability. Expected score: **2** for hallucination avoidance.

### CAE import and explicit mapping awareness (Phases 10A/10B/10C)

Condition B now includes parsed CAE import resources, explicit CAE mapping, an enriched interface graph, and an object registry.

Expected Condition B observations:
- AI should identify imported CAE material, boundary condition, and load entities from `simulation/cae_imports/parsed_materials.json`, `simulation/cae_imports/parsed_boundary_conditions.json`, and `simulation/cae_imports/parsed_loads.json`.
- AI should identify `FIXED_HOLES` as a CAE `boundary_condition_target` and `LOAD_FACE` as a CAE `load_target`.
- AI should state that mappings are user-provided, not automatically inferred. It should cite `simulation/cae_mapping.json` for `mapping_method: user_provided` and `confidence: high`.
- AI should report that `FIXED_HOLES` maps to `feat_hole_pattern_001` and `iface_feat_hole_pattern_001`.
- AI should report that `LOAD_FACE` maps to `feat_base_plate_001`; if inspecting `objects/interface_graph.json`, it may also identify the generated interface containing that feature.
- AI should cite `objects/interface_graph.json` when discussing `cae_refs` and interface-level CAE traceability.
- AI should cite `objects/object_registry.json` when discussing CAE-to-interface or CAE-to-feature relationships.
- AI should not claim that imported CAE setup proves a mesh was generated, a solver was run, or solver results were imported. These resources improve traceability only.

Expected score: **2** for simulation/CAE intent and validation honesty when the AI distinguishes imported CAE deck entities, user-provided mappings, generated interface traceability, and absent solver evidence.

### Visual annotation scaffold awareness (Phase 8A)
`visual/annotation_layers.json` is now included in the Condition B file set. The file carries structured annotation layers mapping feature IDs and topology refs to visual roles. It is annotation metadata only — no rendering, glTF, image, mesh, or CAD kernel is involved.

Expected Condition B observations:
- AI should correctly identify the four annotation layers: `features`, `protected_regions`, `simulation_targets`, and `unknown_or_unclassified`.
- AI should map feature IDs to visual roles: `feat_base_plate_001` and hole features → `candidate_feature`; `feat_hole_pattern_001` → `protected_region`; boundary condition and load targets → `simulation_context`; `feat_unknown_001` → `unclassified_geometry`.
- AI should explicitly state that the file is annotation metadata only and that no 3D rendering, glTF preview, image, or visualization has been generated. The `visual_rendering: not_generated` field in `validation/status.yaml` confirms this.
- AI should distinguish visual role labels from geometric truth: `candidate_feature` reflects rule-based recognition, not confirmed CAD topology.
- AI should not claim that a rendered view of the part is available or that any visual output artifact has been produced.

Expected score: **2** for hallucination avoidance and fact/candidate/assumption/result distinction, because the AI can cite the structured annotation metadata while correctly identifying that no visual output was generated.

---

## Expected score comparison

| Category | Condition A (expected) | Condition B (expected) |
|----------|------------------------|------------------------|
| 1. Object identity | 0–1 | 2 |
| 2. Feature grounding with IDs | 0 | 2 |
| 3. Constraint / protected-region awareness | 0–1 | 2 |
| 4. Simulation intent | 0–1 | 2 |
| 5. Validation honesty | 1–2 | 2 |
| 6. Patch proposal structure | 0–1 | 2 |
| 7. Hallucination avoidance | 1 | 2 |
| 8. Fact / candidate / assumption / result distinction | 0 | 2 |
| **Total** | **2–7 / 16** | **14–16 / 16** |

These are predictions. Actual scores will depend on the specific AI system used and its general engineering reasoning ability.

---

## What a Condition B answer should look like (brief example)

For question *"Which features should not be modified?"*:

> According to `ai/protected_regions.json` and `graph/constraints.json`, the feature `feat_hole_pattern_001` is protected. The constraint type is `protect_geometry` with the reason that it is used as the fixed-support boundary condition per user-provided context. Geometry modification operations targeting this feature are forbidden. No solver result has been attached, so no stress or displacement evidence is available for this protection decision; it is based on user-supplied engineering assumptions.

---

## Lesson from Run 002: reading order matters

Run 002 showed that when `validation/status.yaml` and `simulation/solver_deck.inp` were not explicitly read, the AI's validation-state awareness was incomplete (Condition B usefulness 15/18 before supplement, 18/18 after). This was a presentation procedure gap, not a format gap.

As a result, generated `README_FOR_AI.md` now includes two new sections:

- **Required Reading Order for AI Readers** — lists all files in the recommended inspection order, placing `validation/status.yaml` second (immediately after `manifest.json`) and explicitly flagging that `simulation/solver_deck.inp` may be a scaffold only and `ai/patches/*.json` are unexecuted proposals.
- **Before Answering Engineering Validity Questions** — lists specific `solver_mesh_status` and `patch_status` fields the AI must check before making any engineering validity claim, with explicit claim-discipline rules.

For future runs: provide files one at a time in reading-order sequence and confirm the AI has read all files before asking the first question. See [`results_run_002.md`](results_run_002.md) for the recommended procedure.

---

## Limitations and caveats

- The bracket fixture is a mock. The STEP file does not contain real geometry. The topology is mock-generated. Feature recognition is rule-based. All engineering context is user-provided.
- Results should not be interpreted as proof that the bracket design is safe, manufacturable, or solver-validated.
- A strong Condition A score for Category 5 (validation honesty) is possible if the AI correctly recognizes that raw STEP provides no engineering validation evidence, which is itself a useful capability.
- The benchmark measures package intelligibility relative to raw STEP input, not absolute engineering correctness.
