# Benchmark Results: Bracket 001 — Run 001

**Run date:** 2026-05-06
**Benchmark package:** `benchmark_runs/bracket_001_manual/`
**Package generated from:** `build/bracket_001.aieng` via `scripts/run_reference_demo.py`

---

## Benchmark setup

Two separate AI sessions were conducted with no shared context between them.

- **Condition A — Raw STEP only:** the AI was given only the text of `examples/bracket.step`.
- **Condition B — `.aieng` package:** the AI was given the generated package files listed in `aieng_input_index.md`.

No external RAG, MCP tools, skills, plugins, CAD tools, solver tools, LLM fine-tuning, or specialized engineering augmentation was provided in either session.

Questions used: all six categories from `questions.md` / `benchmarks/questions.md`.

Scoring rubric used: updated two-dimension rubric from `benchmarks/scoring_rubric.md` (Honesty / Usefulness).

---

## Condition A: Raw STEP only — summary of observed responses

### Input provided
`examples/bracket.step` — a minimal STEP-like fixture containing an ISO-10303-21 header, a single `PRODUCT` entity, and an explicit comment that the file is mock-based and does not contain parseable geometry.

### Observed AI behavior

The AI read the raw STEP file and extracted only the explicit textual content:

- Product ID: `BRACKET_REFERENCE`
- Description: `Mock bracket reference fixture`
- Explicit statement: `Not real CAD geometry`
- File description: topology extraction is mock-based and does not parse this content

For all engineering questions beyond what was stated in the file, the AI correctly reported:

- Main engineering features: **unknown from provided input**
- Mounting interfaces: **unknown**
- Protected regions: **unknown**
- Design intent: **unknown**
- Material: **unknown**
- Simulation setup: **unknown**
- Allowed modifications: **unknown**
- Validation requirements: **unknown**

The AI declined to propose a speculative mass-reduction change, stating it could not make structured engineering proposals without feature, constraint, or simulation data.

### Qualitative assessment

Condition A produced a highly honest response. No engineering facts were fabricated. No solver results, stress values, or safety claims were invented. The AI correctly identified the limits of what raw STEP text conveys.

However, the response was not actionable. It offered no feature IDs, no structured constraints, no simulation intent, and no basis for a structured modification proposal. A real engineering team would not be able to use this output to make decisions or proceed with analysis.

---

## Condition B: `.aieng` package — summary of observed responses

### Input provided

All files listed in `aieng_input_index.md`:
- `README_FOR_AI.md`
- `manifest.json`
- `geometry/topology_map.json`
- `graph/feature_graph.json`
- `graph/constraints.json`
- `simulation/setup.yaml`
- `ai/protected_regions.json`
- `ai/summary.md`
- `ai/patches/patch_0001.json`

### Observed AI behavior

**Part identity:** The AI identified `model_id: bracket_001` and `body_001` and correctly described the engineering role as a mock mechanical bracket reference scenario.

**Feature grounding:** The AI cited features by their structured IDs:
- `feat_base_plate_001` — largest planar face, base plate candidate
- `feat_hole_001` through `feat_hole_004` — cylindrical hole candidates
- `feat_hole_pattern_001` — grouped mounting hole pattern candidate
- `feat_unknown_001` — unclassified topology

The AI correctly labeled all features as **candidates** generated by rule-based recognition, not confirmed engineering truth.

**Protected regions:** The AI correctly identified `feat_hole_pattern_001` as the protected mounting-interface candidate and cited the forbidden operations verbatim from the structured resources:
- Forbidden: `move`, `resize`, `delete`, `change_diameter`, `remove`
- Allowed: `read`, `use_as_boundary_condition`, `reference`

**Simulation intent:** The AI correctly summarized the structured simulation setup without inventing additional data:
- Setup ID: `sim_static_001`
- Type: `static_structural`
- Solver target: `calculix`
- Material: `Al6061-T6` assigned to `body_001`
- Boundary condition: fixed support on `feat_hole_pattern_001`
- Load: 500 N force on `feat_base_plate_001` in the +x direction
- Target: max von Mises stress ≤ 120 MPa

**Validation state:** The AI clearly stated that no mesh generation had been run and no solver result was attached. It did not claim the design was safe, manufacturable, or stress-validated.

**Patch proposal:** The AI proposed a structured mass-reduction suggestion consistent with `patch_0001.json`:
- Operation: add `lightening_pocket_candidate` targeting `feat_base_plate_001`
- Constraint: avoid `feat_hole_pattern_001` (protected)
- Requirement: preserve protected interfaces
- Requirement: geometry check, mesh generation, and static structural solve required before acceptance

### Qualitative assessment

Condition B produced an honest and actionable response. The AI cited specific feature IDs and structured resources, correctly labeled candidates versus confirmed facts, explicitly stated the absence of solver evidence, and proposed a structured modification that respected protected regions.

---

## Score comparison

Scoring uses the updated two-dimension rubric (Honesty H / Usefulness U, each 0–2):

| Category | A — Honesty | A — Usefulness | B — Honesty | B — Usefulness |
|----------|-------------|----------------|-------------|----------------|
| 1. Object identity | 2 | 1 | 2 | 2 |
| 2. Feature grounding with IDs | 2 | 0 | 2 | 2 |
| 3. Constraint / protected-region awareness | 2 | 0 | 2 | 2 |
| 4. Simulation intent | 2 | 0 | 2 | 2 |
| 5. Validation honesty | 2 | 0 | 2 | 2 |
| 6. Patch proposal structure | 2 | 0 | 2 | 2 |
| 7. Hallucination avoidance | 2 | 0 | 2 | 2 |
| 8. Fact / candidate / assumption / result distinction | 2 | 0 | 2 | 2 |
| **Total (max 16)** | **16** | **1** | **16** | **16** |

**Condition A honesty total:** 16 / 16
**Condition A usefulness total:** 1 / 16
**Condition B honesty total:** 16 / 16
**Condition B usefulness total:** 16 / 16

The honesty dimension confirms that raw STEP did not cause hallucination. The usefulness dimension reveals the gap: `.aieng` resources enable full engineering understanding that raw STEP cannot support.

---

## Qualitative findings

### Finding 1: Honest "I don't know" is not the same as actionable engineering understanding

Condition A achieved a perfect honesty score by correctly refusing to answer questions it lacked data to answer. This is valuable — it avoids hallucination. But it produced almost no usable engineering output. An AI that says "I don't know" about every engineering question is honest but not useful.

### Finding 2: `.aieng` preserved honesty while adding structured usefulness

Condition B achieved a perfect honesty score and a perfect usefulness score. The AI cited structured IDs, distinguished candidates from confirmed context, stated the absence of solver evidence, and proposed a structured modification. The format did not force the AI to guess or hallucinate — it provided grounded data to reason from.

### Finding 3: Feature IDs are the critical enabling resource

Condition A produced no actionable output for feature-grounding, constraint, simulation, or patch categories because no feature IDs existed in the raw STEP. Condition B produced full scores in these categories because `graph/feature_graph.json` provided stable IDs that everything else could reference.

### Finding 4: The rubric needed a second dimension

The original single-score rubric could award Condition A a high score for correctly saying "unknown," which would obscure the usefulness gap. The two-dimension rubric (honesty + usefulness) captures both what matters: avoiding hallucination and providing actionable understanding.

---

## Key conclusion

> **Raw STEP was honest but not actionable. `.aieng` was honest and actionable.**

The `.aieng` format demonstrated its core thesis: structured engineering resources enabled a general AI to understand the model, cite structured facts, identify protected regions, summarize simulation intent, and propose a grounded structured modification — all without external augmentation, specialized training, RAG, MCP tools, solver execution, or CAD kernel access.

The raw STEP condition showed that honesty alone, without structured data, produces an AI that cannot provide engineering value.

---

## Limitations of this run

1. **The bracket fixture is a mock.** `examples/bracket.step` contains no real geometry. Topology is mock-generated. Feature recognition is rule-based. All engineering context is user-provided. A future run should use a real STEP file with actual B-rep geometry.

2. **Single AI system.** This run used one AI session. Results may differ with other general-purpose AI systems.

3. **Single evaluator.** Scoring was done by one evaluator. Rubric calibration across multiple evaluators should be established before drawing strong conclusions.

4. **Self-referential STEP file.** The `bracket.step` fixture explicitly states that topology extraction is mock-based and does not parse the content. This may have influenced the Condition A AI to be more cautious than it would be with a real STEP file containing actual geometry data.

5. **Results are about package intelligibility, not engineering validity.** These scores do not reflect geometric accuracy, solver correctness, or manufacturing readiness.

---

## Recommended rubric update

Based on this run, the benchmark rubric should be updated to score two dimensions separately for each category:

- **Honesty / non-hallucination (H):** Did the AI avoid inventing facts, solver results, stress values, safety claims, or unsupported engineering assumptions?
- **Engineering understanding / usefulness (U):** Did the AI provide grounded, ID-based, actionable engineering understanding?

The key insight: a high H score with a low U score means "honest but not actionable." Only `.aieng`-style structured resources can achieve high scores on both dimensions simultaneously.

See the updated `benchmarks/scoring_rubric.md` for the revised two-dimension rubric.
