# Validation Matrix

This matrix records verified environments and evidence for Abaqus Agent's
Simulation QA kernel. It distinguishes real Abaqus execution from local dry-run
or unit-test coverage.

## Current Evidence

| Date | Environment | Abaqus | Case / Workflow | Result | Evidence |
|---|---|---|---|---|---|
| 2026-06-02 | GitHub Actions, Ubuntu, Python 3.10/3.11/3.12 | Not installed | Unit tests + package build | PASS | CI run `26803610038`, `260 passed`, build passed |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | `run_benchmark.py --dry-run` over 5 public cases | PASS | `5 / 5` dry-run validation |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | `cantilever` full orchestrator run | PASS | `status=COMPLETED`, `U_tip=-0.0019039579201489687`, `MISES_MAX=0.6528551578521729`, KPI errors `[]` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | `plate_hole` full orchestrator run | PASS | `status=COMPLETED`, `MISES_HOLE_EDGE=295.57122802734375`, `U_X_MAX=0.05009150877594948`, `SCF=264.1424560546875`, regression PASS, KPI errors `[]`, exported `mises_contour.png` and `u_magnitude.png` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | `modal` full orchestrator run | PASS | `status=COMPLETED`, `freq_1=210.24`, `freq_2=416.36`, `freq_3=1304.0`, regression PASS, KPI errors `[]` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | `explicit_impact` full orchestrator run | PASS | `status=COMPLETED`, syntaxcheck warnings `4`, `RF_Z_MAX=31817.26953125`, `U_Z_MIN=-2.0`, regression PASS, KPI errors `[]`, exported `u_magnitude.png` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | `blast_plate` full orchestrator run | PASS | `status=COMPLETED`, syntaxcheck warnings `2`, `U_MAX_DEFLECTION=-93.54364776611328`, `PEEQ_MAX=0.026396118104457855`, `ALLPD_MAX=125353184.0`, regression PASS, KPI errors `[]`, exported `u_magnitude.png` and `peeq_contour.png` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Physics Contracts on `cantilever` | PASS | `tip_deflects_downward`, `tip_displacement_regression_band`, `mises_max_reasonable` all PASS |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Simulation Diff: baseline load `-1.0` vs candidate load `-1.1` | PASS / Expected WARNING | Diff report showed `bc_load.value -1 -> -1.1`, `U_tip` and `MISES_MAX` changed by `10.00%`, contracts stayed PASS |
| 2026-06-02 | macOS local dev, Python 3.12 venv, headless Chrome UI | Not installed | Simulation Diff structured summary smoke | PASS | Targeted tests verified `summary.sections.kpis.WARNING`, `summary.sections.spec.WARNING`, and Markdown `Change Summary` for run-dir diffs across CLI/API/MCP paths; headless Chrome rendered Diff panel `Change Summary`, `Total Rows=3`, `Warnings=2`, and section counts without layout overflow |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Simulation Diff per-KPI tolerance smoke | PASS | Targeted tests verified `--tolerances-json '{"MISES": 0.3}'`, `/api/diff` tolerances, run-id `tolerances_json`, and MCP `tolerances_json`; `MISES` changed from WARNING to PASS while `U_tip` remained WARNING under default rtol |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Simulation Diff Markdown download smoke | PASS | `/api/run/{baseline}/diff/{candidate}/diff.md` returned `200 text/markdown` with `Simulation Diff Report`, `Change Summary`, and an `abaqus-diff-*-vs-*.md` attachment filename |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Simulation Diff artifact evidence smoke | PASS | Unit coverage verified artifact diff rows expose `before_sha256`, `after_sha256`, `before_bytes`, `after_bytes`, ADDED/REMOVED statuses, and Markdown `Artifact Diff` reason fields |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | ODB Lens extractor on real `Cantilever.odb` | PASS | custom recipe with lowercase `u2` returned `U_tip_lower=-0.0019039579201489687`, `MISES_LAST=0.6528551578521729`, errors `[]` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Case Memory search controls against real capsule history | PASS | `python cli.py memory search cases --query PlateHole --sort-by run_id --sort-order asc --min-score 0.5 --json` indexed `4`, matched `1`, returned real `plate_hole` run `95712d55bf05f243`, score `1.25`, reasons `text:platehole`, `contracts passed`, `completed run`; `--artifact mises_contour.png --min-score 0.5` returned the same run, score `1.1`, reason `artifact:mises_contour.png` |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Case Memory free-text match mode smoke | PASS | Unit/API/CLI/MCP coverage verified `match_mode=all` requires every query token while default `any` keeps broad matching, and the selected mode is echoed in search result metadata |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Case Memory result facets smoke | PASS | Unit/UI coverage verified search results include `facets` for status, geometry type, solver, material, and contract result distributions, and the web UI renders top facet values under the search summary |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Report HTML preview and bundle export from real `plate_hole` capsule | PASS | `GET /api/run/real_plate_hole_report/report.html?template=client_summary&download=false` returned `200 text/html` inline with `PlateWithHole` and embedded image data; `GET /api/run/real_plate_hole_report/report.zip?template=client_summary` returned `200 application/zip`, `1,892,410` bytes, with `report.md`, `report.html`, `capsule.json`, `artifact_manifest.json`, `mises_contour.png`, and `27` included artifacts |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Environment preflight command | PASS | `python cli.py validate env --json` returned `status=ready`, resolved `C:\SIMULIA\Commands\abaqus.BAT`, ran `abaqus information=release` with return code `0`, and detected release `2021` |
| 2026-06-02 | Windows 11 23H2 `DESKTOP-PH97BKO`, Tailscale SSH | Abaqus 2021 | Offline report export CLI from real `plate_hole` capsule | PASS | `python cli.py report export cases\plate_hole\runs\95712d55bf05f243 --template client_summary --out ...\offline_report.html --json` wrote `42,799` byte HTML; zip export wrote `1,893,638` bytes with `32` entries, included `report.md`, `report.html`, `capsule.json`, `result.json`, `artifact_manifest.json`, `artifacts/mises_contour.png`, and `artifacts/u_magnitude.png`; HTML contained `Simulation QA Summary`, `PlateWithHole`, and embedded image data |
| 2026-06-02 | macOS local dev, Python 3.12 venv, browser UI | Not installed | Offline report export API/UI smoke | PASS | `/api/report/export` loaded a synthetic run directory into the Report panel; UI rendered `ui_offline_report`, model `UiOffline`, KPI `U_tip=-0.002`, and client-summary Markdown without starting a new analysis run |
| 2026-06-02 | macOS local dev, Python 3.12 venv, headless Chrome UI | Not installed | Optional PDF export API/UI smoke | PASS | `/api/report/export.pdf` returned `501` with explicit `abaqus-agent[pdf]` / `playwright install chromium` guidance when Playwright was absent; headless Chrome loaded synthetic offline report `ui_pdf_report`, rendered model `UiPdf`, and showed the `下载 .pdf` action without layout overflow |
| 2026-06-02 | macOS local dev, Python 3.12 venv, headless Chrome UI | Not installed | Engineering delivery report template smoke | PASS | `python cli.py report export <synthetic-run> --template engineering_delivery --out delivery.html --json` wrote a `5,297` byte HTML report containing `Engineering Delivery Report`, `Delivery verdict: PASS`, `Traceability`, and `Artifact Inventory`; headless Chrome selected `engineering_delivery` in the Report panel and rendered `ui_delivery_report`, `Delivery verdict: PASS`, and `Traceability` without layout overflow |
| 2026-06-02 | macOS local dev, Python 3.12 venv, headless Chrome UI | Not installed | Case Memory contract/count controls smoke | PASS | `python cli.py memory search <synthetic-root> --contract energy --contracts-passed failed --sort-by artifact_count --sort-order desc --json` returned `memory_contract_fail`, `contract=energy`, `contracts_passed=failed`, `sort_by=artifact_count`, `artifact_count=2`; headless Chrome used the same controls and rendered only `ui_memory_contract_fail` with `1 contracts` and `2 artifacts` |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Engineering delivery evidence checklist smoke | PASS | `python cli.py report export <synthetic-run> --template engineering_delivery --out delivery.html --json` wrote a `6,803` byte HTML report containing `Delivery Verdict`, `Evidence Checklist`, `KPI regression`, `Physics contracts`, and PASS evidence for capsule/result/KPI/regression/contract/artifact handoff |
| 2026-06-02 | macOS local dev, Python 3.12 venv | Not installed | Engineering delivery manifest smoke | PASS | Unit coverage verified `Delivery Manifest` appears in Markdown/HTML engineering reports with bundle contents, run identity, artifact payload byte summary, and visual evidence rows |
| 2026-06-02 | macOS local dev, Python 3.12 venv, headless Chrome UI | Not installed | Environment preflight expected-release smoke | PASS | `python cli.py validate env --expected-release 2026 --skip-release-check --json` returned `status=missing_abaqus`, `expected_release=2026`, `release_match.status=unknown`, and `abaqus_release_target` evidence without requiring Abaqus; headless Chrome Environment panel rendered `Expected 2026`, `Release Match unknown`, and the target check without layout overflow |
| 2026-05-06 | External contributor validation | Abaqus 2026 | Cantilever GUI / Windows compatibility | PASS | GLY2024 contribution and case study, see `docs/CASE_STUDY_GLY2024.md` |
| 2026-06-02 | Windows 11 23H2 DESKTOP-PH97BKO, Tailscale SSH, latest source 6ada851 | Abaqus 2021 | Environment preflight readiness | PASS | status=ready, resolved C:\SIMULIA\Commands\abaqus.BAT, detected release 2021, workdir writable, runner_cfg cpus=2/mp_mode=threads/timeout=600 |
| 2026-06-02 | Windows 11 23H2 DESKTOP-PH97BKO, Tailscale SSH, latest source 6ada851 | Abaqus 2021 | cantilever full orchestrator run | PASS | run_id=6ad82dcf3a6fa88c, status=COMPLETED, U_tip=-0.0019039579201489687, MISES_MAX=0.6528551578521729, regression PASS |
| 2026-06-02 | Windows 11 23H2 DESKTOP-PH97BKO, Tailscale SSH, latest source 6ada851 | Abaqus 2021 | Case Memory facet filters on real cantilever capsule | PASS | query Cantilever with geometry_type=cantilever, solver=standard, material_name=Steel matched run_id=6ad82dcf3a6fa88c; facets status COMPLETED=1, geometry cantilever_block=1, material Steel=1 |
| 2026-06-02 | Windows 11 23H2 DESKTOP-PH97BKO, Tailscale SSH, latest source 6ada851 | Abaqus 2021 | 5 public cases full benchmark | PASS | blast_plate, cantilever, explicit_impact, modal, plate_hole all COMPLETED; pipeline 5/5, regression 5/5; report benchmark_20260602_185758.json |
| 2026-06-02 | Windows 11 23H2 DESKTOP-PH97BKO, Tailscale SSH, latest source 6ada851 | Abaqus 2021 | Case Memory solver facet filters on 5 real capsules | PASS | status=COMPLETED --solver standard matched 3 runs; --solver explicit matched 2 runs; facets reported geometry/material distributions correctly |

## Public Case Coverage

These specs validate without an Abaqus installation, and have now also passed
full Abaqus 2021 execution on Windows:

| Case | Dry-run | Real Abaqus 2021 | Notes |
|---|---|---|---|
| `blast_plate` | PASS | PASS | Explicit blast workflow, PEEQ/U/history KPI extraction |
| `cantilever` | PASS | PASS | Static baseline, contracts, diff, ODB Lens recipe |
| `explicit_impact` | PASS | PASS | Explicit displacement workflow, RF/U KPI extraction |
| `modal` | PASS | PASS | Frequency workflow, eigenfrequency KPI extraction |
| `plate_hole` | PASS | PASS | Static stress concentration workflow and image export |

## Next Validation Targets

- Before each real validation pass, run `abaqus-agent validate env --workdir <run-dir> --runner-json '{"cpus":4,"mp_mode":"threads","timeout_seconds":900}' --json` to record OS, Python, Abaqus command resolution, `abaqus information=release`, workdir writability, license marker, and runner config evidence.
- After each real validation pass, run `abaqus-agent validate record --environment "<env>" --abaqus "<release>" --workflow "<case/workflow>" --result PASS --evidence "<evidence>"` to append a normalized evidence row.
- Windows + Abaqus 2021: repeat full public-case validation after major changes to build, submit, ODB extraction, or visualization.
- Windows + Abaqus 2026: repeat cantilever and one ODB Lens recipe with contributor environment if available.
- Linux + Abaqus: syntaxcheck and custom `.inp` capsule path.
- Report templates: validate optional Playwright PDF export on a machine with Chromium installed, then polish downstream document templates.
