# Exploit: HPO Phenotype Enrichment for Clinical Symptoms

## Winning Approach

Explore chose a pivot from adding another HPO-like phenotype source to a
two-tier disease phenotype architecture:

- Keep current curated HPO/Monarch phenotype rows as `phenotypes`.
- Add source-native MedlinePlus clinical symptom rows as `clinical_features`.
- Preserve source URL, source-native topic ID, evidence text, tier, rank, and
  confidence-scored HPO mapping instead of forcing all clinical symptoms into a
  required HPO row.
- Prefer direct disease pages before extracting symptoms, because the explore
  MedlinePlus probe showed useful recall but also related-topic noise.

## Full-Scale Results

Full scale for this ticket is the full architecture-spike scope: uterine
fibroid plus two common HPO-sparse diseases, with page selection, extraction,
HPO mapping, regression control, validation, and durable result artifacts.

Result artifact:
`/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_full_scale.json`

Measured summary:

| Metric | Value |
|---|---:|
| Diseases | 3 |
| Candidate MedlinePlus topics | 12 |
| Selected topics | 5 |
| Topic noise reduction | 7 |
| Diseases with direct disease page selection | 2 |
| Clinical features extracted | 15 |
| HPO-mapped features | 15 |
| Unmapped features | 0 |
| Expected symptoms matched | 15/23 |
| Expected symptom recall | 0.652 |
| Mismatch count | 8 |
| Fixture extraction elapsed | 8.3 ms |
| Features per second | 1807.229 |
| Peak RSS | 25324 KB |
| Output checksum | `f08c35ff31306ff4696bd953eaba4b00aeed9e6746a1228469e1479238e3d34f` |

Per-disease output:

| Disease | Selected topics | Features | Recall |
|---|---:|---:|---:|
| Uterine fibroid | 1 direct topic: Uterine Fibroids | 6 | 0.750 |
| Endometriosis | 1 direct topic: Endometriosis | 6 | 0.857 |
| Chronic venous insufficiency | 3 related topics: Deep Vein Thrombosis, Leg Injuries and Disorders, Varicose Veins | 3 | 0.375 |

## Regression Control

Regression artifact:
`/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_regression_control.json`

Explore baselines:

| Control | Explore |
|---|---:|
| Current HPO phenotype rows | 2 |
| Current HPO expected symptom recall | 0/23 = 0.000 |
| MedlinePlus blind topic recall | 14/23 = 0.609 |
| MedlinePlus blind topic mismatch count | 9 |
| MedlinePlus retained topics | 12 |

Exploit controls:

| Control | Exploit | Rule result |
|---|---:|---|
| Exact explore benchmark reproduction | 14/23 recall, 9 mismatches | Passed, equal mismatch count |
| Selected-page clinical feature output | 15/23 recall, 8 mismatches | Passed, mismatch count decreased |
| Current HPO live control | 2 phenotype rows | Passed |
| HPO row checksum | `74f9bd48513f6706c1d05e8ff3baa219b06ccfce2a1e7b16d329aa810d975520` | Passed, unchanged |
| Topic count | 5 selected vs 12 explore retained | Passed, noise reduced |

HPO live control timings were 2745.8 ms total: uterine fibroid 1005.5 ms,
endometriosis 848.8 ms, and chronic venous insufficiency 841.3 ms. Explore HPO
timings summed to 2848.0 ms, so the live control was faster and within the
latency rule.

No regression-rule carveouts were needed. One implementation issue was found
before final reporting: direct-page selection initially removed the related
ectopic-pregnancy page that had supplied explore's endometriosis abdominal-pain
lexical match. The direct Endometriosis page contains the same clinical concept
as lower-abdomen pain, so the extractor added a reviewed synonym instead of
reintroducing related-topic noise. Final selected output improved mismatch count
from explore's 9 to 8.

## Validation

Validation artifact:
`/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_validation.json`

Validation passed.

Checks performed:

| Check | Result |
|---|---|
| Required `clinical_features` schema fields present | Passed |
| Evidence text contains the matched extraction anchor | Passed |
| Recall matches or beats explore | Passed |
| Direct-page selection reduces noise for uterine fibroid and endometriosis | Passed |
| Artifact paths are persistent and not under `/tmp/` | Passed |
| HPO live control is available or documented | Passed |

Focused tests:

```text
pytest -q architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/scripts/test_clinical_features_spike.py
3 passed, 1 warning in 2.81s
```

Exploit artifact generation:

```text
python3 architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/scripts/run_exploit.py --offline
```

The `--offline` mode uses the committed explore MedlinePlus result as a stable
fixture. The same CLI can refresh live MedlinePlus cache files under the
gitignored experiment `work/` directory.

## Produced Artifacts

Committed source:

- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/scripts/run_exploit.py`
- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/scripts/clinical_features_spike/`
- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/scripts/test_clinical_features_spike.py`

Committed JSON results:

- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_full_scale.json`
- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_regression_control.json`
- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_validation.json`
- `architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_contract_numbers.json`

Persistent local paths:

- Results directory:
  `/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results`
- Gitignored working/cache directory:
  `/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/work`

No `/tmp/` artifacts are required.

## Contract Numbers

Downstream spikes can depend on these exploit-measured values:

| Contract | Value |
|---|---:|
| Disease scope | 3 |
| Clinical features | 15 |
| HPO-mapped clinical features | 15 |
| Expected symptom recall | 0.652 |
| Mismatch count | 8 |
| Selected MedlinePlus topics | 5 |
| Topic noise reduction | 7 |
| Direct-page diseases | 2 |
| Fixture extraction elapsed | 8.3 ms |
| Features per second | 1807.229 |
| Peak RSS | 25324 KB |
| Output checksum | `f08c35ff31306ff4696bd953eaba4b00aeed9e6746a1228469e1479238e3d34f` |

The contract-number artifact is:
`/home/ian/workspace/worktrees/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/architecture/experiments/243-architecture-hpo-phenotype-enrichment-for-clinical-symptoms/results/clinical_features_contract_numbers.json`

## Conclusions

The winning architecture works at the ticket's full spike scope. It preserves
the curated HPO section exactly, adds a source-native clinical-feature tier, and
improves expected symptom recall from explore's 14/23 to 15/23 while reducing
MedlinePlus topic noise from 12 retained topics to 5 selected topics.

This enables the next build ticket to add a real `clinical_features` output
contract to `get disease <id> phenotypes` without pretending all clinical
symptoms are first-class HPO annotations. The product integration should keep
the reviewed mapping/allow-deny fixture concept, make MedlinePlus page
selection explicit, and preserve source-native evidence text for auditability.

Remaining risks:

- The HPO mapping fixture is deterministic for the spike, not a production
  mapper.
- Chronic venous insufficiency has no direct MedlinePlus page in this probe, so
  related-topic evidence remains weaker for that disease.
- Production rollout needs Rust schema/rendering changes and broader validation
  beyond the 3-disease architecture sample.
