Pipeline

flowchart TD
    Q["50 canonical-lookup query strings
(Class A + B, in-source)"]:::input
    R["per-query regex pattern
apple-docs://<framework>/<concept>($|/...)"]:::input
    Q --> H[scripts/eval/search-quality-phase1.py]
    R --> H

    BA["Arm A: /opt/homebrew/bin/cupertino (v1.1.0)
+ ~/.cupertino/search.db (schema 13, 285k docs)"]:::arm
    BB["Arm B: Packages/.build/release/cupertino (v1.2.0)
+ ~/.cupertino-dev/search.db (schema 18, 352k docs)"]:::arm
    H -->|"cupertino search --format json --limit 10"| BA
    H -->|"cupertino search --format json --limit 10"| BB

    BA --> JA["top-10 URI list per query
(parsed; strips v1.2.0's ISO timestamp prefix)"]
    BB --> JB["top-10 URI list per query
(parsed)"]

    JA --> SA["per-query score:
first-relevant-rank → RR = 1/rank
P@1, P@5, NDCG@10"]
    JB --> SB["per-query score:
first-relevant-rank → RR = 1/rank
P@1, P@5, NDCG@10"]

    SA --> PAIR["paired comparison:
buckets (Added/Removed/Fixed/Degraded/
Unchanged/Both-suboptimal)"]
    SB --> PAIR
    PAIR --> WX["Paired Wilcoxon signed-rank
on per-query RR vector"]:::stat
    PAIR --> MC["McNemar exact (binomial)
on rank-1 contingency 2×2"]:::stat

    WX --> MD["this audit MD
(dashboard glob picks up)"]:::out
    MC --> MD

    classDef input fill:#0a84ff,stroke:#0040cc,color:#fff
    classDef arm fill:#5856d6,stroke:#3634a3,color:#fff
    classDef stat fill:#ff9500,stroke:#c4730a,color:#fff
    classDef out fill:#34c759,stroke:#1f7a3a,color:#fff