Back to Search-quality version diff: v1.1.0 → v1.2.0

    flowchart TD
        Q["50 canonical-lookup query strings
    (Class A + B, in-source)"]:::input
        R["per-query regex pattern
    apple-docs://<framework>/<concept>($|/...)"]:::input
        Q --> H[scripts/eval/search-quality-phase1.py]
        R --> H

        BA["Arm A: /opt/homebrew/bin/cupertino (v1.1.0)
    + ~/.cupertino/search.db (schema 13, 285k docs)"]:::arm
        BB["Arm B: Packages/.build/release/cupertino (v1.2.0)
    + ~/.cupertino-dev/search.db (schema 18, 352k docs)"]:::arm
        H -->|"cupertino search --format json --limit 10"| BA
        H -->|"cupertino search --format json --limit 10"| BB

        BA --> JA["top-10 URI list per query
    (parsed; strips v1.2.0's ISO timestamp prefix)"]
        BB --> JB["top-10 URI list per query
    (parsed)"]

        JA --> SA["per-query score:
    first-relevant-rank → RR = 1/rank
    P@1, P@5, NDCG@10"]
        JB --> SB["per-query score:
    first-relevant-rank → RR = 1/rank
    P@1, P@5, NDCG@10"]

        SA --> PAIR["paired comparison:
    buckets (Added/Removed/Fixed/Degraded/
    Unchanged/Both-suboptimal)"]
        SB --> PAIR
        PAIR --> WX["Paired Wilcoxon signed-rank
    on per-query RR vector"]:::stat
        PAIR --> MC["McNemar exact (binomial)
    on rank-1 contingency 2×2"]:::stat

        WX --> MD["this audit MD
    (dashboard glob picks up)"]:::out
        MC --> MD

        classDef input fill:#0a84ff,stroke:#0040cc,color:#fff
        classDef arm fill:#5856d6,stroke:#3634a3,color:#fff
        classDef stat fill:#ff9500,stroke:#c4730a,color:#fff
        classDef out fill:#34c759,stroke:#1f7a3a,color:#fff