Search-quality baseline: prose / conceptual (Phase 1.5, v1.2.0 candidate)

Aggregate

This is the second-worst class baseline after acronym (4/22, 18%). For prose, this number represents an upper-bound on the methodology problem and a lower-bound on the ranker problem; the truth is somewhere in between.

What the Regex Says

The harness defined per-query "valid" URIs as a tight enumeration: for "actor reentrancy semantics", valid = apple-docs://swift/actor OR swift-evolution://SE-0306 OR swift-evolution://SE-0327.

Reading the Misses Honestly

Several of the "misses" are arguably correct results that the regex rejected:

What This Audit Measures Vs What It Doesn't

What This Baseline Says About the Ranker

Cupertino's BM25F + RRF configuration is tuned for canonical-lookup and symbol-identifier queries (classes A, D), where it excels (92% P@1, 100% rank-1 fragment recall).

Possible Future Directions (out of Scope for This Audit)

Following the feedback_code_changes_as_ideas_for_future rule:

Implications for Criterion 2 (anti-hallucination)

An AI agent issuing a prose query like "how to make a type Sendable" gets appentity and systemcoordinator at top-3, NOT the Sendable protocol page.

Method Recap

15 prose queries, each with a regex matching a per-query enumerated valid-URI set. For each: run cupertino search "<query>" --limit 10, find first-relevant rank, compute P@3, P@5, any-match-in-top-3 (binary).

Combined Phase 1 Baseline Coverage on V1.2.0

Six of eight Phase 1.x classes from §1.4 now have documented baselines. One remains: H (symbol-attribute). Plus Phase 1.7 (anti-hallucination agent-end-to-end).