Back to dashboard
Headline result
19 / 25 (76.0%)
Method & source Reciprocal Rank Fusion (k=60)Cormack, Clarke, Büttcher (2009), SIGIR

## Read in detail

Each card opens its own page. The headline and charts above are all you need at a glance; the cards are for the why and how.

### Aggregate

The 6 mismatches are not "ranker chose wrong source from candidates"; they are "expected source had nothing in the top-10 at all because apple-docs's source weight outcompetes the lower-weight sources." A deeper finding…

Read details →

### The Clean Wins: 19 / 25

For 19 queries where the expected top-1 source was apple-docs (weight 3.0), the actual top-1 was always apple-docs:

Read details →

### The Interesting 6: Hig and Apple-archive Starve

For each of these queries, the HIG or apple-archive corpus does have a relevant page (verified via cupertino search "<query>" --source hig and --source apple-archive):

Read details →

### Method Recap

25 (query, expected_source, rationale) triples. For each: run cupertino search "<query>" --limit 10, extract top-1 URI, derive its source via the URI prefix, compare against expected.

Read details →

### Implications for Criterion 2 (anti-hallucination)

For the LLM-agent consumer, this baseline is good news: the agent grounding on cupertino's top-1 will always get apple-docs (the API reference) for any Swift-code-generation query that has an apple-docs answer.

Read details →

### Combined Phase 1 Baseline Coverage on V1.2.0

Three of eight query classes from §1.4 now have documented baselines. Five remain: C (acronym), D (CamelCase fragment), G (prose), H (symbol-attribute), and Phase 1.7 agent-end-to-end.

Read details →

## Sources cited in this measurement

Every metric and method this audit relies on, with a link to the foundational source. Auto-collected from the audit text.

### Reciprocal Rank Fusion (k=60)

Cormack, Clarke, Büttcher (2009), SIGIR

Open citation

### Sign / binomial test

Conover (1999), Practical Nonparametric Statistics

Open citation

### Mean Reciprocal Rank

Voorhees (1999), TREC-8 QA Report

Open citation

### P@k (Precision at k)

Manning, Raghavan, Schütze (2008) IIR §8.4

Open citation