Back to dashboard
Headline result
0.2533
Method & source P@k (Precision at k)Manning, Raghavan, Schütze (2008) IIR §8.4

## Read in detail

Each card opens its own page. The headline and charts above are all you need at a glance; the cards are for the why and how.

### Aggregate

This is the second-poorest baseline by P@5 after acronym (which was even more dramatic: 4/22 = 18%). For symbol-attribute, the picture is bimodal: some queries score near 1.0, many score 0.

Read details →

### The Bimodal Split

Read details →

### What This Baseline Says

The split is consistent with cupertino's actual search architecture. The default search path goes through FTS5 over docs_fts columns (uri, source, framework, language, title, content, summary, symbols, symbol_components)…

Read details →

### Why Some Attribute Queries Did Work

@MainActor, Sendable conformance, protocol type, Hashable conformance — these all happen to align with token-presence:

Read details →

### Possible Future Directions (out of Scope for This Audit)

Per the feedback_code_changes_as_ideas_for_future rule:

Read details →

### Implications for Criterion 2 (anti-hallucination)

An AI agent issuing find async throws functions returning Result gets top-10 results that don't constrain on async/throws/return-type.

Read details →

### Method Recap

15 (query, sql_filter) pairs. For each: 1. Compute relevant_uris = {row.doc_uri for row in doc_symbols WHERE <filter>} 2. Run cupertino search "<query>" --limit 10 3.

Read details →

### Combined Phase 1 Baseline Coverage on V1.2.0

All 6 of 6 query classes A-G from §1.4 (excluding Phase 1.7 which is its own design) now have documented baselines on the v1.2.0 candidate DB:

Read details →

## Sources cited in this measurement

Every metric and method this audit relies on, with a link to the foundational source. Auto-collected from the audit text.

### P@k (Precision at k)

Manning, Raghavan, Schütze (2008) IIR §8.4

Open citation

### Mean Reciprocal Rank

Voorhees (1999), TREC-8 QA Report

Open citation