P@k (Precision at k)
Manning, Raghavan, Schütze (2008) IIR §8.4
Open citationThis audit tests symbol-attribute queries — queries that conceptually describe symbols by attribute (@MainActor, @Observable), by signature flag (async throws, public static), by conformance (Sendable conformance, Hashable conformance), or by kind (actor type, protocol type, initializer, subscript, typealias). Unlike previous phases where relevance was defined by a URI regex, here relevance is defined by a SQL filter against the doc_symbols table.
Each card opens its own page. The headline and charts above are all you need at a glance; the cards are for the why and how.
This is the second-poorest baseline by P@5 after acronym (which was even more dramatic: 4/22 = 18%). For symbol-attribute, the picture is bimodal: some queries score near 1.0, many score 0.
Read details →The split is consistent with cupertino's actual search architecture. The default search path goes through FTS5 over docs_fts columns (uri, source, framework, language, title, content, summary, symbols, symbol_components)…
Read details →@MainActor, Sendable conformance, protocol type, Hashable conformance — these all happen to align with token-presence:
Read details →Per the feedback_code_changes_as_ideas_for_future rule:
Read details →An AI agent issuing find async throws functions returning Result gets top-10 results that don't constrain on async/throws/return-type.
Read details →15 (query, sql_filter) pairs. For each: 1. Compute relevant_uris = {row.doc_uri for row in doc_symbols WHERE <filter>} 2. Run cupertino search "<query>" --limit 10 3.
Read details →All 6 of 6 query classes A-G from §1.4 (excluding Phase 1.7 which is its own design) now have documented baselines on the v1.2.0 candidate DB:
Read details →Every metric and method this audit relies on, with a link to the foundational source. Auto-collected from the audit text.
Manning, Raghavan, Schütze (2008) IIR §8.4
Open citationVoorhees (1999), TREC-8 QA Report
Open citation