# Skill-Store Recall Benchmark

End-to-end retrieval-quality + latency benchmark on a single-node YantrikDB.
Measures whether YantrikDB is fit-for-purpose as a **skill substrate** for
agent systems: thousands of typed knowledge records (skills, lessons,
procedures), with a mix of broad ("what skills apply here?") and sharp
("find this specific skill") queries.

## Headline result (5000 skills, single-node, builtin embedder)

| Metric | Value |
|---|---|
| Skills written | 5000 / 5000 (zero failures) |
| Write throughput (server-side embedding) | **26 writes/sec** |
| Recall p50 / p95 | **87 ms / 115 ms** |
| recall@5 — overall | **0.86** |
| recall@5 — broad queries (find any of 10 sibling skills) | **0.96** |
| recall@5 — sharp queries (find this exact skill) | 0.76 |
| SQLite exact-id cache lookup p50 / p95 | **79 µs / 89 µs** |
| Cache disk size for 5000 skills | 4.2 MB |

**Reading the numbers**: at this scale, YantrikDB delivers production-grade
semantic recall under 200 ms for every query, with retrieval quality high
enough that 86% of arbitrary queries find a target skill in the top-5
results — and 96% on the realistic "any from this topic family" use case.
Combined with a thin SQLite side-cache for known-id lookups, the substrate
serves both `skill_recall(query)` (semantic discovery) and `skill_get(id)`
(exact fetch) at the latency budgets agent systems require.

The 0.76 sharp-query recall reveals the limit of pure semantic search at
scale: when many sibling skills share vocabulary, structured metadata
filters (`applies_to`, `min_success_count`, keyword overlap) need to layer
on top — see `bench.py --help` for the filter knobs once added.

## Methodology

### Corpus

5000 synthetic skill records generated by `generate.py`:

- **50 topics** spanning agent infrastructure (recipe authoring, template
  resolution, vector recall, leader election, model routing, telegram
  chunking, ...)
- **10 actions** per topic (diagnose, mitigate, retry, batch, parallelize,
  dedupe, version, audit, recover, escalate)
- **10 variants** per (topic, action) tuple, each with a distinguishing
  trigger phrase + counterexample so sibling skills are *confusably similar
  by design* — this stresses the embedder's ability to discriminate.

Each skill is stored as a `comm.skill` record:

```jsonc
{
  "namespace": "skill_test_substrate",
  "text": "<skill body — embedded server-side>",
  "memory_type": "procedural",
  "domain": "skill",
  "importance": 0.55,
  "metadata": {
    "record_type": "skill",
    "schema_version": 1,
    "skill_id": "skill.<topic>.<action>.v<n>",
    "version": 1, "status": "active",
    "skill_type": "lesson",
    "applies_to": [topic],
    "triggers": [<distinguishing phrases>],
    "topic_family": <topic>, "action_family": <action>,
    "success_count": 0, "failure_count": 0
  }
}
```

### Queries

100 ground-truth queries split into two cohorts:

1. **Family-level** (50 queries) — paraphrased natural-language form ("How
   do I retry safely on transient failures of issues in routing model
   calls to OpenAI/Anthropic/Ollama?"). Target = any of the 10 variant
   skills in that (topic, action) family.

2. **Variant-specific** (50 queries) — trigger-phrase form ("I see symptom
   X during topic Y. What's the right action pattern?"). Target = one
   specific skill out of 5000.

For each query we run `/v1/recall(top_k=10)` and record:
- the smallest rank where any target appears (or None if not in top-10)
- end-to-end HTTP latency (no warm-up; numbers include cold + warm)

### Cache benchmark

`cache_bench.py` populates a side-cache mapping `skill_id → rid + cached_text`
via a single bulk recall, then performs 1000 random `skill_id` lookups
(with 10% intentional misses) against pure SQLite — no network. This
models the agent's "I know which skill to use, give me the body" path,
where semantic search would be wasted work.

## Setup

```bash
# 1. Boot a single-node YantrikDB locally.
cd benchmarks/skill_recall
docker compose up -d

# 2. Mint a token. The server creates a "default" db on startup; we mint a
#    token against it via a sidecar exec so the running server reads the
#    same control.db.
#
# WINDOWS / Git Bash users: prefix `docker exec` calls with
# `MSYS_NO_PATHCONV=1` — without it, MSYS rewrites the absolute Linux
# paths (`/var/lib/yantrikdb`) to a Windows-style path
# (`C:/Program Files/Git/var/lib/yantrikdb`) before they reach docker.
# The CLI then writes to the wrong location inside the container, so
# the token row never lands in the `control.db` the server reads.
# Symptom: token create succeeds but `/v1/remember` returns "invalid or
# revoked token" against the freshly-minted token.
#
docker exec yantrikdb-skill-bench yantrikdb db --data-dir /var/lib/yantrikdb \
  list   # confirm 'default' exists
docker exec yantrikdb-skill-bench yantrikdb token --data-dir /var/lib/yantrikdb \
  create --db default --label bench
# → save the printed token

export YDB_TOKEN=<token>

# 3. Generate corpus + queries (deterministic, seed=42).
python generate.py 5000

# 4. Seed.
python seed.py --nodes 127.0.0.1 --port 17438 --rate-per-sec 50

# 5. Recall benchmark.
python bench.py --nodes 127.0.0.1 --port 17438

# 6. Exact-id cache benchmark.
python cache_bench.py --nodes 127.0.0.1 --port 17438
```

Reports land in `report.md` and `cache_report.md`.

## Files

| File | Purpose |
|---|---|
| `generate.py` | Deterministic synthetic-corpus + ground-truth-query generator |
| `seed.py` | Rate-limited writer; cluster-aware fallback across nodes |
| `bench.py` | 100-query semantic-recall benchmark with recall@K + latency |
| `cache_bench.py` | SQLite skill_id cache populate + lookup latency benchmark |
| `docker-compose.yml` | Single-node yantrikdb container, port 17438 |
| `yantrikdb_single.toml` | Single-node server config (no cluster, builtin embedder) |

## What this benchmark does NOT cover

- **Concurrent reads.** All queries are sequential; under N parallel
  workers latency may degrade. See `tests/chaos/` for cluster behavior
  under load.
- **Scale beyond 5k.** Vector indexes scale logarithmically with corpus
  size, but this benchmark is calibrated for the agent-systems use case
  (1k–10k skills typical). Production deployments at 100k+ should re-run.
- **Decay-weighted ranking.** All synthetic skills are equally fresh;
  `temporal()`-weighted scoring isn't exercised.
- **Cluster replication latency.** Single-node only — replication adds
  microseconds-to-milliseconds depending on quorum size and network.
- **Multi-tenant isolation.** All records in one namespace; cross-tenant
  noise isn't tested.
- **Filter-augmented recall.** The 0.76 sharp-query recall is the pure-
  semantic floor; structured-filter results would land here once the
  `applies_to=...` / `kind=...` query parameters land.

## Why this matters for YantrikDB

YantrikDB is positioned as the persistent memory substrate for agent
systems. A useful skill substrate has to deliver:

1. **Semantic discovery at scale** — given a context query, surface
   relevant typed knowledge in milliseconds.
2. **Exact-id fetch in microseconds** — when the agent already knows
   which skill to use, no network round-trip.
3. **Structured metadata** — filter by skill_type, apply-to-domain,
   success rate; not just text similarity.
4. **Audit + provenance** — every recall traces to written records with
   timestamps, authors, evidence chains.

This benchmark validates (1) and (2) empirically at 5k scale. (3) and
(4) are properties of the schema, not retrieval, and are exercised in
`tests/integration/`.

## Reproduce in CI

The bench is non-flaky and deterministic (random seed = 42 in
`generate.py`). A GitHub Actions matrix could pin it as a regression
gate: any PR that drops recall@5 below 0.80 or pushes p95 above 200 ms
gets flagged.
