# CAS-100 — Cortex Adversarial Suite

**Version:** v1 (2026-04-24)
**Items:** 100
**Categories:** 15
**Reference:** `docs/internal/v060/research/judge-reliability-adversarial-eval.md` §6.3-7.4

The first hand-authored adversarial evaluation suite for Cortex's recall quality. Designed to stress-test the core daemon against failure modes that LongMemEval-S does not adequately cover. Each item is engineered so that at least one adapter (BM25-only, pure-dense, full stack) is expected to fail.

---

## Format

JSONL, one item per line. Schema:

```json
{
  "id": "cas-NNN",                // unique string, monotonic
  "category": "<one of 15>",      // see table below
  "q": "user question",
  "gold": ["answer", "alt 1"],    // array; empty → abstention expected
  "tier": "easy|medium|hard",
  "probes": "what this item tests",
  "context_memories": [           // sequence of memory strings to ingest
    { "text": "...", "context": "optional session id or source" }
  ],
  "notes": "optional: author caveats / known-ambiguous items"
}
```

### Semantics

- **`gold`**: accepted correct answers. Judge uses any-of matching. If empty, the only correct response is an abstention ("I don't know", "no memory", etc.).
- **`context_memories`**: the exact memory stream that must be ingested before answering. Order matters — last-write wins for the judge's temporal interpretation.
- **`tier`**: engineering budget. Easy = 1 hop, lexical match. Medium = paraphrase or 1-2 distractors. Hard = multi-hop, negation, or temporal trick.

---

## Category breakdown

| # | Category | Count | Probes |
|---|----------|------:|--------|
| 1 | paraphrase | 7 | BM25 keyword miss + embedding semantic drift |
| 2 | typo | 7 | Embedding robustness, no spell-correct in core |
| 3 | code-reference | 7 | Domain terms, token boundaries |
| 4 | semantic-drift | 7 | Temporal + semantic resolution |
| 5 | distractor-heavy | 7 | Re-ranking, RRF tuning |
| 6 | multi-hop-distractor | 7 | Entity + event disambiguation |
| 7 | temporal-trick | 7 | Recency vs historical preference resolution |
| 8 | temporal-anchor | 7 | Temporal grounding without explicit date |
| 9 | negation | 6 | Dense retriever collapse under negation |
| 10 | ambiguity | 6 | Disambiguation UI surface or most-recent |
| 11 | near-duplicate | 6 | Family collapse correctness |
| 12 | rolled-back | 6 | Temporal validity (superseded decisions) |
| 13 | conflict | 6 | Last-write-wins vs confidence weighting |
| 14 | abstention-missing | 7 | Correct abstention when no memory exists |
| 15 | abstention-wrong-domain | 7 | Correct abstention outside Cortex scope |
| — | **Total** | **100** | |

## Tier distribution

- Easy: **30** (30%) — single hop, lexical match, zero distractors
- Medium: **45** (45%) — semantic paraphrase, 1-2 distractors
- Hard: **25** (25%) — multi-hop, temporal tricks, heavy distractors

Per category, roughly 2 easy / 3 medium / 2 hard; abstention categories skew easier since the failure mode (hallucinating memory that doesn't exist) is binary.

---

## Authoring protocol (per item)

1. **Drafted** by primary engineer, targeting one category at a time.
2. **Reviewed** for gold-answer uniqueness and equivalent formulations.
3. **Filter pass:**
   - Weak retriever (BM25-only pure) should fail ≥30% of hard items.
   - Strong retriever (full Cortex stack) should fail ≤15% after Phase 2 lands, ≤5% by v0.6.1.
   - Any item both retrievers ace → replace (too easy).
   - Any item both fail → verify a human can solve it; otherwise drop (malformed).

Items failing all three filters are dropped, not published.

---

## How to run

```bash
# Full CAS-100 against cortex-http-pure
bash scripts/benchmark-triad.sh cas-100

# Subset by category
python benchmarking/run_cas.py \
  --suite cas-100.jsonl \
  --filter category=rolled-back \
  --memory-backend cortex-http-pure \
  --output results/cas100-rolledback-YYYYMMDD.json

# Triangle judge (recommended — see `triangle.py`)
python benchmarking/judges/triangle.py \
  --answers results/cas100-answers.jsonl \
  --gold benchmarking/adversarial/cas-100.jsonl \
  --judges gpt-4o,claude-opus-4-6,qwen3-30b-local
```

Scored with: `correct / total`, plus per-category breakdown. Wilson 95% CI half-width at N=100, p=0.85 ≈ ±7pp.

---

## Purity pledge

No item in CAS-100 may be tuned against during development. Items are constructed from first-principles stress-testing, reviewed manually, and frozen. Regeneration of items requires a version bump (`cas-100-v2.jsonl`). Old versions remain in repo for comparison.

Items **must not** be shown to the answerer LLM in raw form. Only memory fragments + question are presented; gold answers are visible only to the judge.

---

## Known limitations

1. **Single-author bias.** Items initially authored by one engineer; second- and third-pass review planned for v0.6.1.
2. **English-only.** Non-English adversarial items deferred to v0.7.0.
3. **100 items ≠ stable at single-point-decision level.** ±7pp Wilson CI; for sub-5pp claims, expand to CAS-200 (v0.6.1 follow-up) or CAS-500 (v0.7.0).
4. **Memory fragments are synthetic.** Constructed to embed the needle. Real-world memory streams have different distractor shapes; add real-corpus sampled items in v0.6.1.
5. **No multi-agent items.** All items assume single-user single-session; multi-tenant abuse items deferred to v0.7.0 with the multi-scope feature.

---

## Revision log

| Version | Date | Items | Notes |
|---------|------|------:|-------|
| v1 | 2026-04-24 | 100 | Initial authoring per §6.4 research recommendation |
