# LoCoMo Benchmark Scripts

Self-contained Modal scripts for benchmarking 8 memory systems on the LoCoMo dataset. Each script runs independently with zero local imports -- all dependencies are installed inside the Modal container.

## Prerequisites

```bash
# Required for all systems
modal secret create openrouter-key OPENROUTER_API_KEY=sk-or-...

# Required for Supermemory only
modal secret create supermemory-key SUPERMEMORY_API_KEY=sm_...
```

## Scripts

### bench_bm25.py -- BM25 Keyword Baseline
Okapi BM25 term-frequency retrieval using `rank-bm25`. No embeddings or neural models. Pure TF-IDF keyword matching with top-100 retrieval.

```bash
modal run --detach bench_bm25.py          # Full run
modal run --detach bench_bm25.py --smoke  # Smoke test
```

### bench_engram.py -- Engram Memory
Engram (`engram-core`) lightweight memory system with built-in SQLite storage. Stores facts with importance scores, retrieves via built-in search with limit=100.

```bash
modal run --detach bench_engram.py          # Full run
modal run --detach bench_engram.py --smoke  # Smoke test
```

### bench_evermemos.py -- EverMemOS
EverMemOS uses pre-computed agentic retrieval data stored on a Modal Volume. The retrieval file (`evermemos_retrieval.json`) contains pre-built context for all 1540 questions, generated by EverMemOS's BM25+Embedding+RRF+Reranker pipeline. This script only performs answer generation and judging.

**Pre-requisite:** Upload the retrieval file first:
```bash
modal volume put locomo-results evermemos_retrieval.json /
```

```bash
modal run --detach bench_evermemos.py          # Full run
modal run --detach bench_evermemos.py --smoke  # Smoke test
```

### bench_mem0.py -- Mem0
Mem0 (`mem0ai`) with LLM-based memory extraction. Uses OpenRouter as LLM backend and HuggingFace sentence-transformers for embeddings. Mem0 extracts structured memories from raw text using an LLM, then retrieves via embedding similarity.

```bash
modal run --detach bench_mem0.py          # Full run
modal run --detach bench_mem0.py --smoke  # Smoke test
```

### bench_truememory_edge.py -- TrueMemory Edge Tier
TrueMemory Edge using Model2Vec (`potion-base-8M`, 256-dim) embeddings and cross-encoder reranker (`cross-encoder/ms-marco-MiniLM-L-6-v2`, 22M params). No HyDE, no GPU required. 3-run mean: 89.6% LoCoMo.

```bash
modal run --detach bench_truememory_edge.py          # Full run
modal run --detach bench_truememory_edge.py --smoke  # Smoke test
```

### bench_truememory_base.py -- TrueMemory Base Tier (Default, GPU recommended)
TrueMemory Base using Qwen3-Embedding-0.6B truncated to 256-dim (Matryoshka) and `Alibaba-NLP/gte-reranker-modernbert-base` (149M params). No HyDE. 3-run mean: 92.0%. Runs on CPU but a T4 GPU is strongly recommended for the reranker.

```bash
modal run --detach bench_truememory_base.py          # Full run
modal run --detach bench_truememory_base.py --smoke  # Smoke test
```

### bench_truememory_pro.py -- TrueMemory Pro Tier (+HyDE, GPU)
TrueMemory Pro uses the same embedder + reranker as Base, plus HyDE (Hypothetical Document Embeddings) via OpenRouter LLM. 3-run mean: 93.0%. Requires a T4 GPU on Modal and an OpenRouter key for the HyDE call.

```bash
modal run --detach bench_truememory_pro.py          # Full run
modal run --detach bench_truememory_pro.py --smoke  # Smoke test
```

### bench_rag.py -- ChromaDB RAG Baseline
Standard dense vector retrieval using ChromaDB with sentence-transformers embeddings. Cosine similarity search with top-100 retrieval.

```bash
modal run --detach bench_rag.py          # Full run
modal run --detach bench_rag.py --smoke  # Smoke test
```

### bench_supermemory.py -- Supermemory Cloud API
Supermemory cloud-hosted memory service. Ingests via API, waits for async indexing (90s + verification), then retrieves via search endpoint. Requires a separate Supermemory API key.

```bash
modal run --detach bench_supermemory.py          # Full run
modal run --detach bench_supermemory.py --smoke  # Smoke test
```

## Utility Scripts

### verify_scores.py
Recomputes accuracy from the raw JSON result files. Zero dependencies beyond Python stdlib.

```bash
python3 verify_scores.py
```

## Downloading Results

All scripts save results to the Modal Volume `locomo-results`. Download them with:

```bash
modal volume get locomo-results / ./results --force
```

## Common Flags

All `bench_*.py` scripts accept:
- `--smoke` -- Run a smoke test (1 conversation, 5 questions) to validate setup.
- `--dataset <path>` -- Override the dataset path (defaults to `../data/locomo10.json`).

## Evaluation Pipeline

Every script uses the identical evaluation pipeline:
- **Answer model:** `openai/gpt-4.1-mini` via OpenRouter, `temperature=0`, `max_tokens=200`
- **Judge model:** `openai/gpt-4o-mini` via OpenRouter, `temperature=0`, `max_tokens=10`
- **Voting:** 3 judge runs per question, majority vote
- **Checkpointing:** Results are saved after each conversation and can resume from interruption
