ATM-Bench

NIAH Evaluation Setup

In addition to end-to-end retrieval + generation evaluation, we provide NIAH (Needle In A Haystack):

• Each question is paired with a fixed evidence pool (niah_evidence_ids) that contains all ground-truth items.
• The rest of the pool is filled with realistic distractors.
• This isolates answer generation/reasoning quality from retrieval quality.

See: - docs/niah.md

<a id="quick-start"></a>

Installation

conda create -n atmbench python=3.11 -y
conda activate atmbench
pip install -r requirements.txt
pip install -e .

🚀 Quick Start

Optional but recommended: preload reverse-geocoding cache

ANSWERER_MODEL env vars).

bash scripts/QA_Agent/MMRAG/run.sh

+ OPENAI_API_KEY set in the environment or api_keys/.openai_key.

bash scripts/QA_Agent/Oracle/run_oracle_gpt5.sh ```

Baseline Compatibility and Environments

• Core baselines (MMRAG, Oracle, NIAH) are tested in the main atmbench environment.
• Third-party memory-system baselines in this repo include:
• A-Mem
• HippoRAG2
• mem0
• MemoryOS
• MemPalace
• SimpleMem
• MemoryOS and MemPalace are strongly recommended to run in separate conda environments. MemoryOS uses a FAISS / sentence-transformers stack, while MemPalace uses ChromaDB / ONNX-backed local embeddings; isolating them avoids dependency collisions with the core baseline environment and each other.
• A-Mem, HippoRAG2, and mem0 are tested to be compatible with the core baseline environment, but separate environments are still safer for reproducibility and dependency isolation.
• SimpleMem runs against a sibling clone of the upstream repo (LanceDB + Tantivy FTS stack); see memqa/qa_agent_baselines/SimpleMem/README.md. Pinned upstream commit: 094027eca4c890dc9912be8cee1da04428de8076 (verified by scripts/QA_Agent/SimpleMem/run.sh).
• Setup references for the vendored baselines are under third_party/:
• third_party/A-mem/
• third_party/HippoRAG/
• third_party/mem0/
• third_party/MemoryOS/
• MemPalace ships as a PyPI package (mempalace==3.3.5) and is installed via memqa/qa_agent_baselines/Mempalace/requirements.txt — no third_party/ vendoring.
• SimpleMem is not vendored under third_party/. Clone the upstream repo at the pinned commit alongside ATMBench and point SIMPLEMEM_DIR at it (defaults to ../SimpleMem):

  git clone https://github.com/aiming-lab/SimpleMem.git ../SimpleMem
  git -C ../SimpleMem checkout 094027eca4c890dc9912be8cee1da04428de8076
  pip install -r ../SimpleMem/requirements.txt
  pip install -r memqa/qa_agent_baselines/SimpleMem/requirements.txt
  

For detailed setup, data layout, and reproducibility settings, see: - docs/README.md - docs/data.md - docs/reproducibility.md - docs/baseline.md - docs/niah.md

<a id="repository-structure"></a>

API Keys

Set via environment variables:

export OPENAI_API_KEY="your-key"
export VLLM_API_KEY="your-key"

Or use local key files (gitignored): - api_keys/.openai_key - api_keys/.vllm_key

+ a running vLLM endpoint at http://127.0.0.1:8000/v1/chat/completions

serving Qwen/Qwen3-VL-8B-Instruct-FP8 (override with VLLM_ENDPOINT /

+ a running vLLM endpoint serving Qwen/Qwen3-VL-8B-Instruct-FP8.

bash scripts/QA_Agent/Oracle/run_oracle_qwen3vl8b_raw.sh

ATM-Bench: Long-Term Personalized Referential Memory QA

The first benchmark for multimodal, multi-source personalized referential memory QA over long time horizons (~4 years), with evidence-grounded retrieval and answering.

🇬🇧 English • 🇨🇳 中文

🚀 Quick Start • 🤖 Agent Results • 🧠 Memory Systems • 📊 Oracle / NIAH • 🏆 Live Leaderboard • 📖 Citation

</div>

<video src="https://atmbench.github.io/static/videos/ATM-Bench-demo.mp4" controls width="100%"></video>

📄 Paper: According to Me: Long-Term Personalized Referential Memory QA 🌐 Project Page: https://atmbench.github.io/ 🏆 Live Leaderboard: https://atmbench.github.io/leaderboard.html