SQLite
Serverless database for on-disk text storage.
What is it?
SQLite is a self-contained, serverless SQL database engine. It requires no separate process, no configuration, no administration. The entire database lives in a single file. It ships with Python's standard library and is the most widely deployed database engine in the world.
In Membot
The split cartridge format uses SQLite for text storage. The vector index (_index.npz) stays in RAM for fast search, while the raw text lives on disk in a SQLite database (_text.db). After search identifies the top results by vector similarity, Membot fetches only those texts from SQLite — typically taking ~1 ms per lookup.
This architecture means you can search millions of entries without loading all the text into memory.
Format comparison
| Aspect | Standard (.cart.npz) | Split (_index.npz + _text.db) |
|---|---|---|
| Text storage | In-memory (pickle) | On-disk (SQLite) |
| RAM at 2.4M | ~8 GB | ~1.5 GB (vectors only) |
| Text lookup | Instant (in RAM) | ~1 ms (disk) |
| Portability | Single file | Two files |
| Max practical scale | ~500K entries | Millions+ |
Why SQLite?
- Zero setup — no database server to install or manage
- Built-in — ships with Python, no pip install needed
- Portable — the .db file works on any platform
- Concurrent reads — multiple processes can read simultaneously
- Paged access — only loads the data pages actually needed from disk
Further reading
Brain cartridges — the file format overview
Sign-Zero Encoding — how the index is compressed