# Graph / Vector Data Migrations

ONE alembic-style revision chain for cognee's **data migrations** — each
migration is a cross-store transformation that may touch the graph database,
vector database and relational delete-ledger together (the relational schema
has its own Alembic setup under `cognee/alembic/`). Each database pair records
the slug of the last migration applied to it; on startup the runner walks the
chain forward, stamping after every step.

## Where state lives

| Mode | Revision | Written by |
|---|---|---|
| Access control ON (default) | `dataset_database.migration_revision` — one row per dataset's database pair | head-stamped at row creation (`get_or_create_dataset_database`); advanced per applied step by `runner.run_database_migrations` |
| Access control OFF | `global_database_version.global_migration_revision` — one row, one global pair | `runner._run_global_migrations` |

`cognee_version` columns are **audit-only** (which release last migrated /
last started); nothing gates on them. Revisions are the only gates.

## When migrations run

- FastAPI lifespan (`api/client.py`) — every server start, once per worker
- the first `remember()` or `cognify()` call in an SDK process (both write
  new-scheme ids, so both migrate first; once-per-process guarded)
- explicitly: `await cognee.run_migrations()`

Steady state is cheap: an in-memory revision comparison per database, nothing
opened, nothing written.

## Operator commands (alembic-style)

```
cognee-cli upgrade [revision]              # head (default) or a slug; partial-chain ok
cognee-cli downgrade <revision> [--dataset ID]   # 'base' or a slug; data-rewriting, confirmed
cognee-cli stamp <revision> [--dataset ID] # set the stored revision WITHOUT running anything
cognee-cli history                         # the chains, newest first
cognee-cli current                         # each database's stamped revision
```

`stamp` is the bookkeeping-repair escape hatch: `stamp base --dataset <id>`
re-arms the chain for a database whose data drifted from its stamp (restored
backup, old-code writes after migration), letting the idempotent chain
converge it on the next upgrade.

## Authoring contract — read before writing migration #2

1. **Slugs are append-only and immutable.** The slug IS the revision stored in
   every deployed database. Renaming or removing a shipped entry orphans every
   stamped database: `pending_migrations` logs a warning and runs NOTHING for
   it, forever. Never "swap" an existing entry — append after it with
   `down_revision=<previous slug>`.
2. **Idempotent, and cheap on empty stores.** Fresh deployments execute the
   whole chain once (revisions start NULL); your migration must no-op quickly
   against an empty database, and re-running it against an already-migrated
   one must change nothing.
3. **Crash-retry safe, derived stores first.** If you rewrite ids that live in
   several stores, compute the map while the source of truth (the graph) still
   holds the source ids; re-key the derived stores (vector points, relational
   ledger) first and rename the graph LAST, so a crash at any point leaves a
   state the next startup finishes. See `versions/namespace_entity_type_node_ids.py`
   for the worked example.
4. **Freeze everything you derive.** Vendor private copies of id derivations,
   normalization rules and collection names into the migration module
   (`_frozen_*`). A revision must mean one deterministic transformation
   forever — never import live models (`Entity`, `id_for`, `index_fields`)
   into a migration; when the live scheme changes, that is a NEW migration.
5. **`MigrationContext`** carries `graph_engine`, `vector_engine`, and
   `dataset_id` (`None` in global mode → ledger updates unscoped). Get the
   relational engine yourself via `get_relational_engine()` (lazy import).
6. **Downgrades** are optional per migration (`down=`), explicit-only
   (`runner.downgrade_database_migrations`), and a chain can only be
   downgraded through a contiguous span where every entry defines `down`.
7. **Validate at import.** The registry validates the chain (linearity,
   unique slugs, reserved keywords) at module bottom so a typo'd
   `down_revision` fails in CI, not at customer startup. The unit tests
   additionally pin the shipped slugs.

## Landmine index (things migration #1 had to learn the hard way)

- Vector payloads are **`IndexSchema`-shaped** (`{id, text, belongs_to_set}`),
  written by `index_data_points` — never a dump of the source model. Re-insert
  through `index_data_points` with a small carrier; never reconstruct models
  from payloads.
- Re-keys must be **merge-safe**: the target id may already exist (an SDK
  process wrote new-scheme data before migrating). Delete the old row instead
  of moving it onto a duplicate key.
- Ladybug fabricates `(id, id, "SELF")` placeholder edges for edgeless graphs —
  filter them or you persist fake relationships.
- cognify embeds `source_node_id`/`target_node_id` INSIDE edge properties and
  retrieval prefers them over topology — remap them with the edge.
- `Triplet_text` point ids are hashed from edge endpoint ids — entity renames
  move triplet points too.
- The relational delete-ledger (`nodes.slug`, `edges.source/destination`) and
  the legacy `graph_relationship_ledger` key on graph node ids.
- Engines arrive wrapped (`_VectorEngineHandle`, leased proxies) — they spoof
  `__class__`, so dispatch on `engine.__class__.__name__`, never `type(engine)`.
- lance 0.32's `merge_insert` can panic on tables carrying deletion vectors —
  prefer plain `add` + delete in migrations, compact when the handle allows.
- Hybrid graph+vector backends (Neptune Analytics) store vectors as graph
  nodes sharing the entity id — id migrations must detect
  (`_is_hybrid_provider`) and refuse rather than corrupt.
- `get_graph_data()` loads the ENTIRE graph into memory; fine at current
  scales, budget for it in a migration that targets chunk-scale data.

## Concurrency

The runner serializes migrate-then-stamp per database with a Postgres
advisory lock held on a dedicated connection (no open transaction, so long
migrations don't block anything relational). SQLite has no cross-process
primitive: multi-worker startup against one SQLite metadata store during a
migration window is unsupported. Out-of-process writers (rolling deploys, SDK
scripts on old code) are NOT serialized against — ship id-scheme changes with
a maintenance window.

## Testing a migration

- Unit: drive the store-helpers against the fakes in
  `cognee/tests/unit/modules/migrations/` — they mirror the REAL adapter
  contracts (IndexSchema payloads; no `create_data_points` on the vector fake,
  so model-reconstruction regressions fail loudly).
- Integration: `cognee/tests/migrations/test_migration_lockstep.py` — seed via real cognify,
  downgrade via the migration's own `down`, re-migrate, verify all stores,
  re-cognify, hard-delete. Run per phase in separate processes.
- Cross-version: `cognee/tests/backwards_compatibility/phase{1,2}` against a
  `main`-seeded database.
