# v0.4 to v0.5 migration guide

## What changed

Sparse embedding vectors (SPLADE, BGE-M3-sparse, etc.) are no longer stored
inline on `Node` bytes. They now live in a per-commit Prolly sidecar tree
(`Commit.sparse`, keyed by `NodeCid`). This mirrors the G16 change that moved
dense embeddings to `Commit.embeddings`.

Moving sparse data to the sidecar means `NodeCid` is byte-identical across
machines regardless of which sparse encoder version or vocabulary snapshot was
used. Federated dedup is no longer broken by encoder differences.

## Pre-upgrade backup

```sh
cp -r .mnem .mnem.bak
```

## What happens to v0.4 data when opened by v0.5

v0.5 reads old repos transparently. The DAG-CBOR `sparse_embed` map that was
inline on `Node` in v0.4 is absorbed into `node.extra["sparse_embed"]` by the
forward-compat flatten decoder. `NodeCid` is unchanged - the bytes round-trip
identically.

Nodes are fully queryable (graph traversal, prop lookups, text search, dense
nearest-neighbour retrieval). They just have no entry in the sparse sidecar
yet, so sparse retrieval will not return them until you run one of the two
paths below.

## Path 1 - re-derive from text (default, needs sparse encoder)

```sh
mnem config set sparse.provider splade
mnem config set sparse.model opensearch-doc-v3-distill
mnem reindex
```

This calls the configured sparse encoder for each node and writes fresh
sparse vectors into the sidecar. Safe to run multiple times (idempotent
unless `--force` is passed).

## Path 2 - lift existing sparse vectors without an encoder

```sh
mnem reindex --lift-legacy-sparse
```

Reads the raw `SparseEmbed` bytes from `node.extra["sparse_embed"]` and
promotes them directly into the sidecar without any network or model call.
Works offline and preserves the original vectors exactly. Preferred for
operators who do not have the original encoder available.

`--dry-run` is supported: counts how many nodes have `extra["sparse_embed"]`
without touching storage.

Cannot be combined with `--force` (use one or the other).

## Verifying NodeCid stability after reindex

Pick any node UUID and confirm its CID matches before and after:

```sh
mnem get <uuid>         # note "node_cid: <cid>"
mnem reindex --lift-legacy-sparse
mnem get <uuid>         # must show the same node_cid
```

Alternatively, `mnem cat-file <node_cid>` should decode successfully and
return the same bytes as before.

## Note: nodes without a sparse vector after upgrade

If neither `mnem reindex` nor `mnem reindex --lift-legacy-sparse` is run,
nodes are fully accessible via graph, prop, and dense nearest-neighbour
queries. They will not appear in sparse retrieval results (`mnem retrieve
--sparse`) until reindexed.

## Note: running both dense and sparse lift in one pass

If your v0.4 repo had both dense (`extra["embed"]`) and sparse
(`extra["sparse_embed"]`) inline fields (uncommon but possible), lift each
separately:

```sh
mnem reindex --lift-legacy-extra    # promotes dense embedding sidecar
mnem reindex --lift-legacy-sparse   # promotes sparse embedding sidecar
```

Both operations are safe to run in either order or in sequence. Neither
modifies `NodeCid`.
