---
id: w1/m19/t001
title: Backups first: etcd + OpenBao Raft snapshots, pg_dump every CNPG DB, verified
worker: worker1
status: done
estimate: 45m
depends_on: []
---

## Objective

Take and verify restorable backups of every stateful system before any teardown: etcd (app cluster state), OpenBao (tenant credentials), and all CNPG Postgres databases (control-plane store + Kratos/Hydra + tenant DBs). m17 wired barmanObjectStore backups for TENANT Databases (operator-side), but the platform clusters (bex-db/kratos-db/hydra-db, deploy/gitops) have no continuous backup - for them the dumps here are the only safety net; for tenant DBs they are the belt-and-suspenders.

## Context

- Runbooks exist for two of three: docs/ADR011-etcd-backup-restore.md, docs/ADR015-openbao-backup-restore.md (nightly CronJobs -> object storage).
- CNPG clusters live: `bex-db` (default ns, control-plane store), `kratos-db`/`hydra-db` (auth ns), plus any tenant Databases (`kubectl get clusters.postgresql.cnpg.io -A`).
- Object storage credentials: same bucket the etcd/OpenBao backup CronJobs use (see their manifests in deploy/gitops/).

## Steps

1. Trigger a fresh etcd snapshot (or verify tonight's) and a fresh OpenBao Raft snapshot; confirm both land in object storage with sane sizes.
2. `pg_dump -Fc` every database of every CNPG cluster via `kubectl exec` on the primary; upload to the same bucket under `rearch-2026-07-10/`.
3. Dump the bex CRs that live ONLY in etcd (the control-plane store projects apps, but NOT datastores): `kubectl get apps,databases,keyvalues.app.bex.co -A -o yaml` -> `crs.yaml` in the bucket. Without this, t006's rebuild silently loses every tenant Database/KeyValue (found in the 2026-07-10 pull review: store schema has no databases/keyvalues tables).
4. Valkey data: for each KeyValue instance, `redis-cli BGSAVE` (with its password) + copy `dump.rdb` from the PVC to the bucket.
5. Verify restorability: `pg_restore --list` on each dump (`--no-owner` dumps to dodge role mismatches on the rebuilt clusters); checksum manifest committed to the bucket.

## Files

- scripts/ (one-off backup script is fine to keep as scripts/rearch-backup.sh if reusable)
- no repo files strictly required — this is an operational task

## Acceptance criteria

- [x] Object storage contains dated etcd snapshot, OpenBao snapshot, and one verified `pg_dump` per database, with a checksum manifest. _Done 2026-07-10_, `s3://<tf-state-bucket>/rearch-2026-07-10/`: `bex.dump`/`kratos.dump`/`hydra.dump`/`openfga.dump` (pg 18.4, `--no-owner`, in-pod `pg_restore --list` verified) + `crs.yaml` (3 Apps; 0 Databases; 0 KeyValues) + merged `SHA256SUMS`; fresh etcd snapshot via the CronJob. **OpenBao snapshot: n/a — prod OpenBao was NEVER initialized** (all 3 pods 501/uninitialized; m10's "prod activation runbook" never ran; env-vars API has never worked on prod). **Valkey: n/a** (no KeyValue instances). **Fixed live en route:** `etcd-backup-s3`/`openbao-backup-s3` secrets were missing — etcd backups had been failing ≥3 days; recreated per the runbooks, etcd backup green again. **Gotchas:** dump jobs must run in a netpol-allowed namespace (`bex-system`/`auth` — kube-system is denied by `deny-tenant-ingress`); CP-node `kubectl exec/logs` is dead (kubelet serving cert expired — rearchitecture.md symptom A materialized); use worker-scheduled jobs + termination-log for output.
- [x] `pg_restore --list` succeeds on every dump (run in-pod immediately after each dump).
- [x] `crs.yaml` contains every App/Database/KeyValue CR (3/0/0); Valkey RDB n/a — zero KeyValue instances.

## Out of scope

- Wiring CNPG continuous backup/PITR (that is w1/m17's job; this task is its why-now).
