---
id: w1/m19/t006
title: Execute: teardown -> rebuild via CI -> GitOps converge -> restore data
worker: worker1
status: todo
estimate: 90m
depends_on: [w1/m19/t001, w1/m19/t002, w1/m19/t003, w1/m19/t004, w1/m19/t005]
---

## Objective

~~The one-shot rebuild~~ **Executed 2026-07-10/11 (op log below + m19.1).** _Rescoped 2026-07-11 to the quota-gated residue only:_ flip the two `TEMP (m19.1)` knobs (KCP 1→3, platform max 2→3) once the Hetzner quota lands, let the machines self-materialize, then **join + unseal `openbao-1`** (keys already in `.env`/GH secrets — OpenBao was first-initialized 2026-07-11 on a 2-member raft; do NOT run `bao-init.sh` init again, and there is no snapshot to restore).

## Context

- Order inside this task: verify t001 backups readable FIRST; then `kubectl delete cluster bex` on the mgmt cluster (CAPH deprovisions machines + kube-api LB); delete the CCM-created `bex-traefik` LB via hcloud API; `terraform apply` (firewall); merge/push the t002-t005 PR -> app-cluster.yml runs: CAPH creates network `bex` + LBs + 3 CP + platform pool; addons (Cilium+WireGuard, CCM, CSI); deploy.yml bootstraps Argo -> platform converges from deploy/gitops/overlays/prod.
- Restore order matters: OpenBao — **no snapshot to restore** (t001 finding: prod was never initialized); instead run `scripts/bao-init.sh` fresh on the new cluster (the m10 prod-activation runbook, finally) + `scripts/bao-k8s-auth.sh`; **re-apply t001's `crs.yaml` FIRST** (Database/KeyValue CRs exist only in etcd - the operator then recreates CNPG clusters + Valkey instances); then `pg_restore` t001's dumps into the recreated clusters (postgres:18 client — servers run PG 18.4) and restore each `dump.rdb` into its Valkey PVC; etcd snapshot itself NOT restored (new cluster; Apps re-project from the store, datastores come from `crs.yaml`).
- The App CRs for tenant apps re-project from the control-plane store (bex-db) once bex-api + projector are up - verify tenant apps return; `crs.yaml` doubles as the App fallback if projection disagrees. Note: CNPG regenerates credential Secrets on recreate - tenant connection strings keep working because bex-api reads live Secrets, but restored roles/ownership need `--no-owner` dumps (t001).

## Steps

1. Run the sequence; keep a timestamped operation log in the PR/issue for the post-mortem trail.

## Files

- No new repo files - execution of t002-t005's declarations
- operational: infra/local/ kubeconfigs regenerated

## Acceptance criteria

- [ ] app-cluster.yml green end-to-end on the rebuild run.
- [ ] 3 CP Ready+tainted, platform pool 2 nodes, OpenBao 3/3 Ready after restore+unseal, CNPG clusters healthy with restored data, tenant App CRs re-projected and Running.
- [ ] `kubectl get pods -A -o wide`: zero platform pods outside the platform pool.

## Operation log (UTC, 2026-07-10)

- 21:31 — `c291304` pushed (prep merged). `infra.yml` **green**: `hcloud_firewall.app_nodes` applied. `app-cluster.yml` **red as predicted**: `HetznerCluster` rejected the immutable `hcloudNetwork` change; partial apply had configured KCP→3 + created the `bex-platform` MachineDeployment on the doomed cluster (2 machines reached Pending, never became servers). `deploy.yml` run cancelled manually.
- 21:36 — pre-teardown gate: S3 `rearch-2026-07-10/` re-verified readable (4 dumps + crs.yaml + SHA256SUMS) + fresh etcd snapshot `etcd-20260710-203800.db.gz` (5.1 MB).
- 21:38:00 — `kubectl delete cluster bex` on the mgmt cluster (downtime window opens). `bex-traefik` LB (CCM-owned, id 7086655) deleted via hcloud API (204).
- 21:41 — CAPH controller found wedged (deletionTimestamps set, zero reconcile activity; pre-restart logs spinning on the half-born platform machines' missing OwnerRefs). Pod restarted; ~20 s of self-webhook `no endpoints` races, then deletion proceeded.
- 21:42:51 — Cluster object fully gone; Hetzner shows only `bex-infra`. Old kube-api LB deprovisioned by CAPH.
- 21:43:33 — rebuild run `29125596222` (`app-cluster.yml` via workflow_dispatch) started: fresh CAPH create — network `bex` 10.10.0.0/16, 3×CP tainted, platform+tenant pools, addons with the t003 scheduling.
- 21:45 — verified: network `bex` created **labeled `caph-cluster-bex: owned`**; first CP up with private IP 10.10.0.3 attached at creation (the property the old substrate could never have). KCP scaling 1→3; both platform machines provisioning; tenant pool at min 0 as designed.
- 21:50 — run failed at `cluster-autoscaler (infra cluster)`: helm `--wait` 3m timeout. Cause: the 13h-old bootstrap-autoscaler pod (pre-teardown) crashlooping against the dead old-cluster endpoint; `bex-kubeconfig` was deleted with the old cluster and the new one's secret (regenerated 21:44:01) appeared mid-upgrade — the old pod never remounted it. Fix: bounced the pod (fresh secret mounts, pod Ready in seconds); `gh run rerun --failed` at 21:52 (steps idempotent).
- 21:56 — rerun failed at `install Hetzner CCM`: the unpinned hccm chart renders env values unquoted, so `--set …value="false"` type-coerced to bool → Deployment rejected (EnvVar.value is string). Fixed with `--set-string` (`ecb146e`); push auto-triggered run `29126129756`.
- 22:10 — run failed at `wait for app cluster nodes Ready`: platform nodes churning in a create→join→kill loop. Two structural findings, fixed in `880506c` → run `29127049164`: (1) **MHC Ready timeout 180s** remediated healthy nodes mid-bring-up — fresh nodes stay NotReady >3m while Cilium pulls images and its config init container retries; now 600s on all three pools. (2) **`kubelet-preferred-address-types` still preferred ExternalIP** (old-world leftover) — apiserver→kubelet is dead under the t005 firewall (public :10250 closed), breaking logs/exec/metrics-server; now InternalIP-first. Also noted: CP #1 Ready with CCM+cilium-operator correctly pinned on it; KCP scaling serially (normal); platform nodes ship no sshd (debug path is CP-node SSH + private net).

- 22:14 — **the deep one** (run `29127049164`, nodes still NotReady with MHC patience): cilium's config init container hangs on `https://10.96.0.1:443` from every non-CP node. kubeadm advertised the default-route (public) interface, so the `kubernetes` Service endpoint was CP-public `:6443` and etcd peer URLs public `:2380` — both closed by the t005 firewall. Same cause stalled every CP join (KCP 1/3, MHC churning CP replacements). Diagnosed via CP-node SSH + kubelet API over the private net (public logs/exec path already dead — the address-types fix rides the same push). Fix `53112c2`: placeholder `advertiseAddress` in KCP init+join, rewritten to the machine's 10.10.0.0/16 address by preKubeadmCommands (loud boot failure if the interface never appears). Cluster torn down again 22:18:43 (still empty — nothing to lose) for a pristine bring-up: run `29127365918`.

- 22:20–22:25 — race note: run `29127365918` (auto-fired by the `53112c2` push) had grabbed the DYING cluster's kubeconfig; failed at first app-cluster contact (old LB EOF). Rerun on the clean slate → **green end-to-end**: `kubernetes` endpoint = `10.10.0.3` (private ✓), 1 CP + 2 platform Ready, tenant pool min 0.
- 22:31 — post-Ready regression: cilium agents crashloop on all nodes — `unable to determine direct routing device`. Unpinned cilium chart drifted to 1.19.5 (second unpinned-chart bite today); on the new two-NIC topology auto-detection keys off node InternalIP, which the CCM never set because it can't schedule past cilium's own `agent-not-ready` taint (circular). deploy.yml first attempt cancelled mid-flight (Argo partially installed; idempotent). Fix `d18e791`: `devices`/`direct-routing-device` pinned to `enp7s0` (the CAPH private NIC — where east-west belongs) + version pins for cilium/hccm/hcloud-csi; applied in place by run `29128073501` (no teardown — the substrate itself is fine).

- 22:41–23:05 — deploy.yml round: dashboard image build failed on a cache-miss runner — `corepack` fell back to yarn classic (no `packageManager` pin) against the yarn-4 lockfile; previous builds had ridden cached install layers. Fixed `0624c2e` (pin `yarn@4.17.0`, verified in node:22-alpine). Cilium fix confirmed live: agents Running on all nodes after a DS restart (chart doesn't auto-roll on ConfigMap change), CCM scheduled, node InternalIPs populated, **CP #2 joined Ready — etcd-over-private proven**. Next deploy attempt stalled: `argocd-application-controller-0` Pending — pod predated the platform-scheduling patch and the StatefulSet didn't roll it (it had also triggered a wasteful tenant scale-up); deleted pod-0 → recreated from the patched template → Argo synced 8 apps immediately. Deploy job rerun for the imperative steps.

- 23:10–23:20 — deploy rerun stalled again: `argocd-application-controller-0` predated the platform patch (sts didn't roll a Pending pod); deleted → Argo synced. Then the app-of-apps sync had **failed with retries exhausted**: base NetworkPolicies land in `monitoring`/`secrets`, created only by child apps born in the same sync — invisible before because old-cluster namespaces pre-existed. Fixed `0c1d5e3` (Namespace objects in the base) + created live → **all 23 Applications converged**; `deploy.yml` rerun **green end-to-end** (auth secrets ✓ bootstrap client ✓ OpenFGA model ✓ operator ✓ rolls ✓).
- 23:25 — autoscaler proof: openbao-1's anti-affinity demanded a 3rd platform node → `TriggeredScaleUp bex-platform 2→3` — the elasticity this milestone exists for, firing on the first real demand.
- 23:30 — **external blocker: Hetzner account server limit (5) reached** — CAPH `resource_limit_exceeded` on CP #3, platform #3, and the tenant node; also explains the morning's pre-teardown Pending machines. Blocks: bao 3/3 (openbao-1), App pods + ACME solvers (tenant pool), KCP 3/3, and the target shape post-pivot (needs ≥7–9). **User action: raise the limit in Hetzner Cloud Console.**
- 23:35 — restores done on what fits: kratos/hydra/openfga dumps clean (0 errors); bex restored via drop-schema + fresh pg_restore (first pass raced bex-api's live connections — scale to 0 first), rows verified (1 tenant, 1 member; `apps=0` is faithful — old prod's Apps never went through the store). `crs.yaml` re-applied: 3 Apps Deploying (pods Pending on tenant nodes → limit). `etcd-backup-s3` + `openbao-backup-s3` secrets recreated per runbooks — the silent-backup-failure class closed from day 0. Traefik LB up: `49.12.20.236`.
- Notes for t007/t008: in-cluster autoscaler Application crashloops pre-pivot (no CAPI CRDs — expected to settle after `clusterctl init` + move; verify then). DNS is manually managed in Cloudflare (no API token in .env/CI): cut `dashboard/api/auth/oauth/schemas.bex.co` + `*.onbex.co` (+ the DNS-only `*.db.bex.co`/`*.kv.bex.co` passthroughs) to `49.12.20.236` **after** LE certs issue (solvers need a tenant node → after the limit bump). `bao-init.sh` runs after openbao-1 schedules (keys → local `.env`, then rotate into CI secrets per t008).

- 2026-07-11 01:30–03:20 — **m19.1 rescue executed** (full log in `../m19.1/`): desired state aligned to the 5-server quota (KCP webhook forbids even counts → interim 1 CP), pivot done 01:39Z (`clusterctl move`, in-cluster autoscaler healed instantly, mgmt neutralized), CI rewritten to SSH-to-CP (both workflows green; deploy's first red was a restore artifact — old-prod dump carried `schema_migrations` 6-dirty + missing 0004 table, patched live to 7/clean), `bex-infra` destroyed 01:47Z, tenant node born by the in-cluster autoscaler, apps revived via the deploy-from-local runbook (fresh amd64 builds from clean `git archive` contexts, `ctr` import, CR patches — spec carries no repo; old flow wrote built tags into spec.image), DNS cut in two Cloudflare zones (bex.co then onbex.co — separate zones, the second initially missed), all LE certs issued, content-verified end-to-end (`<title>Beancount.io` / `<title>AgentMarketCap` from the public edge). **Quota work parked in `.pm/FUTURE-MAYBE.md` by user decision** — this task's OpenBao/3-CP residue reopens with it.

## Out of scope

- Pivot (t007) and DNS (t008) - immediately follow.
