# Prod KCP cannot manage the control plane — m7 LB/private-net aftermath (blocks m3/t008)

Found 2026-07-10 while running m3's prod acceptance. The app cluster **serves fine**, but Cluster API's management of its control plane is dead:

- `KubeadmControlPlane` preflight fails permanently: `EtcdMemberHealthy … Failed to connect to etcd: context deadline exceeded` — the CP node's InternalIP is now the Hetzner private-net address `10.0.1.2`, which the **infra cluster cannot route to** (the two clusters share no L3). Any KCP change (e.g. m3's scheduler config, k8s version bumps, remediation) queues forever: `RollingOut=True`, no replacement machine ever created.
- Kubelet serving-cert CSRs are **denied in a loop** (`the IP address "10.0.1.2" is not allowed`) — CAPH doesn't know the private network, so its CSR validator rejects the node's address. Serving cert will eventually expire → metrics/exec/logs against the CP node break.
- `kubectl apply` of the CAPH overlay fails: `HetznerCluster.spec.hcloudNetwork … field is immutable` — the declared network (`10.0.0.0/16`, added in m7 t005) can't be applied to the live cluster (which got attached to a different subnet, `10.0.1.0/24`, out-of-band). **`app-cluster.yml` has been red since 2026-07-09** for this reason; on 2026-07-10 it also stopped before the new autoscaler-install step (run manually instead, see m3/t008).

Repair direction (m7's call, not m3's): either make CAPH own the network (recreate the HetznerCluster/CP with `hcloudNetwork` declared — a controlled CP replacement) or detach/repoint the node addressing so the mgmt cluster can reach etcd again; then the queued m3 KCP rollout (scheduler `MostAllocated`) completes on its own and `app-cluster.yml` goes green.

Note: CAPH-created **workers** are unaffected (public-IP CSRs approved; m3's scale-from-zero verified live 2026-07-10).

---

**Promoted 2026-07-10** → **w1/m19** (Rearchitecture: rebuild the Hetzner substrate right), designed in [docs/rearchitecture.md](../../../docs/rearchitecture.md). Resolution rides m19/t009's acceptance.

---

**Resolution (2026-07-11):** root cause removed by the w1/m19 rebuild + m19.1 pivot — CAPH owns the network (10.10.0.0/16, private-IP advertise), KCP healthy and self-managed in-cluster, CSR approval chain clean, CI green end-to-end on the SSH-to-CP path. Evidence: m19 t009 pre-verified list + `scripts/verify-substrate.sh` (ALL PASS). This note closes the loop m3/t008 was gated on.
