---
id: w3/m6/t003
title: Rule pack — platform invariants (CrashLoop, node NotReady, PVC near-full, cert expiry)
worker: worker3
status: todo
estimate: 30m
depends_on: [w3/m6/t001, w3/m6/t002]
---

## Objective

The generic "is the platform healthy" pack, kept deliberately small and high-signal: container CrashLoopBackOff / Deployment not-Ready persisting >10m in platform namespaces (`bex-system`, `argocd`, `traefik`, auth/infra namespaces), node NotReady >5m, PVC >85%, cert-manager Certificate not Ready / expiring <14d. _2026-07-11: the original "etcd/Loki/OpenBao/Prometheus volumes are all single-host disks" rationale described the dead single-host cluster — prod is now 4 nodes across 3 pools with PVs on hcloud-csi network volumes. The PVC and NotReady alerts stay and matter MORE: the single control-plane node is the highest blast radius until the quota lift reverts the TEMP (m19.1) knobs to 3 CP nodes — make node NotReady severity=page for the control-plane node specifically._

## Context

- Rules as PrometheusRule-equivalent config in the chart's `serverFiles`/rule values (server-only chart — no operator CRDs; follow whatever rule-file mechanism the pinned chart exposes).
- Every alert carries a one-line `description` with the kubectl command to start debugging — alerts nobody can act on are noise.
- Cert-expiry data source: kube-state-metrics does not export cert-manager Certificate CRs — `certmanager_certificate_expiration_timestamp_seconds` / `_ready_status` come from cert-manager's own `:9402` endpoint, scraped by t002's cert-manager job. Without that job this rule has no series.

## Steps

1. Author the rules with `for:` durations and severity labels (page vs warn; control-plane-node NotReady = page while the CP pool is one node).
2. `promtool check rules` locally; sync; confirm they load in the Prometheus UI.
3. While touching `prometheus.yaml`, fix its stale storage comment ("local-path on both the CAPD mock and prod" — prod's default StorageClass is `hcloud-volumes` now; local-path is mock-only).

## Files

- `deploy/gitops/base/values/` (rule files)

## Acceptance criteria

- [ ] Rules load; a deliberately-broken Deployment in bex-system fires the not-Ready alert and resolves on fix.

## Out of scope

- bex-specific rules (t004); tenant-app alerting (never platform-paged — tenants get their own story later).
