---
id: w3/m6/t004
title: Rule pack — bex invariants (backup staleness, OpenBao any-member sealed, bex-api down, 5xx spike, stranded images)
worker: worker3
status: todo
estimate: 30m
depends_on: [w3/m6/t003]
---

## Objective

The rules that encode bex's own promises: `etcd-backup` and `openbao-backup` CronJob last-success age > 26h (the nightly jobs' staleness window — this is the "silent backup rot = data loss" alert), OpenBao sealed (its env-vars API 503s tenants; alert when ANY member reports sealed), bex-api down >5m, a Traefik 5xx-rate spike (>5% of requests for 10m — Traefik metrics are already scraped for the request-metrics feature), and stranded node-local images (below). _2026-07-11: the original sealed-detection options don't fit the live deployment — no blackbox exporter exists (adding one fights t002's minimal-exporter stance), `openbao.values.yaml` carries no telemetry stanza, and seal state is PER-MEMBER on the now-initialized multi-member raft: the `openbao` Service round-robins across sealed+unsealed members (`scripts/bao-endpoints.sh`), so a Service-level `/v1/sys/health` probe would flap or miss a sealed follower. Likewise nothing can HTTP-probe bex-api's health endpoint today — use t002's KSM instead._

## Context

- CronJob staleness: `kube_cronjob_status_last_successful_time` from t002's kube-state-metrics (cronjobs collector over `kube-system` + `secrets`).
- OpenBao sealed-on-restart is a documented failure mode (docs/ADR013-secrets.md); CI unseals on deploy, but a mid-cycle pod restart seals silently. Detection: enable listener telemetry with `unauthenticated_metrics_access = true` in `deploy/gitops/base/values/openbao.values.yaml` + t002's per-POD scrape job; alert when any member reports sealed. Keep the sealed alert's fix path: "run scripts/bao-init.sh unseal path / see docs/ADR013-secrets.md runbook".
- bex-api down: `kube_deployment_status_replicas_available{namespace="bex-system"} == 0` from t002's KSM — no HTTP probe, no new exporter.
- Traefik metrics labels: reuse what `BEX_PROM_URL` request-metrics queries already select on (`lego/backend/internal/metrics/`).
- Stranded node-local images (new-architecture failure mode): app images are node-local ctr imports (no registry — zot is empty), so tenant-node replacement or autoscaler scale-down strands them and App pods sit in ImagePullBackOff/ErrImageNeverPull. That is a PLATFORM defect (the node-local-image design), not tenant paging — it does not violate t003's no-tenant-alerting scope.

## Steps

1. Author the five rules with actionable descriptions (sealed → the bao-init.sh/docs/ADR013-secrets.md runbook line; stranded images → the `/deploy-app-from-local` re-import runbook): backup staleness ×2, any-member-sealed, bex-api replicas==0, 5xx spike, and `kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImageNeverPull"} > 0` for 10m in tenant namespaces.
2. Values: the OpenBao telemetry stanza (`unauthenticated_metrics_access = true`) so t002's per-pod scrape job has an endpoint.
3. `promtool check rules`; verify each series exists on prod (the mock lacks the backup CronJobs and GitOps entirely — note prod-only ones honestly).

## Files

- `deploy/gitops/base/values/` (rule files)
- `deploy/gitops/base/values/openbao.values.yaml` (listener telemetry stanza)

## Acceptance criteria

- [ ] All five rules load and reference live series; the backup-staleness alert fires when the CronJob is suspended past its window (verified in t005); the sealed rule is per-member, not per-Service.

## Out of scope

- Auto-remediation (alerts page humans; CI-driven unseal stays the fix path).
