---
id: w3/m6/t002
title: Enable kube-state-metrics + explicit scrape jobs (KSM, kubelet /metrics, cert-manager :9402, OpenBao telemetry)
worker: worker3
status: todo
estimate: 40m
depends_on: []
---

## Objective

The rule packs need object-state series (`kube_pod_container_status_waiting_reason`, `kube_node_status_condition`, `kube_job_status_succeeded`, `kube_cronjob_status_last_successful_time`, PVC usage) that the server-only Prometheus doesn't collect. Enable kube-state-metrics in the chart — and only it; keep the no-pushgateway/no-node-exporter stance. _2026-07-11: "PVC usage comes from kubelet, already scraped" was wrong — `prometheus.yaml` wholesale-replaces the chart's default scrape config with exactly two jobs (traefik + cadvisor via the apiserver proxy), so neither kubelet `/metrics` (`kubelet_volume_stats_*`) nor a newly-enabled KSM gets scraped; no annotation-based discovery survives the override. Explicit scrape jobs are part of this task (estimate bumped 25m → 40m)._

## Context

- `deploy/gitops/base/prometheus.yaml` comment documents the deliberate exporter-free posture — amend the comment to reflect the new, still-minimal scope and why.
- Retention is 3d; alerting evaluates on recent data, so no retention change needed.
- KSM collector scope: the `cronjobs` collector must cover `kube-system` (etcd-backup) and `secrets` (openbao-backup) namespaces or t004's staleness alert has no series.
- Scheduling (w1/m19 t003 invariant): the kube-state-metrics subchart does not inherit the `server:` block's platform-pool pinning — pin it explicitly or it lands on the autoscaler-elastic tenant pool.
- Verification target: the mock has no Argo CD/GitOps out of the box (`scripts/mock-cluster.sh` installs neither) — either bootstrap Argo CD + `deploy/gitops/bootstrap/local.yaml` first, or verify against prod where the whole stack (and both backup CronJobs) already exists.

## Steps

1. Values: enable kube-state-metrics (cronjobs collector over `kube-system` + `secrets` included); pin the subchart to the platform pool (`kube-state-metrics.nodeSelector: { bex.co/pool: platform }` + the `bex.co/platform=true:NoSchedule` toleration, mirroring `server:`).
2. Add explicit scrape jobs to `serverFiles:` (nothing is discovered automatically — the config replaces the chart default): (a) the kube-state-metrics service endpoint; (b) kubelet `/metrics` via the same apiserver-proxy pattern as the existing cadvisor job (for `kubelet_volume_stats_*`); (c) cert-manager pods on `:9402` (t003's cert-expiry rule — KSM does not export Certificate CRs); (d) OpenBao per-pod telemetry (t004's sealed alert; the values-side telemetry stanza is t004's to add).
3. Verify the needed series exist via the Prometheus API (prod, or a mock bootstrapped per Context).
4. Update the manifest's posture comment.

## Files

- `deploy/gitops/base/prometheus.yaml`, `deploy/gitops/base/values/`

## Acceptance criteria

- [ ] The series required by t003/t004 return data (including `kube_cronjob_status_last_successful_time` for both backup jobs, `kubelet_volume_stats_*`, and `certmanager_certificate_expiration_timestamp_seconds`); no other exporters were enabled.
- [ ] KSM pod scheduled on the platform pool.

## Out of scope

- node-exporter (add only if a rule proves impossible without it — record the decision).
