---
id: w3/m4/t001
title: "GitOps: scrape kubelet/cAdvisor in prometheus.yaml (container CPU/memory, pod labels, RBAC)"
worker: worker3
status: done
estimate: 40m
depends_on: []
---

## Objective

Extend the in-cluster Prometheus to scrape kubelet's cAdvisor endpoint so `container_cpu_usage_seconds_total` and `container_memory_working_set_bytes` (with pod/namespace labels) are queryable — the raw series the bex-api resource-metrics source will read.

## Context

- `deploy/gitops/base/prometheus.yaml` currently has a single scrape job (`traefik`, line ~49). Argo CD reconciles this file into the app cluster.
- cAdvisor is served by each kubelet at `/metrics/cadvisor`; the standard pattern is `kubernetes_sd_configs: role: node`, scheme https, kubelet CA/token from the ServiceAccount, `insecure_skip_verify` as needed for the CAPD cluster.
- Prometheus's ServiceAccount needs RBAC for node discovery and the kubelet metrics path: `nodes`, `nodes/metrics`, `nodes/proxy` (get/list/watch as applicable).
- App pods carry the `app.bex.co/app=<name>` label — keep (or relabel in) the pod label so series can be matched per App; keep `container!=""` to drop pod-sandbox series.

## Steps

1. Add a `kubernetes-cadvisor` scrape job to the Prometheus config in `deploy/gitops/base/prometheus.yaml` (node SD, https, bearer token, metric/label relabeling for pod + namespace + container).
2. Extend the Prometheus RBAC in the same file with the node discovery/metrics rules.
3. Apply (or let Argo sync) and confirm `container_memory_working_set_bytes{pod=~"<app>.*"}` returns series in the Prometheus API.

## Files

- `deploy/gitops/base/prometheus.yaml`

## Acceptance criteria

- [ ] `curl '$PROM/api/v1/query?query=container_cpu_usage_seconds_total{container!=""}'` returns non-empty series with `pod` and `namespace` labels after sync.
- [ ] Existing `traefik` job still up (`up{job="traefik"} == 1`).

## Out of scope

- kube-state-metrics (limits come from the pod spec, as today).
- Any bex-api code changes (t002).
