---
id: w1/m19/t003
title: GitOps platform scheduling: tolerations + nodeSelector onto bex-platform for every platform chart
worker: worker1
status: done
estimate: 60m
depends_on: []
---

## Objective

With the control plane tainted and the tenant pool default-schedulable, every platform component must explicitly target the bex-platform pool: toleration for `bex.co/platform=true:NoSchedule` + nodeSelector `bex.co/pool: platform`. Otherwise platform pods land among tenant workloads (or nowhere).

## Context

- Components (deploy/gitops/base/ + values/): traefik, kratos, hydra, openfga, openbao, cnpg-operator + the CNPG Cluster manifests (bex-postgres, auth-dbs - CNPG has its own affinity stanza), prometheus, loki, log-shipper (DaemonSet - needs toleration only, it must run on ALL nodes including tenant pool for App logs), zot, metrics-server, cert-manager, sealed-secrets, opensandbox-controller, bex operator+api (lego/operator/config/), dashboard, Argo CD itself (its install manifests in deploy.yml).
- DaemonSets (log-shipper, cilium, CCM/CSI) must tolerate BOTH pools' taints - they run everywhere.
- The etcd-backup/openbao-backup CronJobs also need the platform toleration.

## Steps

1. Sweep every Application/values file adding tolerations+nodeSelector (chart-appropriate keys differ: `tolerations`/`nodeSelector` vs CNPG `affinity.tolerations`/`affinity.nodeSelector`).
2. `bash scripts/gitops-validate.sh` green.
3. Grep-audit: every base/*.yaml either targets bex-platform, is a DaemonSet tolerating all taints, or has a written exemption comment.

## Files

- deploy/gitops/base/*.yaml
- deploy/gitops/base/values/*.yaml
- deploy/gitops/charts/bex-postgres/cluster.yaml
- lego/operator/config/api/deployment.yaml + manager deployment (tolerations/nodeSelector)
- .github/workflows/deploy.yml (Argo install), .github/workflows/app-cluster.yml (CCM/CSI/Cilium addons tolerations)

## Acceptance criteria

- [x] gitops-validate.sh passes; audit list in the PR description shows every platform component's scheduling decision. _2026-07-10: PASS (full tree renders); audit below._
- [ ] After t006, `kubectl get pods -A -o wide` shows zero platform pods on control-plane or tenant nodes. _Post-t006 check — t009 asserts it._

## Audit — every component's scheduling decision (2026-07-10)

**Platform pool** (`nodeSelector bex.co/pool=platform` + toleration `bex.co/platform=true:NoSchedule`):

- values files: traefik, kratos, hydra, openfga, openbao, opensandbox-controller
- base manifests: cert-manager (controller+webhook+cainjector), cnpg-operator, prometheus, loki, metrics-server, sealed-secrets, zot
- CNPG Clusters (`affinity.tolerations`/`affinity.nodeSelector`): bex-postgres, kratos-db, hydra-db, openfga-db
- CronJob: openbao-backup
- bex workloads: operator manager, bex-api, activator (lego/operator/config), dashboard
- Argo CD itself: post-install `kubectl patch` loop in deploy.yml (upstream install carries no scheduling; patch keeps the stock manifest)
- cluster-autoscaler: `helm.valuesObject` on the Argo Application only — NOT the shared values file (bootstrap-phase install targets the mgmt cluster, which has no bex.co labels)
- hcloud-csi controller: `--set` in app-cluster.yml (not bootstrap-critical; PVs first needed once platform charts run)

**DaemonSets — run on ALL nodes** (tolerate everything):

- log-shipper (platform toleration added, deliberately no nodeSelector — tenant App logs)
- cilium, cilium-envoy, hcloud-csi-node (chart defaults, verified by render)

**Control-plane exemptions** (bootstrap-critical; tolerate the CP taint by upstream design, pinned to CP so they never ride a pool node):

- hccm (initializes nodes — cannot wait for a worker pool)
- cilium-operator (no CNI anywhere until it runs)
- etcd-backup CronJob (hostNetwork to etcd on 127.0.0.1:2379 — must run on CP)

**No pods (exempt):** cert-manager-issuers, tenant-namespace, tenant-quotas, network-policies, auth-dbs.yaml/bex-postgres.yaml/etc. (Application pointers — scheduling lives in the pointed-at chart/values). `cluster-api.yaml` is a decision record (providers stay clusterctl-owned); upstream CAPI provider manifests tolerate/prefer the CP — confirm placement at t007.

## Out of scope

- Tenant-pod scheduling policy (tenant pods already default to the untainted pool).
