---
id: w3/m6/t001
title: Enable Alertmanager + one receiver (webhook/email), secret out-of-band
worker: worker3
status: todo
estimate: 30m
depends_on: []
---

## Objective

Flip `alertmanager: { enabled: false }` to a minimal single-replica Alertmanager in the Prometheus chart values and configure exactly one real notification channel — with the secret delivered out-of-band (SealedSecret, the w1/m7 pattern), never in values. _2026-07-11: no credential to mint — prod already holds a working SendGrid SMTP credential (`BEX_SMTP_ADDR`/`BEX_SMTP_FROM` + PLAIN auth, the w4/m12 invite-email relay, same relay as the Kratos courier), already under `.env`/GH-secrets custody. Point the receiver at it via an `email_config` sealed secret._

## Context

- `deploy/gitops/base/prometheus.yaml` + `deploy/gitops/base/values/`; the chart bundles alertmanager — enabling is a values change, not a new app.
- Secret pattern: `docs/ADR016-sealed-secrets.md` (controller + `kubeseal` workflow shipped in w1/m7; controller already synced via `deploy/gitops/base/sealed-secrets.yaml`).
- Route/grouping: keep the default route simple (one receiver, group by alertname); tuning is not this milestone.
- Scheduling (w1/m19 t003 invariant): subcharts do NOT inherit the `server:` block's platform-pool pinning — without its own nodeSelector/tolerations the alertmanager pod lands on the untainted, autoscaler-elastic tenant pool, exactly the nodes that churn.

## Steps

1. Values: enable alertmanager, wire `alertmanagers` scrape/notify config, `email_config` receiver on the existing SendGrid relay, credential referenced from the sealed secret.
2. Pin the subchart to the platform pool, mirroring the `server:` block: `alertmanager.nodeSelector: { bex.co/pool: platform }` + toleration `{ key: bex.co/platform, operator: Equal, value: "true", effect: NoSchedule }`.
3. Seal + commit the SMTP secret; local overlay may point at a dummy receiver.
4. `kustomize build` both overlays; Argo sync; `amtool`/UI check the receiver loads.

## Files

- `deploy/gitops/base/prometheus.yaml`, `deploy/gitops/base/values/`
- sealed secret manifest under `deploy/gitops/`

## Acceptance criteria

- [ ] Alertmanager runs; a hand-fired test alert (amtool) reaches the channel; no plaintext secret in git.
- [ ] Alertmanager pod scheduled on the platform pool (nodeSelector + toleration set on the subchart, not just `server:`).

## Out of scope

- Rules (t003/t004); multi-channel routing, on-call schedules, silencing policy.
