---
id: w3/m6/t005
title: Acceptance — synthetically break two rules; notifications arrive and resolve
worker: worker3
status: todo
estimate: 25m
depends_on: [w3/m6/t004]
---

## Objective

Prove the loop end-to-end (receiver may be a capture webhook locally): break two monitored invariants — the etcd-backup staleness window and a bex-system Deployment rolled to a bad image — then watch both alerts fire on the channel and resolve after the fix. _2026-07-11: the staleness break cannot run on the mock as originally written — the local overlay `$patch: delete`s BOTH backup Applications (`deploy/gitops/overlays/local/kustomization.yaml`; disposable cluster, no S3 creds), so no `etcd-backup` CronJob and no `kube_cronjob_status_last_successful_time` series exist there. Either apply a stub CronJob named `etcd-backup` in `kube-system` on the mock (same name/namespace, dummy command) before shortening the rule window, or run that one break against prod out-of-band and record it. The bad-image bex-system break stays on the mock._

## Context

- Mock prerequisite: `scripts/mock-cluster.sh` installs neither Argo CD nor the gitops tree — bootstrap argocd + `deploy/gitops/bootstrap/local.yaml` first, or run the whole verification against prod, where the full stack (and both backup CronJobs) already exists.

## Steps

1. Fire: bad-image rollout → not-Ready alert within one evaluation window; fix → resolved notification.
2. Fire: backup-staleness — on the mock via the stub `etcd-backup` CronJob + window-shortened test rule (or timestamp fixture), or against prod out-of-band (suspend the real CronJob, restore it after) → alert; restore → resolved.
3. Record the observed end-to-end latency (eval interval + group_wait) in the milestone notes.

## Files

- `scripts/` (a small `alerts-verify.sh` if it earns its keep; else documented manual steps in the milestone README)

## Acceptance criteria

- [ ] Two distinct alerts observed firing and resolving on the real channel path; latency recorded.

## Out of scope

- Chaos testing every rule; prod game-days.
