<!-- GENERATED FILE — do not edit. Source: tools/observability/lib/docs.mjs
     (rules table derives from lib/alerts.mjs ALERT_SPECS). Regenerate with:
     node tools/observability/generate-observability.mjs -->

# Grafana alerting — live setup on the observability server

> **Status (2026-07-02):** everything below is **provisioned and healthy** on the
> production observability Grafana (the dedicated server, Loki 3.x native-OTLP
> stack — see RUNBOOK.md). This file documents it so it can be recreated from
> scratch. Webhook URLs are **secrets** — they live only in Grafana contact
> points, never in this repo.

## Routing model — one Slack channel per signal

| Slack channel | Contact point | Route matchers |
|---|---|---|
| `#node-logs` | `DKG node logs (Slack)` | `team=dkg`, `signal=logs` |
| `#node-metrics` | `DKG node metrics (Slack)` | `team=dkg`, `signal=metrics` |
| `#node-traces` | `DKG node traces (Slack)` | `team=dkg`, `signal=traces` |

Routes are appended as children of the root notification policy
(`group_wait 30s`, `group_interval 5m`, `repeat_interval 4h`). Grouping is
signal-aware: logs group by `alertname, service_instance_id, deployment_environment`; metrics group by `alertname, service_instance_id, instance, deployment_environment`; traces group by `alertname` (the metrics set covers both the
Loki- and VictoriaMetrics-sourced rules). Every rule carries labels
`team=dkg` + `signal=<x>`; add those two labels to any new rule and it routes
itself.

The webhooks belong to the Slack app **"DKG Grafana Alerts"** (workspace
OriginTrail) — manage/rotate them at *api.slack.com/apps → DKG Grafana Alerts →
Incoming Webhooks*.

## The 10 rules (folder "DKG V10 Node Observability", group `dkg-node-telemetry`)

> **Machine-importable copy:** `alert-rules.provisioning.json` in this directory
> holds the exact provisioning-API payloads (rules, contact-point templates with
> placeholder webhooks, policy routes). Both it and the dashboard JSONs are
> **generated by `generate-observability.mjs`** — edit that script and
> regenerate, never the JSON by hand. The committed payload keeps
> `<VM_DATASOURCE_UID>` as a placeholder (instance-specific); render an
> importable copy with
> `node generate-observability.mjs /tmp/render --vm-uid <your-uid>`.
> This section is the human explanation.

All queries are **range queries** — on Loki 3.x an *instant* metric query over a
range ≥ a few hours is split internally and dies with `maximum of series (500)
reached`; range+reduce avoids that class of failure entirely.

| # | Alert | Channel | Datasource | Fires when | for | noData |
|---|---|---|---|---|---|---|
| 1 | Node silent — seen in last 3h, quiet 15m (per node) | #node-logs | Loki | `> 0` | 5m | OK |
| 2 | Fleet blackout — NO node logs reaching Loki | #node-logs | Loki | `< 1` | 5m | Alerting |
| 3 | Error spike on a node (>10 ERROR / 10m) | #node-logs | Loki | `> 10` | 5m | OK |
| 4 | Warn spike on a node (>150 WARN / 10m) | #node-logs | Loki | `> 150` | 10m | OK |
| 5 | RPC credit burn spike (armed — needs nodes on a post-#1409 build) | #node-metrics | Loki | `> 6000` | 5m | OK |
| 6 | Log pipeline export failing (collector cannot ship to Loki) | #node-metrics | VictoriaMetrics | `> 0` | 10m | OK |
| 7 | Collector exporter queue near capacity (>80%) | #node-metrics | VictoriaMetrics | `> 0.8` | 10m | OK |
| 8 | Publish failures on a node (armed — needs node metrics enabled) | #node-metrics | VictoriaMetrics | `> 0.02` | 5m | OK |
| 9 | Chain RPC failover exhausted on a node (armed — needs node metrics enabled) | #node-metrics | VictoriaMetrics | `> 0` | 5m | OK |
| 10 | Errored spans rate (armed — needs traces + spanmetrics enabled) | #node-traces | VictoriaMetrics | `> 0.05` | 5m | OK |

**Queries** (range queries, reduced with `last`, evaluated against the condition above):

1. Node silent — seen in last 3h, quiet 15m (per node)

```
count by (service_instance_id, deployment_environment) (count_over_time({service_name="dkg-node"}[3h] offset 15m)) unless count by (service_instance_id, deployment_environment) (count_over_time({service_name="dkg-node"}[15m]))
```

2. Fleet blackout — NO node logs reaching Loki

```
count(sum by (service_instance_id) (count_over_time({service_name="dkg-node"}[15m])))
```

3. Error spike on a node (>10 ERROR / 10m)

```
sum by (service_instance_id, deployment_environment) (count_over_time({service_name="dkg-node"} | severity_text=`ERROR` [10m]))
```

4. Warn spike on a node (>150 WARN / 10m)

```
sum by (service_instance_id, deployment_environment) (count_over_time({service_name="dkg-node"} | severity_text=`WARN` [10m]))
```

5. RPC credit burn spike (armed — needs nodes on a post-#1409 build)

```
sum by (service_instance_id, deployment_environment) (sum_over_time({service_name="dkg-node"} |= `rpc_usage` | logfmt | method != `` | unwrap count [1h]))
```

6. Log pipeline export failing (collector cannot ship to Loki)

```
sum(rate(otelcol_exporter_send_failed_log_records[10m]))
```

7. Collector exporter queue near capacity (>80%)

```
max(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity)
```

8. Publish failures on a node (armed — needs node metrics enabled)

```
sum by (instance) (rate(dkg_publish_total{outcome=~"failed|error"}[15m]))
```

9. Chain RPC failover exhausted on a node (armed — needs node metrics enabled)

```
sum by (instance) (rate(dkg_chain_rpc_failover_total{reason="exhausted"}[15m]))
```

10. Errored spans rate (armed — needs traces + spanmetrics enabled)

```
sum(rate(traces_spanmetrics_calls_total{status_code="STATUS_CODE_ERROR"}[15m]))
```

### Design notes (the "why" behind the table)

- **Node silent** keeps **node identity**: the `unless` form returns one series
  per silent node, so the alert names the node and is immune to churn — a new
  node joining cannot mask another node dying, which a count-vs-count
  comparison would miss. Empty result (noData) = every known node reporting =
  healthy, hence noData OK.
- **Fleet blackout** complements it: the per-node `unless` form cannot fire
  when *both* sides are empty, so total-outage detection (pipeline down, Loki
  down, zero nodes) lives in its own rule with **noData = Alerting**.
- **RPC credit burn** reads the `rpc_usage` log lines (delta counts, so
  `sum_over_time` is exact) — needs nodes on a post-#1409 build.
- **Failover exhausted** filters `reason="exhausted"`: the metric contract also
  documents `reason="recovered"`, and without the filter a recovery event would
  page as an outage the moment that emission is added.
- **"Armed" rules** (publish failures, failover, errored spans) evaluate
  healthy with **noData OK** — zero noise now, they fire automatically once
  nodes export the signal. If the eventual spanmetrics label names differ,
  adjust the errored-spans matchers in `generate-observability.mjs`.

## Re-provisioning from scratch (API recipe)

With an admin session, `X-Disable-Provenance: true` header keeps everything
UI-editable:

```bash
# 1. Contact points (one per channel; $HOOK_* from your password manager)
POST /api/v1/provisioning/contact-points
  { "name": "DKG node logs (Slack)", "type": "slack",
    "settings": { "url": "$HOOK_NODE_LOGS" } }          # ×3, one per signal

# 2. Notification policy — ALWAYS GET the tree first and APPEND child routes;
#    a PUT replaces the WHOLE tree and would clobber other teams' routes.
GET  /api/v1/provisioning/policies
PUT  /api/v1/provisioning/policies    # tree + appended routes (matchers above)

# 3. Rules
POST /api/v1/provisioning/alert-rules # one per rule; folderUID of the
                                      # dashboards folder, ruleGroup dkg-node-telemetry
```

Legacy note: an earlier 4-rule single-channel version of this setup exists on
polaris (Loki 2.5 query shapes — `level=` label instead of `severity_text`
metadata). The rules above are the current reference.
