// example-alerts.md as a first-class generated artifact: the WHOLE document
// is rendered from this template (rules table interpolated from the alert
// catalog), so render/--check treat it exactly like the JSON artifacts — no
// in-place mutation of a pre-existing file, no split source of truth.
import { alertDatasource } from './alerts.mjs';

// Markdown renderers for the alert specs (documentation concerns only — the
// Grafana payload derivation lives in alerts.mjs):
// - the summary table carries only plain-text fields; free-text cells are
//   pipe-escaped so a future title containing '|' cannot break the table
// - each query is emitted as its own FENCED code block (fence length adapts
//   to the longest backtick run), so no valid LogQL/PromQL is constrained by
//   GFM table escaping
const fenced = (t) => {
  const runs = t.match(/`+/g) ?? [];
  const fence = '`'.repeat(Math.max(3, ...runs.map(r => r.length + 1)));
  return `${fence}\n${t}\n${fence}`;
};
const mdCell = (t) => String(t).replaceAll('|', '\\|');
const rulesTable = (specs) => [
  '| # | Alert | Channel | Datasource | Fires when | for | noData |',
  '|---|---|---|---|---|---|---|',
  ...specs.map((s, i) =>
    `| ${i + 1} | ${mdCell(s.title)} | #node-${s.signal} | ${alertDatasource(s).name} | \`${s.condition.op} ${s.condition.value}\` | ${s.forDur} | ${s.noData} |`),
  '',
  '**Queries** (range queries, reduced with `last`, evaluated against the condition above):',
  '',
  ...specs.flatMap((s, i) => [`${i + 1}. ${s.title}`, '', fenced(s.expr), '']),
].join('\n').replace(/\n+$/, '');

const routesTable = (routes) => [
  '| Slack channel | Contact point | Route matchers |',
  '|---|---|---|',
  ...routes.map(r => `| \`${r.channel}\` | \`${r.contactPoint}\` | ${r.matchers.map(([k, , v]) => `\`${k}=${v}\``).join(', ')} |`),
].join('\n');
const groupingSentence = (routes) =>
  routes.map(r => `${r.signal} group by \`${r.groupBy.join(', ')}\``).join('; ');

export const buildDocs = ({ specs, routes }) => ({
  'example-alerts.md': `<!-- GENERATED FILE — do not edit. Source: tools/observability/lib/docs.mjs
     (rules table derives from lib/alerts.mjs ALERT_SPECS). Regenerate with:
     node tools/observability/generate-observability.mjs -->

# Grafana alerting — live setup on the observability server

> **Status (2026-07-02):** everything below is **provisioned and healthy** on the
> production observability Grafana (the dedicated server, Loki 3.x native-OTLP
> stack — see RUNBOOK.md). This file documents it so it can be recreated from
> scratch. Webhook URLs are **secrets** — they live only in Grafana contact
> points, never in this repo.

## Routing model — one Slack channel per signal

${routesTable(routes)}

Routes are appended as children of the root notification policy
(\`group_wait 30s\`, \`group_interval 5m\`, \`repeat_interval 4h\`). Grouping is
signal-aware: ${groupingSentence(routes)} (the metrics set covers both the
Loki- and VictoriaMetrics-sourced rules). Every rule carries labels
\`team=dkg\` + \`signal=<x>\`; add those two labels to any new rule and it routes
itself.

The webhooks belong to the Slack app **"DKG Grafana Alerts"** (workspace
OriginTrail) — manage/rotate them at *api.slack.com/apps → DKG Grafana Alerts →
Incoming Webhooks*.

## The ${specs.length} rules (folder "DKG V10 Node Observability", group \`dkg-node-telemetry\`)

> **Machine-importable copy:** \`alert-rules.provisioning.json\` in this directory
> holds the exact provisioning-API payloads (rules, contact-point templates with
> placeholder webhooks, policy routes). Both it and the dashboard JSONs are
> **generated by \`generate-observability.mjs\`** — edit that script and
> regenerate, never the JSON by hand. The committed payload keeps
> \`<VM_DATASOURCE_UID>\` as a placeholder (instance-specific); render an
> importable copy with
> \`node generate-observability.mjs /tmp/render --vm-uid <your-uid>\`.
> This section is the human explanation.

All queries are **range queries** — on Loki 3.x an *instant* metric query over a
range ≥ a few hours is split internally and dies with \`maximum of series (500)
reached\`; range+reduce avoids that class of failure entirely.

${rulesTable(specs)}

### Design notes (the "why" behind the table)

- **Node silent** keeps **node identity**: the \`unless\` form returns one series
  per silent node, so the alert names the node and is immune to churn — a new
  node joining cannot mask another node dying, which a count-vs-count
  comparison would miss. Empty result (noData) = every known node reporting =
  healthy, hence noData OK.
- **Fleet blackout** complements it: the per-node \`unless\` form cannot fire
  when *both* sides are empty, so total-outage detection (pipeline down, Loki
  down, zero nodes) lives in its own rule with **noData = Alerting**.
- **RPC credit burn** reads the \`rpc_usage\` log lines (delta counts, so
  \`sum_over_time\` is exact) — needs nodes on a post-#1409 build.
- **Failover exhausted** filters \`reason="exhausted"\`: the metric contract also
  documents \`reason="recovered"\`, and without the filter a recovery event would
  page as an outage the moment that emission is added.
- **"Armed" rules** (publish failures, failover, errored spans) evaluate
  healthy with **noData OK** — zero noise now, they fire automatically once
  nodes export the signal. If the eventual spanmetrics label names differ,
  adjust the errored-spans matchers in \`generate-observability.mjs\`.

## Re-provisioning from scratch (API recipe)

With an admin session, \`X-Disable-Provenance: true\` header keeps everything
UI-editable:

\`\`\`bash
# 1. Contact points (one per channel; $HOOK_* from your password manager)
POST /api/v1/provisioning/contact-points
  { "name": "DKG node logs (Slack)", "type": "slack",
    "settings": { "url": "$HOOK_NODE_LOGS" } }          # ×3, one per signal

# 2. Notification policy — ALWAYS GET the tree first and APPEND child routes;
#    a PUT replaces the WHOLE tree and would clobber other teams' routes.
GET  /api/v1/provisioning/policies
PUT  /api/v1/provisioning/policies    # tree + appended routes (matchers above)

# 3. Rules
POST /api/v1/provisioning/alert-rules # one per rule; folderUID of the
                                      # dashboards folder, ruleGroup dkg-node-telemetry
\`\`\`

Legacy note: an earlier 4-rule single-channel version of this setup exists on
polaris (Loki 2.5 query shapes — \`level=\` label instead of \`severity_text\`
metadata). The rules above are the current reference.
`,
});
