---
title: "Monitoring"
description: "Set up the OpenSearch + OpenSearch Dashboards + Prometheus monitoring stack for agent observability"
icon: "chart-mixed"
---

Clawker includes an optional monitoring stack that collects telemetry from Claude Code agents running in containers. It provides dashboards for logs and metrics — giving you visibility into what every agent is doing, how much it costs, what tools it's calling, and whether it's making progress.

## What You Get

The stack ingests logs into OpenSearch and exposes them through **OpenSearch Dashboards** for ad-hoc querying and dashboarding. **Prometheus** backs metrics with its own UI. You can see per-agent and per-project activity for:

- **Cost and token usage** — API call costs, input/output tokens, rate limiting
- **Coding activity** — file edits, tool invocations, command executions
- **Tool usage** — which tools the agent calls, how often, and how long they take
- **Session detail** — full event timeline for individual agent sessions
- **Egress traffic** — Envoy access logs and CoreDNS query logs from the firewall stack (when enabled)

## Architecture

The monitoring stack runs as four long-lived Docker Compose services on the `clawker-net` network, plus one short-lived bootstrap container that runs once per `monitor up` to preconfigure the cluster:

```
Claude Code (in container)
    |
    | OTLP (HTTP/gRPC) — logs, metrics, traces
    v
OTEL Collector ──┬──> OpenSearch (logs + traces) ──> OpenSearch Dashboards
                 └──> Prometheus (metrics + spanmetrics) ──> Prometheus UI
                                                       \\
                                                        \\── (also read by OpenSearch Dashboards
                                                              via a preconfigured datasource)
```

Claude Code emits logs, metrics, AND traces. Span export gates on the `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` flag (baked into the image alongside `CLAUDE_CODE_ENABLE_TELEMETRY=1`). The OTEL Collector's `spanmetrics` connector also produces RED (rate/error/duration) metrics from incoming spans and feeds them into the same Prometheus pipeline.

### Services

| Service | Image | Default Port | Purpose |
|---------|-------|:---:|---------|
| **OTEL Collector** | `otel/opentelemetry-collector-contrib` | 4318 (HTTP), 4317 (gRPC) | Receives telemetry from agents and routes it to backends |
| **OpenSearch** | `opensearchproject/opensearch` | 9200 | Stores telemetry across the `claude-code` (Claude Code OTLP), `clawker-cli` (host CLI), `clawker-cp` (clawker-cp mTLS push), `clawker-envoy` (firewall HTTP/TLS/TCP access logs), `clawker-coredns` (firewall DNS query logs) log indices, plus the `traces`/`clawker` (SS4O) trace index |
| **OpenSearch Dashboards** | `opensearchproject/opensearch-dashboards` | 5601 | UI for querying logs, dashboards, and (via the preconfigured `clawker_prometheus` direct-query datasource) Prometheus metrics |
| **Prometheus** | `prom/prometheus` | 9090 | Time-series metrics storage and UI |
| **clawker-opensearch-bootstrap** | `curlimages/curl` | — | One-shot init container. Runs after OpenSearch + Prometheus are reachable, before the collector starts; applies templates, ISM, datasource, workspace, saved objects, then exits |

All containers are pre-configured with labels (`dev.clawker.purpose=monitoring`) and attached to the `clawker-net` network. The OpenSearch security plugin is disabled by default for local development — Dashboards is reachable at `http://localhost:5601` with no login required. Set `OPENSEARCH_JAVA_OPTS` via the `opensearch_heap_mb` setting if you need more heap.

### Service Hostnames Are Constants

Individual service hostname constants (`MonitoringServiceOtelCollector`, `MonitoringServicePrometheus`, `MonitoringServiceOpenSearchNode`, `MonitoringServiceOpenSearchDashboards`) are defined in `internal/consts/monitoring.go`. The `MonitoringServiceHostnames` slice — consumed by CoreDNS `internalHosts` forward zones — intentionally contains only `otel-collector` and `prometheus`: agents push telemetry through the collector and never query OpenSearch directly, so `opensearch-node` and `opensearch-dashboards` are excluded from the forwarded set (containers on `clawker-net` that do need those reach them via Docker's embedded resolver).

### How Agents Connect

OTEL endpoints are baked into every Clawker image at build time. At container creation, Clawker checks whether the monitoring stack is running and conditionally enables telemetry:

- `OTEL_EXPORTER_OTLP_ENDPOINT` — single base URL pointing at the collector's OTLP/HTTP listener (baked into image). The Claude Code SDK appends `/v1/metrics`, `/v1/logs`, `/v1/traces` per signal
- `OTEL_METRICS_EXPORTER` / `OTEL_LOGS_EXPORTER` / `OTEL_TRACES_EXPORTER` — all set to `otlp`
- `CLAUDE_CODE_ENABLE_TELEMETRY` — set to `1` when the `otel-collector` container is detected as running on `clawker-net`, set to `0` otherwise
- `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA` — set to `1` (gates Claude Code's span export; both this and `CLAUDE_CODE_ENABLE_TELEMETRY` must be `1` for traces to flow)
- `OTEL_RESOURCE_ATTRIBUTES` — tags with `project=` and `agent=` for filtering

**Key behavior:** Telemetry is **only active when the monitoring stack is running** at the time the container is created. If the stack isn't up, Clawker explicitly disables telemetry so agents don't waste time attempting exports to an unreachable collector. You can force-enable telemetry via `agent.env` in your `.clawker.yaml` even when the stack is down, but Claude Code won't retry failed collector connections.

**Start the monitoring stack before starting your agents** to ensure telemetry is captured.

## Egress Traffic Visibility

When the firewall is enabled, Envoy and CoreDNS access logs flow into OpenSearch alongside agent telemetry.

- **Envoy access logs** — domain, protocol (`tls`/`tls_mitm`/`http`/`tcp`/`deny`), response codes, response flags. `response_flags` containing `UF` (upstream failure) signals blocked/denied traffic.
- **CoreDNS query logs** — queried domain, qtype (A/AAAA), rcode (NOERROR / NXDOMAIN), resolution duration. `rcode=NXDOMAIN` means the domain was not in the allow list.

## Setup

### 1. Initialize Configuration

```bash
clawker monitor init
```

This scaffolds configuration files in your data directory (`~/.local/share/clawker/monitor/`):

- `compose.yaml` — Docker Compose definition for all four services
- `otel-config.yaml` — OpenTelemetry Collector pipeline configuration
- `prometheus.yaml` — Prometheus scrape targets

Use `--force` to regenerate files if they already exist.

### 2. Start the Stack

```bash
clawker monitor up
```

The stack runs in the background. Service URLs are printed on startup:

```
OpenSearch Dashboards: http://localhost:5601
OpenSearch API:        http://localhost:9200
Prometheus:            http://localhost:9090
```

### 3. Run Agents

Start agents as usual — telemetry flows automatically:

```bash
clawker run -it --agent dev @
```

### 4. Explore in OpenSearch Dashboards

Open OpenSearch Dashboards at `http://localhost:5601`. From the splash / welcome screen, under the **Analytics** panel on the far right, click **Clawker** to enter the auto-created workspace.

Inside the workspace, the left navbar has an **Explore** section — click **Logs** or **Metrics** to browse. The preconfigured index patterns (`claude-code`, `clawker-cli`, `clawker-cp`, `clawker-envoy`, `clawker-coredns`) and the `clawker_prometheus` direct-query datasource are already wired, so logs and Prometheus metrics are both reachable from inside OSD. Raw Prometheus is also still available at `http://localhost:9090` if you prefer it.

**Dashboards.** Robust pre-made dashboards are on the roadmap. For now, build your own from the index patterns + the Prometheus datasource. An example **Claude Code** dashboard with KPI visualizations (sessions, cost, input/output/cache tokens) ships under the workspace's **Dashboards** view for inspiration or direct use.

<Warning>
**Prometheus label naming: use `kind` instead of `type`**

Claude Code's metrics docs reference a `type` label on a few counters (`claude_code_token_usage_tokens_total`, `claude_code_active_time_seconds_total`, `claude_code_lines_of_code_count_total`). The monitoring stack stores these as `kind` instead — same values (`input`/`output`/`cacheRead`/`cacheCreation`, `cli`/`user`, `added`/`removed`), only the key is renamed.

Why: a known bug in the current OpenSearch SQL plugin's direct-query Prometheus connector causes the OSD Explore "Metrics" UI to error out (`missing type id property 'type'`) on any Prom series that carries a label literally named `type`. The OTel collector renames the label at ingest so the UI works. The native Prometheus UI at `http://localhost:9090` is unaffected either way — `type` would work there if it existed.

When writing your own PromQL or saved searches against these metrics, query with `kind=` rather than `type=`.
</Warning>

## Bootstrap

The stack is preconfigured every `monitor up`. A one-shot `clawker-opensearch-bootstrap` compose service runs after OpenSearch reports healthy, applies:

- **Component templates** with shared mappings (`@timestamp` as date, `service.name` / `ingest_source` / `project` / `agent` as keyword)
- **Index templates** with per-source field mappings for each log index plus the SS4O `traces` mapping (so dynamic mapping never locks the wrong type at first ingest). The `claude-code` template carries the full Claude Code log + span field set.
- **ISM retention policy** (7-day rollover-to-delete, auto-attached to all indices via `ism_template.index_patterns`)
- **Empty index pre-creation** for the log indices so OSD Discover and dashboards don't error on initial load before the first record arrives
- **`clawker_prometheus` direct-query datasource** registered via the OpenSearch SQL plugin so OSD can read Prometheus metrics without a federated proxy
- **OSD `Clawker` workspace** with `features: ["use-case-all"]` so the explore-flavor `metrics` / `logs` / `traces` nav groups all mount
- **Saved objects** imported INTO the Clawker workspace — index patterns for every log index plus an example `Claude Code` dashboard with KPI visualizations (sessions, cost, input/output/cache tokens, filter-controls bar) for inspiration or direct use. Robust pre-made dashboards are planned; users build their own off the index patterns + Prometheus datasource until then.

`otel-collector` gates on bootstrap completing successfully — it never starts until the cluster is preconfigured. Prometheus starts in parallel; bootstrap depends on Prometheus being up so the `clawker_prometheus` datasource registration can validate the configured URI. Bootstrap failure surfaces in `docker logs clawker-opensearch-bootstrap` and leaves the stack half-up by design (so wrong-mapped indices can't be silently created). The throwaway-stack model means picking up template/policy edits requires `monitor down --volumes && monitor up`; templates only apply at index creation.

## Telemetry Controls

Fine-tune what telemetry is collected via `settings.yaml`:

```yaml
monitoring:
  telemetry:
    log_tool_details: true       # Include tool invocation details
    log_user_prompts: true       # Include user prompts in logs
    include_account_uuid: true   # Include account identifier
    include_session_id: true     # Include session identifier
```

## Port Configuration

Override default ports in `settings.yaml` if they conflict with other services:

```yaml
monitoring:
  otel_collector_port: 4318
  otel_grpc_port: 4317
  opensearch_port: 9200
  opensearch_dashboards_port: 5601
  opensearch_heap_mb: 512        # JVM -Xms/-Xmx for the OpenSearch node
  prometheus_port: 9090
  prometheus_metrics_port: 8889
```

After changing ports, regenerate config files and restart:

```bash
clawker monitor init --force
clawker monitor down
clawker monitor up
```

## Checking Status

```bash
clawker monitor status
```

Shows container status (running/stopped) and service URLs for running services.

## Teardown

```bash
# Stop the stack (preserves data)
clawker monitor down

# Stop and remove all data volumes
clawker monitor down --volumes
```

Without `--volumes`, monitoring data persists across restarts (named volume `clawker-opensearch` for indices, `clawker-prometheus` for TSDB). The `clawker-net` network is preserved for other Clawker services (firewall, agents).
