---
title: Observability API
description: Effect layers, Prometheus exposition, and OpenTelemetry export for a Smithers control plane.
icon: "chart-line"
---

Observability ships as a set of [Effect](https://effect.website) layers. One call
to `createSmithersObservabilityLayer` installs the logger, the metrics registry,
the tracing service, and (when enabled) an OTLP exporter. Metrics are always
collected in-process and read back as a Prometheus exposition string; traces and
logs are exported over OTLP only when `enabled` is set.

```ts
import {
  createSmithersObservabilityLayer,
  createSmithersRuntimeLayer,
  createSmithersOtelLayer,
  resolveSmithersObservabilityOptions,
  trackSmithersEvent,
  renderPrometheusMetrics,
  prometheusContentType,
  smithersMetrics,
  SmithersObservability,
} from "smithers-orchestrator";
```

<Note>
Metric collection is global. `renderPrometheusMetrics()` and `trackSmithersEvent`
read and write the process-wide Effect registry, so they work whether or not you
have built a layer. Building a layer adds logging, tracing, and OTLP export on
top.
</Note>

## SmithersObservability

An Effect `Context.Tag` for the resolved observability service. Provided by
`createSmithersObservabilityLayer`; depend on it to read the resolved options or
to open a span without importing `withSmithersSpan` directly.

```ts
import { Effect } from "effect";
import { SmithersObservability } from "smithers-orchestrator";

const program = Effect.gen(function* () {
  const obs = yield* SmithersObservability;
  yield* obs.withSpan("my-step", Effect.log("inside the span"));
});
```

<ResponseField name="SmithersObservabilityService" type="object">
  The shape behind the tag.

  <Expandable title="members">
    <ResponseField name="options" type="ResolvedSmithersObservabilityOptions">
      The fully resolved options the layer was built with.
    </ResponseField>
    <ResponseField name="annotate" type="(attributes) => Effect<void>">
      Attach attributes to the current trace span.
    </ResponseField>
    <ResponseField name="withSpan" type="(name, effect, attributes?) => Effect">
      Run `effect` inside a named child span.
    </ResponseField>
  </Expandable>
</ResponseField>

**Source** [`SmithersObservability.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersObservability.js) · [`SmithersObservabilityService.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersObservabilityService.ts) · **Tests** [`observability.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability.test.js) · **See also** [Server integration](/integrations/server)

## Layers

### createSmithersObservabilityLayer

Builds the full observability layer: logger, `MetricsService`, `TracingService`,
the `SmithersObservability` service, and `BunContext`. The OTLP exporter is
folded in only when the resolved options have `enabled: true`.

```ts
function createSmithersObservabilityLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<MetricsService | TracingService | SmithersObservability | BunContext>;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Partial options, resolved through `resolveSmithersObservabilityOptions`. See
  [`resolveSmithersObservabilityOptions`](#resolvesmithersobservabilityoptions)
  for fields and defaults.
</ParamField>

```ts
import { Effect, Layer } from "effect";
import { createSmithersObservabilityLayer } from "smithers-orchestrator";

const layer = createSmithersObservabilityLayer({
  serviceName: "my-control-plane",
  logFormat: "json",
});

await Effect.runPromise(myProgram.pipe(Effect.provide(layer)));
```

### createSmithersRuntimeLayer

Alias of `createSmithersObservabilityLayer`, exported under the name used by the
runtime. Same signature, same return type. Prefer it in runtime/server wiring so
the intent reads clearly.

```ts
import { createSmithersRuntimeLayer } from "smithers-orchestrator";

const layer = createSmithersRuntimeLayer({ logLevel: "debug" });
```

### createSmithersOtelLayer

The OTLP-export slice on its own. Returns `Layer.empty` unless `enabled` resolves
to true, so it is safe to merge unconditionally. Use it when you already have a
metrics/logging layer and only want to add OpenTelemetry traces.

```ts
function createSmithersOtelLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<never>;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Same options object. Only `enabled`, `endpoint`, and `serviceName` affect this
  layer.
</ParamField>

```ts
import { createSmithersOtelLayer } from "smithers-orchestrator";

// No-op unless SMITHERS_OTEL_ENABLED=1 (or enabled: true).
const otel = createSmithersOtelLayer({
  endpoint: "http://localhost:4318",
  serviceName: "smithers",
});
```

**Source** [`createSmithersObservabilityLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersObservabilityLayer.js) · [`createSmithersOtelLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersOtelLayer.js) · [`createSmithersRuntimeLayer.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/createSmithersRuntimeLayer.js) · **Tests** [`observability.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability.test.js) · **See also** [Agent trace OTel verification](/guides/agent-trace-otel-verification)

## resolveSmithersObservabilityOptions

Merge partial options with environment variables and defaults. The layer
factories call this for you; call it directly to inspect what a deployment
resolves to.

```ts
function resolveSmithersObservabilityOptions(
  options?: SmithersObservabilityOptions,
): ResolvedSmithersObservabilityOptions;
```

<ParamField path="options" type="SmithersObservabilityOptions">
  Every field is optional. Unset fields fall back to env vars, then defaults.

  <Expandable title="SmithersObservabilityOptions">
    <ParamField path="enabled" type="boolean">
      Enable OTLP export. Defaults from `SMITHERS_OTEL_ENABLED` (`1`/`true`),
      otherwise `false`.
    </ParamField>
    <ParamField path="endpoint" type="string">
      OTLP HTTP endpoint. Defaults from `OTEL_EXPORTER_OTLP_ENDPOINT`, otherwise
      `http://localhost:4318`.
    </ParamField>
    <ParamField path="serviceName" type="string">
      Resource service name. Defaults from `OTEL_SERVICE_NAME`, otherwise
      `smithers`.
    </ParamField>
    <ParamField path="logFormat" type='"json" | "pretty" | "string" | "logfmt"'>
      Logger format. Defaults from `SMITHERS_LOG_FORMAT`, otherwise `logfmt`.
    </ParamField>
    <ParamField path="logLevel" type="LogLevel | string">
      Minimum log level. Accepts an Effect `LogLevel` or a string
      (`none`/`trace`/`debug`/`info`/`warning`/`error`/`fatal`/`all`). Defaults
      from `SMITHERS_LOG_LEVEL`, otherwise `Info`.
    </ParamField>
    <ParamField path="installLogger" type="boolean" default="true">
      Replace the default Effect logger and set the minimum level. Set `false` to
      keep your own logger.
    </ParamField>
  </Expandable>
</ParamField>

<ResponseField name="ResolvedSmithersObservabilityOptions" type="object">
  Every field present and concrete.

  <Expandable title="ResolvedSmithersObservabilityOptions">
    <ResponseField name="enabled" type="boolean" />
    <ResponseField name="endpoint" type="string" />
    <ResponseField name="serviceName" type="string" />
    <ResponseField name="logFormat" type='"json" | "pretty" | "string" | "logfmt"' />
    <ResponseField name="logLevel" type="LogLevel" />
    <ResponseField name="installLogger" type="boolean" />
  </Expandable>
</ResponseField>

```ts
import { resolveSmithersObservabilityOptions } from "smithers-orchestrator";

const resolved = resolveSmithersObservabilityOptions({ serviceName: "ci" });
// { enabled: false, endpoint: "http://localhost:4318", serviceName: "ci", ... }
```

**Source** [`resolveSmithersObservabilityOptions.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/resolveSmithersObservabilityOptions.js) · [`ResolvedSmithersObservabilityOptions.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/ResolvedSmithersObservabilityOptions.ts) · **Tests** [`observability-options.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability-options.test.js)

## trackSmithersEvent

Map a [`SmithersEvent`](/reference/event-types) onto the metric registry. Returns
an `Effect<void>` that increments the relevant counters, updates gauges, and
records durations for that event type. Every event also bumps the
`smithers_events_emitted_total` counter. The runtime calls this on each emitted
event; call it yourself only when feeding events from outside the engine.

```ts
function trackSmithersEvent(event: SmithersEvent): Effect.Effect<void>;
```

<ParamField path="event" type="SmithersEvent" required>
  A discriminated union keyed by `type`. See [Event types](/reference/event-types).
</ParamField>

```ts
import { Effect } from "effect";
import { trackSmithersEvent } from "smithers-orchestrator";

await Effect.runPromise(
  trackSmithersEvent({
    type: "NodeFinished",
    runId: "RUN_ID",
    nodeId: "NODE_ID",
    iteration: 0,
    attempt: 1,
    timestampMs: Date.now(),
  }),
);
```

**Source** [`trackEvent.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/metrics/trackEvent.js) · [`SmithersEvent.ts`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/SmithersEvent.ts) · **Tests** [`effect-metrics-track.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/effect-metrics-track.test.js) · **See also** [Event types](/reference/event-types)

## Prometheus

### renderPrometheusMetrics

Render the entire metric registry as a Prometheus exposition string, with
`# HELP`/`# TYPE` headers and every catalog metric present at zero even before it
fires. Refreshes the process gauges (uptime, RSS, heap) on each call. This is
what the server's `GET /metrics` route returns.

```ts
function renderPrometheusMetrics(): string;
```

### prometheusContentType

The exact `Content-Type` for a Prometheus response:
`text/plain; version=0.0.4; charset=utf-8`. Pair it with the rendered body.

```ts
import { renderPrometheusMetrics, prometheusContentType } from "smithers-orchestrator";

// Inside any HTTP handler:
function handleMetrics() {
  return new Response(renderPrometheusMetrics(), {
    headers: { "Content-Type": prometheusContentType },
  });
}
```

<Note>
The built-in server already exposes this at `GET /metrics`. See
[Server integration](/integrations/server). Use these directly only when wiring
metrics into your own HTTP layer.
</Note>

**Source** [`renderPrometheusMetrics.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/renderPrometheusMetrics.js) · [`prometheusContentType.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/prometheusContentType.js) · **Tests** [`observability-prometheus.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/observability-prometheus.test.js) · **See also** [Server integration](/integrations/server)

## Metrics catalog

`smithersMetrics` is the registry of every metric, an object keyed by the
camelCase names below whose values are the underlying Effect `Metric`s. Each
metric renders under its Prometheus name; histograms also emit `_bucket`,
`_sum`, and `_count` series. The table groups the core metrics by area; the
registry holds more (gateway, devtools, agent, alert, supervisor families).

```ts
import { smithersMetrics } from "smithers-orchestrator";

const everyMetric = Object.values(smithersMetrics); // all Effect Metrics
```

### Runs and nodes

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_runs_total` | counter | Runs started |
| `smithers_runs_finished_total` | counter | Runs finished |
| `smithers_runs_failed_total` | counter | Runs failed |
| `smithers_runs_cancelled_total` | counter | Runs cancelled |
| `smithers_runs_resumed_total` | counter | Runs resumed |
| `smithers_runs_continued_total` | counter | Runs continued-as-new |
| `smithers_nodes_started` | counter | Nodes started |
| `smithers_nodes_finished` | counter | Nodes finished |
| `smithers_nodes_failed` | counter | Nodes failed |
| `smithers_node_retries_total` | counter | Node retries |
| `smithers_errors_total` | counter | Errors |
| `smithers_events_emitted_total` | counter | Events emitted |
| `smithers_runs_active` | gauge | Active runs |
| `smithers_nodes_active` | gauge | Active nodes |
| `smithers_scheduler_queue_depth` | gauge | Scheduler queue depth |
| `smithers_scheduler_concurrency_utilization` | gauge | Scheduler concurrency utilization |
| `smithers_run_duration_ms` | histogram | Run duration |
| `smithers_node_duration_ms` | histogram | Node duration |
| `smithers_attempt_duration_ms` | histogram | Per-attempt duration |
| `smithers_scheduler_wait_duration_ms` | histogram | Scheduler wait time |
| `smithers_snapshots_captured` | counter | Snapshots captured |
| `smithers_forks_created` | counter | Run forks created |
| `smithers_replays_started` | counter | Replays started |
| `smithers_snapshot_duration_ms` | histogram | Snapshot duration |

### Approvals and timers

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_approvals_requested` | counter | Approvals requested |
| `smithers_approvals_granted` | counter | Approvals granted |
| `smithers_approvals_denied` | counter | Approvals denied |
| `smithers_approval_pending` | gauge | Pending approvals |
| `smithers_external_wait_async_pending` | gauge | Pending external waits |
| `smithers_approval_wait_duration_ms` | histogram | Approval wait duration |
| `smithers_timers_created` | counter | Timers created |
| `smithers_timers_fired` | counter | Timers fired |
| `smithers_timers_cancelled` | counter | Timers cancelled |
| `smithers_timers_pending` | gauge | Pending timers |
| `smithers_timers_delay_ms` | histogram | Timer fire delay |

### Cache and tokens

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_cache_hits` | counter | Cache hits |
| `smithers_cache_misses` | counter | Cache misses |
| `smithers_tokens_input_total` | counter | Input tokens |
| `smithers_tokens_output_total` | counter | Output tokens |
| `smithers_tokens_cache_read_total` | counter | Cache read tokens |
| `smithers_tokens_cache_write_total` | counter | Cache write tokens |
| `smithers_tokens_reasoning_total` | counter | Reasoning tokens |
| `smithers_tokens_input_per_call` | histogram | Input tokens per call |
| `smithers_tokens_output_per_call` | histogram | Output tokens per call |
| `smithers_tokens_context_window_per_call` | histogram | Context window per call |

### Database

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_db_retries` | counter | DB retries |
| `smithers_db_transaction_retries` | counter | DB transaction retries |
| `smithers_db_transaction_rollbacks` | counter | DB transaction rollbacks |
| `smithers_db_query_ms` | histogram | DB query duration |
| `smithers_db_transaction_ms` | histogram | DB transaction duration |

### HTTP and hot reload

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_http_requests` | counter | HTTP requests |
| `smithers_http_request_duration_ms` | histogram | HTTP request duration |
| `smithers_hot_reloads` | counter | Hot reloads |
| `smithers_hot_reload_failures` | counter | Hot reload failures |
| `smithers_hot_reload_duration_ms` | histogram | Hot reload duration |

### Tools

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_tool_calls_total` | counter | Tool calls |
| `smithers_tool_calls_errors_total` | counter | Tool call errors |
| `smithers_tool_output_truncated_total` | counter | Tool outputs truncated |
| `smithers_tool_duration_ms` | histogram | Tool duration |

### Sandbox

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_sandbox_created_total` | counter | Sandboxes created |
| `smithers_sandbox_completed_total` | counter | Sandboxes completed |
| `smithers_sandbox_active` | gauge | Active sandboxes |
| `smithers_sandbox_duration_ms` | histogram | Sandbox duration |
| `smithers_sandbox_bundle_size_bytes` | histogram | Sandbox bundle size |
| `smithers_sandbox_transport_duration_ms` | histogram | Sandbox transport duration |
| `smithers_sandbox_patch_count` | histogram | Sandbox patch count |

### Memory

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_memory_fact_reads` | counter | Memory fact reads |
| `smithers_memory_fact_writes` | counter | Memory fact writes |
| `smithers_memory_recall_queries` | counter | Memory recall queries |
| `smithers_memory_message_saves` | counter | Memory messages saved |
| `smithers_memory_recall_duration_ms` | histogram | Memory recall duration |

### OpenAPI tools

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_openapi_tool_calls` | counter | OpenAPI tool calls |
| `smithers_openapi_tool_call_errors` | counter | OpenAPI tool call errors |
| `smithers_openapi_tool_duration_ms` | histogram | OpenAPI tool duration |

### VCS and process

| Metric | Kind | Meaning |
| --- | --- | --- |
| `smithers_vcs_duration_ms` | histogram | VCS operation duration |
| `smithers_process_uptime_seconds` | gauge | Process uptime |
| `smithers_process_memory_rss_bytes` | gauge | Process RSS memory |
| `smithers_process_heap_used_bytes` | gauge | Process heap used |

**Source** [`smithersMetrics.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/smithersMetrics.js) · [`metrics`](https://github.com/smithersai/smithers/blob/main/apps/observability/src/metrics) · **Tests** [`effect-metrics-definitions.test.js`](https://github.com/smithersai/smithers/blob/main/apps/observability/tests/effect-metrics-definitions.test.js) · **See also** [Server integration](/integrations/server), [Event types](/reference/event-types)
