# Telemetry & error reporting

Pythinker Code can emit anonymous telemetry to help us spot crashes,
regressions, and silent failures. Telemetry is on by default; this page
documents exactly what's collected, where it's sent, and how to turn it off.

## Backends

| Stack | Endpoint | What it carries |
|---|---|---|
| **OpenTelemetry → SigNoz** (logs / metrics / traces) | `https://otel.pythinker.com` | Structured events emitted via `track(...)`, periodic metric snapshots, and (when sampled) trace spans. |
| **Sentry-compatible Bugsink** (errors) | `https://errors.pythinker.com/1` | Unhandled exceptions, asyncio task failures, and explicitly-reported handled exceptions. |
| **Hosted feedback endpoint** | `<platform>/feedback` | Free-form text submitted via the `/feedback` slash command. |

The OTel collector and Bugsink are pythinker-operated. The OTel ingest token
embedded in the binary is public by design (same pattern used by Datadog RUM
and PostHog public keys); rate limiting and PII scrubbing happen server-side
at the collector edge.

## What's collected

### OTel events

Emitted from `pythinker_code.telemetry.track(event_name, **properties)`. Every
event carries:

| Field | Source |
|---|---|
| `event_id` | UUIDv4 generated client-side. |
| `device_id` | Stable per-install hash (no user identity). |
| `session_id` | Per-process random ID. |
| `event` | The event name (e.g. `session_started`, `feedback_submitted`, `error`). |
| `properties.*` | Primitive enum-like attributes the call site passed. |
| `context.*` | Static attributes added by the sink (version, OS, Python, locale). |
| `timestamp` | Wall-clock time on the client. |

Property values **must** be primitives (bool/int/float/str/None). The call
sites are coded to never pass user input, file paths, or code snippets.

### Sentry events

Emitted from `pythinker_code.telemetry.sentry.capture_exception(exc)` (and
implicitly via `sys.excepthook` and the asyncio exception handler). Carries:

- Exception class, message, and stack trace.
- Release tag (`pythinker-code@<version>`).
- Lifecycle phase (`startup` / `runtime` / `shutdown`).
- A handful of static tags (`ui_mode`, `model`).

The `before_send` hook in `telemetry/sentry.py`:

- Strips file-path prefixes above `site-packages/` or `pythinker_code/`,
  collapsing them to `<env>/` so home directories don't leak.
- Removes `server_name` (which is usually the user's hostname).
- Disables `send_default_pii` and request-frame variable capture.

### Reported handled errors

`pythinker_code.telemetry.errors.report_handled_error(exc, *, site, tool=None, **attrs)`
is the canonical helper for any `except Exception:` block that catches an
exception, renders a graceful failure to the user (`ToolError`, red TUI line,
warning), and currently has no monitoring visibility.

It:

1. Emits an OTel `error` event with `{site, exc_class, expected, tool, **attrs}`.
2. Calls `sentry.capture_exception(exc)` **only when the error is not expected**.
   Expected user-environment failures — bad/expired credentials, exhausted
   quotas, rate limits, request timeouts, offline network, abandoned OAuth
   flows, MCP servers lacking an optional capability (see
   `errors.is_expected_error`) — still flow to the OTel `error` stream with
   `expected=True`, but are withheld from Sentry/Bugsink, which is reserved for
   actionable defects.

Both calls are wrapped in `contextlib.suppress(Exception)` so monitoring can
never break the host program.

Common `site` values (extend with care — these are dashboard query keys):

| Site | Where |
|---|---|
| `tool.read` | `tools/file/read.py` |
| `tool.read_media` | `tools/file/read_media.py` |
| `tool.write` | `tools/file/write.py` |
| `tool.replace` | `tools/file/replace.py` |
| `tool.glob` | `tools/file/glob.py` |
| `tool.grep` | `tools/file/grep_local.py` |
| `tool.grep.rg_exec` | `tools/file/grep_local.py` (ripgrep subprocess failure) |
| `tool.shell.exec` | `tools/shell/__init__.py` (foreground command) |
| `tool.shell.background_start` | `tools/shell/__init__.py` (background spawn) |
| `tool.agent.foreground` | `tools/agent/__init__.py` |
| `tool.ask_user` | `tools/ask_user/__init__.py` |
| `tool.plan.enter` | `tools/plan/enter.py` |
| `tool.plan.exit` | `tools/plan/__init__.py` |
| `tool.plan.handoff` | `tools/plan/__init__.py` (`ExitPlanMode` handoff) |
| `auth.keyring.read` | `auth/oauth.py` (`_load_from_keyring`) |
| `auth.oauth.device_authorize` | `auth/oauth.py` (device-code request) |
| `auth.oauth.device_poll` | `auth/oauth.py` (device-code poll loop) |
| `auth.models.fetch` | `auth/oauth.py` (`list_models` post-login) |
| `auth.oauth.refresh` | `auth/oauth.py` (foreground token refresh) |
| `auth.oauth.refresh.background` | `auth/oauth.py` (background refresh loop) |
| `auth.platforms.refresh.pre_sync` | `auth/platforms.py` (pre-sync refresh) |
| `auth.platforms.refresh.after_401` | `auth/platforms.py` (refresh-on-401) |
| `auth.platforms.sync` | `auth/platforms.py` (model sync fallback) |
| `auth.openai.discover_chatgpt_models` | `auth/openai/login.py` |
| `auth.openai.browser_login` | `auth/openai/login.py` |
| `auth.openai.device_start` | `auth/openai/login.py` |
| `auth.openai.device_poll` | `auth/openai/login.py` |
| `soul.btw.execute` | `soul/btw.py` (side question) |
| `soul.btw.run_wire` | `soul/btw.py` (wire-based side question) |
| `soul.toolset.register_external` | `soul/toolset.py` (external tool registration) |
| `soul.toolset.mcp.connect` | `soul/toolset.py` (MCP server connect) |
| `soul.toolset.mcp.call` | `soul/toolset.py` (MCP tool call) |
| `soul.toolset.mcp.call.timeout` | `soul/toolset.py` (MCP tool call — timeout-classified) |
| `soul.toolset.external_tool` | `soul/toolset.py` (`WireExternalTool`) |
| `soul.injection.get` | `soul/pythinkersoul.py` (injection provider get) |
| `soul.injection.on_context_compacted` | `soul/pythinkersoul.py` |
| `soul.injection.on_auto_changed` | `soul/pythinkersoul.py` |
| `soul.context.compact` | `soul/pythinkersoul.py` (compaction inside step) |
| `soul.context.prune` | `soul/pythinkersoul.py` (context prune failure) |
| `soul.step.error` | `soul/pythinkersoul.py` (any step exception) |
| `soul.chat.recover` | `soul/pythinkersoul.py` (provider recovery) |
| `soul.deliberation.advisor` | `soul/deliberation.py` (advisor deliberation failure) |
| `acp.session.prompt` | `acp/session.py` |
| `acp.session.approval` | `acp/session.py` |
| `acp.host.terminal` | `acp/host.py` |
| `hooks.runner` | `hooks/runner.py` |
| `hooks.engine.run` | `hooks/engine.py` (top-level fail-open wrap) |
| `hooks.engine.triggered_cb` | `hooks/engine.py` (HookTriggered callback) |
| `hooks.engine.resolved_cb` | `hooks/engine.py` (HookResolved callback) |
| `hooks.engine.wire` | `hooks/engine.py` (wire hook dispatch) |
| `subagents.run.soul` | `subagents/runner.py` (soul-run wrapper) |
| `subagents.run.background` | `subagents/runner.py` (background failure) |

When you add a new catch site, prefer reusing an existing prefix and add a
specific suffix. Do not put free-form messages in `site` — it must remain a
small, stable enumeration for dashboard slicing.

### `/feedback` slash command

Distinct from automatic telemetry. The user types feedback explicitly; it is
POSTed to the configured feedback endpoint along with `session_id`, version,
OS, and current model. No code or file context is attached.

Endpoint resolution order:

1. `PYTHINKER_FEEDBACK_URL` environment variable.
2. `feedback.endpoint_url` in `config.toml`.
3. The built-in Pythinker platform endpoint.

When using a configured endpoint, set `PYTHINKER_FEEDBACK_API_KEY` or
`feedback.api_key` to send an `Authorization: Bearer ...` header. `feedback.custom_headers`
can add static headers. The built-in Pythinker provider remains supported but
OAuth is no longer required for self-hosted/dev feedback routing.

### `/report-error` slash command

A user-invoked complement to automatic Sentry/OTel capture, designed for the
case where the user *knows* something is wrong but the code path didn't
raise — or where they want to attach a comment to a recently-seen failure.

The runtime keeps a **process-local ring buffer** of the last 10 errors that
were forwarded through `report_handled_error` (`pythinker_code/telemetry/errors.py`).
Each entry stores: `timestamp`, `site`, `exc_class`, a 200-char-truncated
`message`, and the optional `tool` name. The buffer is in-memory only — it
does not persist across processes and does not back the OTel/Sentry pipeline
(those already received the full event when it occurred).

When the user runs `/report-error`:

1. The TUI prints the buffer's contents (site + class + tool).
2. The user types a free-form description and hits enter.
3. The runtime POSTs to `<platform>/feedback` with `type=error`, the comment,
   the same context as `/feedback` (version, OS, model, session_id), and a
   `recent_errors[]` array constructed from the ring buffer.
4. On success, the buffer is cleared.

If submission is unavailable or fails, the slash falls back to opening the
GitHub issue tracker in a browser.

## Turning it off

Telemetry is on by default. Use any of the following to turn off Sentry and
OTel emission:

- `pythinker --no-telemetry` — disables it for that invocation.
- `PYTHINKER_DISABLE_TELEMETRY=1` — set before launching `pythinker`; kills
  both Sentry and OTel emission for the process. The `/feedback` slash command
  is **not** affected (it only fires on explicit user invocation).
- `config.telemetry = false` in your config file (TOML) — same effect,
  persisted across runs.

Explicit feedback settings are independent of automatic telemetry:

- `PYTHINKER_FEEDBACK_URL` — overrides where explicit `/feedback` and
  `/report-error` submissions are sent.
- `PYTHINKER_FEEDBACK_API_KEY` — optional bearer token for that feedback endpoint.

Override individual endpoints when running your own collectors:

| Variable | Default |
|---|---|
| `PYTHINKER_SENTRY_DSN` | `https://...@errors.pythinker.com/1` |
| `PYTHINKER_OTEL_ENDPOINT` | `https://otel.pythinker.com` |
| `PYTHINKER_OTEL_TOKEN` | (embedded) |
| `PYTHINKER_OTEL_TRACE_SAMPLE_RATE` | `1.0` (always-on) |

`PYTHINKER_OTEL_TRACE_SAMPLE_RATE` accepts a float in `[0.0, 1.0]`. Values are
clamped; malformed input falls back to the default. Sampling is `ParentBased`
on top of `TraceIdRatioBased`, so a parent's sample decision (e.g. an
upstream ACP/wire request that already chose to sample) is honored —
otherwise the rate decides at the root.

## Spans

Currently emitted (when traces are sampled):

| Span | Where | Key attributes |
|---|---|---|
| `pythinker.turn` | `soul/pythinkersoul.py` | `session.id`, `agent.role`, `model`, `plan_mode`, `turn.stop_reason`, `turn.step_count` |
| `pythinker.llm` | `soul/pythinkersoul.py` | `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.id`, `llm.tool_calls` |
| `pythinker.tool` | `soul/toolset.py` | `tool.name`, `tool.success`, `tool.error_type` |
| `pythinker.mcp.call` | `soul/toolset.py` | `mcp.server`, `mcp.tool`, `mcp.timeout_ms`, `mcp.is_error` |

## What's *not* collected

- Prompts, completions, or any LLM input/output.
- File contents, paths above `site-packages/` (Sentry path-scrubbing), or
  command arguments.
- Local environment variable values.
- The user's hostname or username.
- Anything from the `/feedback` slash command unless the user explicitly
  typed it.
