# PtcRunner MCP — Phase 6 benchmark

> Generated by `mcp_server/bench/mcp_bench.exs` per spec § 15 Phase 6.

## Environment

- **Runs per cell:** 20
- **LLM mode:** stubbed (canned responses, 150 ms simulated latency)
- **Larger workload N:** 1000 records
- **Elixir:** 1.19.3
- **OTP:** 28
- **Schedulers:** 10


## Methodology

For each workload, three modes are exercised:

1. **`llm_only`** — the LLM is asked the question with no tool. Either a
   real OpenRouter call (when `--real-llm` is set and an API key is
   present) or a canned-response stub that returns a plausible-but-not-
   always-correct answer (smaller workload: correct; larger workload:
   off by a fixed ratio to model that the LLM cannot accurately sum
   N unseen numbers in its head). The stub injects a fixed 150 ms
   sleep so wall-clock numbers are not absurdly low.
2. **`in_process_ptc`** — the same fixed PTC-Lisp program is run via
   `PtcRunner.Lisp.run/2` in this BEAM node. No LLM round-trip.
3. **`mcp`** — the same program is sent through the production
   `PtcRunnerMcp.Stdio` GenServer as NDJSON-framed JSON-RPC 2.0,
   exactly as an OS-level MCP client would. The bench drives the
   GenServer in-BEAM via a `StringIO` device because BEAM-on-BEAM
   `Port.open` has long-standing stdin-pipe issues on macOS / Linux
   that prevent reliable byte-flow from the parent BEAM into a child
   release; the workaround is the same pattern the test suite uses
   (`mcp_server/test/support/jsonrpc_harness.ex`). End-to-end
   OS-process boundary cost is reported separately as a one-shot
   startup measurement (see *OS-process startup* below).

The point is *not* a horse-race: the program text is identical between
mode 2 and mode 3. The numbers measure how much overhead the JSON-RPC /
stdio code path adds over the deterministic-compute primitive.


## Prompts and programs

### count_r_in_raspberry

- **Prompt (mode 1):** `How many letter 'r' is in the word 'raspberry'? Reply with a single integer only.`
- **Program (modes 2, 3):** `(count (filter #(= \r %) "raspberry"))`
- **Expected:** `3`

### sum_amount_over_1000

- **Prompt (mode 1):** `I have 1000 records, each with integer field 'amount'. What is the total sum of 'amount' across all records? Reply with a single integer only.`
- **Program (modes 2, 3):** `(reduce + (map #(get % "amount") data/records))`
- **Expected:** `499500`


## Results

| Workload | Mode | Pass | Median | Mean | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| count_r_in_raspberry | llm_only | 0/20 | 151 ms | 151 ms | 150 ms | 152 ms |
| count_r_in_raspberry | in_process_ptc | 20/20 | 125 µs | 2.6 ms | 118 µs | 49 ms |
| count_r_in_raspberry | mcp | 20/20 | 197 µs | 1.3 ms | 187 µs | 23 ms |
| sum_amount_over_1000 | llm_only | 0/20 | 150 ms | 151 ms | 150 ms | 154 ms |
| sum_amount_over_1000 | in_process_ptc | 20/20 | 1.2 ms | 1.7 ms | 1.2 ms | 11 ms |
| sum_amount_over_1000 | mcp | 20/20 | 6.2 ms | 6.3 ms | 5.7 ms | 7.3 ms |

## Observations

### count_r_in_raspberry

- LLM-only pass rate: **0/20** (median 151 ms).
- In-process PTC pass rate: **20/20** (median 125 µs).
- MCP cross-process pass rate: **20/20** (median 197 µs).
- MCP overhead vs in-process PTC: median **72 µs** added per call.

### sum_amount_over_1000

- LLM-only pass rate: **0/20** (median 150 ms).
- In-process PTC pass rate: **20/20** (median 1.2 ms).
- MCP cross-process pass rate: **20/20** (median 6.2 ms).
- MCP overhead vs in-process PTC: median **5.0 ms** added per call.


## OS-process startup

A separate one-shot probe pipes a single `initialize` frame to the
released binary at:

    /Users/andreasronge/projects/ptc_runner/.claude/worktrees/agent-a9e9b1bddf86bae48/mcp_server/_build/prod/rel/ptc_runner_mcp/bin/ptc_runner_mcp

via `printf '...' | <release> start`, and times the round-trip
until the handshake reply is parsed.

- **Wall-clock for one OS-process startup + handshake:** **1613 ms**

This is the *fixed* cost a real MCP client pays once when it opens
the server connection (Claude Desktop, Cursor, Cline, …). Subsequent
`tools/call` requests reuse the same connection, so per-call cost
is what the `mcp` row in the results table reports.


## Caveats

- Stub mode is deterministic by construction; real-LLM mode introduces
  variance that this report does not quantify (no confidence intervals).
  Re-run with `--real-llm --runs=20` for a directional comparison; use
  `mix ablation` (in `demo/`) for proper statistical analysis.
- The `mcp` mode runs in-BEAM through `StringIO` and therefore does
  **not** include OS-pipe / kernel-scheduler overhead. That overhead
  is dominated by handshake-time process startup, captured by the
  *OS-process startup* probe above. Per-call OS-pipe transport
  overhead on a hot connection is on the order of microseconds and
  not separately measured here.
- The `in_process_ptc` cell shares the parent BEAM node's scheduler
  and process pool; the `mcp` cell pays the cost of NDJSON encoding,
  JSON parsing, and per-call worker / monitor / concurrency-gate
  bookkeeping on every call.
- The benchmark runs in `:dev`, not `:prod`. Hot-path timings will be
  slightly faster under `:prod` due to compiler optimizations.

