# OpenChrome v1.12.3

Release date: 2026-05-17

## Summary

v1.12.3 is the live-benchmark execution and publication-gate hardening
release. It takes the v1.12.2 competitive benchmark infrastructure and
adds the missing runtime seams needed to run real-world benchmark episodes
against live or recorded-real evidence without overstating mock/scaffold
results.

The release focuses on four outcomes:

1. live LLM/tool-use loops can be injected and measured;
2. native competitor task loops can be exercised for playwright-mcp and
   browser-use;
3. live Chrome/CDP benchmark execution can use managed Chrome endpoints
   consistently; and
4. generated benchmark reports now fail closed before making headline
   claims without live/recorded-real postcondition evidence.

```
v1.12.2  →  v1.12.3   (this release)

  10 PRs integrated (#1326, #1328–#1336)
  1  browser startup cleanup regression fixed
  3  live/native benchmark execution seams added
  1  live throughput executor added
  1  live token-extraction wiring path added
  1  episode fault-injection integration layer added
  1  recorded-real corpus validator added
  1  final headline report gate added
  0  npm publish performed by this release-prep step
```

This is still a patch release for the OpenChrome runtime API. No MCP tool is
removed or renamed, and existing CLI/server consumers do not need to change
configuration. The new behavior is concentrated in benchmark execution,
validation, and reporting paths.

---

## Upgrade

```bash
npm install -g openchrome-mcp@1.12.3
openchrome check
openchrome doctor
```

Fresh install:

```bash
npm install -g openchrome-mcp@latest
openchrome check
openchrome info
```

If you only use OpenChrome as an MCP server or CLI, this should be a drop-in
patch upgrade from v1.12.2. If you run competitive benchmarks, review the new
live/recorded-real gates below before publishing results.

---

## Breaking changes

**None.** v1.12.3 does not introduce source-level breaking changes for MCP
clients, CLI users, or existing benchmark consumers.

Operational notes:

- Headline benchmark report generation is stricter. Diagnostic, scaffold,
  dry-run, and mock-only rows are still useful, but they cannot pass the new
  headline gate unless they include eligible live or recorded-real evidence.
- Live benchmark lanes remain opt-in and may require browser runtime setup,
  competitor packages, and API credentials.
- `npm publish` was intentionally not run as part of this preparation; publish
  from a separate trusted terminal/session.

---

## Runtime fix

### Startup blank-tab cleanup (#1326)

Managed headed Chrome launches can create a transient startup tab. v1.12.3
narrows cleanup so OpenChrome closes only startup-owned blank/new-tab pages and
preserves user-created tabs that appear concurrently in attach mode.

Why it matters:

- avoids closing unrelated user tabs in attach-mode sessions;
- keeps the startup cleanup behavior useful for managed launches; and
- adds regression coverage for concurrently created `chrome://newtab` pages.

---

## Live LLM and native competitor execution

### Injected OpenAI Responses tool-use loop (#1328)

The benchmark suite now has an injectable OpenAI Responses tool-use loop. It
normalizes function calls, dispatches them through an MCP adapter, preserves
reasoning/function-call context across tool turns, and keeps budget accounting
inside the benchmark harness.

Key details:

- client and adapter are injected, so CI can test the loop without paid API
  calls;
- live execution is gated by explicit environment checks;
- reasoning model output context is retained with function-call outputs; and
- tests cover dispatch, final text handling, and fail-closed live env checks.

### Live real-world episode runner seam (#1329)

Real-world task completion can now distinguish deterministic/scaffold runs from
live or recorded-real measurement. The runner preserves canonical task/library
identity, rejects empty or unknown task selections, and requires explicit
pinning/evidence before headline eligibility.

This keeps benchmark claims honest: dry or scaffold rows are still useful for
harness regression, but they do not become publishable competitive evidence by
accident.

### playwright-mcp native task loop (#1330)

A native task loop for `playwright-mcp` now performs tool discovery,
navigation, snapshot collection, and postcondition checking through the native
MCP tool surface.

The implementation intentionally stays small: it does not introduce a new agent
framework, and it accepts both array and iterable/set-style tool discovery so it
can work with existing stdio MCP clients.

### browser-use native task bridge loop (#1331)

The browser-use path now has a native task bridge loop with explicit lifecycle,
timeout, and failure classification.

Improvements include:

- shared transport startup handling;
- unique bridge request IDs;
- timeout results reported as timeouts instead of generic failures;
- default native timeout aligned with the bridge transport timeout; and
- Python bridge syntax coverage through `py_compile`.

---

## Live benchmark measurement paths

### Live token extractor wiring (#1332)

Token/field-retention measurement can now use live MCP payloads. The extractor
opens a tab, calls `read_page` in the selected mode, parses structured JSON or
OpenChrome serialized DOM snapshot entries, and closes the created tab in a
`finally` path.

This avoids two common benchmark mistakes:

- treating raw substring matches as structured extraction evidence; and
- assuming OpenChrome DOM snapshots are always nested HTML instead of
  serialized snapshot lines like `<tag .../>text`.

### Live throughput executor (#1333)

The throughput benchmark now has a live executor that can launch managed Chrome,
thread the exact CDP endpoint into competitor adapters, and configure
OpenChrome's live adapter with the same host/port.

Notable safeguards:

- Playwright, Puppeteer, and OpenChrome all receive the selected CDP endpoint;
- OpenChrome receives an explicit derived port and `CHROME_HOST`/`CHROME_PORT`
  environment;
- managed Chrome is closed on normal completion and benchmark failure; and
- CI-safe tests verify endpoint/port propagation without requiring a live
  browser.

### Episode fault-injection hooks (#1334)

Reliability testing can now inject controlled fault plans into real-world
episode execution. The hook layer validates plan shape, rejects null or invalid
plans, and records injected fault events as part of episode evidence.

This moves reliability work closer to the primary benchmark axis: fault recovery
can be measured inside task episodes rather than only in isolated synthetic
micro-cells.

---

## Recorded-real evidence and publication gates

### Recorded corpus validator (#1335)

Recorded-real corpora now have a validator for publication readiness. It checks
manifest structure, competitor version pins, LLM settings, runtime metadata,
artifact references, postcondition evidence, and secret-like payloads.

The validator is intentionally deterministic and local. It does not require an
external scanner, but it catches common publication blockers:

- missing or malformed manifest fields;
- unsupported competitor version provenance;
- unpinned recorded libraries;
- missing final postcondition evidence;
- malformed artifact references;
- invalid failure categories; and
- likely secrets such as `sk-*`, Anthropic/OpenAI project keys, AWS access keys,
  and credential-shaped bearer tokens.

False-positive risk was reduced for benign prose such as “secret menu” or
“bearer authentication”.

### Headline report eligibility gate (#1336)

The final benchmark report path now includes a headline gate. Report generators
can require that every headline row is backed by eligible live, live-LLM, or
recorded-real measurement and final postcondition evidence.

This gate protects release claims by failing closed when:

- no headline-eligible rows exist;
- diagnostic rows are mixed into headline output;
- aggregate rows include runs without per-run postcondition evidence; or
- the generated real-world section still marks the headline gate as blocked.

A new `bench:realworld:headline` script records the intended release path for
headline real-world reports using a recorded-real corpus and
`--require-headline`.

---

## Benchmark direction and merge-readiness review

Before this release was prepared, the open benchmark PR stack was reviewed for
alignment with the repository direction and over-engineering risk. The accepted
shape is:

- benchmark logic stays under `tests/benchmark/` and `benchmark/`;
- default CI remains deterministic and credential-free;
- live or paid-provider paths are explicit and opt-in;
- publication claims require structured evidence; and
- runtime MCP compatibility is preserved.

The review found the merged work consistent with the v1.12 benchmark roadmap:
real-world task completion is the primary evidence axis, while token,
throughput, reliability, auth, and DX lanes remain supporting or diagnostic
unless tied to headline-eligible live/recorded-real task completion evidence.

---

## Notable PRs

- `#1326` — protect attach-mode user tabs during startup blank-tab cleanup.
- `#1328` — add injected OpenAI Responses tool-use loop.
- `#1329` — add live/recorded-real real-world episode runner seam.
- `#1330` — add playwright-mcp native task loop.
- `#1331` — add browser-use native task bridge loop.
- `#1332` — add live MCP token extractor wiring.
- `#1333` — add live throughput executor and managed Chrome endpoint wiring.
- `#1334` — add real-world episode fault-injection hooks.
- `#1335` — add recorded corpus validator for benchmark publication.
- `#1336` — add headline report eligibility gate and release report wiring.

---

## Verification

Release preparation verification performed locally after merging the PR stack:

```bash
gh pr list --state open
npm version 1.12.3 --no-git-tag-version
npm run build
npm test -- --runInBand
# Jest reported 557/557 non-skipped suites passing; the process then remained alive due existing open-handle warnings.
npm test -- tests/src/cdp-connect-coalescing.test.ts tests/utils/budget.test.ts --runInBand --forceExit
```

Targeted checks used during the PR review/fix loop included:

```bash
npx jest tests/src/session-manager-ttl.test.ts --runInBand
npx jest tests/benchmark/llm-provider/openai-loop.test.ts --runInBand
npx jest tests/benchmark/realworld-task-completion/live-runner.test.ts --runInBand
npx jest tests/benchmark/native/playwright-mcp-native.test.ts --runInBand
npx jest tests/benchmark/native/browser-use-native.test.ts --runInBand
npx jest tests/benchmark/extractors/live-token-wiring.test.ts --runInBand
npx jest tests/benchmark/run-throughput-live.test.ts tests/benchmark/run-throughput.test.ts tests/benchmark/benchmark-runner.test.ts --runInBand
npx jest tests/benchmark/recordings/validator.test.ts --runInBand
npx jest tests/benchmark/realworld-task-completion/recordings.test.ts --runInBand
node benchmark/headline-gate.test.mjs
node benchmark/generate-benchmark-report.mjs
python3 -m py_compile tests/benchmark/bridges/browser_use_bridge.py
```

Recommended post-install smoke checks:

```bash
openchrome check
openchrome doctor
npm run benchmark:ci
```

Run live benchmark commands only in an environment with the required browser
runtime, competitor packages, and API credentials configured. Do not publish
headline claims from mock-only, scaffold-only, or diagnostic-only rows.
