# v0.3.0 Runtime-Integrated Benchmark Evidence Architecture

Status: design proposal for `v0.3.0-alpha`

This document defines the next benchmark evidence architecture for KORA. It does not implement a new benchmark runner and does not expand the current public claim boundary.

## Current Evidence Baseline

The current public benchmark evidence is the deterministic-heavy alpha benchmark path:

- workload: `experiments/workloads/deterministic_heavy_v1_100.json`
- generator: `experiments/generate_workload.py`
- benchmark runner: `experiments/run_benchmark.py`
- summary generator: `experiments/summarize_benchmark_results.py`
- artifact policy: `docs/reports/benchmark_artifact_policy.md`
- raw artifact decision: `docs/reports/v0.2.0-alpha-raw-artifact-freeze-decision.md`

Safe claim:

> KORA reduced model invocations by 80% in a reproducible deterministic-heavy benchmark workload.

Current counters:

| Metric | Value |
|---|---:|
| Total tasks | `100` |
| Deterministic/no-model tasks | `80` |
| Fallback/model-candidate tasks | `20` |
| Direct-baseline simulated model invocations | `100` |
| KORA-controlled simulated model invocations | `20` |
| Avoided simulated model invocations | `80` |
| Avoided invocation rate | `80%` |
| Deterministic outputs checked | `80` |
| Mismatches | `0` |
| Fallback/model-candidate skipped | `20` |

## Current Claim Boundary

The current evidence is valid for the tracked deterministic-heavy workload and the simulated benchmark runner. It does not claim:

- production cost reduction proof
- real API-cost reduction proof
- production benchmark proof
- full runtime-integrated benchmark evidence
- broad workload superiority proof
- energy reduction evidence

Simulated invocation counters are benchmark counters. They are not real API billing counters.

## Why Simulated/Offline Evidence Is Not Enough For v0.3.0-alpha

The current benchmark validates the workload, deterministic resolvers, expected outputs, and simulated direct-vs-KORA invocation accounting. It is useful evidence, but it does not prove that the same accounting is wired through the actual KORA runtime path.

For `v0.3.0-alpha`, stronger evidence should show that benchmark execution flows through `kora.executor.run_graph`, produces runtime events, can be summarized by `kora.telemetry`, and preserves the same claim boundaries. The next benchmark should connect:

- workload definitions
- TaskGraph construction
- deterministic handlers
- adapter/model-candidate tasks
- skip/escalation behavior
- runtime events
- telemetry summaries
- reviewed Markdown reports

The goal is closer runtime integration, not broader claims.

## Definition: Runtime-Integrated Benchmark Evidence

For KORA, runtime-integrated benchmark evidence should mean:

1. The benchmark builds or loads KORA `TaskGraph` objects instead of only counting workload records.
2. The KORA-controlled path executes through `kora.executor.run_graph`.
3. Runtime output includes `events`, `stage_timings`, `outputs`, `final`, and `ok` status.
4. Model-candidate work is represented as adapter-stage runtime events, including skipped adapter events where applicable.
5. Deterministic route outcomes are checked against expected outputs.
6. The benchmark writes raw JSON to `/tmp` or ignored paths by default.
7. The benchmark can generate a Markdown summary with the same claim boundary and artifact policy.
8. The generated run JSON can be summarized by `python3 -m kora telemetry`.

Runtime-integrated does not mean production traffic, real user workloads, real API billing, or energy measurement.

## Proposed Benchmark Layers

### Layer 0: Current Reproducible Offline/Simulated Benchmark

Layer 0 is the existing `v0.2.0-alpha` benchmark path in `experiments/`.

Purpose:

- preserve the current safe evidence baseline
- keep deterministic-heavy workload regeneration stable
- provide a low-cost, fully reproducible benchmark gate

Expected commands remain the current `/tmp` flow documented in `experiments/README.md` and `docs/reports/benchmark_artifact_policy.md`.

Layer 0 claim boundary remains the current safe claim only.

### Layer 1: CLI-Driven Runtime Path Benchmark

Layer 1 should add a CLI-visible benchmark path that executes KORA-controlled workload items through `kora.executor.run_graph`.

Purpose:

- verify that benchmark accounting is connected to the runtime path
- measure runtime events and stage timings from actual graph execution
- compare a direct simulated baseline with a KORA-controlled runtime path

Recommended shape:

```bash
python3 -m kora run runtime_benchmark -- \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --offline \
  --json-out /tmp/kora_runtime_benchmark_v0_3_0.json
```

This can be implemented either as a new example under `examples/runtime_benchmark/` or as a future dedicated CLI subcommand. For the first implementation task, an example is lower risk and follows the current `python3 -m kora run <example>` pattern.

Layer 1 should use mock/offline adapter behavior by default. Any online mode must be opt-in and must not be used for public claims without a separate review.

### Layer 2: Telemetry-Connected Benchmark Summary

Layer 2 should prove that Layer 1 raw JSON can be summarized through the existing telemetry path:

```bash
python3 -m kora telemetry \
  --input /tmp/kora_runtime_benchmark_v0_3_0.json \
  --json-out /tmp/kora_runtime_benchmark_v0_3_0.telemetry.json \
  --md-out /tmp/kora_runtime_benchmark_v0_3_0.telemetry.md
```

Purpose:

- connect benchmark output to `kora.telemetry.summarize_run`
- expose runtime event counts, skipped events, adapter events, and stage counts
- keep benchmark evidence inspectable through one public telemetry path

Layer 2 should not estimate real billing unless the input contains reviewed model and token fields. Even then, estimates are not proof of real API cost.

### Layer 3: Future Real-App Harness

Layer 3 is explicitly future/non-claim for `v0.3.0-alpha`.

The existing `examples/real_workload_harness/` and `docs/benchmark-real-app.md` point toward production-adjacent structure, but a real-app harness needs separate design and review before it can support stronger claims.

Layer 3 should require:

- reviewed workload source and selection criteria
- fixed request set or replayable fixtures
- direct and KORA-controlled execution definitions
- environment and model configuration record
- privacy review
- raw artifact retention decision
- independent claim review

Until those requirements exist, Layer 3 remains a future research and validation path, not a public evidence claim.

## Proposed Runtime Path Instrumentation Points

The current runtime already exposes useful instrumentation through `kora.executor.run_graph`:

| Location | Current signal | Future benchmark use |
|---|---|---|
| Scheduler | `order`, `scheduler_total_s` | Confirm graph execution order and scheduler status. |
| Deterministic task execution | `stage: DETERMINISTIC`, `status`, `time_ms` | Count deterministic route executions and failures. |
| Adapter/model task execution | `stage: ADAPTER`, `usage`, `meta`, `skipped` | Count KORA-controlled model invocations and skipped model-candidate tasks. |
| Verification | `verify_total_s`, failure contracts | Track deterministic checks and mismatch/failure status. |
| Runtime result | `ok`, `events`, `outputs`, `final`, `stage_timings` | Feed benchmark summary and telemetry summary. |
| Telemetry summarizer | `total_llm_calls`, `events_ok`, `events_fail`, `events_skipped`, `stage_counts` | Produce a consistent telemetry-connected report. |

Future implementation should avoid duplicating counter logic across scripts. Runtime-integrated benchmark counters should be derived from runtime events wherever possible.

## Proposed Metrics

The `v0.3.0-alpha` runtime-integrated benchmark should report:

| Metric | Meaning |
|---|---|
| `total_tasks` | Number of workload tasks selected for the benchmark. |
| `deterministic_route_count` | Number of workload tasks routed to deterministic graph execution. |
| `fallback_model_candidate_route_count` | Number of workload tasks requiring adapter/model-candidate handling. |
| `simulated_baseline_model_invocations` | Simulated model invocations for the naive direct baseline. |
| `kora_controlled_model_invocations` | Non-skipped adapter events from the KORA-controlled runtime path. |
| `avoided_simulated_model_invocations` | Difference between simulated baseline invocations and KORA-controlled invocations. |
| `avoided_simulated_model_invocation_rate` | Avoided simulated invocations divided by simulated baseline invocations. |
| `deterministic_output_checks` | Deterministic outputs checked against expected outputs. |
| `mismatch_count` | Deterministic expected-output mismatches. |
| `runtime_path_execution_status` | Aggregate `ok`/failure status for runtime graph execution. |
| `telemetry_event_count` | Count of runtime events emitted and summarized. |
| `benchmark_summary_path` | Path to the generated Markdown summary, usually under `/tmp` unless selected for reviewed docs. |

Additional useful fields:

- `workload_path`
- `source_commit`
- `offline`
- `raw_json_path`
- `telemetry_json_path`
- `telemetry_markdown_path`
- `claim_boundary`

## Proposed Files And Scripts For Future Implementation

Recommended future implementation files:

| Path | Purpose |
|---|---|
| `examples/runtime_benchmark/run.py` | CLI-driven runtime benchmark example that uses `run_graph`. |
| `examples/runtime_benchmark/README.md` | Example-specific usage and claim boundary. |
| `experiments/runtime_workload_to_graph.py` | Helper for converting workload records into KORA `TaskGraph` inputs, if this becomes shared code. |
| `experiments/summarize_runtime_benchmark.py` | Markdown summary generator for runtime-integrated raw JSON, if the current summary generator cannot stay generic. |
| `tests/test_runtime_integrated_benchmark.py` | Tests for runtime path counters, event accounting, and claim-boundary text. |
| `docs/reports/v0.3.0-alpha-runtime-benchmark-readiness.md` | Future readiness report before any claim update. |

Do not add a package-version change for this work. Do not upload release assets. Do not commit raw benchmark JSON outputs by default.

## Artifact Policy For v0.3.0-alpha

The default artifact policy should continue the `v0.2.0-alpha` rule:

- raw benchmark JSON outputs use `/tmp` or ignored paths
- raw outputs are not committed by default
- generated Markdown summaries may be committed only when selected as reviewed evidence
- frozen raw artifacts require explicit release review

For runtime-integrated evidence, any frozen raw artifact decision should additionally record:

- source commit
- CLI command
- workload path
- runtime benchmark layer
- offline/online mode
- adapter configuration
- telemetry summary command
- generated summary path
- claim boundary and non-claims

## Safe And Non-Safe Claim Table

| Claim type | Status for `v0.3.0-alpha` design | Notes |
|---|---|---|
| KORA reduced model invocations by 80% in a reproducible deterministic-heavy benchmark workload | Safe current claim | Current approved public claim. |
| Runtime path benchmark executed through `kora.executor.run_graph` | Future safe claim only after implementation and validation | Requires Layer 1 evidence. |
| Runtime events were summarized by KORA telemetry | Future safe claim only after implementation and validation | Requires Layer 2 evidence. |
| Production cost reduction proof | Not safe | Requires production-grade evidence and billing data review. |
| Real API-cost reduction proof | Not safe | Simulated/offline counters are not billing counters. |
| Production benchmark proof | Not safe | Requires production or production-adjacent benchmark design. |
| Full runtime-integrated benchmark evidence | Not safe today | This document designs the path; it does not provide the evidence. |
| Broad workload superiority proof | Not safe | Current workload is deterministic-heavy and scoped. |
| Energy reduction evidence | Not safe | No energy measurement exists. |

## Acceptance Criteria For Future Implementation

A future implementation task should be accepted only when:

1. The runtime benchmark runs from a public CLI path.
2. The KORA-controlled path uses `kora.executor.run_graph`.
3. The direct baseline is explicitly simulated/offline unless separately reviewed.
4. Runtime JSON includes `ok`, `events`, `stage_timings`, counter fields, workload metadata, and claim boundary text.
5. Deterministic outputs are checked against expected outputs.
6. KORA-controlled model invocations are counted from non-skipped adapter events.
7. Skipped adapter events are counted separately.
8. Telemetry can summarize the raw JSON with `python3 -m kora telemetry`.
9. Tests cover the counter contract and at least one mismatch/failure path.
10. Docs state that the evidence is not production/API-cost/energy proof.
11. Raw JSON artifacts remain in `/tmp` or ignored paths unless explicitly selected for freeze.
12. Validation includes `python3 -m pytest`, the runtime benchmark command, and telemetry summary generation.

## Risks And Mitigations

| Risk | Mitigation |
|---|---|
| Counter drift between benchmark scripts and runtime events | Derive Layer 1 counters from `run_graph` events wherever possible. |
| Accidental production or API-cost overclaiming | Keep claim-boundary text in raw JSON, generated summaries, docs, and tests. |
| Runtime benchmark becomes too large for routine validation | Keep the first Layer 1 implementation offline and bounded to the existing 100-task workload. |
| Raw artifacts get committed accidentally | Continue `/tmp` commands and ignored result paths; add tests or review checklist if needed. |
| Telemetry summary loses benchmark-specific meaning | Include benchmark-specific fields in raw JSON while letting telemetry summarize generic runtime events. |
| Future real-app harness is mistaken for production proof | Label Layer 3 as future/non-claim until separately designed, reviewed, and validated. |

## Recommended Next Task

Task 160 should implement the smallest Layer 1 prototype:

- create `examples/runtime_benchmark/run.py`
- convert the existing deterministic-heavy workload records into KORA runtime graph executions
- write raw runtime benchmark JSON to `/tmp`
- count KORA-controlled invocations from runtime adapter events
- preserve the current claim boundary
- add focused tests for the runtime benchmark counter contract
- validate with `python3 -m pytest`, the new CLI example, and `python3 -m kora telemetry`

Task 160 should not update the package version, create a tag, create a release, upload assets, or claim production/API-cost/energy evidence.
