# v0.3.0-alpha Runtime Evidence Reviewer Guide

Status: reviewer-facing guide for the bounded `v0.3.0-alpha` runtime evidence flow.

## Purpose

This guide explains how reviewers can reproduce KORA's `v0.3.0-alpha` runtime evidence flow locally. It covers the initial runtime-path benchmark harness, generated JSON output, Markdown evidence packet, and telemetry-connected runtime benchmark summary.

## What This Evidence Is

- An initial runtime-path benchmark harness using the tracked deterministic-heavy workload.
- A reproducible local evidence-generation path.
- A runtime benchmark JSON output path.
- A Markdown runtime benchmark evidence packet.
- A telemetry-connected runtime benchmark summary.

The workload is:

```text
experiments/workloads/deterministic_heavy_v1_100.json
```

## What This Evidence Is Not

- This is not production benchmark evidence.
- This does not prove real API-cost reduction.
- This does not prove production cost reduction.
- This does not prove broad workload superiority.
- This does not prove energy reduction.
- This is not full runtime-integrated benchmark evidence.
- This is not formal government validation.
- This is not signed partner validation.
- This does not guarantee adoption or funding.

## Reproduction Commands

Run the test suite:

```bash
python3 -m pytest
```

Run the initial runtime-path benchmark harness:

```bash
python3 -m kora run runtime_integrated_benchmark -- --offline
```

Generate runtime benchmark JSON under `/tmp`:

```bash
python3 -m kora run runtime_integrated_benchmark -- \
  --offline \
  --json-out /tmp/kora_runtime_integrated_benchmark.json
```

Generate the Markdown evidence packet from the runtime benchmark JSON:

```bash
python3 examples/runtime_integrated_benchmark/report.py \
  --input /tmp/kora_runtime_integrated_benchmark.json \
  --md-out /tmp/kora_runtime_integrated_benchmark.md
```

Generate the telemetry summary:

```bash
python3 -m kora telemetry \
  --input /tmp/kora_runtime_integrated_benchmark.json \
  --json-out /tmp/kora_runtime_integrated_benchmark.telemetry.json \
  --md-out /tmp/kora_runtime_integrated_benchmark.telemetry.md
```

## Expected Runtime Benchmark Counters

| Counter | Expected value |
|---|---:|
| `total_tasks` | `100` |
| `deterministic_route_count` | `80` |
| `fallback_or_model_candidate_route_count` | `20` |
| `simulated_baseline_model_invocations` | `100` |
| `kora_controlled_model_invocations` | `20` |
| `avoided_simulated_model_invocations` | `80` |
| `avoided_simulated_model_invocation_rate` | `0.8` |
| `deterministic_outputs_checked` | `80` |
| `mismatch_count` | `0` |
| `runtime_path_execution_status` | `ok` |
| `telemetry_event_count` | `100` |

## Expected Telemetry Counters

| Counter | Expected value |
|---|---:|
| `total_llm_calls` | `20` |
| `events_ok` | `100` |
| `events_fail` | `0` |
| `events_skipped` | `0` |
| `stage_counts.ADAPTER` | `20` |
| `stage_counts.DETERMINISTIC` | `80` |

## Safe Current Claim

> KORA reduced model invocations by 80% in a reproducible deterministic-heavy benchmark workload.

This wording remains the approved bounded public claim for this evidence path. The simulated baseline counters above explain the benchmark methodology and should not be rewritten as production cost reduction, real API-cost reduction, or real API-call reduction proof.

## Artifact Policy

- Generated JSON and Markdown evidence outputs should stay in `/tmp` or user-provided output paths.
- Raw generated benchmark outputs are not committed.
- No release assets should be uploaded unless explicitly approved in a future task.
- If a future task selects frozen evidence, it must also update the artifact policy and release notes.

## Reviewer Checklist

- [ ] Repository is clean before running validation.
- [ ] `python3 -m pytest` passes.
- [ ] Runtime benchmark command passes.
- [ ] Runtime benchmark JSON is generated under `/tmp` or another user-provided output path.
- [ ] Markdown evidence packet is generated under `/tmp` or another user-provided output path.
- [ ] Telemetry summary JSON and Markdown are generated under `/tmp` or another user-provided output path.
- [ ] Runtime benchmark counters match the expected values above.
- [ ] Telemetry counters match the expected values above.
- [ ] Generated JSON and Markdown outputs are not staged or committed.
- [ ] Claim boundaries are preserved in any public text or release notes.

## Related Docs

- [Documentation index](../README.md)
- [Experiments guide](../../experiments/README.md)
- [Runtime-integrated benchmark evidence architecture](../design/v0.3.0-runtime-integrated-benchmark-evidence-architecture.md)
- [Benchmark artifact policy](benchmark_artifact_policy.md)
- [Telemetry and observability counters](../telemetry-and-observability.md)
