# KORA Experiments

This directory contains the current reproducible benchmark workload, benchmark scripts, and generated-output placeholder directory for KORA experiments.

## Current Canonical Workload

The current canonical deterministic-heavy benchmark workload is:

```text
experiments/workloads/deterministic_heavy_v1_100.json
```

It is generated by:

```text
experiments/generate_workload.py
```

Current benchmark metadata:

| Metric | Value |
|---|---:|
| Seed | `42` |
| Total tasks | `100` |
| Deterministic/no-model tasks | `80` |
| Fallback/model-candidate tasks | `20` |
| Direct-baseline simulated model invocations | `100` |
| KORA-controlled simulated model invocations | `20` |
| Avoided simulated model invocations | `80` |
| Avoided invocation rate | `80%` |
| Deterministic outputs checked | `80` |
| Mismatches | `0` |
| Fallback/model-candidate skipped | `20` |

## Regeneration Commands

Use `/tmp` paths for routine generated outputs. Do not commit raw benchmark JSON artifacts unless a later release process explicitly selects frozen evidence.

Regenerate the workload with seed `42`:

```bash
python3 experiments/generate_workload.py \
  --count 100 \
  --seed 42 \
  --benchmark-name deterministic_heavy_v1 \
  --output /tmp/kora_deterministic_heavy_v1_100.regenerated.json
```

Compare the regenerated workload with the tracked workload:

```bash
cmp experiments/workloads/deterministic_heavy_v1_100.json \
  /tmp/kora_deterministic_heavy_v1_100.regenerated.json
```

Run the dry-run benchmark:

```bash
python3 experiments/run_benchmark.py \
  --mode dry-run \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.dry_run.json
```

Run the direct-baseline benchmark:

```bash
python3 experiments/run_benchmark.py \
  --mode direct-baseline \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json
```

Run the KORA-controlled benchmark:

```bash
python3 experiments/run_benchmark.py \
  --mode kora-controlled \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json
```

Generate a Markdown summary from `/tmp` artifacts:

```bash
python3 experiments/summarize_benchmark_results.py \
  --direct-baseline /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json \
  --kora-controlled /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json \
  --output /tmp/kora_deterministic_heavy_v1_100.summary.md
```

## Initial Runtime-Path Harness

The initial `v0.3.0-alpha` runtime-path benchmark harness runs the tracked deterministic-heavy workload through the KORA executor path with offline/mock behavior:

```bash
python3 -m kora run runtime_integrated_benchmark -- --offline
```

To write raw JSON for local inspection, use an explicit `/tmp` path:

```bash
python3 -m kora run runtime_integrated_benchmark -- \
  --offline \
  --json-out /tmp/kora_runtime_integrated_benchmark_v0_3_0.json
```

To generate a local Markdown evidence packet from the saved runtime benchmark JSON, use the report generator:

```bash
python3 examples/runtime_integrated_benchmark/report.py \
  --input /tmp/kora_runtime_integrated_benchmark_v0_3_0.json \
  --md-out /tmp/kora_runtime_integrated_benchmark_v0_3_0.md
```

The harness can also write the Markdown packet during the runtime run with `--md-out`, but the separate report command is the reproducible evidence-packet path for reviewing an existing JSON run.

This harness is an initial runtime-integrated benchmark scaffold. It is not production benchmark evidence, does not call external APIs in offline mode, and does not change the current public claim boundary. Raw generated JSON and generated Markdown evidence packets should stay in `/tmp` or another ignored path unless a later release process explicitly selects frozen evidence.

For reviewer-facing reproduction steps and expected counters, see the [v0.3.0-alpha runtime evidence reviewer guide](../docs/reports/v0.3.0-alpha-runtime-evidence-reviewer-guide.md).

## Artifact Policy

Raw benchmark JSON artifacts are reproducible outputs and are not frozen or committed for `v0.2.0-alpha`.

See:

- [KORA documentation index](../docs/README.md)
- [CHANGELOG.md](../CHANGELOG.md)
- [Benchmark artifact policy](../docs/reports/benchmark_artifact_policy.md)
- [v0.2.0-alpha raw artifact freeze decision](../docs/reports/v0.2.0-alpha-raw-artifact-freeze-decision.md)

The tracked source artifacts are the workload definition, generator, benchmark runner, summary generator, tests, and documentation. The `experiments/results/` directory is intentionally kept empty except for `.gitkeep`.

## Claim Boundary

Safe claim:

> In a reproducible 100-task deterministic-heavy benchmark workload, KORA-controlled execution avoided 80 of 100 simulated model invocations versus a naive direct baseline.

This benchmark evidence does not claim:

- production cost reduction proof
- real API-cost reduction proof
- production benchmark proof
- full runtime-integrated benchmark evidence
- broad workload superiority proof
- energy reduction evidence

## Historical Workload

`experiments/workloads/deterministic_heavy_v0.json` is preserved as an earlier benchmark skeleton. New readers should start with `deterministic_heavy_v1_100`.
