# KORA v0.2.0-alpha Release Notes Draft

Version: `v0.2.0-alpha`

Status: draft / not yet released

Historical status at time of writing; `v0.2.0-alpha` has since been published: https://github.com/Krako-Labs/KORA/releases/tag/v0.2.0-alpha

Previous published release: `v0.1.1-alpha`

## Summary

KORA v0.2.0-alpha is planned as an alpha benchmark-evidence expansion release.

This draft focuses on the deterministic-heavy benchmark evidence path added after `v0.1.1-alpha`: stronger deterministic expected-output checks, automated Markdown summary generation from result JSON, a raw benchmark artifact policy, and expanded correctness/error/fallback test coverage.

This is a draft document only. It does not create a Git tag, GitHub Release, or final public release asset.

## What Changed Since v0.1.1-alpha

- Expanded the deterministic-heavy workload evidence path from the original 20-task benchmark skeleton to the reproducible 100-task `deterministic_heavy_v1_100` workload.
- Added deterministic expected-output correctness checks for benchmark tasks marked `requires_model: false`.
- Added correctness counters for deterministic output checks, mismatches, and fallback/model-candidate skips.
- Added automated Markdown summary generation from raw benchmark result JSON artifacts.
- Added a raw benchmark artifact policy for deciding which generated outputs remain ephemeral and which summaries may be committed.
- Added focused correctness/error/fallback coverage for deterministic success, deterministic mismatch, fallback skip behavior, unsupported categories, malformed deterministic input, and stable `v1_100` evidence counters.

## Scope

This release draft covers:

- benchmark evidence expansion
- deterministic expected-output checks
- summary generation automation
- raw artifact policy
- expanded correctness/error/fallback benchmark coverage

This release draft does not cover production runtime integration, real model/API measurements, production cost analysis, or energy measurement.

## Benchmark Evidence Summary

Current deterministic-heavy benchmark evidence:

| Metric | Value |
|---|---:|
| Workload | `experiments/workloads/deterministic_heavy_v1_100.json` |
| Generator | `experiments/generate_workload.py` |
| Seed | `42` |
| Total tasks | `100` |
| Deterministic/no-model tasks | `80` |
| Fallback/model-candidate tasks | `20` |
| Direct-baseline simulated model invocations | `100` |
| KORA-controlled simulated model invocations | `20` |
| Avoided simulated model invocations | `80` |
| Avoided invocation rate | `80%` |
| Deterministic outputs checked | `80` |
| Deterministic expected-output mismatches | `0` |
| Fallback/model-candidate tasks skipped for deterministic output validation | `20` |

## Safe Claim Boundary

Safe claim:

> In a reproducible 100-task deterministic-heavy benchmark workload, KORA-controlled execution avoided 80 of 100 simulated model invocations versus a naive direct baseline.

Use `simulated` when describing model invocations in this evidence path.

## Explicit Non-Claims

This draft does not claim:

- production cost reduction proof
- real API-cost reduction proof
- production benchmark proof
- full runtime-integrated benchmark evidence
- broad workload superiority proof
- energy reduction evidence

## Regeneration

Raw benchmark result artifacts should generally be generated under `/tmp` or another ignored path. See:

```text
docs/reports/benchmark_artifact_policy.md
```

That policy includes commands to regenerate:

- the `deterministic_heavy_v1_100` workload from seed `42`
- dry-run result JSON
- direct-baseline result JSON
- KORA-controlled result JSON
- generated Markdown summary output

## Suggested Release Readiness Checklist

- [ ] Confirm Tasks 101-105 are merged into `origin/main` in order.
- [ ] Run a clean validation from the final `origin/main` release candidate.
- [ ] Regenerate `deterministic_heavy_v1_100` under `/tmp` and confirm it matches the tracked workload.
- [ ] Run `dry-run`, `direct-baseline`, and `kora-controlled` benchmark modes under `/tmp`.
- [ ] Generate the Markdown benchmark summary from raw `/tmp` result artifacts.
- [ ] Confirm expected-output counters remain: 80 checked, 0 mismatches, 20 fallback/model-candidate skips.
- [ ] Review `docs/reports/benchmark_artifact_policy.md` for consistency with release evidence handling.
- [ ] Review this draft for claim boundary compliance.
- [ ] Decide whether a generated Markdown benchmark summary should be committed as public evidence.
- [ ] Update final release note location if this draft is promoted.
- [ ] Create a Git tag only after explicit release approval.
- [ ] Create a GitHub Release only after explicit release approval.

## Validation Commands

Recommended validation before promoting this draft:

```bash
python3 -m pytest
python3 -m kora --help
python3 -m kora examples list
python3 -m kora run hello_kora
python3 -m kora run retry_demo
python3 -m kora run direct_vs_kora -- --offline

python3 experiments/generate_workload.py \
  --count 100 \
  --seed 42 \
  --benchmark-name deterministic_heavy_v1 \
  --output /tmp/kora_deterministic_heavy_v1_100.regenerated.json

cmp experiments/workloads/deterministic_heavy_v1_100.json \
  /tmp/kora_deterministic_heavy_v1_100.regenerated.json

python3 experiments/run_benchmark.py \
  --mode dry-run \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.dry_run.json

python3 experiments/run_benchmark.py \
  --mode direct-baseline \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json

python3 experiments/run_benchmark.py \
  --mode kora-controlled \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json

python3 experiments/summarize_benchmark_results.py \
  --direct-baseline /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json \
  --kora-controlled /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json \
  --output /tmp/kora_deterministic_heavy_v1_100.summary.md
```

## Limitations

- The benchmark evidence is simulated.
- No real model calls are made.
- No external APIs are used for benchmark execution.
- No real API-cost measurement is included.
- No production benchmark evidence is included.
- No full runtime-integrated benchmark evidence is included.
- The workload is deterministic-heavy by design.
- The current result should be treated as alpha-stage reproducible benchmark evidence, not production proof.
