# KORA v0.2.0-alpha Merge Readiness PR Packet

Status: merge-readiness packet / not released

Historical status at time of writing; `v0.2.0-alpha` has since been published: https://github.com/Krako-Labs/KORA/releases/tag/v0.2.0-alpha

Source branch: `task107_merge_readiness_pr_packet`

Base branch for this packet: `origin/task106_v020_alpha_readiness_check`

Target branch for eventual PR merge: `main`

Current public truth checked: `origin/main 17d0665b6e9eb48bd54a83ae2dda2e59f416a40f`

## PR Title Suggestion

Prepare v0.2.0-alpha benchmark evidence expansion

## PR Summary

This PR packet prepares the Task 101-106 branch chain for review and eventual merge. The chain expands the deterministic-heavy benchmark evidence path while keeping claims limited to reproducible simulated benchmark evidence.

The chain adds deterministic expected-output validation, benchmark summary generation, raw artifact policy documentation, expanded correctness/error/fallback test coverage, a draft `v0.2.0-alpha` release note, and a readiness-check report.

This packet does not merge to `main`, create a Git tag, create a GitHub Release, update package version metadata, or commit raw benchmark JSON artifacts.

## Included Task Chain

| Task | Commit | Summary |
|---|---|---|
| Task 101 | `04d6dd6710ff202765d6debc034c2dfa2557cdad` | Added deterministic expected-output correctness checks and counters. |
| Task 102 | `c35fd7a6f5f6bed40e65c76217ee453f066cdddb` | Added Markdown benchmark summary generation from result JSON artifacts. |
| Task 103 | `b7e9d59533056e900fcaa3b272878138936f099a` | Documented raw benchmark artifact policy and narrowed generated-output ignores. |
| Task 104 | `37b4cbd71b086e2b161a874dbc255972d1bb28c9` | Expanded correctness/error/fallback benchmark case coverage. |
| Task 105 | `7fc346976be6502d87cb8108a2670ea163215659` | Drafted `v0.2.0-alpha` release notes. |
| Task 106 | `5eeba65f1c17901c16a978f3615f2882a03e2049` | Added `v0.2.0-alpha` readiness-check report. |

The chain is linear on top of `origin/main 17d0665b6e9eb48bd54a83ae2dda2e59f416a40f`.

## Expected Files Added or Modified

| Path | Status | Purpose |
|---|---|---|
| `.gitignore` | Modified | Keeps raw generated benchmark outputs ignored while allowing Markdown reports to be tracked. |
| `docs/reports/benchmark_artifact_policy.md` | Added | Documents raw artifact and summary policy. |
| `docs/reports/v0.2.0-alpha-release-notes-draft.md` | Added | Draft release notes only; not a release. |
| `docs/reports/v0.2.0-alpha-readiness-check.md` | Added | Readiness report only; not a release. |
| `experiments/run_benchmark.py` | Modified | Adds deterministic expected-output validation and clearer errors. |
| `experiments/summarize_benchmark_results.py` | Added | Generates Markdown summaries from raw benchmark result JSON. |
| `tests/test_benchmark_runner.py` | Modified | Adds correctness/error/fallback and stable counter coverage. |
| `tests/test_benchmark_summary.py` | Added | Tests summary generation and claim boundary text. |

## Branch-Chain Integrity Checks

- `origin/main` remained at `17d0665b6e9eb48bd54a83ae2dda2e59f416a40f` during this packet preparation.
- `origin/task106_v020_alpha_readiness_check` resolved to `5eeba65f1c17901c16a978f3615f2882a03e2049`.
- The chain from `origin/main` to this packet includes Task 101 through Task 106 commits in order.
- No merge to `main` was performed.
- No tag was created.
- No GitHub Release was created.
- No package version update was performed.

## Raw Artifact Tracking Check

No generated benchmark result JSON artifacts are expected to be tracked.

Tracked benchmark-related JSON files remain source fixtures or workload definitions:

- `experiments/workloads/deterministic_heavy_v0.json`
- `experiments/workloads/deterministic_heavy_v1_100.json`
- `docs/reports/sample_telemetry_input.json`
- example graph JSON files

Generated raw benchmark outputs remain ignored by `.gitignore`:

- `experiments/results/*.json`
- `experiments/results/*.jsonl`
- `docs/reports/*.json`
- `docs/reports/*.jsonl`
- `docs/reports/generated/`

Markdown reports under `docs/reports/` remain trackable.

## Benchmark Evidence Counters

| Metric | Value |
|---|---:|
| Workload | `experiments/workloads/deterministic_heavy_v1_100.json` |
| Generator | `experiments/generate_workload.py` |
| Seed | `42` |
| Total tasks | `100` |
| Deterministic/no-model tasks | `80` |
| Fallback/model-candidate tasks | `20` |
| Direct-baseline simulated model invocations | `100` |
| KORA-controlled simulated model invocations | `20` |
| Avoided simulated model invocations | `80` |
| Avoided invocation rate | `80%` |
| Deterministic outputs checked | `80` |
| Deterministic expected-output mismatches | `0` |
| Fallback/model-candidate tasks skipped for deterministic output validation | `20` |

## Validation Commands

Run before opening or merging the PR:

```bash
python3 -m pytest
python3 -m kora --help
python3 -m kora examples list
python3 -m kora run hello_kora
python3 -m kora run retry_demo
python3 -m kora run direct_vs_kora -- --offline

python3 experiments/generate_workload.py \
  --count 100 \
  --seed 42 \
  --benchmark-name deterministic_heavy_v1 \
  --output /tmp/kora_deterministic_heavy_v1_100.regenerated.json

cmp experiments/workloads/deterministic_heavy_v1_100.json \
  /tmp/kora_deterministic_heavy_v1_100.regenerated.json

python3 experiments/run_benchmark.py \
  --mode dry-run \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.dry_run.json

python3 experiments/run_benchmark.py \
  --mode direct-baseline \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json

python3 experiments/run_benchmark.py \
  --mode kora-controlled \
  --workload experiments/workloads/deterministic_heavy_v1_100.json \
  --output /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json

python3 experiments/summarize_benchmark_results.py \
  --direct-baseline /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json \
  --kora-controlled /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json \
  --output /tmp/kora_deterministic_heavy_v1_100.summary.md
```

## Safe Claim Boundary

Safe claim:

> In a reproducible 100-task deterministic-heavy benchmark workload, KORA-controlled execution avoided 80 of 100 simulated model invocations versus a naive direct baseline.

Use `simulated` when describing model invocations in this evidence path.

## Explicit Non-Claims

This PR packet does not claim:

- production cost reduction proof
- real API-cost reduction proof
- production benchmark proof
- full runtime-integrated benchmark evidence
- broad workload superiority proof
- energy reduction evidence

## Suggested PR Body

```markdown
## Summary

This PR prepares the v0.2.0-alpha benchmark evidence expansion for review.

It expands the deterministic-heavy benchmark evidence path while keeping claims limited to reproducible simulated benchmark evidence. It adds deterministic expected-output validation, Markdown summary generation from benchmark result JSON, a raw artifact policy, expanded correctness/error/fallback coverage, and draft release/readiness documentation.

## Included work

- Deterministic expected-output checks for `requires_model: false` benchmark tasks
- Correctness counters for checked deterministic outputs, mismatches, and fallback/model-candidate skips
- Markdown summary generation from direct-baseline and KORA-controlled result JSON artifacts
- Raw benchmark artifact policy covering ephemeral `/tmp` outputs and frozen evidence requirements
- Expanded benchmark tests for success, mismatch, fallback skip, unsupported category, malformed input, and stable `v1_100` counters
- Draft `v0.2.0-alpha` release notes
- `v0.2.0-alpha` readiness-check report

## Benchmark evidence

Current deterministic-heavy benchmark evidence:

| Metric | Value |
|---|---:|
| Workload | `experiments/workloads/deterministic_heavy_v1_100.json` |
| Seed | `42` |
| Total tasks | `100` |
| Deterministic/no-model tasks | `80` |
| Fallback/model-candidate tasks | `20` |
| Direct-baseline simulated model invocations | `100` |
| KORA-controlled simulated model invocations | `20` |
| Avoided simulated model invocations | `80` |
| Avoided invocation rate | `80%` |
| Deterministic outputs checked | `80` |
| Mismatches | `0` |
| Fallback/model-candidate skipped | `20` |

Safe claim:

> In a reproducible 100-task deterministic-heavy benchmark workload, KORA-controlled execution avoided 80 of 100 simulated model invocations versus a naive direct baseline.

## Non-claims

This PR does not claim production cost reduction proof, real API-cost reduction proof, production benchmark proof, full runtime-integrated benchmark evidence, broad workload superiority proof, or energy reduction evidence.

## Validation

- [ ] `python3 -m pytest`
- [ ] `python3 -m kora --help`
- [ ] `python3 -m kora examples list`
- [ ] `python3 -m kora run hello_kora`
- [ ] `python3 -m kora run retry_demo`
- [ ] `python3 -m kora run direct_vs_kora -- --offline`
- [ ] Regenerate `deterministic_heavy_v1_100` under `/tmp` and compare with tracked workload
- [ ] Run dry-run, direct-baseline, and kora-controlled benchmark modes
- [ ] Generate Markdown summary from `/tmp` benchmark artifacts

## Release actions not performed

- No merge to `main`
- No Git tag
- No GitHub Release
- No package version update
- No raw benchmark JSON artifacts committed
```

## Release Blockers Remaining After This PR Packet

- Task 101-107 chain still needs PR review and merge into `main`.
- Final validation still needs to be run from merged public `origin/main`.
- `CHANGELOG.md` still needs a reviewed `v0.2.0-alpha` update after merge readiness is confirmed.
- Any decision to freeze raw benchmark artifacts remains open.
- Git tag and GitHub Release still require explicit approval.

## Post-Merge Checklist

- [ ] Validate merged `origin/main`.
- [ ] Update `CHANGELOG.md`.
- [ ] Run final release smoke from public HEAD.
- [ ] Decide whether to freeze any raw artifacts.
- [ ] Create tag only after final approval.
- [ ] Create GitHub Release only after tag approval.
