# KORA Benchmark Result Summary - v1 100-Task Workload

Date: 2026-05-04

## Benchmark Status

This document records the current simulated benchmark result for KORA's reproducible 100-task deterministic-heavy workload.

This is a controlled benchmark artifact for the current alpha evidence path. It is not a production benchmark, does not use real model calls, and does not measure real API cost.

## Repository Context

| Item | Value |
|---|---|
| Public HEAD at original benchmark recording | `origin/main b7c555e0dfe0687cdf05177c03eacac0743079a5` |
| Generator | `experiments/generate_workload.py` |
| Workload | `experiments/workloads/deterministic_heavy_v1_100.json` |
| Benchmark runner | `experiments/run_benchmark.py` |
| Workload expansion plan | `docs/benchmarks/workload_expansion_plan_v0_2_0.md` |

## Workload Generation Details

The workload is generated by:

```text
experiments/generate_workload.py
```

Generation parameters:

| Parameter | Value |
|---|---|
| Workload file | `experiments/workloads/deterministic_heavy_v1_100.json` |
| Seed | `42` |
| Count | `100` |
| Benchmark name | `deterministic_heavy_v1` |
| Regeneration comparison | Byte-for-byte match |

The workload contains 100 tasks across 10 deterministic-heavy categories:

| Category | Task count |
|---|---:|
| arithmetic | 10 |
| cache_lookup | 10 |
| config_validation | 10 |
| date_normalization | 10 |
| json_validation | 10 |
| policy_rule_match | 10 |
| routing_decision | 10 |
| schema_check | 10 |
| string_normalization | 10 |
| unit_conversion | 10 |

## Reproducibility Check

The committed workload was regenerated with the same seed, count, and benchmark name, then compared against the tracked workload file. The comparison passed with a byte-for-byte match.

## Benchmark Modes

| Mode | Description | Real model calls |
|---|---|---|
| `dry-run` | Loads the workload, validates task shape, counts categories, and counts `requires_model` values. | No |
| `direct-baseline` | Simulates a naive baseline where every task causes one model invocation. | No |
| `kora-controlled` | Simulates deterministic-first KORA execution by counting `requires_model: false` tasks as deterministic resolutions and `requires_model: true` tasks as fallback/model-candidate tasks. | No |

## Result Table

| Metric | Value |
|---|---:|
| Total tasks | 100 |
| Deterministic/no-model tasks | 80 |
| Fallback/model-candidate tasks | 20 |
| Direct-baseline simulated model invocations | 100 |
| KORA-controlled simulated model invocations | 20 |
| Avoided model invocations | 80 |
| Avoided invocation rate | 80% |

## Approved Public Claim

KORA reduced model invocations by 80% in a reproducible deterministic-heavy benchmark workload.

## Safe Interpretation

This should be interpreted as a reproducible deterministic-heavy alpha benchmark result. The simulated baseline and KORA-controlled counters explain the measurement method: the direct baseline counted 100 simulated model invocations while the KORA-controlled path counted 20.

This evidence is suitable for alpha validation and further runtime-integrated testing. It is not production cost reduction proof or real API-cost reduction proof.

## Non-Claims

This result does not claim:

- this is production cost reduction
- this is real API-cost reduction
- this is a production benchmark
- this is full KORA runtime integration
- this is a broad workload benchmark
- this does not prove general performance superiority
- this does not measure energy reduction
- this proves latency improvement
- this is paper-grade evaluation evidence

## Validation Commands

```bash
python3 experiments/generate_workload.py --count 100 --seed 42 --benchmark-name deterministic_heavy_v1 --output /tmp/kora_deterministic_heavy_v1_100.regenerated.json
cmp experiments/workloads/deterministic_heavy_v1_100.json /tmp/kora_deterministic_heavy_v1_100.regenerated.json

python3 experiments/run_benchmark.py --mode dry-run --workload experiments/workloads/deterministic_heavy_v1_100.json --output /tmp/kora_deterministic_heavy_v1_100.dry_run.json
python3 experiments/run_benchmark.py --mode direct-baseline --workload experiments/workloads/deterministic_heavy_v1_100.json --output /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json
python3 experiments/run_benchmark.py --mode kora-controlled --workload experiments/workloads/deterministic_heavy_v1_100.json --output /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json

./scripts/release_smoke.sh
python3 -m pytest -q
```

Optional JSON validation:

```bash
python3 -m json.tool experiments/workloads/deterministic_heavy_v1_100.json > /tmp/kora_generated_workload_check.json
python3 -m json.tool /tmp/kora_deterministic_heavy_v1_100.dry_run.json > /tmp/kora_v1_dry_run_check.json
python3 -m json.tool /tmp/kora_deterministic_heavy_v1_100.direct_baseline.json > /tmp/kora_v1_direct_baseline_check.json
python3 -m json.tool /tmp/kora_deterministic_heavy_v1_100.kora_controlled.json > /tmp/kora_v1_kora_controlled_check.json
```

Current validation status:

| Check | Result |
|---|---|
| Regeneration comparison | Passed, byte-for-byte match |
| Generated workload JSON validation | Passed |
| Dry-run benchmark | Passed |
| Direct-baseline benchmark | Passed |
| KORA-controlled benchmark | Passed |
| Benchmark output JSON validation | Passed |
| Release smoke | Passed |
| Pytest | Passed, 42 tests |

## Limitations

- The benchmark is simulated.
- No real model calls are made.
- No external API calls are made.
- No real API-cost measurement is included.
- No production cost reduction claim is supported.
- No full KORA runtime integration is implemented yet.
- The workload remains deterministic-heavy by design.
- The avoided invocation rate is based on workload metadata and simulated benchmark modes.
- Correctness scoring against deterministic `expected_output` fields is not yet reported as a benchmark metric.

## Next Steps

1. Add stronger correctness checks for deterministic tasks.
2. Add benchmark summary generation from structured result JSON.
3. Decide whether raw generated benchmark outputs remain generated-only or become committed artifacts.
4. Expand workload variants beyond the first 100-task deterministic-heavy workload.
5. Update `docs/technical_preview_results.md` with the 100-task benchmark evidence once reviewed.
6. Preserve explicit non-claims until runtime-integrated and real model/API measurements exist.
