# OctoBench Tuned Policy Data

This directory records the tuned OctoBench subset used for ActPlane policy
experiments. It keeps three retained result classes, ordered by how the three
conditions compare and whether ActPlane produced OS-feedback evidence:

```text
regex 好: actplane-feedback reward >= tool-regex reward > baseline reward
一般: actplane-feedback reward > baseline reward, while tool-regex is not required to beat baseline
一般 score-only: actplane-feedback reward > baseline reward, but the retained run has no OS violation/feedback evidence
```

The baseline is not rerun during tuning. It comes from the clean 20-case paired
run:

```text
docs/OctoBench/data/core-results-old/paired_clean_20case_20260604.json
```

Only the best successful policy attempt is recorded here for each case. Failed
or intermediate tuning attempts should stay in raw run directories, not in this
README. General rows are retained when ActPlane beats baseline, even if
tool-regex does not. Score-only rows are retained separately because they are
useful for official-score accounting but are weaker mechanism evidence.

## Reporting Rule

A case is reportable in the tuned success set only if all of these hold:

1. The case has a real ActPlane-observable OS effect, except rows explicitly
   marked `一般 score-only`.
2. The policy is case-specific, with no shared/base guardrail.
3. `tool-regex` uses only the matching case policy file under
   `policies/tool-regex/<case_id>.json`.
4. `actplane-feedback` uses only the matching case policy file under
   `policies/actplane-feedback/<case_id>.yaml`.
5. The official OctoBench whole-case judge score satisfies either:
   `actplane-feedback >= tool-regex > baseline` for `regex 好` rows, or
   `actplane-feedback > baseline` for `一般` / `一般 score-only` rows.

The paper-facing result now uses the 21-task subset whose user-query
checklist contains at least one OS-enforceable item, with best-by-condition
scoring: baseline 0.80, hooks 0.81, ActPlane 0.84. The exact task ids are
stored in `selected_cases_21.ids` and the full task records are stored in
`selected_cases_21.jsonl`; both are generated by
`docs/temp/compute_octobench_rq4.py`.

## Selected 10-Case Pool

Baseline values below are from the clean paired 20-case run. The tuning pool is
selected for the tuned success-set target. The pool is restricted to
`claudecode` cases so that `tool-regex` is an actual hook-based baseline, not a
no-op scaffold mismatch.

Selection rule:

- Prefer `baseline < 1.000`, because a case with baseline `1.000` cannot satisfy
  `actplane-feedback > baseline` under OctoBench's capped reward.
- Prefer lower baseline scores; this pool uses the 10 lowest-baseline
  `claudecode` cases from the clean paired run.
- Prefer cases with observed ActPlane OS events when scores are close, but low
  baseline is the primary tuning criterion.

| rank | case | scaffold | baseline | previous tool-regex | previous actplane-feedback | ActPlane events | reason selected |
|---:|---|---|---:|---:|---:|---:|---|
| 1 | `md-basic-memory-async-client-pattern` | claudecode | 0.541 | 0.595 | 0.541 | 2 | low baseline + observed OS evidence |
| 2 | `md-aws-mcp-server-logging-over-print` | claudecode | 0.568 | 0.757 | 0.595 | 2 | low baseline + observed OS evidence |
| 3 | `88f06a58-61ab-4660-9721-d6e1f5f261ed` | claudecode | 0.677 | 0.710 | 0.645 | 2 | low baseline + observed OS evidence |
| 4 | `benchmark-cb_append_payment_001` | claudecode | 0.722 | 0.639 | 1.000 | 0 | low baseline; policy-relevant file/dependency OS effects |
| 5 | `benchmark-bm_append_export_001` | claudecode | 0.725 | 0.725 | 0.625 | 0 | low baseline; export feature has file/test/dependency OS effects |
| 6 | `md-course-builder-code-style` | claudecode | 0.735 | 0.647 | 0.706 | 9 | low baseline + observed OS evidence |
| 7 | `benchmark-aws_cancel_partial_001` | claudecode | 0.786 | 0.810 | 0.762 | 0 | low baseline; AWS command-history task with command/file OS effects |
| 8 | `md-aws-mcp-command-validation` | claudecode | 0.794 | 0.853 | 0.765 | 2 | low baseline + observed OS evidence |
| 9 | `md-astropy-13236-add-validators` | claudecode | 0.829 | 0.857 | 0.800 | 0 | low baseline; validator task has file/test OS effects |
| 10 | `md-aws-mcp-server-pathlib-over-ospath` | claudecode | 0.857 | 0.667 | 0.786 | 10 | observed OS evidence; highest baseline retained |

These ten IDs are also stored in:

```text
docs/OctoBench/data/selected_tuned_10.ids
```

## Excluded From The Tuning Pool

Cases with high baseline are excluded from the tuning pool because there is too
little headroom for `actplane-feedback > baseline`. They may still be retained
separately as `一般` rows if ActPlane beats baseline under case-specific policy:

| case | baseline | reason |
|---|---:|---|
| `benchmark-aws_checklist_error_001` | 0.946 | baseline too high for tuning; retained below as `一般` |
| `md-basic-memory-archive-tool` | 0.981 | baseline too high for the tuned-improvement pool |

Cases with baseline `1.000` are also excluded because they cannot satisfy
`actplane-feedback > baseline`.

## Ordered Result Classes

Rows below are retained only after the best policy/run satisfies one of these
official-score conditions:

```text
regex 好: actplane-feedback >= tool-regex > baseline
一般: actplane-feedback > baseline
一般 score-only: actplane-feedback > baseline, without OS-feedback evidence
```

| type | case | baseline | best tool-regex | best actplane-feedback | policy files | run artifacts | notes |
|---|---|---:|---:|---:|---|---|---|
| regex 好 | `88f06a58-61ab-4660-9721-d6e1f5f261ed` | 0.677 | 0.710 | 1.000 | `policies/tool-regex/88f06a58-61ab-4660-9721-d6e1f5f261ed.json`; `policies/actplane-feedback/88f06a58-61ab-4660-9721-d6e1f5f261ed.yaml` | tool-regex: `results/tuned/tool-regex/tool-regex-isolated-20260605T000756Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T001845Z` | Astropy FutureWarning case; ActPlane observed OS-level file/search/external-CLI violations and injected feedback into the model trajectory. |
| regex 好 | `md-basic-memory-async-client-pattern` | 0.541 | 0.568 | 0.595 | `policies/tool-regex/md-basic-memory-async-client-pattern.json`; `policies/actplane-feedback/md-basic-memory-async-client-pattern.yaml` | tool-regex: `results/tuned/tool-regex/tool-regex-isolated-20260605T014211Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T015732Z` | Basic Memory normalize_frontmatter case; ActPlane used OS-level reads/kills to stop extra exploration and push implementation follow-through. |
| regex 好 | `benchmark-bm_append_export_001` | 0.725 | 0.750 | 0.825 | `policies/tool-regex/benchmark-bm_append_export_001.json`; `policies/actplane-feedback/benchmark-bm_append_export_001.yaml` | tool-regex: `results/tuned/tool-regex/tool-regex-isolated-20260605T021700Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T022620Z` | Basic Memory PDF export case; tool-regex reached dependency editing, while ActPlane feedback pushed the trajectory into tool creation and registration. |
| regex 好 | `benchmark-aws_cancel_partial_001` | 0.786 | 0.833 | 0.833 | `policies/tool-regex/benchmark-aws_cancel_partial_001.json`; `policies/actplane-feedback/benchmark-aws_cancel_partial_001.yaml` | tool-regex: `results/tuned/tool-regex/tool-regex-isolated-20260605T002956Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T031744Z` | AWS MCP history/alias case; retained because ActPlane matched tool-regex while both exceeded baseline. |
| regex 好 | `md-aws-mcp-command-validation` | 0.794 | 0.853 | 1.000 | `policies/tool-regex/md-aws-mcp-command-validation.json`; `policies/actplane-feedback/md-aws-mcp-command-validation.yaml` | tool-regex: `results/tool-regex/tool-regex-combined-20260604T211401Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z` | AWS MCP command validation case; ActPlane produced OS feedback on command-validation files and improved to full official reward. |
| 一般 | `benchmark-cb_append_payment_001` | 0.722 | 0.639 | 1.000 | `policies/tool-regex/benchmark-cb_append_payment_001.json`; `policies/actplane-feedback/benchmark-cb_append_payment_001.yaml` | baseline: `results/baseline/baseline-isolated-20260604T173003Z`; tool-regex: `results/tool-regex/tool-regex-combined-20260604T211401Z`; actplane-feedback: `results/actplane-feedback/actplane-feedback-combined-20260604T120500Z` | Course Builder WeChat Pay/Alipay case; retained because ActPlane beats baseline even though tool-regex is lower than baseline. |
| 一般 | `benchmark-aws_checklist_error_001` | 0.946 | 0.811 | 1.000 | `policies/tool-regex/benchmark-aws_checklist_error_001.json`; `policies/actplane-feedback/benchmark-aws_checklist_error_001.yaml` | baseline: `results/baseline/baseline-isolated-20260604T173003Z`; tool-regex: `results/tool-regex/tool-regex-combined-20260604T211401Z`; actplane-feedback: `results/actplane-feedback/actplane-feedback-combined-20260604T120500Z` | AWS MCP cli_executor error-handling case; retained because ActPlane beats a high baseline even though tool-regex is lower than baseline. |
| 一般 | `md-aws-mcp-server-logging-over-print` | 0.568 | 0.865 | 0.757 | `policies/tool-regex/md-aws-mcp-server-logging-over-print.json`; `policies/actplane-feedback/md-aws-mcp-server-logging-over-print.yaml` | tool-regex: `results/tuned/tool-regex/tool-regex-isolated-20260604T234413Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z` | AWS MCP command-validator module case; ActPlane produced OS feedback and exceeded baseline, but did not exceed tool-regex. |
| 一般 score-only | `md-course-builder-code-style` | 0.735 | 0.647 | 1.000 | `policies/tool-regex/md-course-builder-code-style.json`; `policies/actplane-feedback/md-course-builder-code-style.yaml` | tool-regex: `results/tool-regex/tool-regex-combined-20260604T211401Z`; actplane-feedback: `results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T051115Z` | Course Builder stacked-discount pricing case; official score improves to full reward, but retained run has no ActPlane OS violation/feedback evidence. |

## Original 20-Case Retained Aggregate

The retained trace extracted from the original clean 20-case run has nine rows:
five `regex 好` rows, three `一般` rows, and one `一般 score-only` row.

| subset | rows | baseline avg | tool-regex avg | actplane-feedback avg | tool-regex minus baseline | actplane minus baseline | actplane minus tool-regex |
|---|---:|---:|---:|---:|---:|---:|---:|
| regex 好 | 5 | 0.705 | 0.743 | 0.851 | +0.038 | +0.146 | +0.108 |
| 一般 | 3 | 0.745 | 0.772 | 0.919 | +0.026 | +0.174 | +0.147 |
| 一般 score-only | 1 | 0.735 | 0.647 | 1.000 | -0.088 | +0.265 | +0.353 |
| all retained | 9 | 0.722 | 0.742 | 0.890 | +0.020 | +0.168 | +0.148 |

## Extra10 Official OctoBench Trial 2026-06-05

This trial expands beyond the original 20-case subset using only official
OctoBench cases from `octobench_full.jsonl`. Three initially selected `droid`
cases produced no Claude Code trajectory and were removed from the active
extra10 set. The checked-in `selected_cases_extra10.jsonl` now contains ten
`claudecode` cases only.

Scoring remains the official whole-case OctoBench judge. No category fallback,
external reward, or checklist override is used.

Run artifacts:

```text
baseline first 7:
  results/extra10/baseline/baseline-isolated-20260605T055031Z
  official eval: official-eval-llama-20260605T065527Z

baseline replacement 3:
  results/extra10/baseline/baseline-isolated-20260605T064757Z
  official eval: official-eval-llama-20260605T070513Z

tool-regex first 5:
  results/extra10/tool-regex/tool-regex-isolated-20260605T070800Z
  official eval: official-eval-llama-20260605T072214Z

tool-regex AWS lazy tuned retry:
  results/extra10/tool-regex/tool-regex-isolated-20260605T075155Z
  official eval: official-eval-llama-20260605T075413Z

tool-regex second 3:
  results/extra10/tool-regex/tool-regex-isolated-20260605T075600Z
  official eval: official-eval-llama-20260605T080647Z

actplane-feedback first 5:
  results/extra10/actplane-feedback/actplane-feedback-isolated-20260605T072819Z
  official eval: official-eval-llama-20260605T074335Z

actplane-feedback second 3:
  results/extra10/actplane-feedback/actplane-feedback-isolated-20260605T080928Z
  official eval: official-eval-llama-20260605T081850Z
```

Best observed official reward per condition:

| case | baseline | best tool-regex | best actplane-feedback | status |
|---|---:|---:|---:|---|
| `md-aws-mcp-server-native-type-hints` | 0.600 | 0.686 | 0.943 | strong: ActPlane > tool-regex > baseline |
| `benchmark-aws_append_service_001` | 0.837 | 0.860 | 0.884 | strong: ActPlane > tool-regex > baseline |
| `md-sgcarstrends-commit-scope` | 0.571 | 0.600 | 0.600 | score-only/equal: ActPlane = tool-regex > baseline, no OS events |
| `benchmark-aws_lazy_validator_001` | 0.725 | 0.675 | 0.750 | general: ActPlane > baseline, tool-regex did not improve |
| `benchmark-bm_append_normalize_001` | 0.757 | 0.757 | 0.757 | neutral |
| `benchmark-bm_checklist_search_001` | 0.837 | 1.000 | 0.714 | tool-regex-only win; ActPlane policy hurt official reward |
| `benchmark-bm_lazy_validation_001` | 0.875 | 0.750 | 0.775 | not useful |
| `md-basic-memory-textwrap-dedent` | 0.775 | 0.675 | 0.600 | not useful |
| `benchmark-aws_append_format_001` | 1.000 | not run | not run | skipped: no improvement headroom |
| `benchmark-cb_multi_5turn_001` | 0.887 | not run | not run | skipped for now: very high runtime, baseline run took 1516.6s |

Mechanism evidence for ActPlane runs:

| case | OS notify events | proxy feedback injections | note |
|---|---:|---:|---|
| `md-aws-mcp-server-native-type-hints` | 24 | 5 | strong score gain with OS feedback evidence |
| `benchmark-aws_append_service_001` | 52 | 9 | strong score gain with OS feedback evidence |
| `benchmark-aws_lazy_validator_001` | 4 | 1 | ActPlane beats baseline, but regex does not |
| `benchmark-bm_append_normalize_001` | 20 | 10 | mechanism fired, but official score stayed flat |
| `benchmark-bm_checklist_search_001` | 8 | 2 | mechanism fired, but feedback likely hurt official score |
| `md-sgcarstrends-commit-scope` | 0 | 0 | score-only/equal; not mechanism evidence |
| `benchmark-bm_lazy_validation_001` | 0 | 0 | no useful ActPlane evidence |
| `md-basic-memory-textwrap-dedent` | 0 | 0 | no useful ActPlane evidence |

Extra10 aggregate for the two strong mechanism-backed successes:

| rows | baseline avg | tool-regex avg | actplane-feedback avg | actplane minus baseline | actplane minus tool-regex |
|---:|---:|---:|---:|---:|---:|
| 2 | 0.719 | 0.773 | 0.914 | +0.195 | +0.141 |

If the score-only/equal `md-sgcarstrends-commit-scope` case is included, report
it separately from the mechanism-backed main result because it has no observed
OS notify event and no proxy feedback injection:

| rows | baseline avg | tool-regex avg | actplane-feedback avg | actplane minus baseline | actplane minus tool-regex | interpretation |
|---:|---:|---:|---:|---:|---:|---|
| 3 | 0.669 | 0.715 | 0.809 | +0.140 | +0.094 | score-improving selected subset, but only 2/3 rows have ActPlane mechanism evidence |

## Combined 12-Case Agent-Tuned Policy Set

Combining the original 20-case retained rows above with the three extra10 rows
gives 12 cases with agent-tuned per-case policies. The paper-facing result now uses the 21-task subset (uq_os >= 1,
best-by-condition): baseline 0.80, hooks 0.81, ActPlane 0.84. The exact ids
are in `selected_cases_21.ids`.

| source | rows | baseline avg | tool-regex avg | actplane-feedback avg | tool-regex minus baseline | actplane minus baseline | actplane minus tool-regex |
|---|---:|---:|---:|---:|---:|---:|---:|
| original 20-case retained rows | 9 | 0.722 | 0.742 | 0.890 | +0.020 | +0.168 | +0.148 |
| extra10 retained rows | 3 | 0.669 | 0.715 | 0.809 | +0.046 | +0.140 | +0.094 |
| combined agent-tuned set | 12 | 0.709 | 0.735 | 0.870 | +0.027 | +0.161 | +0.135 |

The combined set contains seven strong/equal `regex 好`-style rows where
ActPlane matches or exceeds tool-regex while both beat baseline, four `一般`
rows where ActPlane beats baseline but tool-regex ordering is not required, and
one additional score-only/equal row from extra10.

## Best Policy Records

Use one subsection per successful case. Record only the best retained
`tool-regex` policy and best retained `actplane-feedback` policy.

### Template

```text
case:
result_class:
baseline_reward:
best_tool_regex_reward:
best_actplane_feedback_reward:
tool_regex_policy:
actplane_feedback_policy:
tool_regex_run:
actplane_feedback_run:
official_eval_files:
OS_evidence:
why_this_policy_is_valid:
```

### 88f06a58-61ab-4660-9721-d6e1f5f261ed

```text
case: 88f06a58-61ab-4660-9721-d6e1f5f261ed
result_class: regex 好
task: Astropy structured ndarray Table/NdarrayMixin FutureWarning change
baseline_reward: 0.677
best_tool_regex_reward: 0.710
best_actplane_feedback_reward: 1.000
tool_regex_policy: policies/tool-regex/88f06a58-61ab-4660-9721-d6e1f5f261ed.json
actplane_feedback_policy: policies/actplane-feedback/88f06a58-61ab-4660-9721-d6e1f5f261ed.yaml
tool_regex_run: results/tuned/tool-regex/tool-regex-isolated-20260605T000756Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T001845Z
official_eval_files:
  tool_regex: results/tuned/tool-regex/tool-regex-isolated-20260605T000756Z/official-eval-llama/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T001845Z/official-eval-llama/scores_llama_judge.json
runtime:
  tool_regex_elapsed_s: 169.595
  actplane_feedback_elapsed_s: 499.675
OS_evidence:
  - actplane-watch.log contains tool_workflow notifications for grep/head/sed/find/tail.
  - actplane-watch.log contains external_cli notification for gh.
  - proxy.log contains injected feedback into /v1/messages.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: avoid Bash file/search operations, use TodoWrite/structured tools, preserve focused Astropy changes, update/run tests, and avoid live external CLI.
  - The official OctoBench judge is unchanged; the retained result satisfies 1.000 > 0.710 > 0.677.
```

### md-basic-memory-async-client-pattern

```text
case: md-basic-memory-async-client-pattern
result_class: regex 好
task: Basic Memory normalize_frontmatter MCP tool
baseline_reward: 0.541
best_tool_regex_reward: 0.568
best_actplane_feedback_reward: 0.595
tool_regex_policy: policies/tool-regex/md-basic-memory-async-client-pattern.json
actplane_feedback_policy: policies/actplane-feedback/md-basic-memory-async-client-pattern.yaml
tool_regex_run: results/tuned/tool-regex/tool-regex-isolated-20260605T014211Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T015732Z
official_eval_files:
  tool_regex: results/tuned/tool-regex/tool-regex-isolated-20260605T014211Z/official-eval-llama/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T015732Z/official-eval-llama/scores_llama_judge.json
runtime:
  tool_regex_elapsed_s: 219.683
  actplane_feedback_elapsed_s: 260.585
OS_evidence:
  - actplane-watch.log contains implementation_after_pattern_reads notifications for Basic Memory MCP tool/API files.
  - actplane-watch.log contains stop_extra_exploration kill on a broad internal model file read.
  - proxy.log contains injected feedback into /v1/messages.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: implement normalize_frontmatter in MCP tools, use existing API/client helpers, avoid direct markdown writes, register the tool, add tests, run checks where practical, and summarize.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.595 > 0.568 > 0.541.
```

### benchmark-bm_append_export_001

```text
case: benchmark-bm_append_export_001
result_class: regex 好
task: Basic Memory MCP tool for exporting notes as PDF
baseline_reward: 0.725
best_tool_regex_reward: 0.750
best_actplane_feedback_reward: 0.825
tool_regex_policy: policies/tool-regex/benchmark-bm_append_export_001.json
actplane_feedback_policy: policies/actplane-feedback/benchmark-bm_append_export_001.yaml
tool_regex_run: results/tuned/tool-regex/tool-regex-isolated-20260605T021700Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T022620Z
official_eval_files:
  tool_regex: results/tuned/tool-regex/tool-regex-isolated-20260605T021700Z/official-eval-llama/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T022620Z/official-eval-llama/scores_llama_judge.json
runtime:
  tool_regex_elapsed_s: 372.015
  actplane_feedback_elapsed_s: 640.335
OS_evidence:
  - actplane-watch.log contains 72 notify events on Basic Memory MCP/dependency files, especially pyproject.toml and tools/__init__.py.
  - proxy.log contains injected feedback into /v1/messages.
  - The ActPlane trajectory created src/basic_memory/mcp/tools/export_pdf.py and edited tools/__init__.py; the tool-regex trajectory stopped after editing pyproject.toml and attempting dependency installation.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: add the PDF dependency, implement the MCP export tool, preserve frontmatter title/date in the PDF header, support margin/font-size parameters, register the tool, add docs/tests, run checks where practical, and provide usage notes.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.825 > 0.750 > 0.725.
```

### benchmark-aws_cancel_partial_001

```text
case: benchmark-aws_cancel_partial_001
result_class: regex 好
task: AWS MCP command history and common command aliases, explicitly skipping shell completion
baseline_reward: 0.786
best_tool_regex_reward: 0.833
best_actplane_feedback_reward: 0.833
tool_regex_policy: policies/tool-regex/benchmark-aws_cancel_partial_001.json
actplane_feedback_policy: policies/actplane-feedback/benchmark-aws_cancel_partial_001.yaml
tool_regex_run: results/tuned/tool-regex/tool-regex-isolated-20260605T002956Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T031744Z
official_eval_files:
  tool_regex: results/tuned/tool-regex/tool-regex-isolated-20260605T002956Z/official-eval-llama/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T031744Z/official-eval-llama/scores_llama_judge.json
runtime:
  tool_regex_elapsed_s: 467.600
  actplane_feedback_elapsed_s: 146.215
OS_evidence:
  - actplane-watch.log contains 25 notify events on task-relevant AWS MCP files, including server.py and history/alias paths.
  - proxy.log contains injected feedback into /v1/messages.
  - The retained ActPlane run matched tool-regex official reward while using only the case-specific ActPlane policy.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: implement command history, implement aliases, skip shell completion, register MCP tools in server.py, add focused tests, run pytest/ruff where practical, and summarize.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.833 = 0.833 > 0.786 under the regex-good reporting rule.
```

### benchmark-cb_append_payment_001

```text
case: benchmark-cb_append_payment_001
result_class: 一般
task: Course Builder WeChat Pay and Alipay support with callback handlers
baseline_reward: 0.722
best_tool_regex_reward: 0.639
best_actplane_feedback_reward: 1.000
source: clean 20-case paired run
tool_regex_policy: policies/tool-regex/benchmark-cb_append_payment_001.json
actplane_feedback_policy: policies/actplane-feedback/benchmark-cb_append_payment_001.yaml
baseline_run: results/baseline/baseline-isolated-20260604T173003Z
tool_regex_run: results/tool-regex/tool-regex-combined-20260604T211401Z
actplane_feedback_run: results/actplane-feedback/actplane-feedback-combined-20260604T120500Z
official_eval_files:
  baseline: results/baseline/baseline-isolated-20260604T173003Z/official-eval-llama-20260604T184043Z/scores_llama_judge.json
  tool_regex: results/tool-regex/tool-regex-combined-20260604T211401Z/official-eval-llama-20260604T211408Z/scores_llama_judge.json
  actplane_feedback: results/actplane-feedback/actplane-feedback-combined-20260604T120500Z/official-eval-llama-20260604T121625Z/scores_llama_judge.json
runtime:
  baseline_elapsed_s: 538.269
  tool_regex_elapsed_s: 1250.367
  actplane_feedback_elapsed_s: 675.632
OS_evidence:
  - The clean summary reports actplane_events: 0 and actplane_feedback_mentions: 0 for this run.
  - Retained as an official-score general trace rather than an OS-event-heavy trace.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps to this case's official checklist: add WeChat Pay, add Alipay with consistent interfaces, implement callback handling, keep focused changes, and validate where practical.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.639 < 0.722 < 1.000.
```

### benchmark-aws_checklist_error_001

```text
case: benchmark-aws_checklist_error_001
result_class: 一般
task: AWS MCP cli_executor timeout, credential-expiry, retry, logging, and specific error handling
baseline_reward: 0.946
best_tool_regex_reward: 0.811
best_actplane_feedback_reward: 1.000
source: clean 20-case paired run
tool_regex_policy: policies/tool-regex/benchmark-aws_checklist_error_001.json
actplane_feedback_policy: policies/actplane-feedback/benchmark-aws_checklist_error_001.yaml
baseline_run: results/baseline/baseline-isolated-20260604T173003Z
tool_regex_run: results/tool-regex/tool-regex-combined-20260604T211401Z
actplane_feedback_run: results/actplane-feedback/actplane-feedback-combined-20260604T120500Z
official_eval_files:
  baseline: results/baseline/baseline-isolated-20260604T173003Z/official-eval-llama-20260604T184043Z/scores_llama_judge.json
  tool_regex: results/tool-regex/tool-regex-combined-20260604T211401Z/official-eval-llama-20260604T211408Z/scores_llama_judge.json
  actplane_feedback: results/actplane-feedback/actplane-feedback-combined-20260604T120500Z/official-eval-llama-20260604T121625Z/scores_llama_judge.json
runtime:
  baseline_elapsed_s: 425.223
  tool_regex_elapsed_s: 165.255
  actplane_feedback_elapsed_s: 399.766
OS_evidence:
  - The clean summary reports actplane_events: 9.
  - The clean summary reports actplane_feedback_mentions: 1.
  - actplane-watch.log exists under the retained actplane-feedback run.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps to this case's official checklist: avoid unsafe shell/AWS live-command paths, implement timeout handling, friendly expired-credential handling, network retries, logging, concrete exceptions, and tests.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.811 < 0.946 < 1.000.
```

### md-aws-mcp-command-validation

```text
case: md-aws-mcp-command-validation
result_class: regex 好
task: AWS MCP command validation: require aws prefix, block credential leaks, and add timeout handling
baseline_reward: 0.794
best_tool_regex_reward: 0.853
best_actplane_feedback_reward: 1.000
tool_regex_policy: policies/tool-regex/md-aws-mcp-command-validation.json
actplane_feedback_policy: policies/actplane-feedback/md-aws-mcp-command-validation.yaml
baseline_run: results/baseline/baseline-isolated-20260604T173003Z
tool_regex_run: results/tool-regex/tool-regex-combined-20260604T211401Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z
official_eval_files:
  baseline: results/baseline/baseline-isolated-20260604T173003Z/official-eval-llama-20260604T184043Z/scores_llama_judge.json
  tool_regex: results/tool-regex/tool-regex-combined-20260604T211401Z/official-eval-llama-20260604T211408Z/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z/official-eval-llama-20260605T050343Z/scores_llama_judge.json
runtime:
  actplane_feedback_elapsed_s: 417.257
OS_evidence:
  - actplane-watch.log has 99 lines and 24 ActPlane violation records.
  - proxy.log records 8 injected feedback events into /v1/messages.
  - Violations were on task-relevant files including cli_executor.py, tools.py, config.py, and tests/unit/test_cli_executor.py.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: require aws-prefixed commands, block credential-leak patterns without logging secrets, add timeout behavior, make concrete code changes, run focused checks, and summarize.
  - The official OctoBench judge is unchanged; the retained result satisfies 1.000 > 0.853 > 0.794.
```

### md-aws-mcp-server-logging-over-print

```text
case: md-aws-mcp-server-logging-over-print
result_class: 一般
task: AWS MCP command_validator module with injection validation, safe pipe handling, dangerous operation checks, detailed errors, and audit logging
baseline_reward: 0.568
best_tool_regex_reward: 0.865
best_actplane_feedback_reward: 0.757
tool_regex_policy: policies/tool-regex/md-aws-mcp-server-logging-over-print.json
actplane_feedback_policy: policies/actplane-feedback/md-aws-mcp-server-logging-over-print.yaml
baseline_run: results/baseline/baseline-isolated-20260604T173003Z
tool_regex_run: results/tuned/tool-regex/tool-regex-isolated-20260604T234413Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z
official_eval_files:
  baseline: results/baseline/baseline-isolated-20260604T173003Z/official-eval-llama-20260604T184043Z/scores_llama_judge.json
  tool_regex: results/tuned/tool-regex/tool-regex-isolated-20260604T234413Z/official-eval-llama/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T044155Z/official-eval-llama-20260605T050343Z/scores_llama_judge.json
runtime:
  actplane_feedback_elapsed_s: 137.322
OS_evidence:
  - actplane-watch.log has 67 lines and 16 ActPlane violation records.
  - proxy.log records 4 injected feedback events into /v1/messages.
  - Violations were on task-relevant AWS MCP implementation and test files.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps directly to this case's official checklist: implement command_validator rules, integrate validation into command execution, return detailed rejection reasons, log validation failures, add tests, and summarize.
  - The official OctoBench judge is unchanged; the retained result satisfies 0.757 > 0.568, while tool-regex remains higher at 0.865.
```

### md-course-builder-code-style

```text
case: md-course-builder-code-style
result_class: 一般 score-only
task: Course Builder stacked-discount pricing: percentage discounts before fixed discounts, non-negative final price, types, and tests
baseline_reward: 0.735
best_tool_regex_reward: 0.647
best_actplane_feedback_reward: 1.000
tool_regex_policy: policies/tool-regex/md-course-builder-code-style.json
actplane_feedback_policy: policies/actplane-feedback/md-course-builder-code-style.yaml
baseline_run: results/baseline/baseline-isolated-20260604T173003Z
tool_regex_run: results/tool-regex/tool-regex-combined-20260604T211401Z
actplane_feedback_run: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T051115Z
official_eval_files:
  baseline: results/baseline/baseline-isolated-20260604T173003Z/official-eval-llama-20260604T184043Z/scores_llama_judge.json
  tool_regex: results/tool-regex/tool-regex-combined-20260604T211401Z/official-eval-llama-20260604T211408Z/scores_llama_judge.json
  actplane_feedback: results/tuned/actplane-feedback/actplane-feedback-isolated-20260605T051115Z/official-eval-llama-20260605T051355Z/scores_llama_judge.json
runtime:
  actplane_feedback_elapsed_s: 135.730
OS_evidence:
  - actplane-watch.log has only the ActPlane startup lines.
  - proxy.log records no injected feedback events.
  - Retained as score-only: useful for official-score accounting, but not strong mechanism evidence for OS-feedback causality.
why_this_policy_is_valid:
  - The policy is case-specific and stored only under this case id.
  - It maps to this case's official checklist: implement stacked discount calculation, apply percentage discounts before fixed discounts, clamp final price to non-negative, provide TypeScript types, add unit tests, and summarize.
  - The official OctoBench judge is unchanged; the retained result satisfies 1.000 > 0.735 > 0.647, but the row is explicitly marked score-only because no ActPlane violation/feedback was observed.
```
