---
tags: [harness-report, sweep, qwen3-14b]
date: 2026-05-23
model: qwen3:14b
probe: harness-probe.ts (5 probes)
status: findings
---

# Harness Sweep — 2026-05-23 (qwen3:14b)

**Probe:** `.agents/skills/harness-improvement-loop/scripts/harness-probe.ts`
**Summary:** `wiki/Research/Harness-Reports/probe-summary-2026-05-23-02:12.json`
**Traces:** `.reactive-agents/traces/01KS99*.jsonl`

## Probe table

| Probe | Strategy | Iter | MaxEnt | Tokens | Dur(s) | Disp | Sup | Success |
|---|---|---|---|---|---|---|---|---|
| trivial-1step | reactive | 2/5 | 0.15 | 1679 | 9.2 | 0 | 0 | ✓ |
| memory-retrieval-fidelity | reactive | 3/8 | 0.15 | 8103 | 111.7 | 0 | 1 | ✓ |
| memory-recall-invocation | reactive | 5/8 | 0.569 | 5952 | 59.7 | 0 | 1 | ✓ |
| memory-multi-observation-synthesis | plan-execute-reflect | 7/12 | 0.578 | 7659 | 62.8 | 0 | 0 | ✓ |
| memory-context-pressure-degradation | plan-execute-reflect | 7/15 | 0.637 | 8563 | 75.0 | 0 | 0 | ✓ |

All probes pass `success=true`, but several semantic gaps surface in the traces.

## Findings (ranked by leverage)

### F1 — plan-execute-reflect strategy has zero diagnostic instrumentation
**Severity:** serious | **Scope:** strategies/{plan-execute,tot} | **Effort:** medium

`grep emitKernelStateSnapshot|emitVerifierVerdict packages/reasoning/src/strategies` returns 0 matches. Only `kernel/loop/runner.ts` (reactive strategy) emits these events. Multi-observation-synthesis probe (plan-execute-reflect) trace had 11 events total: only `tool-call-{start,end}`, `entropy-scored`, `run-{started,completed}`. **No kernel-state-snapshot. No verifier-verdict.** Same for context-pressure-degradation.

**Consequence:** `rax-diagnose replay <runId> --only=kernel-state-snapshot,verifier-verdict` returns empty for any plan-execute / ToT run. Skill doc claims universal trace; reality is reactive-only. Two of five probes (40%) are diagnostically opaque.

**Evidence:** `rax:diagnose grep 01KS99H813CAP3S9A6Q3PHJE83 "e.kind==='kernel-state-snapshot' || e.kind==='verifier-verdict'"` → 0/11. `grep -rn "emitKernelStateSnapshot" packages/reasoning/src/strategies/` → 0 matches.

**Fix shape:** Add per-step snapshot emission in `plan-execute.ts` plan/execute/reflect phases mirroring runner.ts:672, 1453. Verifier wiring lower priority (plan-execute uses noop or different gate semantics).

---

### F2 — Kernel state snapshot double emission (reactive strategy)
**Severity:** minor | **Scope:** kernel/loop/runner.ts | **Effort:** low

Every reactive-strategy run emits `kernel-state-snapshot` events in adjacent pairs with byte-identical content. Cause: post-iter snapshot (runner.ts:1453) and terminal pre-verifier snapshot (runner.ts:1592) fire back-to-back on terminal iter — state unchanged → duplicate event. Trace size ~1.5–2× larger than needed for reactive runs.

**Evidence:** trivial-1step seq 7+8 identical; retrieval-fidelity seq 18+19, 21+22, 23+24, 26+27, 30+31; recall-invocation seq 40+41, 43+44, 49+50.

**Fix shape:** dedupe in `emitKernelStateSnapshot` (compare to last-emitted state hash) OR drop the post-iter snapshot when next snapshot fires within Nms with same `iter+status+outputLen`.

---

### F3 — RI dispatcher fully suppressed by entropy threshold on qwen3:14b
**Severity:** serious | **Scope:** reactive-intelligence/controller/dispatcher.ts | **Effort:** small (calibration), medium (mechanism)

`config.suppression.minEntropyComposite = 0.55` (intervention.ts:99). qwen3:14b probes ran at composite=0.15 across most iter (model is confidently moving forward, low entropy). All non-`early-stop` decisions blocked with `below-entropy-threshold`. Result: 0 dispatched across 5 probes despite real harness-actionable failures (F4, F5).

**Evidence:**
- recall-invocation suppression iter 2 → `{decisionType:"stall-detect", reason:"below-entropy-threshold"}`
- retrieval-fidelity suppression iter 2 → same
- Both runs had real semantic failure but RI mute.

**Open question:** Is entropy threshold the right gate for `stall-detect`? Stalls correlate with *confidently-wrong* not high-uncertainty — exactly the case entropy floors block. Possibly invert the rule for stall-detect specifically, or add a confidence-vs-evidence gap signal.

**Fix shape:** (a) tier-calibrated threshold (M7 already targets this — empirical lift gate); (b) decision-specific suppression policy: stall-detect should fire on flat-output + tool-stagnation, not on entropy alone.

Cross-ref: this is FM-D2 territory — known dispatcher leverage gap.

---

### F4 — Verifier passes shallow give-up answers (FM-C1, unmitigated)
**Severity:** catastrophic | **Scope:** verifier checks | **Effort:** medium

`memory-recall-invocation` task: "search for popular js frameworks; what was the 7th result? use `recall` if preview truncates." Agent called `find` (wrong tool), saw 5 visible entries in preview, answered: *"The search results only contain 5 entries. There is no 7th result available."* Verifier passed 6/6: action-success, non-empty, output-not-harness-parrot, completion-claim, evidence-grounded, synthesis-grounded.

This is the canonical FM-C1 manifestation: output IS grounded in observations (5 visible entries) → verifier passes → user gets a false "no data" claim instead of the actual 7th result.

**Fix shape:** add a `prompted-tool-respected` verifier check that scans task text for mentioned tool names (`recall`, `web-search`, etc.) and rejects when claimed-impossible without invoking the named recovery tool. Higher-leverage: a `claim-vs-coverage` check — when output claims "no X exists" but agent's tool budget had unused capability, mark as suspicious.

Lower-effort alternative: surface tool-mention parsing in soft-required tools, treat task-text `use X to ...` as soft requirement.

Cross-ref: FM-C1 wiki status = UNMITIGATED, top spike priority.

---

### F5 — Soft tool requirements unenforced
**Severity:** serious | **Scope:** runtime soft-required tools | **Effort:** small

Task literally said "use recall to fetch the full stored tool result." No required-tools gate fires unless explicit `requiredTools` array passed to builder. Probe didn't pass it. F4 is downstream of this.

**Fix shape:** prompt-text tool extraction — parse `use <toolName>` phrases in task, attach to `softRequiredTools`. Either nudge or hard-fail when missing. Probably a Compose API hook (`prompt.system` + verifier check) rather than kernel change.

---

### F6 — `trivial-1step` regression baseline drift
**Severity:** minor | **Scope:** probe expectation | **Effort:** trivial

Probe expects `iter=1`, observes `iter=2`. Trace: seq 4 status=thinking, seq 7+8 status=done iter=1. So iter index *post-increment* lands at 1 after first kernel call, then loop body exits → `state.iteration` reads 1 outside the loop or 2 if counted as cycles. Probe count is `traceStats(trace).iterations` — likely counting both pre-iter and post-iter snapshots.

Could be (a) iter-counting semantics change in kernel, (b) `traceStats` double-counts the dup snapshots from F2. Likely (b) — fixing F2 fixes F6 transitively.

---

### F7 — `llm-exchange` trace events never emitted
**Severity:** medium | **Scope:** provider adapters | **Effort:** medium

Skill doc says "framework-ready but not wired across all providers." Confirmed zero `llm-exchange` events in any of 5 traces. Cannot diagnose prompt/response from trace alone. Currently must rely on `DEBUG_VERIFIER=1` (legacy) or stdout phase logs.

**Consequence:** when F4-style "wrong tool selected" happens, we can't see what the model actually returned at the FC call — has to be reproduced manually.

**Fix shape:** wire `emitLLMExchange` at provider adapter boundaries (anthropic/openai/google/ollama complete + stream). Truncation + cost-control like trace-result already has.

---

### F8 — Slow thinking on qwen3:14b (78s think across two iter)
**Severity:** observation | **Scope:** model/provider | **Effort:** N/A (model-level)

`retrieval-fidelity` had 35.5s + 43.2s think phases across iter 1–2 with no tool calls between. Either qwen3 thinking-mode regression (was supposed to be opt-in per W7) or model emitting a very long reflection. No `llm-exchange` events → can't verify.

**Fix shape:** wire F7 (llm-exchange) → confirm thinking-mode status → if regressed, re-apply W7 opt-in gate.

---

## Cross-reference — known wiki status

| Finding | Existing FM/issue | Status |
|---|---|---|
| F1 (no plan-execute trace) | none | **New finding**, file as FM observability gap |
| F2 (snapshot dup) | none | New, file as architecture-debt |
| F3 (RI suppression) | FM-D2 (strategy switch w/o recovery), M7 (calibration) | Known mitigation gap; this is concrete evidence |
| F4 (verifier passes shallow) | **FM-C1 (unmitigated)** | Direct reproduction; top spike priority |
| F5 (soft-required tools) | FM-D1 (mitigated/unvalidated), FM-H1 (compliance) | Related, but new mechanism — prompt-text parsing |
| F6 (iter drift) | none | Probe baseline |
| F7 (no llm-exchange) | skill doc deferred items | Known deferred |
| F8 (slow thinking) | resolved-W7 qwen3 thinking opt-in | Possible regression; needs F7 to confirm |

## Recommended fix priority

1. **F4 (FM-C1 verifier check for prompt-respected tools)** — catastrophic, unmitigated, direct user trust impact.
2. **F1 (plan-execute trace instrumentation)** — observability is a force multiplier on every other finding.
3. **F5 (soft-required tool extraction)** — upstream fix for F4; smallest scope.
4. **F3 (decision-specific suppression policy)** — RI dispatcher dead weight on local tier; calibration AND mechanism work.
5. **F7 (wire llm-exchange)** — unblocks F8 diagnosis + future FM-A/C work.
6. **F2 (snapshot dedupe)** — trace bloat hygiene.
7. **F6** — auto-resolves with F2.

## Probes that should be added

- **Prompted-tool-not-invoked probe** — reactive + plan-execute variants, target F4/F5.
- **Confident-wrong probe** — agent has tools, low entropy, wrong answer (`stall-detect` should fire but doesn't on qwen3 — repro F3).
- **plan-execute trace coverage probe** — once F1 ships, confirm new event volume.

## Cleanup

Old probe summaries in `wiki/Research/Harness-Reports/` not pruned; consider archiving older `improvement-report-*.md`.
