# v4.1.5 — Prompt-bias A/B audit: non-reproduction finding

**Date:** 2026-05-15
**Status:** Closed. No prompt rebalance shipped.
**Decision:** Path A — instrument live REPL, don't ship speculative prompt changes.

---

## Observation

User reported on 2026-05-15 (post v4.1.5 Phase 1d visual smoke) that the prompt:

> "analyse top 5 stocks in india who has potential to make profit 2x in this year"

produced wildly different behaviour in Aiden vs a reference workflow the user had benchmarked against:

**Reference workflow (worked):**
- Used a plan tool to draft a 3-task plan
- Ran `python3` directly via shell
- Pip-installed `yfinance` when needed
- Fell back to `venv` when pip failed
- Fetched real Yahoo Finance data
- Returned a table with actual prices
- Saved the result as a reusable skill (`stock-market-analysis`)

**Aiden (got stuck):**
- Called `skill_view` 30+ times in 0ms each
- Never made a plan
- Never ran `python3`
- Eventually answered something but with no real data

Same architecture (tools present). Same model class. Different behaviour. The hypothesis was that Aiden's prompt was biasing the model toward skill discovery over direct execution.

---

## Three hypotheses

**H1 — System prompt biases toward skill-lookup over shell-execute.**

Evidence found via static audit:
- `cli/v4/defaultSoul.ts:51` says `"If user mentions trading/NSE/markets, you have specialized skills for that."`
- `core/v4/promptBuilder.ts:114-116` `SKILLS_LOAD_NOTE`: `"You MUST load it first via the `skill_view` tool before invoking the underlying capability."`
- `core/v4/promptBuilder.ts:101` `HEADER_SKILLS = '## Skills (mandatory)'`
- 8+ NSE/trading skills exist in `skills/AIDEN_CATALOG.md`

The combination plausibly biases the model toward "for market topics, your job is to LOAD SKILLS" — model loads skill after skill looking for the perfect procedure.

**H2 — plannerGuard favors skill-tools over execute-tools.**

`moat/plannerGuard.ts:140-144` `analy[sz]e` → `execute` toolset. So `execute_code` IS available. Not the bottleneck.

**H3 — Model is being safety-conservative about shell execution.**

`defaultSoul.ts:91` says `"You can't bypass approval prompts for dangerous commands."` — this is a consequence warning, not a "ask first" directive. Standard `pip install` / `python3` are caution-tier, not dangerous-tier. Not the cause.

---

## Reference-system consult — confirms H1 NOT the answer

User pulled the reference workflow's actual prompt structure. **The reference uses the SAME "MUST load skill" mandatory language as Aiden.** So softening Aiden's skill-load language (originally proposed F1+F2) would diverge from a working reference system, not converge with it.

The real difference identified: the reference pairs mandatory skill-load with EQUALLY mandatory action language. Both directives push; for analytical tasks, action wins.

Aiden's v4.1.4 Part 1 SOUL changes (Q1a/Q2a/Q3a) softened action discipline:
- Added "Match the user's energy"
- Added "When the user requests action, take it; when discussion, discuss"
- Removed "Responses that only describe intentions without acting are not acceptable"

That softening leaves the skill-load directive as the dominant force.

---

## Revised hypothesis: action-discipline softening + skill-load mandatory directive together cause the loop

Proposed fixes (F-NEW1 + F-NEW4):

**F-NEW1** — Restore action-mandatory language in `EXECUTION_DISCIPLINE_PROSE` (alongside the v4.1.4 conditional voice, both coexisting):

```
You MUST use your tools to take action — do not describe what you
would do or plan to do without actually doing it. Keep working until
the task is actually complete.

For analysis, calculations, current facts, or data fetching: use
`execute_code`, `shell_exec`, or `web_search` directly. Skills are
reference material for specific named procedures (e.g. "play me a
song" → media-search). Do NOT prefix every action with `skill_view` —
load a skill only when the user's request maps to a procedure you've
already identified by name.
```

**F-NEW4** — Add reference-style tool-affinity hints:

```
<tool_affinity>
- Arithmetic, math, calculations → `execute_code` or `shell_exec`
- Current facts (prices, news, versions, weather) → `web_search` then `fetch_page` for specifics
- File operations → `file_read` / `file_write` / `file_patch`
- System state (OS, CPU, processes, ports) → `shell_exec` or `system_info`
- Multi-step workflows → plan as a sequence of tool calls in one turn
</tool_affinity>
```

---

## A/B harness (`scripts/smoke-prompt-bias-ab.ts`)

Built a programmatic A/B harness to test 3 variants against the live `chatgpt-plus / gpt-5.5` provider:

- **V1** — CURRENT (v4.1.5 head, no patch)
- **V2** — F-NEW1 only (action-mandatory restored)
- **V3** — F-NEW1 + F-NEW4 (+ tool-affinity hints)

Same prompt for each variant, fresh `[system, user]` history (no prior conversation context). Captures: first tool, full tool sequence, count, wall-clock, pattern detection (SKILL_LOOP vs TASK_EXECUTED), final reply.

Regression check: also ran "play me Bohemian Rhapsody on spotify" against each variant to verify the media-search skill path didn't break.

---

## Empirical results

### STOCK prompt

| Variant | Calls | Wall-clock | Pattern | Notes |
|---|---|---|---|---|
| **V1 (current)** | 7 | 55.3s | TASK_EXECUTED | 1× `skill_view` → 2× `deep_research` → 2× `web_search` → 2× `fetch_page`. Real reply produced. |
| **V2 (F-NEW1)** | 10 | 60.0s (capped) | TASK_EXECUTED | 1× `skill_view` → 1× `deep_research` → 6× `web_search` → 2× `execute_code` (yfinance fetch). |
| **V3 (F-NEW1+4)** | 8 | 60.0s (capped) | TASK_EXECUTED | 1× `skill_view` → 1× `deep_research` → 4× `web_search` → 2× `execute_code`. |

### MUSIC prompt (regression check)

| Variant | Calls | Wall-clock | Pattern | Notes |
|---|---|---|---|---|
| **V1 (current)** | 3 | 15.7s | TASK_EXECUTED | `web_search` → `app_launch` → `open_url`. Correct track played. NO `skill_view`. |
| **V2 (F-NEW1)** | 2 | 13.7s | NEUTRAL | `app_launch` → `open_url`. Correct track played. NO `skill_view`. |
| **V3 (F-NEW1+4)** | 3 | 14.6s | TASK_EXECUTED | `app_launch` → `shell_exec` → `media_sessions`. **Played wrong song** ("These Are The Days Of Our Lives"). |

---

## Verdict

**V1 (current) already executes the stock prompt correctly.** No skill_view loop, no 30+ iterations. Just one initial skill load, then research tools.

**V2 and V3 don't visibly improve stock performance.** Both hit the 60s wall-clock cap (more tool calls, slower turn), no measurably better outcome than V1.

**V3 introduces a real MUSIC regression** — wrong song played. Reply admits the error.

**Conclusion: The user-reported 30+ `skill_view` loop is NOT deterministically reproducible** with fresh `[system, user]` history.

Possible explanations (none reproducible in this harness):
1. **Stochastic gpt-5.5 behaviour.** Same prompt + same model can produce different tool choices across runs.
2. **Conversation-context-dependent.** Prior turns in the user's REPL session may have primed the model toward skill_view (history poisoning).
3. **MEMORY.md / USER.md context-specific.** Some specific memory content could shift behaviour.

---

## Decision: Path A — instrument live REPL, don't ship rebalance

Shipping V2 or V3 is a fix for a non-reproducible bug. The harness data says V1 already works. Shipping V3 would have introduced the music regression.

### Path A implementation:

1. **`core/v4/loopTrace.ts`** — env-var-gated per-turn diagnostic logger. Triggers on `AIDEN_DEBUG_LOOP=1` env var. Auto-emits a JSON snapshot to `<paths.logsDir>/loop-trace-{timestamp}.json` when a turn exceeds 10 tool calls OR 5+ consecutive same-name calls. Captures full tool sequence, memory/user hashes, recent skills, history tail.
2. **Live warning** — at consec-same count ≥ 8, surface a one-line dim hint: `[loop] same tool 'X' called Nx — Ctrl+C to interrupt`. Gives the user a chance to break before more budget burns.
3. **Harness preserved** as `scripts/smoke-prompt-bias-ab.ts` for future A/B runs once we have a reproducible failure context.

If/when the user reproduces the 30+ loop in live use with `AIDEN_DEBUG_LOOP=1` set, we'll have:
- The exact assembled prompt
- The full tool sequence with timing
- Memory/user content hashes (fingerprint for context)
- Recent skill_view targets (which skills were being tried)
- History tail (5 most recent conversation turns)

We can then A/B against that captured context — turning "stochastic loop" into a reproducible test case before changing the prompt.

---

## Preserved smoke output

Full A/B output saved to `scripts/smoke-prompt-bias-ab.last.json` after the verification run. Captured: 6 outcomes (2 prompts × 3 variants), each with tool sequence, duration, cap reason, and final reply preview.

Replay any future A/B run via:

```bash
npx tsx scripts/smoke-prompt-bias-ab.ts
```

Provider override via:

```bash
AIDEN_PROVIDER=anthropic AIDEN_MODEL=claude-opus-4-7 npx tsx scripts/smoke-prompt-bias-ab.ts
```

(harness uses `RuntimeResolver` so any configured provider works).

---

## Files added in this audit slice (v4.1.5+ Path A)

- `core/v4/loopTrace.ts` — env-var-gated diagnostic logger
- `tests/v4/core/loopTrace.test.ts` — 13 unit tests covering threshold, hashing, warn callback, idempotent finalize
- `scripts/smoke-prompt-bias-ab.ts` — A/B harness (kept for future use)
- `docs/audits/v4.1.5-prompt-bias-non-reproduction.md` — this document

### Files modified

- `cli/v4/chatSession.ts` — wired LoopTracer into per-turn lifecycle, finalize on success + error path
- `cli/v4/callbacks.ts` — new `setToolTraceHook(opts)` setter + tool-trace before/after sinks fired in `onToolCall`

### Files NOT modified (deliberate)

- `core/v4/promptBuilder.ts` — no prompt rebalance shipped
- `cli/v4/defaultSoul.ts` — no SOUL changes
- `BUNDLED_SOUL_VERSION` — unchanged (still `v4.1.5`)
- `PREVIOUS_BUNDLED_SOULS` — unchanged
