# Code Generation Validation — backtesting.py Case Study

> **Aggregated copy** of `~/own/amonic/minimax/api-patterns/code-generation-validation.md` (source-of-truth — read-only, source iter-33). Sibling-doc refs resolve within `references/api-patterns/`; fixture refs use absolute source paths (most fixtures are not aggregated per [`../INDEX.md`](../INDEX.md)). Aggregated 2026-04-29 (iter-11).

Verified 2026-04-29 with `MiniMax-M2.7-highspeed`. **Headline finding: M2.7 generates Python code that compiles cleanly and LOOKS idiomatic but FAILS at runtime due to hallucinated library APIs.** All 3 generated `backtesting.py` strategies failed with `ImportError` for fabricated indicator imports (`SMA`, `RSI`, `BollingerBands`) that don't exist in the actual library. Even with explicit API hints in the system prompt, subtler runtime bugs persist (pandas-vs-numpy type assumptions). **Production rule: end-to-end execution validation is mandatory for any M2.7-generated code — never trust without running in a sandbox.**

Closes F5 with a definitive negative result on autonomous code generation reliability.

## Test setup

3 strategies of varying complexity, generated by M2.7-highspeed without library API hints in the prompt (testing genuine library knowledge, not spec-following):

| Strategy              | Description                                        |
| --------------------- | -------------------------------------------------- |
| S1-sma-crossover      | 10/30-day SMA crossover (long when fast > slow)    |
| S2-rsi-meanreversion  | 14-period RSI mean-reversion (oversold/overbought) |
| S3-bollinger-breakout | 20/2-stdev Bollinger Bands breakout                |

4-layer validation harness:

| Layer | Check                                                          |
| ----- | -------------------------------------------------------------- |
| L1    | Code extraction — can we extract a Python block from response? |
| L2    | Syntactic validity — `compile()` succeeds                      |
| L3    | Runtime execution — Backtest.run() returns without exception   |
| L4    | Sensible metrics — n_trades > 0 (strategy actually traded)     |

The harness wraps the generated Strategy class with synthetic OHLCV data (deterministic, 500 bars) and runs `Backtest(data, strategy_class).run()` in a subprocess.

## Results

### Phase 1 — initial run (NO API hints in prompt)

| Strategy              | L1 extracted | L2 compiles | L3 runs | L4 sensible | Failure mode                                                              |
| --------------------- | ------------ | ----------- | ------- | ----------- | ------------------------------------------------------------------------- |
| S1-sma-crossover      | ✅           | ✅          | ❌      | —           | `ImportError: cannot import name 'SMA' from 'backtesting.lib'`            |
| S2-rsi-meanreversion  | ✅           | ✅          | ❌      | —           | `ImportError: cannot import name 'RSI' from 'backtesting.lib'`            |
| S3-bollinger-breakout | ✅           | ✅          | ❌      | —           | `ImportError: cannot import name 'BollingerBands' from 'backtesting.lib'` |

**3/3 ImportError on identical class of failure** — all three strategies tried to import indicator helpers (SMA, RSI, CrossOver, BollingerBands) from `backtesting.lib` that DO NOT EXIST in the actual library.

The actual `backtesting.lib` exports: `crossover` (lowercase), `cross`, `barssince`, `SignalStrategy`, `TrailingStrategy`, `resample_apply`, `compute_drawdown_duration_peaks`, `random_ohlc_data`, `OHLCV_AGG`, `plot_heatmaps`. **No SMA, no RSI, no BollingerBands.**

### Phase 2 — retry S1 with explicit API hints

System prompt augmented with:

```
CRITICAL API NOTES:
- backtesting.lib does NOT export indicator functions like SMA, RSI, BollingerBands, CrossOver.
- For SMA, define it inline: `def SMA(arr, n): return pd.Series(arr).rolling(n).mean()`
- For crossover, use `from backtesting.lib import crossover` (LOWERCASE) — this IS exported.
```

M2.7's revised output:

```python
from backtesting import Strategy
from backtesting.lib import crossover

def SMA(series, n):
    """Simple moving average (inline definition)."""
    return series.rolling(n).mean()  # ← BUG: assumes Series input

class SmaCrossover(Strategy):
    def init(self):
        self.sma10 = self.I(SMA, self.data.Close, 10)
        self.sma30 = self.I(SMA, self.data.Close, 30)

    def next(self):
        if crossover(self.sma10, self.sma30):
            self.buy()
        elif crossover(self.sma30, self.sma10):
            self.position.close()
```

**Runtime error**: `RuntimeError: Indicator "SMA(C,10)" error.` because `self.data.Close` is a numpy-like `_Array`, not a pandas Series — it lacks `.rolling()`. Correct fix: `return pd.Series(arr).rolling(n).mean()` (with explicit `pd.Series()` wrapping).

The hint corrected the major hallucinated imports but a SUBTLER type-assumption bug persisted. The model uses `series.rolling()` directly, assuming the input is already a pandas Series.

## Headline findings

### Finding 1: 🚨 M2.7 hallucinates library APIs with HIGH CONFIDENCE

The 3 generated strategies failed with the same class of error: imports of non-existent indicator helpers. The code structure was perfect (Strategy subclass, init/next methods, self.I() lazy registration), suggesting M2.7 has accurate knowledge of `backtesting.py`'s overall design — but invents specific helper names that don't exist.

Possible causes:

1. **Stale training data**: an older version of `backtesting.py` may have exported these helpers (unlikely given the library has been stable since ~2019).
2. **Library confusion**: M2.7 may be conflating `backtesting.py` with other Python finance libraries (`talib` exports `SMA`, `RSI`, `BollingerBands`; `bt` library has different idioms; `pyalgotrade` etc.).
3. **Pattern completion**: the model "expects" a finance library to export named indicators, so it fills in the most plausible ones without verification.

This adds a 5th instance to the campaign's "M2.7 fills in plausible details rather than admit uncertainty" pattern (alongside iter-9 JSON, iter-13 vision deliberation, iter-30 confidence handling, iter-32 Item attributions).

### Finding 2: 🚨 Even with explicit API hints, subtler bugs persist

Phase 2's hinted retry corrected the import hallucination but introduced a type-assumption bug: `series.rolling(n).mean()` assumes Series input, fails at runtime with the actual `_Array` type that backtesting.py provides.

The root issue: **M2.7 doesn't deeply understand the type contracts of the libraries it writes against**. It pattern-matches plausible Python at the syntactic and import level but doesn't reason about whether `self.data.Close` is a Series or an Array.

Production implication: hint-based prompting helps but doesn't eliminate the need for execution validation. The "M2.7 + system prompt = production-ready code" path is brittle.

### Finding 3: 🎯 Production rule — END-TO-END EXECUTION VALIDATION is mandatory

The L1 → L4 validation cascade revealed that pass-rates DROP STEEPLY at deeper layers:

| Layer                 | Pass rate  |
| --------------------- | ---------- |
| L1 code extraction    | 3/3 (100%) |
| L2 syntactic validity | 3/3 (100%) |
| L3 runtime execution  | 0/3 (0%)   |
| L4 sensible metrics   | 0/3 (0%)   |

**A linter or `python -c "compile(...)"` validation is INSUFFICIENT.** The code looks right at the syntax level. Only running the code in a sandbox with realistic data reveals the failure.

This is universal for LLM-generated code; iter-33 just confirms it for `backtesting.py` specifically. Any amonic service that uses M2.7 to generate runnable code MUST include a sandbox-execution harness:

````python
async def safe_codegen_with_validation(prompt: str, validation_harness: str) -> dict:
    """Generate code via M2.7, then VALIDATE END-TO-END before deploying.

    Returns a dict with code + validation results. NEVER trust code that hasn't
    passed L4 (sensible runtime behavior).
    """
    # Generate code from M2.7
    response = await call_minimax(prompt, max_tokens=4096, temperature=0.1)
    code_match = re.search(r"```(?:python)?\s*\n([\s\S]*?)\n```", response)
    if not code_match:
        return {"verdict": "FAIL_L1", "reason": "no code block extracted"}
    code = code_match.group(1).strip()

    # L2 — syntactic
    try:
        compile(code, "<llm-gen>", "exec")
    except SyntaxError as e:
        return {"verdict": "FAIL_L2", "code": code, "error": str(e)}

    # L3 + L4 — sandbox execution
    sandbox_path = Path("/tmp/llm_gen_sandbox.py")
    sandbox_path.write_text(validation_harness.format(generated_code=code))

    try:
        proc = subprocess.run(
            ["uv", "run", "--python", "3.13", "--with", "...required-deps...",
             "python", str(sandbox_path)],
            capture_output=True, text=True, timeout=60, check=False,
        )
        if proc.returncode == 0:
            return {"verdict": "PASS", "code": code, "stdout": proc.stdout}
        return {"verdict": "FAIL_L3", "code": code, "stderr": proc.stderr}
    except subprocess.TimeoutExpired:
        return {"verdict": "FAIL_TIMEOUT", "code": code}
````

Use this pattern for ANY M2.7-generated code that will run in production.

### Finding 4: 🆕 Library knowledge has a "depth ceiling" — surface idioms learned, deep API not

M2.7 clearly knows the general SHAPE of `backtesting.py`:

- ✅ Strategy subclass pattern
- ✅ `init()` for setup, `next()` for per-bar logic
- ✅ `self.I(indicator_func, ...)` for lazy indicator registration
- ✅ `self.buy()`, `self.sell()`, `self.position.close()` for trade actions
- ✅ Capitalized OHLCV column names

But fails on specific API details:

- ❌ Specific indicator imports (SMA, RSI, BollingerBands don't exist)
- ❌ Type contracts (`self.data.Close` is `_Array`, not Series)
- ❌ Crossover capitalization (`crossover` lowercase, not `CrossOver`)

This is consistent with the model having absorbed surface idioms from documentation/blog posts but lacking deep API knowledge from actual library source code.

**Production implication**: M2.7 is suitable for SCAFFOLDING code (overall structure, idiomatic shape) but NOT for finalized library-specific code. Use it to draft, then have a human or execution-validator finalize.

### Finding 5: 🆕 Reasoning tokens scale with code complexity (~1200-1900 tokens for 30-line strategy)

iter-33 measured:

| Strategy | Latency | Completion | Reasoning |
| -------- | ------- | ---------- | --------- |
| S1 SMA   | 33.8s   | 1956       | 1776      |
| S2 RSI   | 24.8s   | 1366       | 1198      |
| S3 BB    | 40.1s   | 2208       | 1916      |

Avg ~1600 reasoning tokens for ~30-line code outputs. This is much heavier than F2 trade signal (~464 reasoning tokens for JSON output) but lighter than F1 Sharpe ratio computation (~10K reasoning tokens). Code generation sits in the middle of the reasoning-budget spectrum.

**Production implication**: latency for M2.7-generated production code is ~30-40s per call. Acceptable for offline batch (overnight strategy library generation) but too slow for interactive REPL-style coding assistance.

## Implications

### For amonic strategy-prototyping workflow

```python
# ❌ WRONG — trust M2.7-generated code without validation
async def generate_strategy_dangerous(spec: str) -> str:
    response = await ask_minimax(f"Write a backtesting.py strategy for: {spec}")
    return extract_code(response)  # Could fail at runtime!


# ✅ RIGHT — generate, validate end-to-end, return ONLY if passes
async def generate_strategy_safe(spec: str) -> dict:
    response = await ask_minimax(
        f"Write a backtesting.py strategy for: {spec}",
        system_prompt=BACKTESTING_API_HINTS,  # Provide API context to reduce hallucination
    )
    code = extract_code(response)

    # End-to-end validation in sandbox
    result = await validate_in_sandbox(code, harness=BACKTEST_HARNESS)
    if result["verdict"] != "PASS":
        # Try ONE retry with the specific error in the prompt
        retry_response = await ask_minimax(
            f"The previous code failed with: {result['error']}\n\nFix the code:\n{code}",
            system_prompt=BACKTESTING_API_HINTS,
        )
        retry_code = extract_code(retry_response)
        retry_result = await validate_in_sandbox(retry_code, harness=BACKTEST_HARNESS)
        if retry_result["verdict"] == "PASS":
            return {"code": retry_code, "took_retry": True}
        return {"verdict": "FAIL", "errors": [result["error"], retry_result["error"]]}

    return {"code": code, "took_retry": False}
```

### For combining F5 with F1+F2+F3+F4 (full agentic flow extension)

The Tier F primitives now extend further:

- **F1**: Python does the math (don't ask M2.7 to compute)
- **F2**: M2.7 emits structured judgment (JSON trade signals)
- **F3**: M2.7 explains theory (textbook-quality grounding)
- **F4**: M2.7 retrieves from long context (SEC filings)
- **F5**: M2.7 SCAFFOLDS code (don't trust without execution validation)

The full pattern:

```python
async def end_to_end_strategy_research(thesis: str, market_data) -> dict:
    """Generate, validate, and explain a strategy from a research thesis."""
    # F4: extract relevant facts from research filings/literature
    facts = await retrieve_relevant_facts(thesis)

    # F3: generate theoretical justification
    theory = await ask_quant_question(f"What's the theoretical basis for {thesis}?")

    # F5: generate strategy code (with execution validation)
    strategy_result = await generate_strategy_safe(thesis)
    if strategy_result["verdict"] == "FAIL":
        return {"verdict": "STRATEGY_GEN_FAILED", "errors": strategy_result["errors"]}

    # F1: backtest the validated code (Python does the math)
    backtest_metrics = run_backtest(strategy_result["code"], market_data)

    # F2: emit structured judgment about backtest quality
    judgment = await emit_strategy_signal(backtest_metrics)

    return {
        "thesis": thesis,
        "theory": theory,         # F3 output
        "facts": facts,           # F4 output
        "code": strategy_result["code"],  # F5 output (validated)
        "metrics": backtest_metrics,      # F1 output (Python math)
        "judgment": judgment,             # F2 output
    }
```

This 5-leg orchestration is the canonical agentic flow for amonic-quant research workflows.

## Open questions for follow-up

- **Does providing the actual `backtesting.lib` API in context (e.g., paste the README) eliminate hallucinations?** Likely yes — but this is a "RAG over library docs" pattern, not pure model knowledge.
- **Does plain `MiniMax-M2.7` behave better or worse for code gen?** Untested. Per iter-28's finding that plain is faster for short outputs, plain might be a better dev-iteration choice if it has similar code quality.
- **Multi-shot iterative repair**: send the runtime error back to M2.7 and ask for a fix. How many rounds before convergence? Worth a follow-up probe.
- **Does the same hallucination pattern affect other Python libraries** (pandas, numpy, scipy)? Or is it specific to less-popular libs like backtesting.py? Test with a more mainstream library.
- **Cost vs Llama 3.1 70B local**: bigblack has GPU; local code-generation models (DeepSeek Coder, Llama Code) may produce more reliable code at lower cost. Comparative benchmark would be valuable.
- **Real EDGAR filing test for F4-extended**: combines F4 retrieval + F5 codegen — extract a strategy from a research note, generate the code, validate end-to-end.

## Provenance

| Probe                 | http_status | latency | completion_tokens | reasoning_tokens | L1  | L2  | L3  | L4  | Failure mode                                                |
| --------------------- | ----------- | ------- | ----------------- | ---------------- | --- | --- | --- | --- | ----------------------------------------------------------- |
| S1-sma-crossover      | 200         | 33.8s   | 1956              | 1776             | ✅  | ✅  | ❌  | —   | ImportError: SMA from backtesting.lib                       |
| S2-rsi-meanreversion  | 200         | 24.8s   | 1366              | 1198             | ✅  | ✅  | ❌  | —   | ImportError: RSI from backtesting.lib                       |
| S3-bollinger-breakout | 200         | 40.1s   | 2208              | 1916             | ✅  | ✅  | ❌  | —   | ImportError: BollingerBands from backtesting.lib            |
| S1-retry-hinted       | 200         | 25.2s   | (unrecorded)      | (unrecorded)     | ✅  | ✅  | ❌  | —   | RuntimeError: SMA indicator (Series.rolling AttributeError) |

Wall-clock for 3 parallel codegen + sequential validation: 43.9s.

Fixtures:

- [`fixtures/backtesting-codegen-iter33-2026-04-29.json`](~/own/amonic/minimax/api-patterns/fixtures/backtesting-codegen-iter33-2026-04-29.json) — initial 3 probes with full error traces
- (Retry not saved as fixture — diagnostic-only)

Verifier: autonomous-loop iter-33. 4 API calls (3 initial + 1 retry).
