\section{Implementation and Experiments}\label{sec:experiments}

\subsection{Reference Implementation}

The correspondence is implemented in the Operon framework (\texttt{operon-ai}, MIT license). Key components:

\begin{itemize}[leftmargin=*]
\item \textbf{Architecture extraction}: \texttt{extract\_architecture(organism)} produces the $(G, \mathrm{Know}, \Phi)$ triple from any \texttt{SkillOrganism}.
\item \textbf{Compiler functors}: Five \texttt{CompilerFunctor} instances with automatic preservation verification.
\item \textbf{Certificate framework}: Self-verifying certificates with \texttt{certify()} and \texttt{verify()} methods on certifiable components (ATP budget, quorum sensing, mTOR scaling).
\item \textbf{Atomic skills catalog}: \texttt{seed\_library\_from\_atomic\_skills()} registers the five skills with correct topology mapping.
\item \textbf{LangGraph compiler}: \texttt{organism\_to\_langgraph()} creates one LangGraph node per stage, each calling \texttt{run\_single\_stage()}.
\end{itemize}

\subsection{Escalation Experiment}

To test whether structural guarantees are genuinely harness-level, we designed an experiment that changes the model while keeping the harness constant.

\paragraph{Setup.} A single-stage code review organism with:
\begin{itemize}[leftmargin=*]
\item \textbf{Fast model}: Phi-3 Mini (3.8B parameters) via Ollama
\item \textbf{Deep model}: Gemma 4 (27B MoE, 4B active) via Ollama
\item \textbf{VerifierComponent}: Rubric-based quality evaluation using Gemma~4 as judge
\item \textbf{WatcherComponent}: Verifier-threshold escalation after quality evaluation
\item \textbf{Quality threshold}: 0.6 (below this, the watcher fires ESCALATE)
\end{itemize}

The task is \texttt{hard\_par\_08}: a code review containing four subtle bugs (off-by-one pagination, TOCTOU cache race, float precision in financial math, \texttt{except Exception} swallowing \texttt{KeyboardInterrupt}).

\paragraph{Results.}

\begin{center}
\small
\begin{tabular}{@{}lccl@{}}
\toprule
\textbf{Phase} & \textbf{Model} & \textbf{Quality} & \textbf{Action} \\
\midrule
Initial execution & Phi-3 Mini (fast) & 0.50 & EXECUTE \\
Verifier evaluation & Gemma 4 (judge) & --- & Score = 0.50 \\
Watcher decision & --- & $0.50 < 0.60$ & ESCALATE \\
Escalated execution & Gemma 4 (deep) & --- & EXECUTE \\
\bottomrule
\end{tabular}
\end{center}

The escalation fires correctly: Phi-3 Mini produces a review that the judge scores below threshold, the watcher escalates to Gemma~4, and the deep model re-executes the stage. In this two-model, one-task experiment, the quality-based escalation control path is model-parametric: it is represented as harness structure ($\mathrm{Know}$) and executed under the chosen deployment map ($\Phi$).

\paragraph{Discrimination.} Independent validation of the task confirms discrimination between weak and strong models: Phi-3 Mini scores 0.72 (identifies bug classes but not concrete failure scenarios), Gemma~4 scores 1.00 (explains each failure scenario in detail). Quality delta = 0.28, exceeding the 0.20 threshold.

\subsection{Certificate Preservation Measurement}

We compile organisms with multiple certificates (ATP priority gating + quorum sensing no-false-activation + mTOR no-oscillation) across all four non-LangGraph compilers and measure preservation:

\begin{center}
\small
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Compiler} & \textbf{Certificates Preserved} & \textbf{Verified} \\
\midrule
Swarms & 3/3 (100\%) & 3/3 \\
DeerFlow & 3/3 (100\%) & 3/3 \\
Ralph & 3/3 (100\%) & 3/3 \\
Scion & 3/3 (100\%) & 3/3 \\
\bottomrule
\end{tabular}
\end{center}

Full certificate identity (theorem + parameters + source) is preserved across all four measured compilers, providing empirical evidence for the operational certificate-replay invariant. The LangGraph compiler preserves certificates via per-stage execution of \texttt{run\_single\_stage()} (Section~\ref{sec:preservation}).

\subsection{SWE-bench-lite: 8B Format Discipline Is the Ceiling}
\label{sec:swebench-phase2}

To probe whether the categorical harness delivers end-to-end value beyond certificate preservation, we ran an apples-to-apples comparison on SWE-bench-lite: 10 real Python bug-fix instances from \texttt{astropy} and \texttt{django}, with patches applied inside the official SWE-bench Docker harness and evaluated via \texttt{FAIL\_TO\_PASS} + \texttt{PASS\_TO\_PASS} test suites. An initial 2026-04-16 run was inconclusive: 28 of 30 submissions failed at \texttt{git apply}, mixing model weakness with prompt-format/pathing issues that we could not separate. We therefore built a \emph{patch-apply pipeline} (sanitizer + repository grounding) and reran. With grounding active, the failure mode shifts from \texttt{git apply} errors to honest \texttt{empty\_patch} rejections, and the run \emph{does} support a capability conclusion: at 8B / Q4\_K\_M, the model's diff-format discipline is below what SWE-bench-lite requires.

\paragraph{Setup.} Gemma~4 via Ollama (tag \texttt{gemma4:latest}, digest \texttt{c6eb396dbd59}, blob \texttt{sha256:4c27e0f5}, architecture \texttt{gemma4}, 8.0B parameters, Q4\_K\_M quantization, 131{,}072 context). The immutable digest is recorded in \texttt{eval/results/swebench\_phase2.json}. Three conditions:
\begin{itemize}[leftmargin=*]
\item \textbf{baseline}: direct single prompt, request unified diff.
\item \textbf{organism}: three-stage \texttt{SkillOrganism} (\textit{localize} $\to$ \textit{edit} $\to$ \textit{verify}).
\item \textbf{langgraph}: same organism, compiled via \texttt{organism\_to\_langgraph()} and executed as a LangGraph \texttt{StateGraph}.
\end{itemize}

The pipeline applied to all three conditions in this rerun:
\begin{itemize}[leftmargin=*]
\item Each instance's repository is shallow-cloned at its \texttt{base\_commit} into a local cache; up to five candidate file snippets selected by issue-text heuristics are injected into the prompt as repository context.
\item Model output is sanitized: paths normalized (e.g.\ stripping the \texttt{owner/repo/} prefix the model often hallucinates), hunk headers checked for placeholder line numbers (\texttt{@@ -XXX,N +XXX,N @@} style), hunk bodies validated against declared counts, and patches whose paths don't exist in the cloned tree fuzzy-corrected against unique basename matches or rejected. Patches that the sanitizer cannot make safe are dropped (recorded as \texttt{empty\_patch}, never forwarded to \texttt{git apply}).
\end{itemize}

\paragraph{Results (per-condition outcome distribution, 10 instances per condition).}

\begin{center}
\small
\begin{tabular}{@{}lccccccc@{}}
\toprule
\textbf{Condition} & \textbf{Resolved} & \textbf{Unresolved} & \textbf{Sanitizer-rejected} & \textbf{Runtime error} & \textbf{Evaluated} & \textbf{Mean latency$^*$} \\
\midrule
baseline    & 0 & 1 & 7  & 2 & 1/10 & 131~s \\
organism    & 0 & 0 & 10 & 0 & 0/10 & 170~s \\
langgraph   & 0 & 0 & 10 & 0 & 0/10 & 172~s \\
\bottomrule
\end{tabular}
\end{center}
\noindent$^*$Mean latency is computed over predictions that completed (the model returned text); the two baseline runtime errors were API timeouts where the call never returned and have no meaningful latency.

Categories: \texttt{resolved} (patch applied, target tests pass), \texttt{unresolved} (patch applied, tests fail), \texttt{sanitizer-rejected} (the model returned a diff-shaped output but the patch sanitizer dropped it for placeholder hunk headers, malformed counts, or invented paths --- harness sees this as \texttt{empty\_patch}), \texttt{runtime error} (the model call itself raised before returning, e.g.\ Ollama API timeout). \textbf{Evaluated} counts only instances that reached a pass/fail harness verdict (resolved + unresolved). Zero \texttt{git apply} errors occurred: sanitizer pre-rejection ensures no malformed patch reaches the harness.

\paragraph{Comparison with the 2026-04-16 run.} The earlier (un-grounded, un-sanitized) run had baseline 1/10 evaluated with 9 \texttt{error}, organism 1/10 with 5 \texttt{error} + 4 \texttt{empty\_patch}, langgraph 0/10 with 6 \texttt{error} + 4 \texttt{empty\_patch}. Two changes are notable. First, every \texttt{error} category collapsed to zero: the sanitizer refuses to forward patches that \texttt{git apply} would reject. Of the 27 model returns that reached the sanitizer in the rerun, 27 were dropped (placeholder hunks, malformed counts, or invented paths); the remaining 3 submissions across conditions either produced one applicable patch (django-11001 baseline) or never returned (the 2 baseline API timeouts on astropy-12907 and astropy-14995 --- these are recorded as \texttt{runtime\_error} rather than \texttt{empty\_patch} so a downstream reader can distinguish "model produced invalid output" from "model never produced output"). Second, the organism/baseline \texttt{empty\_patch} gap from the original run (4 vs.\ 0) closed: the tightened ``\texttt{[edit]} stage emits a single fenced diff and nothing else'' instruction eliminated the previous multi-stage format leak. The remaining 1-vs-0 gap (baseline produces one applicable diff, organism and LangGraph produce none) is consistent with the localize/edit decomposition asking the model to write a self-contained diff while juggling stage outputs, a discipline tax the single-shot baseline avoids.

\paragraph{Interpretation: 8B format discipline, not file selection, is the ceiling.} The original run confounded two failure modes: (a)~the model selecting nonexistent or wrong file paths, (b)~the model writing hunk headers and bodies that \texttt{git apply} cannot consume. Grounding reduced (a) but did not eliminate it: every prompt now contains the actual files at \texttt{base\_commit}, yet some outputs still name paths that do not exist in the cloned tree. The dominant remaining rejection is (b), hunk-format errors---placeholder hunk headers and mismatched header counts---which is separable from residual path errors. At this model scale, providing the actual files in the context window does \emph{not} translate to format-correct unified diffs. The 2 runtime errors are noise from the local Ollama API and do not refute this conclusion: those instances also failed via organism and langgraph (which completed normally), so the model never produced a usable diff for them either.

The single survivor, \texttt{django/django-11001} (baseline), reached the harness as \texttt{unresolved}: the patch applied cleanly but the test suite still failed. That same instance was the one survivor in the 2026-04-16 run as well, suggesting a structural property of that issue (likely a small, localized hunk) makes it more amenable to small-model output.

\paragraph{Latency cost of grounding.} Baseline mean latency (over the 8 completed calls) rose from 44~s in the original run to 131~s with grounding, organism from 88~s to 170~s, langgraph from 90~s to 172~s. The 30~KB of repository context in each prompt roughly doubles to triples per-call wall-clock with no compensating gain in evaluated instances at this model scale. Grounding's cost-benefit changes with stronger models, but at 8B it is overhead.

\paragraph{What this rerun does and does not say.} It does say: at 8B / Q4\_K\_M, the model's ability to emit \texttt{git apply}-clean unified diffs is the binding constraint on SWE-bench-lite resolution, and three-stage decomposition does not relax it. It does not say that the organism architecture is intrinsically worse than direct prompting --- with 1 vs.\ 0 evaluated instances, the experiment is still under-powered to discriminate. The harness-level \emph{structural} guarantees reported in Sections~\ref{sec:preservation} and in the escalation experiment are unaffected: they are $\mathrm{Know}$-level properties verified independently of SWE-bench task resolution.

This motivates Section~\ref{sec:conclusion}: the demonstrated transfer value of categorical harness engineering in this paper is in \emph{preservation} (certificate identity across compilers) and \emph{primitive reuse} (escalation), not in task-resolution gains on SWE-bench-lite at this model scale.

\subsubsection{Phase C: Cross-Model Check with Format-Correction Retry}
\label{sec:swebench-phase-c}

The v0.34.5 result above named two follow-up wedges: a stronger model and a format-correction retry loop. Paper 5's released code (v0.35.0) implements the retry and runs a cross-model check against a second locally-available 8B model, answering both wedges at once. The aim is not a better SWE-bench score --- prior results already place this outside the reach of 8B-class local models --- but a test of whether the 8B format-discipline ceiling reported above generalizes across training regimes, and whether a reason-coded retry prompt can recover any of the sanitizer-rejected submissions.

\paragraph{Setup.} \texttt{deepseek-r1:8b} via Ollama (tag \texttt{deepseek-r1:8b}, digest \texttt{6995872bfe4c}, blob \texttt{sha256:e6a7edc1}, architecture \texttt{qwen3} --- the Ollama weight is the DeepSeek-R1 distillation onto a Qwen3-8B base, 8.2B parameters, Q4\_K\_M quantization, 131{,}072 context). Same 10 SWE-bench-lite instances as the v0.34.5 run, same grounded prompt pipeline, three conditions (baseline, organism, langgraph). The new flag \texttt{-{}-retry-on-reject} is active: when the sanitizer rejects a submission, the runner re-prompts the model once with the specific rejection reason code (one of \texttt{placeholder\_hunk}, \texttt{truncated\_hunk}, \texttt{overlong\_hunk}, \texttt{path\_not\_found}, \texttt{ambiguous\_path}, \texttt{empty\_extraction}, \texttt{malformed\_metadata}) embedded in the retry prompt, followed by the failed output verbatim. Artifact at \texttt{eval/results/swebench\_phase2\_deepseek\_retry.json}.

\paragraph{Results (deepseek-r1 with retry, per-condition, 10 instances each).}

\begin{center}
\small
\begin{tabular}{@{}lccccccc@{}}
\toprule
\textbf{Condition} & \textbf{Resolved} & \textbf{Unresolved} & \textbf{Sanitizer-rejected} & \textbf{Runtime error} & \textbf{Evaluated} & \textbf{Mean latency} \\
\midrule
baseline    & 0 & 0 & 10 & 0 & 0/10 & 1084~s (18~min) \\
organism    & 0 & 0 & 10 & 0 & 0/10 & 1073~s (18~min) \\
langgraph   & 0 & 0 & 10 & 0 & 0/10 & 1130~s (19~min) \\
\bottomrule
\end{tabular}
\end{center}

Zero submissions reached the harness under any condition. Zero retry-recovered patches across all 30 submissions, even with targeted reason-code prompts. Zero runtime errors (the 900~s timeout introduced in v0.34.5 absorbs the long \texttt{<think>} blocks reasoning models produce; the mean per-call latency is 18--19~min vs gemma4's 2--3~min, but every call completed cleanly).

\paragraph{Comparison with gemma4 v0.34.5.}

\begin{center}
\small
\begin{tabular}{@{}lccc@{}}
\toprule
 & \textbf{gemma4:latest (v0.34.5)} & \textbf{deepseek-r1:8b (v0.35.0)} \\
\midrule
Total evaluated                   & 1/30 & 0/30 \\
Sanitizer-rejected                & 27   & 30 \\
Runtime errors                    & 2    & 0 \\
Resolved                          & 0    & 0 \\
Retry active                      & no   & yes \\
Retry-recovered patches           & ---  & 0 \\
Mean latency (baseline, completed)& 131~s & 1084~s \\
\bottomrule
\end{tabular}
\end{center}

Two observations.

First, \textbf{the ceiling is consistent across the two tested local 8B Q4\_K\_M models.} Both models (gemma4 instruction-tuned, deepseek-r1 reasoning-distilled-onto-Qwen3) produce zero resolved instances. Gemma4 crossed the sanitizer once; deepseek-r1 crossed it zero times. The single v0.34.5 ``survivor'' (\texttt{django-11001} baseline, \texttt{unresolved}) was gemma4-specific: deepseek-r1 also could not produce a \texttt{git apply}-clean diff for that instance under any of the three conditions. Within this two-model local setup, the 8B-class format-discipline ceiling reported in v0.34.5 is not a single-model artifact.

Second, \textbf{retry-with-reason-code did not break the ceiling}, and the reason distribution itself is informative. The v0.35 artifact records each submission's final sanitizer reason (per-instance \texttt{sanitize\_reason} field). For deepseek-r1:8b across 30 submissions: 26 \texttt{empty\_extraction} (the model's output contained no diff-shaped content at all), 3 \texttt{overlong\_hunk} (body content exceeded declared counts), 1 \texttt{truncated\_hunk} (body shorter than declared counts); zero submissions hit \texttt{placeholder\_hunk}, \texttt{path\_not\_found}, or \texttt{ambiguous\_path} (so the grounding-specific reason codes covering file-selection failure didn't fire even once --- consistent with the v0.34.5 claim that file selection is not the bottleneck). The retry mechanism itself is sound: the callback is invoked exactly when expected, the retry prompts embed the reason code and the failed output verbatim with reason-specific guidance (``use real integer line numbers'' for \texttt{placeholder\_hunk}; ``use exact paths from the repository context'' for \texttt{path\_not\_found}; ``respond with a single fenced diff block and nothing else'' for \texttt{empty\_extraction}; etc.), and the artifact records every submission's \texttt{retry\_attempted=true} with \texttt{retry\_recovered=false}. What the retry cannot do is move the model across the capability boundary. At 8B, the dominant failure mode is ``doesn't produce diff-shaped output in the first place'' (26/30), not ``produces diff-shaped output with fixable errors'' (4/30). Retry-with-guidance is calibrated for the second regime but the model sits mostly in the first. This is a sharper negative result than the v0.34.5 paper predicted (we suggested retry ``might recover 20--40\% of sanitizer drops''); at 8B, it recovers zero.

\paragraph{What Phase C does and does not say.} It says: (i)~the 8B format-discipline ceiling holds across training regimes, (ii)~reasoning-distilled training does not, on its own, improve diff-format production, (iii)~retry-with-targeted-guidance is the wrong lever for a capability ceiling and requires a model that is at least close to producing correct output --- which 8B models largely are not. It does not say retry is universally useless; stronger models that make occasional format mistakes may recover meaningfully. Cloud-GPU reruns against a 70B class model are the natural next wedge, outside this paper's local-only scope. The patch-apply pipeline (sanitizer + repo grounding + reason-coded retry) is instrumented for exactly that follow-up.
