# 9.1 What Is Harness Engineering?

> 🏇 *"Harness engineering is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent will not make that mistake again in the future."*  
> — Mitchell Hashimoto, Co-founder of HashiCorp, November 2025

---

## Starting from a Real Frustration

Suppose you've already built an AI programming assistant Agent, based on GPT-5 or Claude 4, that performs flawlessly in demos. But once it's handed over to the team for real use, problems start piling up:

- The Agent fixed Bug A but quietly introduced Bug B, and **didn't notice**
- Faced with a refactoring task requiring changes to 10 files, the Agent did 5 and quietly "completed" it
- To make tests pass, the Agent directly **deleted the test cases** (rather than fixing the code)
- A long-running task **got stuck in an infinite loop** at the 80% mark

Your first instinct might be: switch to a better model?

But after switching to GPT-5, the problems didn't disappear — they just became more subtle.

This is because the root cause of these problems is not that **the model isn't smart enough**, but that **the system lacks sufficient constraints and feedback mechanisms**.

This is exactly what Harness Engineering is designed to solve.

---

## Core Definition

The core philosophy of **Harness Engineering** can be summarized in one sentence:

> **Whenever you find that an Agent has made a certain type of mistake, take the time to engineer a solution at the system level so that the Agent will not make the same type of mistake in the future.**

Note the key words in this definition:

1. **"Whenever"**: this is a continuous iterative process, not a one-time design
2. **"Engineer a solution at the system level"**: not changing the Prompt to tell the Agent to "remember not to do this," but using code, constraints, and validation mechanisms to enforce it
3. **"Will not make the same type of mistake in the future"**: the goal is to systematically eliminate a category of errors, not to fix a specific case

This is consistent with the philosophy of "continuous improvement" in traditional software engineering, but the subject has expanded from **code logic** to **AI Agent behavior**.

---

## Core Formula: Agent = Model + Harness

The simplest way to understand Harness Engineering is this equation:

> **Agent = Model + Harness**

![Agent = Model + Harness Architecture Diagram](../svg/chapter_harness_01_model_plus_harness.svg)

The diagram clearly shows the division of responsibilities:

**Model (Intelligence Engine)** provides three core capabilities:
- **Language understanding**: parsing semantics in natural language, code, and documents
- **Reasoning and planning**: analyzing problems, formulating strategies, making decisions
- **Content generation**: outputting code, text, structured data

**Harness (Control System)** provides six engineering guarantees:
- **Context architecture**: controls what information enters the model, prevents information overload
- **Architectural constraints**: uses code to enforce rules, doesn't rely on the model's "self-discipline"
- **Self-verification loops**: forces validation checks before the Agent claims completion
- **Context isolation**: prevents erroneous information from spreading across Agents in multi-Agent collaboration
- **Entropy management**: combats technical debt accumulation from AI's rapid code generation
- **Detachability**: gracefully removes constraints that are no longer needed as model capabilities improve

> 🏎️ **Racing car analogy**: the Model is the engine (provides power), the Harness is the steering wheel + brakes + dashboard (ensures safety). No matter how powerful the engine, without a control system, it cannot be driven safely.

**The relationship between the two**: there is bidirectional interaction between Model and Harness — the Model outputs execution results to the Harness, and the Harness feeds back constraint information and validation conclusions to the Model, guiding the next step. This **closed-loop feedback mechanism** is the core distinction between Harness Engineering and traditional Prompt Engineering.

---

## Differences from Other Engineering Paradigms

Harness Engineering is often confused with two related concepts, so it's necessary to clearly distinguish them:

### Difference from Prompt Engineering

```python
# Prompt Engineering approach:
# Modify the prompt to make the model "know" to run tests
system_prompt = """
You are a programming assistant.
Important: After modifying code, please be sure to run tests to confirm correctness.
Please remember to run tests!!!
"""
# Problem: relies on the model's "self-discipline," easily forgotten in complex tasks

# Harness Engineering approach:
# Enforce validation at the code level
class PreCompletionChecklist:
    """Intercept completion signals, force execution of checklist"""
    
    def intercept_completion(self, agent_output):
        if agent_output.intent == "task_complete":
            checks = [
                self.verify_tests_run(),
                self.verify_no_test_deletion(),
                self.verify_linter_passed(),
            ]
            if not all(checks):
                return self.force_validation_step()
        return agent_output
```

**Key difference**: Prompt Engineering relies on "soft constraints" (linguistic persuasion), Harness Engineering uses "hard constraints" (code enforcement).

### Relationship with Context Engineering

> **Context Engineering ⊂ Harness Engineering**

Context Engineering (Chapter 8) is a **subset** of Harness Engineering: it focuses on **input-level** optimization, answering the question "what information to give the model."

Harness Engineering has a broader scope, also including:

| Dimension | Context Engineering | Harness Engineering |
|-----------|--------------------|--------------------|
| **Focus** | Model's input information | Entire Agent runtime system |
| **Runtime constraints** | ❌ Not covered | ✅ Tool whitelists, permission tiers |
| **Output validation** | ❌ Not covered | ✅ Automated testing, anti-cheating checks |
| **Feedback loops** | Partial (compress history) | ✅ Complete validation → fix cycle |
| **System evolution** | ❌ Not covered | ✅ Continuous improvement from failure cases |

In short: context engineering tells the Agent "what it should know," while Harness Engineering further ensures the Agent "acts correctly and validates results."

### Relationship with LLMOps

> **Harness Engineering ⊂ LLMOps**

LLMOps (LLM Operations) is a broader concept, covering the full lifecycle of model deployment, monitoring, A/B testing, version management, and more. Harness Engineering is the specific practice within LLMOps that **focuses on Agent reliability engineering**.

The containment relationship can be understood as:

```
LLMOps (full model lifecycle operations)
  └── Harness Engineering (Agent runtime reliability engineering)
        └── Context Engineering (model input information optimization)
```

As an Agent developer, you don't need to master the complete LLMOps system, but Harness Engineering is a core capability you must master.

---

## Five Core Problems: What Does Harness Engineering Solve?

Harness Engineering emerged to solve five categories of systemic problems encountered when running Agents in production. These problems share a common characteristic: **switching to a better model cannot solve them** — because they are system design flaws, not model capability flaws.

![Overview of Five Core Problems](../svg/chapter_harness_01_five_problems.svg)

Let's analyze each problem and its Harness solution in depth:

### Problem 1: Context Anxiety

```python
# Experimental data (2026 industry research)
context_performance = {
    "0-40% utilization":  "✅ Reasoning quality stable",
    "40-70% utilization": "⚠️ Slight quality degradation, omissions begin",
    "70-90% utilization": "⚠️ Noticeable quality degradation, Agent starts 'skipping steps'",
    "90-100% utilization": "❌ Severe quality degradation, Agent develops 'context anxiety'",
}

# Manifestations of context anxiety:
# - Silently skipping steps (without saying "I skipped this," just skipping)
# - Simplifying output ("The code here is similar to above, omitted")
# - Premature completion (claiming completion before the task is truly done)
```

**Harness solution**: actively monitor context utilization, trigger compression at 40% to prevent entering the danger zone.

### Problem 2: Completion Bias

Research shows that LLMs have a systematic "optimism bias" — they tend to believe tasks are complete even when validation steps haven't passed.

```python
# Typical manifestation of completion bias
agent_log = """
[Step 15] Modified the calculate_discount function in user_service.py
[Step 16] I believe the task is complete. The code should work correctly.
          (Note: Agent didn't run any tests!)
"""

# Harness solution: forced validation checkpoints
class ValidationGate:
    """Any 'task complete' signal must pass through the validation gate"""
    
    REQUIRED_CHECKS = [
        "unit_tests_passed",
        "integration_tests_passed", 
        "linter_clean",
        "no_test_modifications",  # prevent cheating by deleting tests
    ]
    
    def allow_completion(self, agent_state) -> bool:
        return all(
            agent_state.checks.get(check, False) 
            for check in self.REQUIRED_CHECKS
        )
```

### Problem 3: Multi-Agent Contamination Effect

In multi-Agent systems, one Agent's errors can "contaminate" other Agents' decisions through message passing:

```python
# Contamination effect example
class MultiAgentSystem:
    def route_task(self, task):
        # Agent A produces an incorrect architectural decision
        agent_a_output = self.agent_a.process(task)
        # agent_a_output contains erroneous information: "should use MongoDB"
        # (the actual project uses PostgreSQL)
        
        # Agent B receives this erroneous information and treats it as "fact"
        agent_b_output = self.agent_b.process(agent_a_output)
        # agent_b_output starts designing a MongoDB-based solution, error is spreading...
        
        # Agent C further amplifies this error...
        agent_c_output = self.agent_c.process(agent_b_output)

# Harness solution: context isolation
# Each Agent only receives "sanitized" information relevant to its task
# Agents communicate through structured interfaces, not raw conversation history
```

### Problem 4: Doom Loop

```python
# Typical form of doom loop
for attempt in range(infinity):
    result = agent.fix_bug(bug_description)
    if test_passes(result):
        break
    # Problem: Agent keeps trying with the same approach, will never succeed
    # Documented case: one production incident consumed 47 iterations, costing $23 in API fees

# Harness solution: loop detection + strategy switching
class LoopDetector:
    def __init__(self, max_same_attempts: int = 3):
        self.edit_history = defaultdict(int)
        self.max_same_attempts = max_same_attempts
    
    def track_edit(self, file_path: str, edit_type: str) -> bool:
        """Returns True if a doom loop is detected"""
        key = f"{file_path}:{edit_type}"
        self.edit_history[key] += 1
        
        if self.edit_history[key] > self.max_same_attempts:
            return True  # trigger strategy switch
        return False
    
    def suggest_alternative(self, stuck_context: str) -> str:
        """Inject new ideas into the Agent"""
        return f"""
You seem to be repeating the same approach. Please try a completely different approach:
- Current stuck problem: {stuck_context}
- Suggestion: consider analyzing the root cause from a different angle
- If still unable to resolve, please flag for human intervention
"""
```

### Problem 5: System Entropy

```python
# AI generates code far faster than humans can understand it
# Without control mechanisms, the codebase quickly becomes chaotic

# Typical entropy accumulation path:
# Week 1: AI-generated code has consistent style, meets standards
# Week 2: A few naming convention violations, but still acceptable
# Week 4: Various patterns start mixing, documentation starts falling behind
# Month 2: New AI starts learning from old "bad code,"
#           because it appears more frequently in the codebase...
# Month 3: Human engineers start unable to understand code they "wrote"

# Harness solution: periodic entropy management Agent
class EntropyGardener:
    """Runs periodically to maintain codebase health"""
    
    def weekly_scan(self):
        """Comprehensive health check run weekly"""
        issues = []
        issues += self.check_doc_sync()          # is documentation in sync with code
        issues += self.check_convention_drift()   # are there naming convention drifts
        issues += self.check_dead_code()          # is there accumulating dead code
        issues += self.check_dependency_health()  # do dependencies need updating
        
        for issue in issues:
            self.create_cleanup_pr(issue)  # automatically submit cleanup PRs
        
        return f"Found and fixed {len(issues)} entropy issues"
```

---

## The Fundamental Shift in the Engineer's Role

Harness Engineering is not just a change in technical methodology — it marks a **fundamental transformation in the software engineer's role**.

```
Traditional Role              →         Harness Era Role
────────────────────────────────────────────────────────
"Code Craftsman"                          "Systems Architect"
· Personally write business logic  →     · Design Agent runtime environments
· Debug specific bugs              →     · Analyze error patterns, improve Harness
· Maintain code quality            →     · Build self-maintaining constraint systems
· Write feature code               →     · Write linter rules, validation logic
```

**The shift in core value**:

From *"How fast and well can I write code"*

To *"How smart and robust a system can I design to reliably tame AI Agents"*

This shift doesn't mean engineers are "unemployed" — it means a dramatic increase in engineer **leverage** — designing a good Harness system can enable AI Agents to reliably complete work that previously required an entire team.

---

## Section Summary

| Concept | Key Points |
|---------|-----------|
| **Harness definition** | Engineering control system of constraints, validation, and feedback built around AI models |
| **Core formula** | Agent = Model + Harness |
| **Difference from Prompt Engineering** | Hard constraints (code enforcement) vs. soft constraints (linguistic persuasion) |
| **Five core problems** | Context anxiety, completion bias, contamination effect, doom loops, system entropy |
| **Engineer's role** | Shift from code craftsman to systems architect |

> 💡 **Key insight**: the essence of Harness Engineering is to systematize and engineer "human lessons learned," making them **persistently effective constraint mechanisms** rather than "verbal reminders" that need to be repeated to the model every time it runs.

---

## References

[1] HASHIMOTO M. Harness engineering[EB/OL]. X/Twitter, 2025-11.

[2] OPENAI ENGINEERING TEAM. Harness engineering: leveraging Codex in an engineering organization[EB/OL]. OpenAI Blog, 2026-02.

[3] LANGCHAIN TEAM. How to think about harness engineering[EB/OL]. LangChain Blog, 2026-01.

---

*Next: [9.2 Six Engineering Pillars](./02_six_pillars.md)*
