<!--
  Title: Miniforge.ai
  Author: Christopher Lester (christopher@miniforge.ai)
  Copyright 2025-2026 Christopher Lester. Licensed under Apache 2.0.
-->

# Workflow Optimization: Systems Thinking Foundations

This document explores the theoretical foundations for treating SDLC workflows as
optimizable systems, drawing from petri nets, logistics, supply chain management,
and systems thinking.

## Core Insight

**The SDLC workflow itself is a heuristic to be optimized.**

Just as we version and improve prompts, thresholds, and repair strategies, we should
version and improve the workflow configuration that orchestrates the entire system.

## Theoretical Foundations

### 1. Petri Nets

**What they are:** Mathematical modeling language for distributed systems.

**Components:**

- **Places** (circles): States where tokens (artifacts) can reside
- **Transitions** (bars): Actions that consume/produce tokens
- **Arcs**: Connections showing token flow
- **Tokens**: Items flowing through the network
- **Guards**: Conditions for transition firing

**Mapping to miniforge:**

| Petri Net Concept | miniforge Equivalent |
|-------------------|---------------------|
| Place | Phase (state where artifact exists) |
| Transition | Agent action (transforms artifact) |
| Token | Artifact (code, tests, docs) |
| Arc | phase/next (allowed flows) |
| Guard | Gate (validation before transition) |

**Example:**

```text
[Input Spec] --validate--> [Plan] --design--> [Design] --implement--> [Code]
                             ↓                   ↓                      ↓
                          gates pass         gates pass             gates pass
```

**Analysis techniques:**

1. **Reachability**: Can we reach the "Merged PR" state from "Input"?
   - Useful for validating workflow configs
   - Detect impossible-to-complete workflows

2. **Liveness**: Will the workflow eventually complete?
   - Check for deadlocks (stuck waiting for approval)
   - Ensure every phase can progress

3. **Boundedness**: Is there a max number of artifacts in any phase?
   - Prevents resource exhaustion
   - Ensures bounded token usage

4. **Fairness**: Does every agent get a chance to run?
   - Prevents starvation
   - Ensures balanced workload

**Research Questions:**

- Can we automatically detect deadlocks in workflow configs?
- Can we prove that every workflow configuration is live (eventually completes)?
- Can we calculate the state space (all possible execution paths)?
- Can we find the minimal workflow that guarantees quality?

### 2. Queuing Theory

**What it is:** Mathematical study of waiting lines and service processes.

#### Classic model: M/M/1 queue

- Arrival rate (λ): Tasks entering the system
- Service rate (μ): Tasks completed per unit time
- Utilization (ρ = λ/μ): System load

#### Little's Law

```text
L = λW
```

Where:

- L = Average number of items in system
- λ = Arrival rate
- W = Average time in system

**Mapping to miniforge:**

| Queue Theory Concept | miniforge Equivalent |
|----------------------|---------------------|
| Queue | Review backlog, PR queue |
| Server | Agent (reviewer, implementer) |
| Service time | Phase duration |
| Arrival rate | New tasks per hour |
| Utilization | Agent busy % |

**Analysis:**

1. **Optimal parallelism**: Should we run phases in parallel?
   - More parallelism = lower wait time
   - But higher token cost (multiple agents running)

2. **Review queue size**: How many reviewers should we have?
   - Too few = long wait times
   - Too many = expensive, context switching

3. **Priority queuing**: Should critical bugs skip ahead?
   - Priority queue vs FIFO
   - Weighted fair queuing

**Research Questions:**

- What's the expected wait time at each phase?
- How do we balance speed vs cost (parallel vs sequential)?
- What's the optimal review queue size for our task arrival rate?
- Can we predict completion time based on current queue state?

### 3. Supply Chain Optimization

**What it is:** Managing flow of goods through production and distribution networks.

**Key metrics:**

- **Throughput**: Items processed per unit time
- **Lead time**: Time from order to delivery
- **WIP** (Work In Progress): Items currently being processed
- **Inventory**: Items waiting in queues
- **Quality**: Defect rate, rework rate
- **Cost**: Total cost of production

**Mapping to miniforge:**

| Supply Chain Concept | miniforge Equivalent |
|----------------------|---------------------|
| Station | Phase (processing station) |
| Operation | Agent action (value-add) |
| Inventory | Artifacts in phase queues |
| Throughput | Tasks completed per hour |
| Lead time | Time from spec to merged PR |
| Quality | Bugs escaped to production |
| Rework | Inner loop repair cycles |
| Cost | Token usage + time |

#### The Goal: Minimize cost while maximizing throughput and quality

#### Theory of Constraints (Goldratt)

1. **Identify the constraint** (bottleneck phase)
2. **Exploit the constraint** (max utilization)
3. **Subordinate everything** to the constraint
4. **Elevate the constraint** (add capacity)
5. **Repeat**

**For miniforge:**

1. Which phase takes longest? (bottleneck)
2. How can we make that phase faster?
3. Should we allocate more budget to bottleneck phases?
4. Can we parallelize the bottleneck?
5. Monitor and adjust as bottleneck shifts

**Six Sigma / Lean Manufacturing:**

- **Reduce waste**: Eliminate phases that don't add value
- **Reduce variation**: Make repair cycles more predictable
- **Continuous improvement**: Meta loop = Kaizen

**Research Questions:**

- What's the bottleneck phase in our workflow?
- Can we reduce WIP by limiting parallel phases?
- What's the optimal batch size (how many tasks to process together)?
- How do we balance quality vs speed (coverage threshold vs iteration count)?

### 4. Systems Thinking

**What it is:** Understanding how components interact to produce system behavior.

**Key concepts:**

1. **Stocks and flows**:
   - Stocks: Artifacts at rest (in phases)
   - Flows: Artifacts moving (transitions)
   - Feedback loops: Review loops, repair cycles

2. **Feedback loops**:
   - **Positive feedback**: Repair fails → more iterations → more failures
   - **Negative feedback**: High bugs → stricter gates → fewer bugs

3. **Delays**:
   - Review approval delays
   - CI run delays
   - PR merge delays

4. **Archetypes** (common patterns):
   - **Limits to Growth**: Early phases fast, but bottleneck in review
   - **Shifting the Burden**: Quick fixes instead of addressing root cause
   - **Fixes that Fail**: Lowering coverage threshold temporarily helps, but increases bugs

**Leverage points** (where to intervene):

1. **Parameters** (weak): Adjust thresholds, timeouts
2. **Feedback loops** (medium): Add/remove review loops
3. **Structure** (strong): Change phase order, merge/split phases
4. **Paradigm** (strongest): Rethink what "quality" means

**Research Questions:**

- What are the dominant feedback loops in our workflow?
- Where are the high-leverage intervention points?
- Can we simulate workflow changes before deploying?
- How do delays propagate through the system?

### 5. Constraint Programming

**What it is:** Declarative approach to optimization by stating constraints.

**Approach:**

1. Define **variables**: Phase order, review loops, gates
2. Define **domains**: Allowed values for each variable
3. Define **constraints**: Rules that must hold
4. **Solve**: Find assignment that satisfies all constraints

**Hard constraints** (must be true):

- Every workflow must start with :input phase
- Every workflow must end with :observe phase
- Tests must be implemented before production code (if TDD)
- Review must happen before release
- All gates must pass before transition

**Soft constraints** (preferences):

- Minimize total time
- Minimize total token usage
- Maximize test coverage
- Minimize repair cycles
- Balance reviewer workload

**Research Questions:**

- Can we auto-generate optimal workflows from constraints?
- Can we prove no valid workflow exists for certain constraint sets?
- Can we find the Pareto frontier (time vs quality vs cost tradeoffs)?

### 6. Evolutionary Algorithms

**What it is:** Optimize through mutation, crossover, and selection.

**Genetic algorithm for workflows:**

1. **Population**: Multiple workflow configurations
2. **Fitness**: Success rate, time, tokens, quality
3. **Selection**: Keep best-performing workflows
4. **Crossover**: Combine phases from two workflows
5. **Mutation**: Random changes (add/remove phase, swap order)
6. **Iterate**: Evolve over generations

**Example mutations:**

- Add a review loop to a phase
- Remove a phase entirely
- Swap phase order (if dependencies allow)
- Merge two phases into one
- Split one phase into two
- Adjust budget parameters

**Example crossover:**

```text
Parent 1: [Input → Plan → Design → Test → Code → Review → Release]
Parent 2: [Input → PlanDesign → TDD → Review → Release]

Child:    [Input → Plan → Design → TDD → Review → Release]
          (took Plan+Design split from P1, TDD from P2)
```

**Research Questions:**

- Can we discover novel workflows humans wouldn't design?
- What's the right fitness function (multi-objective)?
- How do we ensure evolved workflows are interpretable?
- Can we constrain evolution to preserve critical phases?

### 7. Reinforcement Learning

**What it is:** Learn optimal policies through trial and error.

**RL formulation:**

- **State**: Current phase, artifact quality, budget remaining
- **Action**: Which next phase to transition to
- **Reward**: +1 for successful merge, -1 for bugs in production
- **Policy**: Which action to take in each state

**Approaches:**

1. **Q-learning**: Learn value of (state, action) pairs
2. **Policy gradient**: Directly learn policy function
3. **Actor-Critic**: Combine both

**Meta-learning (learning to learn):**

- Learn which workflow to use for which task type
- Learn how to adjust workflows based on context
- Transfer learning from one task type to another

**Research Questions:**

- Can RL discover better phase transition policies?
- Can we learn task-specific workflows (feature vs bugfix)?
- Can we learn from failures (bugs in production) to improve workflow?
- How do we balance exploration (try new workflows) vs exploitation (use best known)?

## Integration with Meta Loop

The Meta loop orchestrates workflow optimization:

```text
┌────────────────────────────────────────────────────────┐
│ Meta Loop: Workflow Optimization                       │
│                                                         │
│ 1. Observe: Collect metrics from runs                  │
│    - canonical-sdlc-v1: 80% success, 45min avg, 50k tokens │
│    - lean-sdlc-v1: 75% success, 20min avg, 30k tokens  │
│                                                         │
│ 2. Analyze: Find bottlenecks and patterns              │
│    - Review phase takes 40% of total time               │
│    - Inner loop repairs in implement-code: avg 8 cycles │
│                                                         │
│ 3. Hypothesize: Propose new workflow variant           │
│    - canonical-sdlc-v1.1: Reduce review max-rounds to 2 │
│    - Theory: Saves time without impacting quality      │
│                                                         │
│ 4. Experiment: A/B test new variant                    │
│    - Run 10 tasks on v1.0.0                            │
│    - Run 10 tasks on v1.1.0                            │
│                                                         │
│ 5. Measure: Compare outcomes                           │
│    - v1.1.0: 79% success, 35min avg, 48k tokens       │
│    - Result: Faster, slightly lower success           │
│                                                         │
│ 6. Decide: Keep, promote, or discard                   │
│    - If time is more important: Promote v1.1.0         │
│    - If quality is more important: Keep v1.0.0         │
│                                                         │
│ 7. Store: Version the workflow in heuristic store      │
│    - save-heuristic :workflow-config "1.1.0" {...}    │
└────────────────────────────────────────────────────────┘
```

## Experiments to Run

### Phase 1: Baseline Metrics

Run canonical-sdlc-v1 on diverse tasks:

- 10 small features
- 10 bug fixes
- 10 refactorings
- 5 new components

Collect:

- Success rate per task type
- Time distribution (avg, p50, p95)
- Token usage distribution
- Repair cycles per phase
- Bugs caught in review vs CI vs production

### Phase 2: Variant Testing

Test alternative workflows:

- **lean-sdlc-v1**: Fewer phases, faster
- **rigorous-sdlc-v1**: More review loops, higher quality
- **concurrent-sdlc-v1**: Parallel phases, faster but more tokens

Compare on same task set.

### Phase 3: Optimization

Use optimization techniques:

- **Constraint programming**: Find minimal workflow that achieves 90% success
- **Genetic algorithms**: Evolve workflows over 10 generations
- **RL**: Train agent to choose workflows based on task type

### Phase 4: Novel Discoveries

Look for surprising results:

- Does phase order matter? (Design before plan?)
- Do parallel reviews help? (Multiple reviewers at once)
- Is TDD always better? (Tests first vs tests during)
- Can we skip phases for certain tasks? (No design for simple bugs)

## Success Criteria

1. ✓ Workflows are data (EDN configs), not code
2. ✓ Multiple workflow variants exist (canonical, lean, rigorous)
3. ✓ Can measure and compare workflow effectiveness
4. ✓ Meta loop proposes and tests new variants
5. ✓ Metrics show statistically significant differences
6. ✓ Can explain why certain workflows work better (interpretability)
7. ✓ Foundation exists for advanced techniques (petri nets, RL, genetic algorithms)

## References

### Petri Nets

- Murata, T. (1989). "Petri nets: Properties, analysis and applications"
- Reisig, W. (2013). "Understanding Petri Nets"

### Queuing Theory

- Kleinrock, L. (1975). "Queueing Systems"
- Little, J.D.C. (1961). "A Proof for the Queuing Formula: L = λW"

### Supply Chain

- Goldratt, E. (1984). "The Goal: A Process of Ongoing Improvement"
- Hopp, W. & Spearman, M. (2011). "Factory Physics"

### Systems Thinking

- Meadows, D. (2008). "Thinking in Systems: A Primer"
- Senge, P. (1990). "The Fifth Discipline"

### Optimization

- Russell, S. & Norvig, P. (2020). "Artificial Intelligence: A Modern Approach"
- Sutton, R. & Barto, A. (2018). "Reinforcement Learning: An Introduction"

## Next Steps

1. **Phase 2**: Implement Workflow component
   - Load and validate workflow configs
   - Execute workflows (DAG interpreter)
   - Collect and store metrics

2. **Phase 3**: Integrate with Meta loop
   - Propose workflow variants
   - A/B test workflows
   - Compare and promote

3. **Phase 4**: Advanced optimization
   - Implement petri net analysis
   - Try genetic algorithms
   - Explore RL approaches

4. **Phase 5**: Publish findings
   - Which workflows work best for which tasks?
   - Novel insights about SDLC optimization
   - Open source the approach
