# DD-HOLMESGPT-013: Vendor Local Copy of HolmesGPT SDK

**Date**: October 18, 2025
**Status**: ✅ APPROVED
**Decision Maker**: Architecture Team
**Confidence**: 95%

---

## 📋 **Context**

The `kubernaut-agent` service depends on the **HolmesGPT Python SDK** for AI-powered investigation capabilities. During SDK integration, we discovered critical dependency management challenges that threatened production stability:

### **Triggering Issue: Dependency Specification Bugs**

HolmesGPT SDK has conflicting dependency specifications in `pyproject.toml`:
```toml
supabase = "^2.5"        # Allows ANY 2.x version (2.5, 2.6, ... 2.12)
postgrest = "0.16.8"     # Locked to 0.16.8 for bug workaround
```

**Problem**: When pip resolves dependencies:
1. Picks latest `supabase` (e.g., 2.12.0)
2. `supabase 2.9+` requires `postgrest>=0.17`
3. Conflicts with HolmesGPT's `postgrest==0.16.8` pin
4. **Installation fails**

This demonstrates upstream dependency management is fragile and requires our oversight.

### **Key Requirements**

- **Stability**: Prevent breaking changes in upstream from breaking production
- **Control**: Review and approve SDK updates before integration
- **Reproducibility**: Consistent builds across dev/CI/prod
- **Speed**: Fast installation for developer productivity and CI/CD
- **Security**: Patch vulnerabilities without waiting for upstream
- **Auditability**: Track exactly which SDK version is deployed

---

## 🎯 **Decision**

**Vendor a local copy of HolmesGPT SDK in `dependencies/holmesgpt/` directory.**

### **Implementation**

1. **Local Copy**: Maintain HolmesGPT SDK in `dependencies/holmesgpt/`
2. **Requirements.txt**: Reference local copy: `../dependencies/holmesgpt/`
3. **Dependency Fix**: Pin compatible versions locally:
   ```txt
   supabase>=2.5,<2.8  # Compatible with postgrest 0.16.8
   postgrest==0.16.8   # Match HolmesGPT's pin
   ../dependencies/holmesgpt/  # Local copy
   ```
4. **Docker Build**: Copy SDK from local path during image build
5. **Sync Process**: Monthly review of upstream changes, selective merge

---

## 🔄 **Alternatives Considered**

### **Alternative 1: Install from Git (pip install git+https://...)**

**Approach**: Reference HolmesGPT directly from GitHub

**Pros**:
- ✅ Always get latest upstream changes
- ✅ Simple `requirements.txt` entry

**Cons**:
- ❌ **No breaking change protection**: Upstream breaks production
- ❌ **Slow**: Git clone takes ~2-3 minutes per install
- ❌ **Network dependency**: Requires GitHub access
- ❌ **No audit trail**: Can't see what changed
- ❌ **Can't patch**: Must wait for upstream fixes
- ❌ **Dependency conflicts**: As we experienced with supabase/postgrest

**Speed Impact**: 2m 43s per installation

**Decision**: ❌ **REJECTED** - Too risky, too slow

---

### **Alternative 2: Pin to Specific Git Commit SHA**

**Approach**: Use `git+https://...@<commit-sha>`

**Pros**:
- ✅ Reproducible builds
- ✅ Protection against breaking changes
- ✅ Can review before updating

**Cons**:
- ❌ **Still slow**: Git clone required (~2-3 minutes)
- ❌ **Network dependency**: Requires network access
- ❌ **Manual SHA management**: Track commit SHAs manually
- ❌ **Can't patch**: Must fork to apply patches
- ❌ **No offline dev**: Network required

**Speed Impact**: 2m 43s per installation

**Decision**: ❌ **REJECTED** - Installation speed unacceptable

---

### **Alternative 3: Vendor Local Copy (Selected) ✅**

**Approach**: Maintain local copy in `dependencies/holmesgpt/`

**Pros**:
- ✅ **Full control**: We decide when to update
- ✅ **Fast**: ~8 seconds (vs. 2-3 minutes) = **95% faster**
- ✅ **Offline dev**: No network required
- ✅ **Independent patching**: Apply fixes immediately
- ✅ **Dependency control**: Pin compatible versions locally
- ✅ **Audit trail**: Git history shows all changes
- ✅ **Reproducible**: Same code everywhere
- ✅ **CI/CD speed**: Faster builds

**Cons**:
- ⚠️ **Manual sync**: Monthly review needed
  - **Mitigation**: Automated notifications, monthly process
- ⚠️ **Repo size**: +~50MB
  - **Mitigation**: Acceptable for stability gained
- ⚠️ **Merge conflicts**: Possible with upstream changes
  - **Mitigation**: Minimize patches, contribute upstream

**Speed Impact**: 0m 08s per installation = **95% reduction**

**Decision**: ✅ **APPROVED** - Benefits vastly outweigh costs

---

### **Alternative 4: Fork HolmesGPT Repository**

**Approach**: Fork to our GitHub org, install from fork

**Pros**:
- ✅ Full control
- ✅ Can apply patches
- ✅ Public fork

**Cons**:
- ❌ **Fork maintenance overhead**: Full fork infrastructure
- ❌ **Still slow**: Git clone required (~2-3 minutes)
- ❌ **Network dependency**: Network access needed
- ❌ **Sync complexity**: Manage fork vs upstream

**Decision**: ❌ **REJECTED** - More overhead than vendoring

---

## 📊 **Impact Analysis**

### **Installation Speed Improvement**

| Approach | Time | Improvement |
|---|---|---|
| Git clone (Alternative 1, 2, 4) | 2m 43s | Baseline |
| **Local copy (Alternative 3)** | **0m 08s** | **95% faster** |

**Impact on Productivity**:
- **Developer installs**: 10×/day × 2.6min saved = **26 min/day/developer**
- **CI/CD builds**: 50×/day × 2.6min saved = **2.2 hours/day saved**
- **Container builds**: Faster production deployments

### **Dependency Conflict Resolution**

**Problem Solved**:
```python
# HolmesGPT's conflicting specs
supabase = "^2.5"        # Allows supabase 2.12
postgrest = "0.16.8"     # But 2.12 needs >=0.17

# Our local fix
supabase>=2.5,<2.8       # Pin compatible range
postgrest==0.16.8        # Match SDK requirement
```

**Result**: ✅ Installation works, dependencies stable

### **Control & Stability**

- **Before**: Upstream changes could break production anytime
- **After**: We review and approve all SDK updates
- **Emergency Patches**: Can fix critical bugs same-day
- **Rollback**: Simple `git revert` if issues found

---

## 🔧 **Operational Procedures**

### **Monthly Sync Process**

```bash
# 1. Check for new HolmesGPT releases
cd dependencies/holmesgpt
git fetch origin
git log HEAD..origin/master --oneline  # Review changes

# 2. Selective merge (review each commit)
git merge --no-commit origin/master
# Review changes, test locally

# 3. Update kubernaut with tested SDK version
cd ../..
git add dependencies/holmesgpt/
git commit -m "chore: sync HolmesGPT SDK to v<version> - [changelog]"
```

### **Emergency Patch Process**

```bash
# If critical bug found in HolmesGPT SDK:
cd dependencies/holmesgpt
# Apply minimal patch
git commit -m "fix: critical patch for [issue]"
cd ../..
git add dependencies/holmesgpt/
git commit -m "hotfix: patch HolmesGPT SDK for [issue]"
# Deploy immediately, contribute fix upstream
```

### **Graceful Degradation**

- If SDK has bugs, service falls back to stub implementation (logged)
- Monitoring alerts on stub usage indicate SDK issues
- Can hot-patch SDK and redeploy without upstream dependency

---

## ✅ **Validation Results**

### **Installation Speed Test**

```bash
# Git clone approach
$ time pip install git+https://github.com/robusta-dev/holmesgpt.git@master
real    2m 43s

# Local copy approach ✅
$ time pip install ../dependencies/holmesgpt/
real    0m 08s

# Improvement: 95% faster
```

### **Dependency Conflict Resolution**

- ✅ Successfully resolved supabase/postgrest conflict
- ✅ All dependencies install cleanly
- ✅ Reproducible across dev/CI/prod environments

### **Build Consistency**

- ✅ Same SDK version in all environments
- ✅ No network failures during builds
- ✅ Offline development verified

---

## 📝 **Key Takeaways**

### **Why This Matters**

1. **Stability is Critical**: HolmesGPT API is core to remediation pipeline. Breaking changes cascade to RemediationProcessor, AIAnalysis, and halt workflows.

2. **Speed Amplifies**: Developer time saved multiplies across team. CI/CD speed improvements compound over hundreds of builds/day.

3. **Control > Convenience**: The ~50MB repo cost is trivial vs. stability and speed gained.

4. **Real Issues Validated**: Dependency conflict proved upstream management needs oversight.

5. **Future-Proof**: Can patch security issues immediately without upstream dependency.

---

## 🔗 **Related Decisions**

- **Supports**: DD-HOLMESGPT-012 (Minimal Internal Service Architecture)
- **Enables**: Fast TDD iteration for SDK integration
- **Informs**: Future Python service dependencies (Context API, etc.)

---

## 📈 **Success Metrics**

| Metric | Target | Current | Status |
|---|---|---|---|
| Installation time | < 15s | ~8s | ✅ |
| Sync frequency | Monthly | Monthly | ✅ |
| Breaking change protection | Zero production breaks | Zero | ✅ |
| Patch turnaround | < 24 hours | Same-day capable | ✅ |
| CI/CD speedup | > 50% | 95% | ✅ |

---

## 🔄 **Review Schedule**

**Review When**:
- HolmesGPT SDK stabilizes dependencies (6+ months conflict-free)
- Upstream provides LTS versions
- Repository size becomes problematic (>500MB)
- Sync burden exceeds 2 hours/month

**Current Status**: All metrics green, no review needed.

---

## 📚 **References**

- **HolmesGPT SDK**: https://github.com/robusta-dev/holmesgpt
- **Implementation**: `kubernaut-agent/requirements.txt`, `dependencies/holmesgpt/`
- **Dependency Fix**: `docs/architecture/HOLMESGPT_DEPENDENCY_CONFLICT.md` (if needed)
- **DD-014 Standard**: [14-design-decisions-documentation.mdc](.cursor/rules/14-design-decisions-documentation.mdc)

