# Effectiveness Monitor Service - Documentation Hub

**Service Name**: Effectiveness Monitor Service
**Port**: 8080 (REST API + Health), 9090 (Metrics)
**Docker Image**: `quay.io/jordigilh/monitor-service`
**V1 Status**: ✅ **V1.0 Level 1: Automated Assessment**. Level 2 (AI): V1.1 (DD-017 v2.0)
**Type**: Stateless HTTP API Service
**Last Updated**: October 6, 2025

---

## 🎯 **Quick Links**

| Document | Purpose | Status |
|----------|---------|--------|
| **[Service Clarification](../../EFFECTIVENESS_SERVICE_CLARIFICATION.md)** | Answers "Which service hosts effectiveness logic?" | ✅ Complete |
| **[Effectiveness Triage](../../EFFECTIVENESS_LOGIC_SERVICE_TRIAGE.md)** | Detailed service analysis and V1 inclusion justification | ✅ Complete |
| **[V1 Feasibility Analysis](../../crd-controllers/AI_INSIGHTS_V1_FEASIBILITY_REVISED.md)** | Why moved from V2 to V1 | ✅ Complete |
| **[V2.1 Architecture Update](../../../architecture/V2.1_EFFECTIVENESS_MONITOR_V1_INCLUSION.md)** | Official architecture decision | ✅ Complete |
| **[Business Logic Implementation](../../../../pkg/ai/insights/)** | Core effectiveness assessment code (6,295 lines) | ✅ 98% Complete |
| **[Database Schema](../../../../migrations/001_v1_schema.sql)** | PostgreSQL schema for effectiveness data (squashed into v1 schema) | ✅ Complete |
| **[DD-EFFECTIVENESS-003](../../../architecture/decisions/DD-EFFECTIVENESS-003-RemediationRequest-Watch-Strategy.md)** | Watch RemediationRequest (not WorkflowExecution) for future-proofing | ✅ Approved (92% confidence) |

---

## 📋 **Service Overview**

### **What Is Effectiveness Monitor Service?**

The Effectiveness Monitor Service is Kubernaut's **performance assessment engine** that evaluates the effectiveness of executed remediation actions through multi-dimensional analysis.

**Core Purpose**:
- Assess how well remediation actions actually solved problems (BR-INS-001)
- Correlate actions with environmental improvements (BR-INS-002)
- Track long-term effectiveness trends (BR-INS-003)
- Detect adverse side effects (BR-INS-005)
- Enable continuous improvement through feedback loops (BR-INS-010)

**Why It Exists**:
Without effectiveness monitoring, Kubernaut would be "flying blind" - executing remediations without knowing if they actually work, which actions consistently succeed, or which cause problems.

---

## 🏗️ **Hybrid Architecture**

**Design Decision**: [DD-EFFECTIVENESS-001: Hybrid Automated + AI Analysis Approach](../../architecture/decisions/DD-EFFECTIVENESS-001-Hybrid-Automated-AI-Analysis.md)

### **Two-Level Assessment**

The Effectiveness Monitor uses a **hybrid approach** combining automated checks (always) with selective AI-powered analysis (high-value cases only).

#### **Level 1: Automated Assessment** (Always Executed)
- **Implementation**: GO service
- **Scope**:
  - Health checks (pod running, OOM errors, latency metrics)
  - Metric comparisons (pre/post execution)
  - Component audit events (health, alert, alert_decay, metrics, spec-hash); DataStorage computes weighted effectiveness score on demand
  - Alert decay detection with multi-probe cross-validation: keeps EA open when all non-alert probes (health, metrics, hash) are positive but the alert is still firing due to Prometheus lookback window decay. Health is re-probed live on each decay pass; metrics score is checked but not re-probed. If any probe turns negative, the decay hypothesis is killed and the alert is accepted at face value. Covers both reactive signals (health as ground truth) and proactive/predictive signals (metrics as ground truth). (Issue #369, BR-EM-012)
  - Anomaly detection (metric changes > thresholds)
- **Cost**: Negligible (computational only)
- **Latency**: <100ms
- **Volume**: Every workflow (~3.67M/year)

#### **Level 2: AI-Powered Analysis** (Selective Execution)
- **Implementation**: Calls HolmesGPT API `POST /api/v1/postexec/analyze`
- **Scope**:
  - Root cause validation ("problem solved" vs "problem masked")
  - Oscillation detection ("fix A caused problem B")
  - Pattern learning (context-aware effectiveness)
  - Lesson extraction (for Context API)
- **Cost**: ~$9K/year (LLM API costs)
- **Latency**: 2-5s (async, doesn't block workflow status updates)
- **Volume**: ~18,000/year (0.5% of workflows)

#### **Decision Triggers** (When AI Analysis is Called)

| Trigger | Volume/Year | Rationale |
|---------|-------------|-----------|
| **P0 Failures** | ~3,650 | Learn from critical failures to prevent recurrence |
| **New Action Types** | ~2,600 | Build pattern library for new actions |
| **Suspected Oscillations** | ~1,825 | Detect "fix A caused problem B" scenarios |
| **Periodic Batch** | ~10,000 | Long-term trend analysis, predictive insights |
| **Routine Successes** | 0 (skip) | No learning value, waste of cost |

**Cost/Benefit**: $9K/year investment for $100K+ value (11x ROI)
- Achieves 85-90% remediation effectiveness (vs 70% current)
- Reduces cascade failures from 30% to <10%
- Enables continuous pattern learning

---

## 🏗️ **Service Architecture**

### **Service Position in V1**

```
WorkflowExecution CRD (completed) → Effectiveness Monitor (8080)
                                    ↓ (5-min stabilization delay)
                                    ├→ Data Storage (8080)
                                    ├→ External Prometheus (9090)
                                    └→ HolmesGPT API (8080) [selective]
                                    ↓
                                    Context API (8080) → Notifications (8080)
```

**Data Flow** (Hybrid Approach):
1. **WorkflowExecution** completes → CRD status = "completed"
2. **Effectiveness Monitor Controller** watches WorkflowExecution CRDs → detects completion (< 1s)
3. **Idempotency Check**: Database lookup via WorkflowExecution.UID → skip if already assessed
4. **Assessment Scheduled**: Record created with **5-minute stabilization delay**
5. **After 5 Minutes** → **Effectiveness Monitor** performs **automated assessment**:
   - Retrieves action history from Data Storage (90-day window)
   - Queries metrics from External Prometheus (before/after comparison)
   - Health checks (pod running, OOM errors, restart count)
   - Metric comparisons (latency, CPU, memory, network)
   - Basic effectiveness scoring (0-1 scale)
   - Anomaly detection (unexpected side effects)
6. **Effectiveness Monitor** makes **decision**: `shouldCallAI()?`
   - P0 failure? → YES (~50/day)
   - New action type? → YES (~10/day)
   - Anomaly detected? → YES (~5/day)
   - Oscillation/recurring failure? → YES (~5/day)
   - Routine success? → NO (~10,000/day)
7. **IF YES** → **Effectiveness Monitor** calls **HolmesGPT API**:
   - `POST /api/v1/postexec/analyze`
   - Root cause validation (was diagnosis correct?)
   - Oscillation detection (repeating failures?)
   - Pattern learning (similar scenarios?)
   - Lesson extraction (what to do differently?)
8. **Effectiveness Monitor** stores **combined results**:
   - Automated metrics (always) → Data Storage PostgreSQL
   - AI lessons (when analyzed) → Context API for future use
9. **Context API** uses effectiveness data for future recommendations

**Key Timing**: WorkflowExecution completion → Assessment complete = **~5-6 minutes total**

### **Dependencies**

| Dependency | Port | Purpose | V1 Status |
|------------|------|---------|-----------|
| **Data Storage Service** | 8080 | Action history, vector DB, effectiveness storage | ✅ V1 |
| **External Prometheus** | 9090 | Metrics scraping, side effect detection | ✅ V1 |
| **HolmesGPT API** | 8090 | AI-powered post-execution analysis (selective) | ✅ V1 |
| **Intelligence Service** | 8080 | Advanced pattern discovery (optional) | 🔴 V2 |

**Result**: ✅ All required dependencies available in V1 (Intelligence Service is optional)

**Integration Pattern**: Effectiveness Monitor calls HolmesGPT API selectively (~18K/year) for intelligent analysis

---

## 📊 **Business Requirements Coverage**

### **BR-INS-001 to BR-INS-010** (10 Requirements)

| Requirement | Capability | Scope |
|------------|-----------|--------|
| **BR-INS-001** | Assess remediation action effectiveness | V1.0 (Level 1) |
| **BR-INS-002** | Correlate action outcomes with environment improvements | V1.0 (Level 1) |
| **BR-INS-005** | Detect adverse side effects | V1.0 (Level 1) |
| **BR-INS-003** | Track long-term effectiveness trends | V1.1 (Level 2) |
| **BR-INS-004** | Identify consistently positive actions | V1.1 (Level 2) |
| **BR-INS-006** | Advanced pattern recognition | V1.1 (Level 2) |
| **BR-INS-007** | Comparative analysis | V1.1 (Level 2) |
| **BR-INS-008** | Temporal pattern detection | V1.1 (Level 2) |
| **BR-INS-009** | Seasonal effectiveness variations | V1.1 (Level 2) |
| **BR-INS-010** | Continuous improvement feedback loop | V1.1 (Level 2) |

**Implementation**: `pkg/ai/insights/service.go` (6,295 lines, 98% complete)

---

## 🎯 **V1.0 Level 1 vs V1.1 Level 2**

### **Level 1 (V1.0): Day-1 Value**

Level 1 automated assessment provides immediate value from the first remediation:
- Dual spec hash capture, health checks via K8s API, pre/post metric comparison via Prometheus
- Alert resolution check via AlertManager; EM emits component audit events; DataStorage computes weighted effectiveness score on demand
- Side-effect detection; EM always emits Normal `EffectivenessAssessed` K8s event on completion
- **No data dependency**: Works from Day 1 without historical accumulation

### **Level 2 (V1.1): 8+ Weeks Post V1.0**

Level 2 AI-powered analysis (HolmesGPT PostExec) requires historical patterns:
- HolmesGPT PostExec analysis, pattern learning, batch processing
- Requires 8+ weeks of Level 1 assessment data for high-confidence analysis (80%+)
- BR-INS-003, BR-INS-004, BR-INS-006 to BR-INS-010

### **Example Responses**

**Level 1 (V1.0)** — Emitted as `effectiveness.assessment.completed` audit event from Day 1:
```json
{
  "event_type": "effectiveness.assessment.completed",
  "effectiveness_score": 0.88,
  "signal_resolved": true,
  "health_checks": { "pod_running": true, "readiness_pass": true },
  "metric_deltas": { "cpu_before": 0.95, "cpu_after": 0.92 },
  "side_effects_detected": []
}
```

**Level 2 (V1.1)** — Enriched with AI analysis when 8+ weeks of data available:
```json
{
  "effectiveness_score": 0.88,
  "pattern_insights": ["Similar actions successful in 87% of production cases"],
  "lessons_learned": [...],
  "root_cause_resolved": true,
  "oscillation_detected": false
}
```

---

## 🔌 **API Endpoints**

### **Assessment Endpoints** (Port 8080)

```yaml
POST /api/v1/assess/effectiveness
  Description: Assess single action effectiveness
  Request:
    {
      "action_id": "act-abc123",
      "wait_for_stabilization": true,
      "assessment_interval": "10m"
    }
  Response:
    {
      "assessment_id": "assess-xyz789",
      "effectiveness_score": 0.88,
      "confidence": 0.91,
      "details": { ... }
    }

GET /api/v1/insights/trends
  Description: Long-term effectiveness trends
  Query Params:
    - timeRange: "7d" | "30d" | "90d"
    - actionType: "restart-pod" | "scale-deployment" | etc.
    - environment: "production" | "staging" | "development"
  Response:
    {
      "trends": [...],
      "patterns": [...],
      "recommendations": [...]
    }

GET /api/v1/insights/patterns
  Description: Pattern recognition results
  Query Params:
    - minConfidence: 0.7
    - category: "temporal" | "seasonal" | "environmental"
  Response:
    {
      "patterns": [...],
      "correlations": [...],
      "anomalies": [...]
    }
```

### **Health Endpoints** (Port 8080)

```yaml
GET /health
  Description: Liveness probe
  Response: { "status": "OK", "timestamp": "..." }

GET /ready
  Description: Readiness probe
  Response: {
    "status": "READY",
    "data_availability": { "weeks": 8, "sufficient": true },
    "dependencies": {
      "data_storage": "healthy",
      "infra_monitoring": "healthy"
    }
  }
```

### **Metrics Endpoint** (Port 9090)

```yaml
GET /metrics
  Description: Prometheus metrics
  Authentication: Kubernetes TokenReviewer (required)
  Metrics:
    - effectiveness_assessments_total
    - effectiveness_score (histogram)
    - assessment_data_availability_weeks
    - insufficient_data_responses_total
    - assessment_duration_seconds
```

---

## 🔐 **Security**

### **Authentication**
- **Method**: Kubernetes TokenReviewer (Bearer Token)
- **Pattern**: Consistent with all Kubernaut services
- **Port**: 8080 (no auth), 9090 (TokenReviewer required)

### **RBAC Permissions**

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: effectiveness-monitor-service
rules:
# TokenReviewer for authentication
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]

# Read action history from Data Storage
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]

# Query metrics from Infrastructure Monitoring
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get"]
```

---

## 🧪 **Testing Strategy**

### **Test Pyramid**
- **Unit Tests** (70%): Business logic, assessment algorithms, pattern recognition
- **Integration Tests** (>50%): Data Storage integration, Infrastructure Monitoring queries, microservices coordination
- **E2E Tests** (10%): Complete assessment flow from action execution to result
- **Rationale**: Effectiveness monitoring requires extensive integration with Data Storage and Infrastructure Monitoring services

### **Key Test Scenarios**
1. ✅ Effectiveness assessment with sufficient data (Week 13+)
2. ✅ Graceful degradation with insufficient data (Week 5)
3. ✅ Side effect detection (CPU spike, memory leak)
4. ✅ Trend analysis (improving, declining, stable)
5. ✅ Temporal pattern detection (business hours vs off-hours)
6. ✅ Seasonal variation detection (Q1 vs Q4)

**Test Location**: `test/unit/ai/insights/`, `test/integration/ai/insights/`

---

## 📊 **Observability**

### **Prometheus Metrics**

```go
// Key Metrics
effectiveness_assessments_total{status="success|insufficient_data|error"}
effectiveness_score{action_type="restart-pod|scale-deployment|..."}
assessment_data_availability_weeks
insufficient_data_responses_total
assessment_duration_seconds
side_effects_detected_total{severity="low|medium|high"}
```

### **Logging** (Zap)

```go
// Example Log Entry
logger.Info("Effectiveness assessment completed",
    zap.String("assessment_id", "assess-xyz789"),
    zap.String("action_id", "act-abc123"),
    zap.Float64("effectiveness_score", 0.88),
    zap.Float64("confidence", 0.91),
    zap.String("trend_direction", "stable"),
    zap.Duration("duration", 450*time.Millisecond),
)
```

---

## 🚀 **Implementation Status**

### **What Exists** (98% complete)

| Component | Location | Lines | Status |
|-----------|----------|-------|--------|
| **Business Logic** | `pkg/ai/insights/service.go` | 6,295 | ✅ 98% |
| **Assessment Algorithm** | `pkg/ai/insights/assessment.go` | 800+ | ✅ Complete |
| **Model Training** | `pkg/ai/insights/model_training_methods.go` | 1,200+ | ✅ Complete |
| **Database Schema** | `migrations/001_v1_schema.sql` (effectiveness tables section) | 150 | ✅ Complete |
| **Integration Tests** | `test/integration/ai/` | 500+ | ✅ Complete |

### **What's Missing** (2% remaining)

| Component | Location | Status |
|-----------|----------|--------|
| **Main Entry Point** | `cmd/monitor-service/main.go` | ⏸️ Needs creation |
| **HTTP API Handlers** | `internal/handlers/effectiveness/` | ⏸️ Needs creation |
| **Health Check Endpoints** | `internal/handlers/health/` | ⏸️ Needs creation |
| **Metrics Exposure** | `internal/handlers/metrics/` | ⏸️ Needs creation |
| **Kubernetes Manifests** | `deploy/effectiveness-monitor-service.yaml` | ⏸️ Needs creation |

**Estimated Effort**: 1-2 weeks to complete HTTP wrapper

---

## 🎯 **Implementation Checklist**

### **Phase 1: HTTP API Wrapper** (1 week)
- [ ] Create `cmd/monitor-service/main.go`
- [ ] Implement REST endpoints (`/api/v1/assess/effectiveness`, `/api/v1/insights/trends`)
- [ ] Add health checks (`/health`, `/ready`)
- [ ] Expose Prometheus metrics (`/metrics` on port 9090)
- [ ] Implement graceful degradation middleware

### **Phase 2: Deployment** (3-5 days)
- [ ] Create Kubernetes deployment manifests
- [ ] Configure RBAC permissions
- [ ] Set up Prometheus ServiceMonitor
- [ ] Deploy to development environment
- [ ] Validate graceful degradation behavior

### **Phase 3: Production Readiness** (3-5 days)
- [ ] Load testing (100 req/s target)
- [ ] Security validation (TokenReviewer, RBAC)
- [ ] Observability validation (metrics, logs, traces)
- [ ] Integration testing with Context API
- [ ] Production deployment

---

## 📚 **Related Documentation**

### **Architecture**
- [APPROVED_MICROSERVICES_ARCHITECTURE.md](../../../architecture/APPROVED_MICROSERVICES_ARCHITECTURE.md) (lines 480-530)
- [V2.1_EFFECTIVENESS_MONITOR_V1_INCLUSION.md](../../../architecture/V2.1_EFFECTIVENESS_MONITOR_V1_INCLUSION.md)

### **Design & Analysis**
- [AI_INSIGHTS_V1_FEASIBILITY_REVISED.md](../../crd-controllers/AI_INSIGHTS_V1_FEASIBILITY_REVISED.md)
- [EFFECTIVENESS_SERVICE_CLARIFICATION.md](../../EFFECTIVENESS_SERVICE_CLARIFICATION.md)
- [EFFECTIVENESS_LOGIC_SERVICE_TRIAGE.md](../../EFFECTIVENESS_LOGIC_SERVICE_TRIAGE.md)

### **Implementation**
- `pkg/ai/insights/service.go` - Core business logic
- `pkg/ai/insights/assessment.go` - Assessment algorithms
- `migrations/001_v1_schema.sql` - Database schema (effectiveness tables)

---

## ✅ **Service Readiness**

| Aspect | Status | Confidence |
|--------|--------|------------|
| **Business Logic** | ✅ 98% Complete | 98% |
| **Architecture** | ✅ Approved for V1 | 99% |
| **Dependencies** | ✅ All available in V1 | 100% |
| **Database Schema** | ✅ Complete | 100% |
| **HTTP API Wrapper** | ⏸️ Needs creation | N/A |
| **Deployment** | ⏸️ Needs manifests | N/A |
| **Overall Readiness** | 🟡 80% Ready for V1 | 90% |

**Next Step**: Create HTTP API wrapper (`cmd/monitor-service/main.go`)

---

## 🔄 **RECENT ARCHITECTURAL UPDATES - October 16, 2025**

### **Major Enhancements**

1. **Restart Recovery & Idempotency Design** (DD-EFFECTIVENESS-002)
   - Database-backed idempotency using WorkflowExecution.UID
   - Automatic catch-up for missed assessments during downtime
   - Race condition handling for HA deployments
   - Zero manual intervention required

2. **5-Minute Stabilization Delay Documented**
   - Allows metrics to stabilize after action execution
   - Improves assessment accuracy from 70% to 85-90%
   - Timing breakdown: completion → assessment = ~5-6 minutes total

3. **Hybrid Approach Formalized** (DD-EFFECTIVENESS-001)
   - Automated assessment: 100% of actions
   - AI analysis: Selective (~0.49% of actions, ~18K/year)
   - Cost/benefit: $9K/year for 11x ROI

### **Related Documentation**

- **Design Decisions**:
  - `/docs/architecture/decisions/DD-EFFECTIVENESS-001-Hybrid-Automated-AI-Analysis.md`
  - `/docs/architecture/decisions/DD-EFFECTIVENESS-002-Restart-Recovery-Idempotency.md`

- **Architecture**:
  - `/docs/architecture/EFFECTIVENESS_MONITOR_RESTART_RECOVERY_FLOWS.md`

- **Technical Details**:
  - `/kubernaut-agent/docs/EFFECTIVENESS_MONITOR_CRD_DESIGN_ASSESSMENT.md`
  - `/kubernaut-agent/docs/EFFECTIVENESS_MONITOR_RESTART_RECOVERY.md`

---

**Document Maintainer**: Kubernaut Documentation Team
**Last Updated**: October 16, 2025 (Architectural Enhancements)
**Status**: ✅ Documentation Hub Complete - Architecturally Validated

