# DD-TEST-011: Mock LLM Fixture Provisioning Pattern for E2E Test Dependency Management

**Version**: 3.0
**Status**: ✅ **IMPLEMENTED & VALIDATED** (File-Based Configuration Pattern)
**Date**: 2026-01-14 (Initial), 2026-01-14 (v2.0 - ConfigMap Pattern), 2026-01-14 (v2.1 - Documentation Enhancement), 2026-01-14 (v3.0 - File-Based Refactoring), 2026-01-14 (v3.0 - Validation Complete)
**Related**: DD-TEST-002 (Integration Test Container Orchestration), DD-TEST-010 (Controller-Per-Process Architecture)
**Applies To**: All tests requiring Mock LLM service (Integration, E2E - AIAnalysis, RemediationOrchestrator, future AI-driven services)
**Confidence**: 100% (validated across all 3 testing tiers - 104/104 tests passing)

---

## Changelog

### Version 4.0 — Go Rewrite & Deterministic UUIDs (2026-03-04)
**Status**: 📋 **PLANNED** (specification phase, implementation pending)

**Decision Change**: File-Based Configuration Pattern → **Deterministic UUID Generation** (eliminates ConfigMap sync entirely)

**Reason for Change**: Issue #531 rewrites the Mock LLM from Python to Go. Issue #548 introduces deterministic UUID generation in DataStorage. Together, these changes make the entire ConfigMap sync infrastructure obsolete.

**Key Insight (v4.0)**: When both DataStorage and Mock LLM use the same deterministic UUID function (`pkg/shared/uuid.DeterministicUUID(workflowName, environment)`), there is no synchronization problem to solve. Both services independently compute identical UUIDs — no ConfigMap, no YAML config file, no rollout restart.

**Impact on v3.0 Infrastructure**:

| v3.0 Component | v4.0 Status | Rationale |
|----------------|-------------|-----------|
| `MOCK_LLM_CONFIG_PATH` env var | **OPTIONAL** | No longer required; retained as optional override for manual testing/debugging (BR-MOCK-033) |
| `load_scenarios_from_file()` | **REMOVED** (as required path) | Deterministic UUIDs are the default; optional YAML overrides still load from file if `MOCK_LLM_CONFIG_PATH` is set |
| `scenarios.yaml` ConfigMap | **REMOVED** | No UUID mapping file needed |
| `UpdateMockLLMConfigMap()` helper | **REMOVED** | No ConfigMap to update |
| `WriteMockLLMConfigFile()` helper | **REMOVED** | No config file to write |
| `SortedWorkflowUUIDKeys()` helper | **REMOVED** | No UUID keys to sort |
| ConfigMap volume mount in deployment | **REMOVED** | No volume needed |
| Rollout restart after DS seeding | **REMOVED** | Mock LLM ready immediately |
| `SYNC_ON_STARTUP` env var | **REMOVED** | No sync mechanism at all |
| `/health` readiness probe | **UNCHANGED** | Still used for liveness/readiness |

**New Architecture**:

```
Test Suite Startup (Simplified):
1. Deploy DataStorage → seed workflows (UUIDs generated deterministically)
2. Deploy Mock LLM (no ConfigMap, no config file)
   - Mock LLM starts → computes UUIDs from shared function
   - /health returns 200 immediately
3. Deploy HAPI → LLM_ENDPOINT=http://mock-llm:8080
4. Tests run — UUIDs match between DataStorage and Mock LLM ✅
```

**Additional Architectural Changes (Go Rewrite)**:

The Go rewrite (#531) introduces several architectural improvements beyond UUID management:

1. **DAG-based conversation engine** (#560): Replaces hardcoded if/else routing with a declarative state machine
2. **Scenario registry** (#564): Self-registering scenarios, one file per scenario
3. **Shared OpenAI types** (#562): Compile-time contract enforcement between Mock LLM and HAPI
4. **HTTP verification API** (#563): Behavioral test assertions from Go tests
5. **Fault injection** (#565): Configurable failure modes for resilience testing
6. **Declarative YAML scenarios** (#566): Non-developers can author scenarios
7. **Pillar composition** (#567): Framework for Threat Remediation (#554) and Cost Optimization (#555)
8. **Prometheus metrics** (#568): Observability for E2E debugging

**Business Requirements**: See [Mock LLM BUSINESS_REQUIREMENTS.md](../../services/test-infrastructure/mock-llm/BUSINESS_REQUIREMENTS.md) (46 BRs across 10 categories)

**Dependencies**:
- #548 — Deterministic UUIDs in DataStorage (must land first or in parallel)
- #433 — HAPI Go rewrite (shared types consumer)

**Backward Compatibility**:
- v4.0 maintains the same HTTP API contract (endpoints, request/response shapes)
- v4.0 maintains the same container contract (port 8080, non-root user 1001, `/health` probe)
- Existing integration and E2E tests pass without modification (beyond image name change)
- v3.0 file-based config pattern still works as an optional override (BR-MOCK-033)

**Confidence**: 95% (specification complete, implementation pending)

**Related Issues**: #531, #548, #560-#568

---

### Version 3.0 - Validation Complete (2026-01-14)
**Status**: ✅ **VALIDATED** - All 3 testing tiers passing (104/104 tests)

**Validation Results**:
- **Python Unit Tests**: 11/11 ✅ (file-based configuration validation)
- **Integration Tests**: 57/57 ✅ (5m 55s runtime - Mock LLM file mounting)
- **E2E Tests**: 36/36 ✅ (6m 3s runtime - ConfigMap delivery pattern)
- **Total**: 104/104 tests passed (100% pass rate)

**Critical Fixes Applied During Validation**:
1. **HAPI Dockerfile.e2e**: Added missing `pip install ./src/clients/datastorage` (fixes ImportError)
2. **Integration Test Config Path**: Changed from `/tmp` to test directory (prevents macOS cleanup issues)
3. **Python Artifacts**: Added `.gitignore` for pytest artifacts (`.hypothesis/`)

**Confidence**: Increased from 98% → **100%** (validated at all levels)

### Version 3.0 (2026-01-14) - File-Based Configuration Refactoring
**Refinement**: ConfigMap Pattern → File-Based Configuration Pattern

**Key Insight**: ConfigMap is just a **deployment mechanism** for E2E tests, not a fundamental requirement.
- **Integration tests**: Write YAML file directly, mount as volume
- **E2E tests**: Use Kubernetes ConfigMap to deliver the same YAML file
- **Mock LLM**: Generic file reader, agnostic to deployment method

**Implementation Changes**:
1. **Mock LLM Server Cleanup**:
   - Removed HTTP `PUT /api/test/update-uuids` endpoint (~40 lines)
   - Removed all HTTP self-discovery code and `requests` dependency
   - Renamed `load_scenarios_from_configmap()` → `load_scenarios_from_file()`
   - Simplified readiness check (no `datastorage_synced` state)

2. **Integration Test Updates**:
   - Added `WriteMockLLMConfigFile()` helper (writes YAML directly to host)
   - Updated `MockLLMConfig` struct with `ConfigFilePath` field
   - Modified `StartMockLLMContainer()` to mount config file if provided
   - Deprecated `UpdateMockLLMWithUUIDs()` (HTTP-based, kept for backward compat)

3. **E2E Test Cleanup**:
   - Removed obsolete `DATA_STORAGE_URL` and `SYNC_ON_STARTUP` env vars from Mock LLM deployment
   - ConfigMap remains as K8s-native delivery mechanism (unchanged from v2.0)

4. **Documentation**:
   - Created comprehensive `test/services/mock-llm/README.md`
   - Clarified file-based architecture with usage patterns for local dev, integration, and E2E
   - Added Python unit tests (`test_config_loading.py`) validating file loading

**Benefits**:
- **Simpler codebase**: 40+ lines removed, no HTTP server complexity
- **Faster tests**: No HTTP roundtrips or client dependencies
- **Clearer semantics**: ConfigMap is delivery, not architecture
- **Better testing**: File-based logic fully unit-tested
- **Easier local dev**: Write YAML file, run Mock LLM directly

**Backward Compatibility**: E2E tests unchanged (ConfigMap pattern still works).

**Confidence**: Increased from 95% to 98% (simplified implementation, fewer failure modes).

**Validation Status (2026-01-14)**: ✅ Complete - All tests passing, ready for production use.

---

### Version 2.1 (2026-01-14) - Documentation Enhancement
**Documentation Improvements**: Added comprehensive examples and specifications for ConfigMap pattern implementation.

**Added Content**:
- **Complete ConfigMap YAML Example**: Full manifest showing structure with 18 workflow UUID mappings (6 base workflows × 3 environments)
- **Complete Mock LLM Deployment Spec**: Full Kubernetes deployment manifest with ConfigMap volume mount, environment variables, and resource limits
- **Key Configuration Summary**: Highlighted critical integration points and configuration settings
- **Example Workflow Mappings**: Concrete UUID examples showing key-value format (`workflow_name:environment` → `UUID`)

**Purpose**: Provide implementers with copy-paste ready examples and clear understanding of ConfigMap structure and deployment configuration.

**No Implementation Changes**: Code remains at v2.0, only documentation enhanced.

### Version 2.0 (2026-01-14) - ConfigMap Pattern Implementation
**Decision Change**: Alternative 3 (Self-Discovery) → Alternative 2 (ConfigMap Pattern)

**Reason for Change**: Alternative 3 (Self-Discovery) implementation encountered timing issues in production E2E tests:
- Mock LLM attempted HTTP sync to DataStorage before workflows were seeded
- DataStorage returned 404 errors, causing Mock LLM readiness probe to fail
- Test suite timed out waiting for Mock LLM to become ready
- Race condition between workflow seeding and Mock LLM startup

**New Approach**: Alternative 2 (ConfigMap Pattern)
- Test suite creates ConfigMap AFTER seeding workflows (deterministic ordering)
- Mock LLM reads ConfigMap at startup (no HTTP calls needed)
- Eliminates timing issues, simplifies Mock LLM logic
- Faster startup, more Kubernetes-native

**Implementation Changes**:
- Disabled `SYNC_ON_STARTUP` (set to `false`)
- Removed `?check=datastorage-sync` readiness probe check
- Added ConfigMap volume mount to Mock LLM deployment
- Added `load_scenarios_from_configmap()` function
- Updated test suite to create ConfigMap after workflow seeding

**Confidence**: Increased from 90% to 95% (deterministic ordering eliminates race conditions)

### Version 1.0 (2026-01-14) - Self-Discovery Pattern (DEPRECATED)
**Initial Decision**: Alternative 3 (Self-Discovery Pattern)
- Mock LLM queries DataStorage /api/workflows at startup
- Readiness probe validates sync completion
- Test suite simplified (no UUID orchestration)

**Deprecated**: Timing issues in E2E environment made this approach unreliable. Kept for reference and potential future use in different contexts.

---

## Context & Problem

### Discovery (January 2026)

During AIAnalysis E2E test development, we encountered a **fixture provisioning architectural challenge**:

**Problem**: How should test infrastructure services (Mock LLM) obtain dynamic configuration data (workflow UUIDs) that is only available after other services (DataStorage) have started?

**Challenge**: DataStorage auto-generates workflow UUIDs for security reasons. Mock LLM needs these exact UUIDs to return matching responses during E2E tests, but:
- DataStorage must be deployed first (workflows don't exist yet)
- Mock LLM must be deployed second (needs UUIDs from DataStorage)
- **Traditional approach**: Test suite orchestrates UUID synchronization AFTER both services are "ready"

**Key Requirements**:
- Test `BeforeSuite` should ONLY provision infrastructure, not orchestrate service dependencies
- Services should manage their own startup dependencies (Kubernetes-native patterns)
- UUID synchronization must be deterministic and restart-safe
- Zero HTTP roundtrip overhead during test execution
- Mock LLM readiness must mean "fully configured and ready to serve requests"

### The Architectural Question

**Where should fixture synchronization logic live?**

| Location | Owner | Implications |
|---------|-------|-------------|
| **Test Suite** | Test code | Tests orchestrate service startup → tight coupling |
| **Mock LLM Service** | Service code | Service manages own dependencies → loose coupling |
| **ConfigMap (K8s)** | Infrastructure | Requires test suite to create ConfigMap → partial coupling |

---

## Alternatives Considered

### Alternative 1: Test Suite Orchestration (REST API Synchronization)

**Approach**: Test `BeforeSuite` calls REST API to update Mock LLM with UUIDs after DataStorage deployment.

**Implementation**:
```go
// test/e2e/aianalysis/suite_test.go - BeforeSuite
var _ = SynchronizedBeforeSuite(func() []byte {
    // 1. Deploy DataStorage
    infrastructure.DeployDataStorage(...)

    // 2. Seed workflows and capture UUIDs
    workflowUUIDs, err := aianalysis.SeedTestWorkflowsInDataStorage(dataStorageURL, GinkgoWriter)

    // 3. Deploy Mock LLM (without UUIDs)
    infrastructure.DeployMockLLM(...)

    // 4. TEST ORCHESTRATES: Update Mock LLM via REST API
    err = aianalysis.UpdateMockLLMWithUUIDs(mockLLMURL, workflowUUIDs, GinkgoWriter)

    // 5. Now tests can run
    return []byte{}
}
```

**Pros**:
- ✅ Works with existing REST API endpoint (`/api/test/update-uuids`)
- ✅ No changes needed to Mock LLM container startup
- ✅ Simple HTTP PUT request for synchronization

**Cons**:
- ❌ **Tight coupling**: Test suite knows about Mock LLM's internal configuration needs
- ❌ **Orchestration complexity**: Test suite manages service startup order AND dependency synchronization
- ❌ **Not restart-safe**: If Mock LLM pod restarts, UUIDs are lost (in-memory only)
- ❌ **Network dependency**: Mock LLM must be reachable via NodePort before tests run
- ❌ **Slow**: Extra HTTP roundtrip adds 1-2 seconds to suite startup
- ❌ **Race conditions**: Tests might start before UUID sync HTTP call completes

**Confidence**: 40% (works but violates architectural principles)

---

### Alternative 2: ConfigMap-Driven Configuration (Kubernetes-Native Fixtures)

**Approach**: Test suite creates a Kubernetes ConfigMap with workflow UUIDs, Mock LLM mounts and reads it at startup.

**Implementation**:
```go
// test/e2e/aianalysis/suite_test.go - BeforeSuite
var _ = SynchronizedBeforeSuite(func() []byte {
    // 1. Deploy DataStorage
    infrastructure.DeployDataStorage(...)

    // 2. Seed workflows and capture UUIDs
    workflowUUIDs, err := aianalysis.SeedTestWorkflowsInDataStorage(dataStorageURL, GinkgoWriter)

    // 3. TEST CREATES: ConfigMap with UUID mapping
    scenariosYAML := fmt.Sprintf(`
scenarios:
  oomkilled: %s
  crashloop: %s
  nodenotready: %s
`, workflowUUIDs["oomkill-increase-memory-v1:production"], ...)

    kubectl.CreateConfigMap("mock-llm-scenarios", scenariosYAML, namespace)

    // 4. Deploy Mock LLM (mounts ConfigMap)
    infrastructure.DeployMockLLM(...) // volumeMount: /config/scenarios.yaml

    // 5. Mock LLM reads /config/scenarios.yaml at startup
    return []byte{}
}
```

```python
# test/services/mock-llm/src/server.py - Startup
def load_scenarios_from_config():
    config_path = os.getenv("MOCK_LLM_CONFIG_PATH", "/config/scenarios.yaml")
    if os.path.exists(config_path):
        with open(config_path, 'r') as f:
            config = yaml.safe_load(f)
            for scenario_key, uuid_override in config.get('scenarios', {}).items():
                if scenario_key in MOCK_SCENARIOS:
                    MOCK_SCENARIOS[scenario_key].workflow_id = uuid_override

if __name__ == "__main__":
    load_scenarios_from_config()  # Before serving HTTP
    start_server()
```

**Pros**:
- ✅ Kubernetes-native configuration management (ConfigMaps are designed for this)
- ✅ Atomic configuration: Mock LLM gets correct UUIDs at startup
- ✅ No network calls: Zero HTTP overhead for UUID sync
- ✅ Restart-safe: ConfigMap persists across Mock LLM pod restarts
- ✅ Faster: Tests start immediately after Mock LLM is ready

**Cons**:
- ❌ **Test orchestration still required**: Test suite must create ConfigMap AFTER seeding workflows
- ❌ **Timing complexity**: ConfigMap must exist BEFORE Mock LLM deployment (deployment order matters)
- ❌ **Partial coupling**: Test suite still knows about Mock LLM's ConfigMap structure
- ❌ **YAML serialization**: Test suite must format YAML correctly

**Confidence**: 65% (better than Alt 1, but still requires test orchestration)

---

### Alternative 3: Mock LLM Self-Discovery Pattern (Service-Managed Dependencies) ✅

**Approach**: Mock LLM automatically discovers and syncs workflow UUIDs from DataStorage at startup, BEFORE marking itself "ready".

**Implementation**:
```go
// test/e2e/aianalysis/suite_test.go - BeforeSuite (SIMPLIFIED)
var _ = SynchronizedBeforeSuite(func() []byte {
    // 1. Deploy DataStorage
    infrastructure.DeployDataStorage(...)

    // 2. Seed workflows (DataStorage now has UUIDs)
    aianalysis.SeedTestWorkflowsInDataStorage(dataStorageURL, GinkgoWriter)

    // 3. Deploy Mock LLM (with DATA_STORAGE_URL env var)
    infrastructure.DeployMockLLM(...) // env: DATA_STORAGE_URL=http://datastorage:8080

    // 4. MOCK LLM HANDLES: Self-discovery and UUID sync at startup
    //    (Test suite just waits for "ready" signal)

    return []byte{}
}
```

```python
# test/services/mock-llm/src/server.py - Startup
def sync_workflows_from_datastorage():
    """
    Sync workflow UUIDs from DataStorage at startup (before serving traffic).

    Mock LLM automatically discovers workflow UUIDs by querying DataStorage's
    workflow catalog API. This eliminates the need for test suite orchestration.
    """
    datastorage_url = os.getenv("DATA_STORAGE_URL", "http://datastorage:8080")
    max_retries = 30  # 30 * 2s = 60s max wait for DataStorage readiness

    print(f"🔄 Mock LLM starting - syncing with DataStorage at {datastorage_url}...")

    for attempt in range(max_retries):
        try:
            # Query DataStorage for all workflows
            response = requests.get(
                f"{datastorage_url}/api/workflows",
                timeout=5,
                headers={"Accept": "application/json"}
            )

            if response.status_code == 200:
                workflows = response.json()
                synced_count = 0

                # Match workflows to scenarios by name+environment
                for workflow in workflows:
                    workflow_key = f"{workflow['name']}:{workflow['environment']}"

                    # Update matching scenarios
                    for scenario in MOCK_SCENARIOS.values():
                        if scenario.workflow_name:
                            # Determine expected environment (production vs test)
                            env = "test" if scenario.name == "test_signal" else "production"
                            expected_key = f"{scenario.workflow_name}:{env}"

                            if workflow_key == expected_key:
                                scenario.workflow_id = workflow['workflow_id']
                                synced_count += 1
                                print(f"  ✅ Synced {scenario.name}: {workflow['workflow_id']}")

                print(f"✅ Mock LLM synced {synced_count} scenarios from DataStorage")
                return True

        except requests.exceptions.RequestException as e:
            print(f"⏳ Waiting for DataStorage (attempt {attempt+1}/{max_retries}): {e}")
            time.sleep(2)

    # Graceful degradation: Log warning but still start
    print("⚠️  Warning: Could not sync with DataStorage, using default scenario UUIDs")
    print("    Tests may fail if DataStorage generated different UUIDs")
    return False

if __name__ == "__main__":
    print("🚀 Mock LLM v2.0 - Self-Discovery Pattern")

    # BEFORE serving HTTP traffic:
    sync_workflows_from_datastorage()

    # NOW ready to serve requests
    start_http_server()
```

```yaml
# test/infrastructure/holmesgpt_api.go - Mock LLM Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-llm
spec:
  template:
    spec:
      containers:
      - name: mock-llm
        image: kubernaut/mock-llm:e2e
        env:
        - name: DATA_STORAGE_URL
          value: "http://datastorage:8080"  # Self-discovery endpoint
        - name: SYNC_ON_STARTUP
          value: "true"
        readinessProbe:
          httpGet:
            path: /health?check=datastorage-sync  # Only ready after sync
            port: 8080
          initialDelaySeconds: 15  # Give DataStorage time to seed workflows
          periodSeconds: 5
          failureThreshold: 12     # 12 * 5s = 60s max wait
```

**Pros**:
- ✅ **Zero test orchestration**: Test suite just provisions infrastructure (Deploy → Wait for Ready)
- ✅ **Loose coupling**: Test suite knows nothing about Mock LLM's internal configuration needs
- ✅ **Kubernetes-native**: Readiness probe enforces "not ready until fully configured"
- ✅ **Restart-safe**: Mock LLM re-syncs on every pod restart automatically
- ✅ **Fast**: Sync happens during pod startup, not after (parallel with other startup tasks)
- ✅ **Deterministic**: Mock LLM won't be "ready" until UUIDs are synced
- ✅ **Graceful degradation**: Mock LLM can start even if DataStorage sync fails (logs warning)
- ✅ **Works everywhere**: Integration tests (REST API fallback), E2E tests (self-discovery primary)
- ✅ **Clean separation**: Test suite provisions infrastructure, services manage dependencies

**Cons**:
- ⚠️ **Startup dependency**: Mock LLM startup time increases by 2-5 seconds (DataStorage discovery)
  - **Mitigation**: Happens in parallel with other pod startup tasks, minimal impact
- ⚠️ **Requires HTTP client**: Mock LLM needs `requests` library for DataStorage queries
  - **Mitigation**: Already included in Python base image, no additional dependencies
- ⚠️ **Network reachability**: Mock LLM must be able to reach DataStorage service
  - **Mitigation**: Kubernetes DNS resolves `datastorage` service name automatically

**Confidence**: 90% (strongly recommended, aligns with Kubernetes best practices)

---

## Decision

**APPROVED: File-Based Configuration Pattern** (v3.0 refinement of Alternative 2)

**Evolution**:
- **v1.0**: Alternative 3 (Self-Discovery) → Deprecated due to timing issues
- **v2.0**: Alternative 2 (ConfigMap Pattern) → Deterministic, Kubernetes-native
- **v3.0**: **File-Based Configuration** → ConfigMap is delivery mechanism, not architecture

**Key Insight (v3.0)**: ConfigMap is just one way to deliver a YAML file to Mock LLM.
- **Integration tests**: Write file directly, mount as volume
- **E2E tests**: Use ConfigMap to deliver the same YAML structure
- **Mock LLM**: Generic file reader (`load_scenarios_from_file()`)

**Note**: Alternative 3 (Self-Discovery Pattern) was initially approved but later replaced with Alternative 2 after discovering timing issues where Mock LLM's self-discovery attempted to query DataStorage before workflows were seeded. v3.0 further simplifies by treating configuration as file-based (ConfigMap is E2E deployment detail).

### Rationale

1. **Deterministic Ordering**: Test suite controls the exact sequence:
   - Deploy DataStorage FIRST → Wait for Ready
   - Seed workflows SECOND → Capture UUIDs
   - Create ConfigMap THIRD → With actual UUIDs
   - Deploy Mock LLM FOURTH → Mounts ConfigMap with correct data
   - No race conditions or timing issues

2. **Kubernetes-Native Configuration**: Uses ConfigMaps for their intended purpose (application configuration data), follows established Kubernetes patterns.

3. **Zero Network Overhead**: Mock LLM reads UUIDs from mounted file at startup, no HTTP calls to DataStorage needed.

4. **Restart Safety**: ConfigMap persists across Mock LLM pod restarts, UUIDs remain correct without re-querying DataStorage.

5. **Simpler Mock LLM Logic**: Eliminates complex retry logic, exponential backoff, HTTP client dependencies, and graceful degradation code. Just reads a YAML file.

6. **Eliminates Timing Issues**: No dependency on DataStorage being reachable or workflows being seeded before Mock LLM starts. ConfigMap is the single source of truth.

**Key Insight**: Test fixture data (workflow UUIDs) is known at deployment time and doesn't change during test execution. ConfigMaps are the Kubernetes-native way to inject configuration data into pods at deployment time, eliminating the need for runtime service discovery.

---

## Implementation

### Phase 1: Test Suite Workflow Seeding and ConfigMap Creation

**File**: `test/infrastructure/aianalysis_e2e.go` - `CreateAIAnalysisClusterHybrid()`

**Sequential deployment with ConfigMap creation**:
```go
// PHASE 7a: Deploy DataStorage FIRST
DeployDataStorageTestServices(ctx, namespace, kubeconfigPath, builtImages["datastorage"], writer)

// PHASE 7b: Seed workflows and create ConfigMap
dataStorageURL := "http://localhost:38080"  // via port-forward
workflowUUIDs, err := SeedAIAnalysisTestWorkflows(dataStorageURL, writer)
createMockLLMConfigMap(ctx, namespace, kubeconfigPath, workflowUUIDs, writer)

// PHASE 7c: Deploy Mock LLM (mounts ConfigMap)
deployMockLLMInNamespace(ctx, namespace, kubeconfigPath, builtImages["mock-llm"], workflowUUIDs, writer)
```

**ConfigMap creation helper**:
```go
func createMockLLMConfigMap(ctx context.Context, namespace, kubeconfigPath string, workflowUUIDs map[string]string, writer io.Writer) error {
    yamlContent := "scenarios:\n"
    for key, uuid := range workflowUUIDs {
        yamlContent += fmt.Sprintf("  %s: %s\n", key, uuid)
    }

    manifest := fmt.Sprintf(`
apiVersion: v1
kind: ConfigMap
metadata:
  name: mock-llm-scenarios
  namespace: %s
data:
  scenarios.yaml: |
%s
`, namespace, yamlContent)

    cmd := exec.CommandContext(ctx, "kubectl", "--kubeconfig", kubeconfigPath, "apply", "-f", "-")
    cmd.Stdin = strings.NewReader(manifest)
    return cmd.Run()
}
```

**Example ConfigMap manifest** (generated by test suite):
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mock-llm-scenarios
  namespace: kubernaut-system
  labels:
    app: mock-llm
    component: test-infrastructure
data:
  scenarios.yaml: |
    scenarios:
      oomkill-increase-memory-v1:production: 42b90a37-0d1b-5561-911a-2939ed9e1c30
      oomkill-increase-memory-v1:staging: 7f3a8c92-4d5e-6b71-a22b-5d48fa7e9c21
      oomkill-increase-memory-v1:test: 9e5c7b31-2f4a-8d63-c44d-8e59ab6f2d42
      crashloop-config-fix-v1:production: 3d8f5a19-7c2b-9e43-b55e-1a73bc8d4f61
      crashloop-config-fix-v1:staging: 6b2e9d47-8f3c-1a54-d66f-2b84cd9e5g72
      crashloop-config-fix-v1:test: 8c4f1b59-9d4e-2b65-e77g-3c95de0f6h83
      node-drain-reboot-v1:production: 5e1a7c28-3d4b-0c52-f88h-4d06ef2g7i94
      memory-optimize-v1:production: 7g2c9e40-4e5d-1d63-g99i-5e17fg3h8j05
      generic-restart-v1:production: 9h3e0f51-5f6e-2e74-h00j-6f28gh4i9k16
      no-workflow-found-v1:production: 0i4f1g62-6g7f-3f85-i11k-7g39hi5j0l27
```

**Key-Value Format**:
- **Key**: `workflow_name:environment` (e.g., `oomkill-increase-memory-v1:production`)
- **Value**: UUID assigned by DataStorage (e.g., `42b90a37-0d1b-5561-911a-2939ed9e1c30`)
- **Total entries**: 18 workflows (6 base workflows × 3 environments)

### Phase 2: Mock LLM ConfigMap Loading Logic

**File**: `test/services/mock-llm/src/server.py`

**Add ConfigMap loading at startup**:
```python
def load_scenarios_from_configmap(config_path: str):
    """Load workflow UUIDs from Kubernetes ConfigMap at startup."""
    try:
        import yaml

        with open(config_path, 'r') as f:
            config = yaml.safe_load(f)

        scenarios_config = config.get('scenarios', {})
        synced_count = 0

        for scenario_name, workflow_uuid in scenarios_config.items():
            if scenario_name in MOCK_SCENARIOS:
                MOCK_SCENARIOS[scenario_name].workflow_id = workflow_uuid
                synced_count += 1
                print(f"  ✅ Loaded {scenario_name}: {workflow_uuid}")

        print(f"✅ Mock LLM loaded {synced_count}/{len(MOCK_SCENARIOS)} scenarios from ConfigMap")
        return True

    except FileNotFoundError:
        print(f"ℹ️  ConfigMap not found, using default scenario UUIDs")
        return False

if __name__ == "__main__":
    config_path = os.getenv("MOCK_LLM_CONFIG_PATH", "/config/scenarios.yaml")
    load_scenarios_from_configmap(config_path)
    start_http_server()
```

### Phase 3: Mock LLM Deployment Manifest

**File**: `test/infrastructure/holmesgpt_api.go` - `deployMockLLMInNamespace()`

**Add ConfigMap volume mount**:
```yaml
spec:
  containers:
  - name: mock-llm
    env:
    - name: MOCK_LLM_CONFIG_PATH
      value: "/config/scenarios.yaml"
    volumeMounts:
    - name: scenarios-config
      mountPath: /config
      readOnly: true
  volumes:
  - name: scenarios-config
    configMap:
      name: mock-llm-scenarios
```

**Simplified readiness probe** (no datastorage-sync check needed):
```yaml
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3
```

**Complete Mock LLM Deployment Example**:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-llm
  namespace: kubernaut-system
  labels:
    app: mock-llm
    component: test-infrastructure
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mock-llm
  template:
    metadata:
      labels:
        app: mock-llm
        component: test-infrastructure
    spec:
      containers:
      - name: mock-llm
        image: kubernaut/mock-llm:e2e
        imagePullPolicy: Never
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        env:
        - name: MOCK_LLM_HOST
          value: "0.0.0.0"
        - name: MOCK_LLM_PORT
          value: "8080"
        - name: MOCK_LLM_CONFIG_PATH
          value: "/config/scenarios.yaml"
        - name: SYNC_ON_STARTUP
          value: "false"  # ConfigMap pattern, no HTTP sync
        volumeMounts:
        - name: scenarios-config
          mountPath: /config
          readOnly: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"
      volumes:
      - name: scenarios-config
        configMap:
          name: mock-llm-scenarios
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: mock-llm
  namespace: kubernaut-system
  labels:
    app: mock-llm
    component: test-infrastructure
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: mock-llm
```

**Key Configuration Points**:
- **ConfigMap Volume**: `scenarios-config` → `/config/scenarios.yaml`
- **No HTTP Sync**: `SYNC_ON_STARTUP=false` (ConfigMap pattern)
- **Config Path**: `MOCK_LLM_CONFIG_PATH=/config/scenarios.yaml`
- **Simple Readiness**: `/health` (no datastorage-sync check)
- **Fast Startup**: 5s initial delay (vs 15s with HTTP sync)

### Data Flow

```
Test Suite Startup (Deterministic Sequential Steps):
1. Test BeforeSuite: Deploy DataStorage → Wait for Ready
2. Test BeforeSuite: Seed workflows via DataStorage API → Capture UUIDs
   - POST /api/workflows → {"workflow_id": "uuid1", "name": "oomkill-increase-memory-v1", ...}
3. Test BeforeSuite: Create ConfigMap with captured UUIDs
   - kubectl apply ConfigMap: scenarios.yaml = {oomkilled: "uuid1", crashloop: "uuid2", ...}
4. Test BeforeSuite: Deploy Mock LLM (mounts ConfigMap)
   - volumeMount: /config/scenarios.yaml
   ↓
Mock LLM Pod Initialization:
5. Mock LLM starts → Executes load_scenarios_from_configmap("/config/scenarios.yaml")
6. Mock LLM reads: {"scenarios": {"oomkilled": "uuid1", "crashloop": "uuid2", ...}}
7. Mock LLM updates: MOCK_SCENARIOS["oomkilled"].workflow_id = "uuid1"
8. Mock LLM readiness: /health returns 200 OK (no sync check needed)
   ↓
Kubernetes Readiness Detection:
9. Kubernetes readiness probe: GET http://mock-llm:8080/health → 200 OK
10. Kubernetes marks: Mock LLM pod as Ready (adds to service endpoints)
11. Test BeforeSuite: Detects Mock LLM Ready → Suite continues
    ↓
Test Execution:
12. HAPI calls Mock LLM: POST http://mock-llm:8080/v1/chat/completions
13. Mock LLM returns: workflow_id="uuid1" (loaded from ConfigMap)
14. Tests validate: Expected UUID matches actual DataStorage UUID ✅
```

---

## Consequences

### Positive

- ✅ **Test suite simplification**: Eliminates 20-30 lines of UUID synchronization orchestration code per test suite
- ✅ **Architectural clarity**: Test suite provisions infrastructure, services manage dependencies
- ✅ **Restart resilience**: Mock LLM automatically re-syncs after pod restarts (no manual intervention)
- ✅ **Deterministic readiness**: Mock LLM "Ready" truly means "ready to serve requests with correct data"
- ✅ **Reusable pattern**: Can be applied to future test services (Mock K8s API, Mock Prometheus, etc.)
- ✅ **Kubernetes-native**: Follows established patterns (service discovery, readiness probes)
- ✅ **Debuggability**: Mock LLM logs show exact sync status (success/failure/degraded)

### Negative

- ⚠️ **Startup latency**: Mock LLM pod becomes Ready 2-5 seconds later (DataStorage query time)
  - **Mitigation**: Acceptable trade-off for architectural clarity, happens during pod startup
- ⚠️ **Network dependency**: Mock LLM must reach DataStorage (Kubernetes DNS required)
  - **Mitigation**: Standard Kubernetes cluster feature, always available in E2E tests
- ⚠️ **Code complexity**: Mock LLM startup logic is more complex (HTTP client, retry logic)
  - **Mitigation**: Well-encapsulated in `sync_workflows_from_datastorage()`, single responsibility

### Neutral

- 🔄 **Alternative approaches still work**: REST API endpoint (`/api/test/update-uuids`) remains for Integration tests
- 🔄 **Hybrid support**: Mock LLM supports BOTH self-discovery (E2E) and REST API updates (Integration)

---

## Validation Results

### Confidence Assessment Progression

**Version 1.0 (Self-Discovery - Deprecated)**:
- **Initial assessment**: 75% confidence (concept validation)
- **After architecture review**: 85% confidence (alignment with Kubernetes patterns confirmed)
- **After user approval**: 90% confidence (approved approach, implementation pending)
- **After E2E testing**: 40% confidence (timing issues discovered, race conditions confirmed)

**Version 2.0 (ConfigMap Pattern - Current)**:
- **After pattern switch**: 95% confidence (deterministic ordering eliminates race conditions)
- **After implementation**: 95% confidence (Phase 1-3 completed, pending E2E validation)

### Key Validation Points (v2.0)

- ✅ **Deterministic ordering**: ConfigMap created AFTER workflow seeding, eliminating race conditions
- ✅ **Architectural simplicity**: Mock LLM reads file at startup, no HTTP client or retry logic needed
- ✅ **Test suite simplification**: Eliminates orchestration complexity (user confirmed requirement)
- ✅ **Restart safety**: ConfigMap persists across pod restarts
- ✅ **Kubernetes-native**: Uses ConfigMaps for their intended purpose (application configuration)
- ✅ **Faster startup**: No HTTP sync delay, Mock LLM ready in 5-10 seconds

### Lessons Learned (v1.0 → v2.0 Transition)

**What Went Wrong with Self-Discovery (v1.0)**:
- Timing assumption was flawed: Mock LLM started before workflows were seeded
- Kubernetes parallel deployment made ordering non-deterministic
- Readiness probe couldn't distinguish "still syncing" from "sync failed"
- HTTP retry logic added complexity without solving root cause

**Why ConfigMap Pattern Succeeded (v2.0)**:
- Test suite controls exact sequence: Deploy → Seed → ConfigMap → Deploy Mock LLM
- No runtime synchronization needed, all configuration known at deployment time
- Simpler logic: file read vs HTTP client with retries
- Better matches Kubernetes patterns: ConfigMaps for static configuration

---

## Validation Results (v3.0 - 2026-01-14)

### Test Execution Summary

**Overall Status**: ✅ **100% PASS RATE** (104/104 tests across all 3 tiers)

#### Tier 1: Python Unit Tests
**Location**: `test/services/mock-llm/tests/test_config_loading.py`
**Results**: **11/11 tests passing** ✅
**Runtime**: < 1 second

**Test Coverage**:
- ✅ Valid YAML file loading
- ✅ Invalid YAML handling
- ✅ Missing `scenarios` key handling
- ✅ Empty file handling
- ✅ Multiple environment matching (`production`, `staging`, `test`)
- ✅ Partial workflow matches
- ✅ Invalid key format handling
- ✅ E2E realistic scenario simulation
- ✅ Integration test file mount simulation
- ✅ ConfigMap mount integration test

**Validation**: File-based configuration loading logic is robust and handles all edge cases.

#### Tier 2: Integration Tests
**Location**: `test/integration/aianalysis/`
**Results**: **57/57 tests passing** ✅
**Runtime**: 5 minutes 55 seconds

**Test Coverage**:
- ✅ Mock LLM file-based config mounting (via host volume)
- ✅ `WriteMockLLMConfigFile()` helper validation
- ✅ DataStorage workflow seeding and UUID capture
- ✅ Mock LLM container startup with config file
- ✅ HAPI integration with Mock LLM (file-based UUIDs)
- ✅ AIAnalysis controller reconciliation
- ✅ Audit trail validation
- ✅ Metrics collection

**Critical Fixes Applied**:
1. **HAPI Dockerfile.e2e**: Added missing `pip install ./src/clients/datastorage`
   - **Issue**: ImportError preventing HAPI container from starting
   - **Fix**: Explicit pip install in Dockerfile (line 38)
   - **Result**: HAPI imports datastorage client successfully ✅

2. **Integration Test Config Path**: Changed from `/tmp` to test directory
   - **Issue**: macOS clearing `/tmp` between test steps
   - **Fix**: Use `filepath.Join(filepath.Dir(GinkgoT().TempDir()), "mock-llm-config")`
   - **Result**: Config file persists across test steps ✅

**Validation**: File-based configuration works correctly in integration tests with direct file mounting.

#### Tier 3: E2E Tests
**Location**: `test/e2e/aianalysis/`
**Results**: **36/36 tests passing** ✅
**Runtime**: 6 minutes 3 seconds

**Test Coverage**:
- ✅ Kind cluster deployment
- ✅ ConfigMap creation with workflow UUIDs
- ✅ Mock LLM ConfigMap volume mount (Kubernetes-native delivery)
- ✅ Mock LLM reading ConfigMap-delivered YAML file
- ✅ HAPI calling Mock LLM with correct UUIDs
- ✅ AIAnalysis E2E workflows (OOMKilled, CrashLoopBackOff, NodeNotReady)
- ✅ Audit trail E2E validation
- ✅ Error handling and edge cases

**Validation**: ConfigMap delivery mechanism works correctly in Kubernetes, Mock LLM reads mounted YAML successfully.

### Code Quality Validation

#### Build Validation
- ✅ Go infrastructure packages build without errors
- ✅ Python Mock LLM server compiles successfully
- ✅ No lint or vet violations

#### Git Artifacts Cleanup
- ✅ Added `.gitignore` for pytest artifacts (`.hypothesis/`)
- ✅ All generated files properly ignored

### Architecture Validation

#### Confirmed Benefits
1. **Simpler Codebase**: 40+ lines of HTTP endpoint code removed ✅
2. **Faster Tests**: No HTTP roundtrips at startup ✅
3. **Better Testability**: 11 comprehensive unit tests ✅
4. **Clearer Semantics**: ConfigMap is delivery mechanism, not core architecture ✅
5. **Fewer Dependencies**: Removed `requests` library ✅
6. **Fewer Failure Modes**: No network dependencies at startup ✅

#### Confidence Assessment Progression
- **v1.0 (Self-Discovery)**: 90% → 40% (timing issues discovered)
- **v2.0 (ConfigMap)**: 95% (deterministic ordering)
- **v3.0 (File-Based)**: 98% (simplified implementation)
- **v3.0 (Validated)**: **100%** (all tests passing) ✅

### Production Readiness

**Status**: ✅ **PRODUCTION READY**

**Evidence**:
- 100% test pass rate across all 3 testing tiers
- All critical bugs fixed (HAPI ImportError, config path)
- Comprehensive documentation (DD-TEST-011 + README)
- Unit test coverage for configuration loading
- Validated in real Kubernetes cluster (Kind)
- Restart-safe (ConfigMap persists)
- Fast startup (no HTTP overhead)

**Ready for**: Immediate use in AIAnalysis testing, extensible to other services requiring Mock LLM

---

## Related Decisions

- **Builds On**: DD-TEST-002 (Integration Test Container Orchestration) - Sequential startup principles
- **Builds On**: DD-TEST-010 (Controller-Per-Process Architecture) - Per-process environment isolation
- **Supports**: BR-AI-001 (AI-Driven Remediation) - Mock LLM must return correct workflow UUIDs for AIAnalysis testing

---

## Review & Evolution

### When to Revisit

- If Mock LLM startup time exceeds 10 seconds (performance regression)
- If DataStorage workflow catalog API changes (breaking change detection)
- If other test services need similar self-discovery patterns (pattern generalization)
- If Kubernetes readiness probe semantics change (unlikely but possible)

### Success Metrics

- **Startup time**: Mock LLM becomes Ready within 20 seconds (including DataStorage sync)
- **Test reliability**: E2E tests pass rate >95% (UUID synchronization errors eliminated)
- **Code simplification**: Test suite UUID orchestration code reduced by >80%
- **Restart recovery**: Mock LLM pod restarts recover automatically within 20 seconds

---

## Implementation Checklist

### v2.0 Implementation (ConfigMap Pattern)
- [x] **Phase 1**: Add `load_scenarios_from_configmap()` to `test/services/mock-llm/src/server.py`
- [x] **Phase 1**: Update `__main__.py` to call `load_scenarios_from_configmap()` before starting HTTP server
- [x] **Phase 1**: Add `pyyaml` dependency to `test/services/mock-llm/requirements.txt`
- [x] **Phase 2**: Add `createMockLLMConfigMap()` function to `test/infrastructure/aianalysis_e2e.go`
- [x] **Phase 2**: Update `CreateAIAnalysisClusterHybrid()` to create ConfigMap after seeding workflows
- [x] **Phase 2**: Update `deployMockLLMInNamespace()` signature to accept `workflowUUIDs` parameter
- [x] **Phase 3**: Add ConfigMap volume mount to Mock LLM deployment manifest
- [x] **Phase 3**: Add `MOCK_LLM_CONFIG_PATH` environment variable to Mock LLM deployment
- [x] **Phase 3**: Simplify readiness probe to `/health` (no datastorage-sync check)
- [x] **Documentation**: Add comprehensive ConfigMap examples and deployment specs to DD-TEST-011 (v2.1)

### v3.0 Refactoring (File-Based Pattern)
- [x] **Refactor**: Renamed `load_scenarios_from_configmap()` → `load_scenarios_from_file()` (generic file reader)
- [x] **Cleanup**: Removed HTTP PUT endpoint `/api/test/update-uuids` (~40 lines)
- [x] **Cleanup**: Removed all HTTP self-discovery code and `requests` dependency
- [x] **Cleanup**: Removed obsolete env vars (`DATA_STORAGE_URL`, `SYNC_ON_STARTUP`) from E2E deployment
- [x] **Integration**: Added `WriteMockLLMConfigFile()` helper for integration tests
- [x] **Integration**: Updated `MockLLMConfig` struct with `ConfigFilePath` field
- [x] **Integration**: Modified `StartMockLLMContainer()` to mount config file if provided
- [x] **Testing**: Created 11 Python unit tests (`test_config_loading.py`)
- [x] **Documentation**: Created comprehensive `test/services/mock-llm/README.md` (467 lines)
- [x] **Documentation**: Updated DD-TEST-011 to v3.0 with changelog and file-based pattern

### v3.0 Validation
- [x] **Python Unit Tests**: 11/11 tests passing ✅
- [x] **Integration Tests**: 57/57 tests passing ✅ (file mounting validated)
- [x] **E2E Tests**: 36/36 tests passing ✅ (ConfigMap delivery validated)
- [x] **Critical Fixes**: HAPI Dockerfile.e2e fix (pip install datastorage client)
- [x] **Critical Fixes**: Integration test config path fix (/tmp → test directory)
- [x] **Git Cleanup**: Added `.gitignore` for pytest artifacts
- [x] **Code Quality**: All code builds without errors
- [x] **Validation Complete**: 100% test pass rate achieved ✅

### Future Enhancements (Optional)
- [ ] **Optimization**: Add caching for repeated YAML file reads (if performance issue arises)
- [ ] **Extensibility**: Add JSON format support alongside YAML (if needed by other services)
- [ ] **Observability**: Add Prometheus metrics for config load success/failure rates (if needed)

---

## Document Metadata

**Document Status**: ✅ **AUTHORITATIVE** (v3.0 validated; v4.0 planned)
**Version**: 4.0 (Go Rewrite & Deterministic UUIDs — Planned)
**Approved By**: User (jordigilh) - 2026-01-14 (v1.0), 2026-01-14 (v2.0), 2026-01-14 (v2.1), 2026-01-14 (v3.0), 2026-03-04 (v4.0 specification)
**Implementation Status**: ✅ v3.0 **COMPLETED & VALIDATED** | 📋 v4.0 **SPECIFICATION PHASE** (#531)
**Documentation Status**: ✅ **COMPREHENSIVE** (DD-TEST-011 + test/services/mock-llm/README.md + Mock LLM BUSINESS_REQUIREMENTS.md)
**Validation Status**: ✅ v3.0 **100% PASS RATE** (Python Unit: 11/11, Integration: 57/57, E2E: 36/36)
**Production Ready**: ✅ v3.0 **YES** | 📋 v4.0 pending implementation
**Next Review**: When #531 implementation begins

**Revision History**:
- **v4.0** (2026-03-04): Go Rewrite & Deterministic UUIDs — eliminates ConfigMap sync via shared UUID function (#531, #548). Adds DAG engine, scenario registry, verification API, fault injection, shared types, pillar composition, metrics. See Mock LLM BUSINESS_REQUIREMENTS.md (46 BRs)
- **v3.0-validated** (2026-01-14): Validation complete - 104/104 tests passing, HAPI fix, integration test path fix, production ready
- **v3.0** (2026-01-14): File-based refactoring - Removed HTTP endpoints, simplified to file reader, added unit tests, created comprehensive README
- **v2.1** (2026-01-14): Documentation enhancement - Added complete ConfigMap YAML examples, Mock LLM deployment specs, and configuration examples
- **v2.0** (2026-01-14): Implementation change - Switched from Self-Discovery (Alt 3) to ConfigMap (Alt 2) - Resolved timing issues
- **v1.0** (2026-01-14): Initial approval of Self-Discovery Pattern (Alt 3) - Deprecated due to timing issues

**Key Artifacts**:
- **Architectural Decision Record**: `docs/architecture/decisions/DD-TEST-011-mock-llm-self-discovery-pattern.md` (this document, 816 lines)
- **Service Documentation**: `test/services/mock-llm/README.md` (467 lines)
- **Unit Tests**: `test/services/mock-llm/tests/test_config_loading.py` (348 lines, 11 tests)
- **Integration Helpers**: `test/integration/aianalysis/test_workflows.go` (`WriteMockLLMConfigFile()`)
- **Infrastructure Support**: `test/infrastructure/mock_llm.go` (`ConfigFilePath` field, volume mounting)
