# Data Storage Service - Business Requirements

**Version**: v1.5
**Date**: March 4, 2026
**Status**: Production-Ready (per ADR-032)
**Service Type**: Stateless HTTP REST API + Database Layer

---

## 📝 **Changelog**

### **v1.5** (March 4, 2026)
- **UPDATED**: BR-STORAGE-038 (Workflow Catalog Create API)
  - `POST /api/v1/workflows` is now an **internal API** consumed exclusively by the AuthWebhook
  - Registration payload changed from OCI pullspec (`containerImage`) to inline content (`content`, `source`, `registeredBy`)
  - User-facing registration is via RemediationWorkflow CRD (BR-WORKFLOW-006, ADR-058)
  - DS validates inline schema content (BR-WORKFLOW-004), action type FK, and uniqueness constraints
  - Duplicate `(workflow_name, version)` triggers re-enable from disabled state
- **References**: [ADR-058](../../../architecture/decisions/ADR-058-webhook-driven-workflow-registration.md), [BR-WORKFLOW-006](../../../requirements/BR-WORKFLOW-006-remediation-workflow-crd.md)

### **v1.4** (December 6, 2025)
- **RESOLVED**: BR numbering gap triage (BR-004, 008, 018, 029)
  - BR-STORAGE-004: Added as "Idempotent Schema Initialization" (was in code but not documented)
  - BR-STORAGE-008: Added as "Embedding Generation" (was in code but not documented)
  - BR-STORAGE-018: Added as "Structured Logging" (was in code but not documented)
  - BR-STORAGE-029: Marked as "Reserved" (intentionally skipped for future use)
- **UPDATED**: Summary statistics corrected to reflect all 45 BRs
- **UPDATED**: Coverage analysis to include all BR categories
- **UPDATED**: Confidence assessment now 100% (all gaps resolved)
- **FIXED**: Last Updated date now matches changelog

### **v1.3** (December 5, 2025)
- **ADDED**: Category 10 - Workflow Catalog CRUD API (BR-STORAGE-038 to BR-STORAGE-042)
  - BR-STORAGE-038: Workflow Catalog Create API (`POST /api/v1/workflows`)
  - BR-STORAGE-039: Workflow Catalog Retrieval API (`GET /api/v1/workflows/{workflow_id}`)
  - BR-STORAGE-040: Workflow Catalog Search API (`POST /api/v1/workflows/search`)
  - BR-STORAGE-041: Workflow Catalog Update API (`PATCH /api/v1/workflows/{workflow_id}`)
  - BR-STORAGE-042: Workflow Catalog Disable API (`PATCH /api/v1/workflows/{workflow_id}/disable`)
- **DOCUMENTED**: BR-STORAGE-039 cross-service integration with HolmesGPT-API (Q17 response)
  - Use case: HAPI `validate_workflow_exists` tool
  - Use case: Parameter schema retrieval for validation
  - Use case: Container image pullspec validation
- **Reference**: AIANALYSIS_TO_HOLMESGPT_API_TEAM.md Q17

### **v1.2** (November 19, 2025)
- **ADDED**: BR-STORAGE-035 - Cursor-based pagination for audit event queries (V1.1 planned)
  - Rationale: Handles real-time data with high write volumes more reliably than offset-based
  - Reference: DD-STORAGE-010 (Query API Pagination Strategy)
- **ADDED**: BR-STORAGE-036 - Parent event lookup index for performance optimization (V1.1 planned)
  - Rationale: Optimize child event lookups and event chain traversal for RCA
  - Reference: FK_CONSTRAINT_IMPLEMENTATION_SUMMARY.md
- **ADDED**: BR-STORAGE-037 - Historical parent-child backfill for data integrity (V1.1 planned)
  - Rationale: Enable historical event chain queries for compliance and analytics
  - Reference: FK_CONSTRAINT_IMPLEMENTATION_SUMMARY.md

### **v1.1** (November 13, 2025)
- **CORRECTED**: BR-STORAGE-012 - Semantic search is for **workflow catalog**, not audit records
  - Changed from: "Generate embeddings from audit text"
  - Changed to: "Generate embeddings from workflow catalog"
  - Rationale: DD-CONTEXT-005 requires workflow semantic search for incident remediation
  - Impact: V1.0 semantic search is for workflow selection; audit embeddings deferred to V2.0 RAR
  - **Terminology**: Per DD-NAMING-001, using "Remediation Workflow" (not "Remediation Playbook")
- **DEFERRED**: BR-STORAGE-009 - Workflow embedding caching deferred to V1.1
  - **V1.0**: No caching (workflows managed via SQL, no cache invalidation control)
  - **V1.1**: Caching enabled (CRD controller provides cache invalidation via REST API)
  - Rationale: DD-STORAGE-006 analysis shows no-cache is acceptable for V1.0 (92% confidence)
  - Performance: 2.5s latency acceptable for AI workflow; < 3% CPU utilization

### **v1.0** (November 8, 2025)
- Initial production-ready business requirements
- 30 BRs with 100% test coverage

---

## 📊 **Summary Statistics**

- **Total BRs**: 45 Data Storage BRs
  - **Active BRs (V1.0)**: 41 (91%)
  - **Planned BRs (V1.1)**: 3 (7%)
  - **Reserved BRs**: 1 (2%) - BR-029 reserved for future use
- **Deprecated BRs**: 0 (0%)
- **V2 Deferred BRs**: 0 (0%)

### **BR Numbering Summary**

| Category | BR Range | Count | Status |
|----------|----------|-------|--------|
| Audit Persistence | 001-004 | 4 | ✅ Active |
| Query API | 005-008 | 4 | ✅ Active |
| Observability | 009, 010, 018, 019 | 4 | ✅ Active (009 deferred V1.1) |
| Security | 011, 025, 026 | 3 | ✅ Active |
| Self-Auditing | 180-182 | 3 | ✅ Active |
| Embedding & Vector | 012, 013 | 2 | ✅ Active |
| Dual-Write | 014-016 | 3 | ✅ Active |
| Error Handling | 017 | 1 | ✅ Active |
| REST API | 020-028 | 9 | ✅ Active |
| Reserved | 029 | 1 | 🔒 Reserved |
| Aggregation API | 030-034 | 5 | ✅ Active |
| V1.1 Enhancements | 035-037 | 3 | 📋 Planned |
| Workflow CRUD | 038-042 | 5 | ✅ Active |
| **Total** | - | **45** | - |

### **Test Coverage by Tier**

| Tier | Test Count | Percentage |
|------|------------|------------|
| **Unit Tests** | ~580 specs | 70%+ coverage |
| **Integration Tests** | 163 specs | Component integration |
| **E2E Tests** | 13 specs | Critical paths |
| **Total** | ~756 specs | Defense-in-depth |

**Overall BR Coverage**: 41/41 Active V1.0 BRs (100%) - All active BRs have test coverage

---

## 🎯 **Service Overview**

The Data Storage Service is the **exclusive database access layer** for Kubernaut, mandated by ADR-032. All services (except Data Storage itself) must access the database exclusively via the Data Storage Service REST API.

### **Core Responsibilities**

1. **Audit Persistence**: Store remediation, notification, and action execution audits
2. **Dual-Write Coordination**: Atomic writes to PostgreSQL + Redis Vector DB
3. **Query API**: REST endpoints for filtering, pagination, and aggregation
4. **Input Validation**: Sanitization and validation of all incoming data
5. **Observability**: Prometheus metrics for all operations
6. **Production Readiness**: Graceful shutdown (DD-007), RFC 7807 errors

### **Key Architecture Decisions**

- **ADR-032**: Exclusive database access layer (no direct PostgreSQL access from other services)
- **DD-007**: Kubernetes-aware graceful shutdown pattern
- **ADR-016**: Podman-based integration testing infrastructure

---

## 📋 **Business Requirements**

### **Category 1: Audit Persistence (BR-STORAGE-001 to BR-STORAGE-003)**

#### **BR-STORAGE-001: Persist Notification Audit**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Persist notification audit records to PostgreSQL with complete metadata (remediation ID, notification ID, recipient, channel, status, delivery status, escalation level)
- **Business Value**: Enable notification delivery tracking and escalation analysis
- **Test Coverage**:
  - Integration: `test/integration/datastorage/repository_test.go:27`
  - Integration: `test/integration/datastorage/http_api_test.go:21`
  - Integration: `test/integration/datastorage/suite_test.go:39`
- **Implementation**: `pkg/datastorage/repository/notification_audit_repository.go`
- **ADR References**: ADR-032 (exclusive database access)

#### **BR-STORAGE-002: Typed Error Handling for Dual-Write**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Provide typed errors for dual-write operations (PostgreSQL failure, Vector DB failure, validation failure, context canceled) to enable precise error handling and recovery
- **Business Value**: Enable intelligent error handling and retry strategies based on failure type
- **Test Coverage**:
  - Unit: `test/unit/datastorage/errors_dualwrite_test.go:18`
  - Unit: `test/unit/datastorage/dualwrite_test.go:186`
- **Implementation**: `pkg/datastorage/dualwrite/errors.go`
- **Related BRs**: BR-STORAGE-014 (atomic dual-write), BR-STORAGE-015 (graceful degradation)

#### **BR-STORAGE-003: Database Version Validation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Validate PostgreSQL and pgvector versions at startup to ensure compatibility (PostgreSQL ≥14, pgvector ≥0.5.0)
- **Business Value**: Prevent runtime failures due to incompatible database versions
- **Test Coverage**:
  - Unit: `test/unit/datastorage/validator_schema_test.go:42` (supported versions)
  - Unit: `test/unit/datastorage/validator_schema_test.go:85` (unsupported PostgreSQL)
  - Unit: `test/unit/datastorage/validator_schema_test.go:117` (unsupported pgvector)
- **Implementation**: `pkg/datastorage/schema/validator.go`
- **ADR References**: ADR-032 (database layer responsibility)

#### **BR-STORAGE-004: Idempotent Schema Initialization**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Ensure database schema initialization is idempotent - repeated migrations do not fail or corrupt data
- **Business Value**: Enable safe service restarts and rolling deployments without manual database intervention
- **Test Coverage**:
  - Unit: `test/unit/datastorage/validator_schema_test.go` (schema validation)
  - Integration: `test/integration/datastorage/schema_validation_test.go`
- **Implementation**: `migrations/*.sql` (Goose migrations with idempotent patterns)
- **Related BRs**: BR-STORAGE-003 (database version validation)
- **Note**: Also referenced in code as "cross-service writes" for audit trail action execution

---

### **Category 2: Query API (BR-STORAGE-005 to BR-STORAGE-008)**

#### **BR-STORAGE-005: Query API with Filtering**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Provide REST API for querying remediation audits with filtering by namespace, status, phase, and combinations thereof
- **Business Value**: Enable targeted audit queries for analysis and troubleshooting
- **Test Coverage**:
  - Unit: `test/unit/datastorage/query_test.go:32` (9 filter combinations)
- **Implementation**: `pkg/datastorage/query/service.go`, `pkg/datastorage/query/builder.go`
- **Related BRs**: BR-STORAGE-006 (pagination), BR-STORAGE-022 (query filtering)

#### **BR-STORAGE-006: Pagination Support**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Support pagination for large result sets with configurable limit (1-1000) and offset
- **Business Value**: Enable efficient handling of large audit datasets without memory exhaustion
- **Test Coverage**:
  - Unit: `test/unit/datastorage/query_test.go:154` (pagination scenarios)
- **Implementation**: `pkg/datastorage/query/builder.go`
- **Related BRs**: BR-STORAGE-023 (pagination validation)

#### **BR-STORAGE-007: Query Performance Tracking**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Track query duration metrics for performance monitoring and optimization
- **Business Value**: Enable identification of slow queries and performance bottlenecks
- **Test Coverage**:
  - Unit: `test/unit/datastorage/metrics_test.go:179`
  - Unit: `test/unit/datastorage/metrics_test.go:354` (benchmark overhead)
- **Implementation**: `pkg/datastorage/metrics/metrics.go`
- **Related BRs**: BR-STORAGE-019 (Prometheus metrics)

#### **BR-STORAGE-008: Embedding Generation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Generate 384-dimensional embeddings using sentence-transformers model for semantic search capabilities
- **Business Value**: Enable semantic similarity search for workflow catalog queries
- **Test Coverage**:
  - Unit: `test/unit/datastorage/embedding_test.go`
  - Unit: `test/unit/datastorage/embedding_client_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
- **Implementation**: `pkg/datastorage/embedding/pipeline.go`, `pkg/datastorage/dualwrite/coordinator.go`
- **Related BRs**: BR-STORAGE-012 (workflow catalog embedding), BR-STORAGE-009 (cache tracking)
- **ADR References**: DD-STORAGE-004 (Embedding Caching Strategy), DD-STORAGE-005 (pgvector String Format)

---

### **Category 3: Observability (BR-STORAGE-009, BR-STORAGE-010, BR-STORAGE-018, BR-STORAGE-019)**

#### **BR-STORAGE-009: Cache Hit/Miss Tracking** ⚠️ DEPRECATED (2026-03)
- **Priority**: ~~P1~~ N/A
- **Status**: ⚠️ **DEPRECATED** — Playbook embedding cache is no longer relevant. The current workflow catalog implementation uses CRD-based registration (BR-WORKFLOW-006) with SQL-backed search (not playbooks or embeddings). This BR was written before the current DataStorage architecture was finalized.
- **Original Description**: ~~Track Redis cache hit and miss rates for playbook embedding cache performance monitoring~~
- **Original Business Value**: ~~Enable cache optimization and identify cache inefficiencies for playbook queries~~
- **Scope Change (v1.1)**:
  - ❌ **OLD**: Cache audit embeddings (INCORRECT - low hit rate, no business value)
  - ✅ **NEW**: Cache playbook embeddings (CORRECT - high hit rate, playbooks queried repeatedly)
- **Cache Strategy**:
  - **V1.0**: ⏸️ **NO CACHING** (deferred to V1.1 per DD-STORAGE-006)
  - **V1.1**: Materialized view or Redis with CRD-triggered invalidation
  - **Rationale**: V1.0 SQL-only playbook management cannot trigger cache invalidation; no-cache avoids stale data risk (92% confidence)
- **Performance (V1.0 No-Cache)**:
  - Latency: 2.5s per query (50 playbooks × 50ms/playbook)
  - CPU Usage: < 3% (acceptable for 1,000 queries/day)
  - Acceptable for AI workflow (total AI decision time: 5-10s)
- **Test Coverage**:
  - Unit: `test/unit/datastorage/metrics_test.go:124` (cache hit) - V1.1
  - Unit: `test/unit/datastorage/metrics_test.go:132` (cache miss) - V1.1
  - Unit: `test/unit/datastorage/metrics_test.go:346` (benchmark overhead) - V1.1
- **Implementation**: `pkg/datastorage/metrics/metrics.go` (V1.1)
- **Related BRs**: BR-STORAGE-012 (playbook embedding generation)
- **Design Decisions**: DD-STORAGE-006 (V1.0 No-Cache Decision)

#### **BR-STORAGE-010: Input Validation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Validate all incoming audit records for required fields, field lengths, and data types before persistence
- **Business Value**: Prevent invalid data from corrupting the database and ensure data quality
- **Test Coverage**:
  - Unit: `test/unit/datastorage/validation_test.go:31` (20+ validation scenarios)
  - Unit: `test/unit/datastorage/metrics_test.go:157` (validation failure tracking)
  - Integration: `test/integration/datastorage/repository_test.go:28`
- **Implementation**: `pkg/datastorage/validation/validator.go`, `pkg/datastorage/validation/rules.go`
- **Related BRs**: BR-STORAGE-011 (input sanitization)

#### **BR-STORAGE-018: Structured Logging**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Implement structured JSON logging following DD-005 Observability Standards for all operations
- **Business Value**: Enable log aggregation, correlation, and debugging in production environments
- **Test Coverage**:
  - Integration: Validated via log output inspection in integration tests
- **Implementation**: `pkg/datastorage/server/server.go`, all handlers
- **Related BRs**: BR-STORAGE-019 (Prometheus metrics)
- **ADR References**: DD-005 (Observability Standards)

#### **BR-STORAGE-019: Prometheus Metrics**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Expose Prometheus metrics for all operations (write total, write duration, dual-write success/failure, cache hits/misses, validation failures)
- **Business Value**: Enable comprehensive observability and alerting for production operations
- **Test Coverage**:
  - Unit: `test/unit/datastorage/metrics_test.go:32` (comprehensive metrics tests)
- **Implementation**: `pkg/datastorage/metrics/metrics.go`
- **ADR References**: ADR-032 (observability requirement), DD-005 (Observability Standards)

---

### **Category 4: Input Sanitization & Security (BR-STORAGE-011, BR-STORAGE-025, BR-STORAGE-026)**

#### **BR-STORAGE-011: Input Sanitization**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Sanitize all text inputs to prevent XSS attacks (strip script tags, event handlers, dangerous HTML)
- **Business Value**: Prevent security vulnerabilities from malicious input
- **Test Coverage**:
  - Unit: `test/unit/datastorage/sanitization_test.go:27` (10+ XSS scenarios)
- **Implementation**: `pkg/datastorage/validation/validator.go`
- **Related BRs**: BR-STORAGE-010 (input validation)

#### **BR-STORAGE-025: SQL Injection Prevention**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Prevent SQL injection attacks through parameterized queries and input validation at handler level
- **Business Value**: Protect database from SQL injection attacks
- **Test Coverage**:
  - Unit: `test/unit/datastorage/handlers_test.go:230` (handler-level protection)
  - Unit: `test/unit/datastorage/query_builder_test.go:50` (query builder protection)
- **Implementation**: `pkg/datastorage/query/builder.go`, `pkg/datastorage/server/handler.go`
- **Related BRs**: BR-STORAGE-010 (input validation)

#### **BR-STORAGE-026: Unicode Support**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Support Unicode characters (Arabic, Chinese, Thai, emoji) in query parameters (namespace, status, phase) without SQL injection or data corruption
- **Business Value**: Enable global deployments with international namespace names and emoji-based identifiers
- **Test Coverage**:
  - Unit: `test/unit/datastorage/query_builder_test.go:69` (Arabic, Chinese, Thai, Emoji, Mixed)
- **Implementation**: `pkg/datastorage/query/builder.go`
- **Related BRs**: BR-STORAGE-025 (SQL injection protection), BR-STORAGE-005 (query filtering)
- **Technical Details**:
  - Parameterized queries prevent SQL injection with Unicode
  - UTF-8 encoding preserved through PostgreSQL
  - Test cases: Arabic (مساحة-الإنتاج), Chinese (生产环境), Thai (สภาพแวดล้อม-การผลิต), Emoji (prod-🚀), Mixed (prod-环境-🔥)

---

### **Category 4.5: Self-Auditing (BR-STORAGE-180, BR-STORAGE-181, BR-STORAGE-182)**

#### **BR-STORAGE-180: Self-Auditing Requirement**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Data Storage Service must generate audit traces for its own operations (meta-audit)
- **Business Value**: Enable compliance tracking and troubleshooting of audit write operations
- **Test Coverage**:
  - Unit: `pkg/audit/internal_client_test.go` (8 test scenarios)
  - Integration: `test/integration/datastorage/audit_self_auditing_test.go` (6 test scenarios)
- **Implementation**: `pkg/audit/internal_client.go`, `pkg/datastorage/server/audit_events_handler.go`
- **Design Decision**: [DD-STORAGE-012](./DD-STORAGE-012-AUDIT-INTEGRATION-PLAN.md)
- **Related BRs**: BR-STORAGE-181 (circular dependency), BR-STORAGE-182 (non-blocking)

#### **BR-STORAGE-181: Circular Dependency Prevention**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Audit traces must not create circular dependencies (Data Storage cannot call its own REST API)
- **Business Value**: Prevent infinite recursion and system instability
- **Test Coverage**:
  - Unit: `pkg/audit/internal_client_test.go:82` (validates direct PostgreSQL writes)
  - Integration: `test/integration/datastorage/audit_self_auditing_test.go:233` (validates no recursion)
- **Implementation**: `pkg/audit/internal_client.go` (InternalAuditClient pattern)
- **Design Decision**: [DD-STORAGE-012](./DD-STORAGE-012-AUDIT-INTEGRATION-PLAN.md)
- **Related BRs**: BR-STORAGE-180 (self-auditing)

#### **BR-STORAGE-182: Non-Blocking Audit Writes**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Audit writes must not block business operations (async buffered writes)
- **Business Value**: Maintain low latency for audit event writes (<5ms overhead)
- **Test Coverage**:
  - Unit: `pkg/audit/internal_client_test.go:138` (validates async writes)
  - Integration: `test/integration/datastorage/audit_self_auditing_test.go:139` (validates non-blocking)
- **Implementation**: `pkg/audit/store.go` (BufferedAuditStore), `pkg/datastorage/server/server.go`
- **Design Decision**: [DD-STORAGE-012](./DD-STORAGE-012-AUDIT-INTEGRATION-PLAN.md)
- **Related BRs**: BR-STORAGE-180 (self-auditing)

---

### **Category 5: Embedding & Vector Operations (BR-STORAGE-012, BR-STORAGE-013)**

#### **BR-STORAGE-012: Workflow Catalog Embedding Generation**
- **Priority**: P0
- **Status**: ✅ Active (v1.1 - Corrected Scope)
- **Description**: Generate 384-dimensional embeddings from workflow catalog content for semantic search (using sentence-transformers/all-MiniLM-L6-v2 model)
- **Business Value**: Enable semantic workflow discovery for incident remediation (DD-CONTEXT-005 "Filter Before LLM" pattern)
- **Terminology**: Per DD-NAMING-001, using "Remediation Workflow" (not "Remediation Playbook")
- **Scope Change (v1.1)**:
  - ❌ **OLD**: Generate embeddings from audit text (INCORRECT - deferred to V2.0 RAR)
  - ✅ **NEW**: Generate embeddings from workflow catalog (CORRECT - V1.0 requirement)
- **Use Case**: HolmesGPT API queries Data Storage to find workflows semantically similar to incident description
- **Test Coverage**:
  - Unit: `test/unit/datastorage/embedding_test.go:69` (embedding generation)
  - Unit: `test/unit/datastorage/query_test.go:236` (semantic search moved to integration)
- **Implementation**: `pkg/datastorage/embedding/pipeline.go`
- **Related BRs**: BR-STORAGE-009 (cache tracking), BR-STORAGE-014 (dual-write)
- **ADR References**: DD-CONTEXT-005 (Minimal LLM Response Schema)

#### **BR-STORAGE-013: Query Performance Metrics**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Track query performance metrics for semantic search and vector operations
- **Business Value**: Enable performance monitoring and optimization of vector operations
- **Test Coverage**:
  - Unit: `test/unit/datastorage/metrics_test.go:179`
- **Implementation**: `pkg/datastorage/metrics/metrics.go`
- **Related BRs**: BR-STORAGE-007 (query performance), BR-STORAGE-012 (embedding generation)

---

### **Category 6: Dual-Write Operations (BR-STORAGE-014 to BR-STORAGE-016)**

#### **BR-STORAGE-014: Atomic Dual-Write Operations**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Perform atomic writes to both PostgreSQL and Redis Vector DB in a single transaction with rollback on failure
- **Business Value**: Ensure data consistency between relational and vector databases
- **Test Coverage**:
  - Unit: `test/unit/datastorage/dualwrite_test.go:152` (atomic operations)
  - Unit: `test/unit/datastorage/metrics_test.go:338` (failure tracking)
- **Implementation**: `pkg/datastorage/dualwrite/coordinator.go`
- **Related BRs**: BR-STORAGE-002 (typed errors), BR-STORAGE-015 (graceful degradation)

#### **BR-STORAGE-015: Graceful Degradation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Continue operation with PostgreSQL-only writes if Vector DB is unavailable, with metrics tracking degraded mode
- **Business Value**: Maintain core audit persistence functionality during Vector DB outages
- **Test Coverage**:
  - Unit: `test/unit/datastorage/dualwrite_test.go:407` (degradation scenarios)
  - Unit: `test/unit/datastorage/metrics_test.go:114` (degradation tracking)
- **Implementation**: `pkg/datastorage/dualwrite/coordinator.go`
- **Related BRs**: BR-STORAGE-014 (atomic dual-write)

#### **BR-STORAGE-016: Context Propagation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Propagate context through all dual-write operations for cancellation, timeouts, and tracing
- **Business Value**: Enable request cancellation and prevent resource leaks from abandoned operations
- **Test Coverage**:
  - Unit: `test/unit/datastorage/dualwrite_context_test.go:159` (context propagation)
  - Unit: `test/unit/datastorage/dualwrite_context_test.go:215` (cancelled context)
  - Unit: `test/unit/datastorage/dualwrite_context_test.go:224` (expired deadline)
- **Implementation**: `pkg/datastorage/dualwrite/coordinator.go`
- **Related BRs**: BR-STORAGE-014 (atomic dual-write)

---

### **Category 7: Error Handling (BR-STORAGE-017)**

#### **BR-STORAGE-017: RFC 7807 Error Responses**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Return RFC 7807 compliant error responses for all API failures (type, title, status, detail, instance)
- **Business Value**: Provide standardized, machine-readable error responses for client error handling
- **Test Coverage**:
  - Integration: `test/integration/datastorage/repository_test.go:29`
- **Implementation**: `pkg/datastorage/server/handler.go`
- **ADR References**: ADR-032 (REST API standards)

---

### **Category 8: REST API Endpoints (BR-STORAGE-020 to BR-STORAGE-028)**

#### **BR-STORAGE-020: Audit Write API**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Provide REST API endpoints for writing audit records (POST /api/v1/audits)
- **Business Value**: Enable other services to persist audit data via REST API (ADR-032 mandate)
- **Test Coverage**:
  - Integration: `test/integration/datastorage/http_api_test.go:21`
  - Integration: `test/integration/datastorage/suite_test.go:39`
- **Implementation**: `pkg/datastorage/server/audit_handlers.go`
- **ADR References**: ADR-032 (exclusive database access via REST API)

#### **BR-STORAGE-021: REST API Read Endpoints**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Provide REST API endpoints for reading audit records (GET /api/v1/incidents with filtering and pagination)
- **Business Value**: Enable audit querying and analysis via REST API
- **Test Coverage**:
  - Unit: `test/unit/datastorage/handlers_test.go:26`
  - Unit: `test/unit/datastorage/handlers_test.go:43`
  - Unit: `test/unit/datastorage/query_builder_test.go:9`
- **Implementation**: `pkg/datastorage/server/handler.go`
- **Related BRs**: BR-STORAGE-005 (filtering), BR-STORAGE-006 (pagination)

#### **BR-STORAGE-022: Query Filtering**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Support query filtering via URL parameters (namespace, status, phase, limit, offset)
- **Business Value**: Enable flexible audit queries via REST API
- **Test Coverage**:
  - Unit: `test/unit/datastorage/query_builder_test.go:9`
  - Unit: `test/unit/datastorage/query_builder_test.go:10`
- **Implementation**: `pkg/datastorage/query/builder.go`
- **Related BRs**: BR-STORAGE-005 (query API), BR-STORAGE-021 (REST API)

#### **BR-STORAGE-023: Pagination Validation**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Validate pagination parameters (limit: 1-1000, offset: ≥0) and reject invalid values with RFC 7807 errors
- **Business Value**: Prevent resource exhaustion from excessive page sizes
- **Test Coverage**:
  - Unit: `test/unit/datastorage/query_builder_test.go:29` (pagination validation)
  - Unit: `test/unit/datastorage/query_builder_test.go:45` (reject limit=0)
  - Unit: `test/unit/datastorage/query_builder_test.go:46` (reject limit>1000)
- **Implementation**: `pkg/datastorage/query/builder.go`
- **Related BRs**: BR-STORAGE-006 (pagination), BR-STORAGE-017 (RFC 7807 errors)

#### **BR-STORAGE-024: RFC 7807 for Not Found**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Return RFC 7807 error response for 404 Not Found scenarios
- **Business Value**: Provide consistent error handling for missing resources
- **Test Coverage**:
  - Unit: `test/unit/datastorage/handlers_test.go:29`
  - Unit: `test/unit/datastorage/handlers_test.go:199`
- **Implementation**: `pkg/datastorage/server/handler.go`
- **Related BRs**: BR-STORAGE-017 (RFC 7807 errors)

#### **BR-STORAGE-027: Large Result Set Handling**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Handle large result sets efficiently with streaming and pagination
- **Business Value**: Prevent memory exhaustion from large queries
- **Test Coverage**:
  - Unit: `test/unit/datastorage/handlers_test.go:153`
- **Implementation**: `pkg/datastorage/server/handler.go`
- **Related BRs**: BR-STORAGE-006 (pagination)

#### **BR-STORAGE-028: Graceful Shutdown (DD-007)**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Implement DD-007 Kubernetes-aware graceful shutdown pattern (4 steps: readiness probe 503, 5-second wait, drain connections, close resources)
- **Business Value**: Achieve ZERO request failures during rolling updates
- **Test Coverage**:
  - Integration: `test/integration/datastorage/graceful_shutdown_test.go:21` (comprehensive DD-007 tests)
  - Integration: `test/integration/datastorage/graceful_shutdown_test.go:31`
  - Unit: `test/unit/datastorage/handlers_test.go:26`
- **Implementation**: `pkg/datastorage/server/server.go`
- **ADR References**: DD-007 (Kubernetes-aware graceful shutdown)

#### **BR-STORAGE-029: Reserved**
- **Priority**: N/A
- **Status**: 🔒 Reserved
- **Description**: Reserved BR number for future REST API endpoint requirements
- **Business Value**: N/A - reserved for future allocation
- **Note**: Intentionally skipped to maintain sequential numbering flexibility between REST API (020-028) and Aggregation API (030-034) categories

---

### **Category 9: Aggregation API (BR-STORAGE-030 to BR-STORAGE-034)**

#### **BR-STORAGE-030: Aggregation API**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Provide REST API endpoints for aggregation queries (success rates, groupings, trends)
- **Business Value**: Enable analytics and reporting on remediation effectiveness
- **Test Coverage**:
  - Integration: `test/integration/datastorage/aggregation_api_test.go:17`
  - Integration: `test/integration/datastorage/aggregation_api_test.go:28`
- **Implementation**: `pkg/datastorage/server/aggregation_handlers.go`
- **Related BRs**: BR-STORAGE-031 to BR-STORAGE-034 (specific aggregations)

#### **BR-STORAGE-031: Success Rate Aggregation**
- **Priority**: P1
- **Status**: ✅ Active (with deprecation notice)
- **Description**: Calculate success rates by incident type and playbook (replaces deprecated workflow_id aggregation per ADR-033)
- **Business Value**: Measure remediation effectiveness by incident type
- **Test Coverage**:
  - Unit: `test/unit/datastorage/aggregation_handlers_test.go:38` (incident-type success rate)
  - Unit: `test/unit/datastorage/aggregation_handlers_test.go:39` (playbook success rate)
  - Unit: `test/unit/datastorage/aggregation_handlers_test.go:77`
- **Implementation**: `pkg/datastorage/server/aggregation_handlers.go`
- **ADR References**: ADR-033 (incident-type aggregation replaces workflow_id)
- **Related BRs**: BR-STORAGE-030 (aggregation API)

#### **BR-STORAGE-032: Namespace Grouping Aggregation**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Aggregate incident counts and success rates by namespace
- **Business Value**: Enable namespace-level analysis and reporting
- **Test Coverage**:
  - Integration: `test/integration/datastorage/aggregation_api_test.go:178`
- **Implementation**: `pkg/datastorage/server/aggregation_handlers.go`
- **Related BRs**: BR-STORAGE-030 (aggregation API)

#### **BR-STORAGE-033: Severity Distribution Aggregation**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Aggregate incident counts by severity level (critical, high, medium, low)
- **Business Value**: Enable severity-based analysis and prioritization
- **Test Coverage**:
  - Integration: `test/integration/datastorage/aggregation_api_test.go:250`
- **Implementation**: `pkg/datastorage/server/aggregation_handlers.go`
- **Related BRs**: BR-STORAGE-030 (aggregation API)

---

#### **ARCHITECTURAL CLARIFICATION: Workflow Severity vs Signal Severity**

**📋 Design Decision**: [DD-SEVERITY-001 v1.1](../../../architecture/decisions/DD-SEVERITY-001-severity-determination-refactoring.md) | ✅ **Separate Domains**

**Context**: DataStorage maintains TWO distinct severity domains that are intentionally separate:

1. **Workflow Severity** (`critical`, `high`, `medium`, `low`)
   - **Domain**: Remediation procedures in workflow catalog
   - **Meaning**: Urgency/criticality of executing the remediation workflow
   - **Source**: Workflow catalog entries (manually curated by operators)
   - **Usage**: Workflow search filtering, aggregation (BR-STORAGE-033), trending
   - **Example**: "Scale deployment workflow" has `severity: high` because scaling operations require careful validation

2. **Signal Severity** (`critical`, `high`, `medium`, `low`, `unknown`)
   - **Domain**: Incoming alerts/signals from monitoring systems
   - **Meaning**: Urgency/criticality of the alert itself
   - **Source**: SignalProcessing Status.Severity (normalized by Rego policy per DD-SEVERITY-001)
   - **Usage**: Incident aggregation, trend analysis
   - **Example**: Prometheus alert has `severity: critical` indicating production outage

**Why Separate?**
- ✅ **Different semantics**: Workflow urgency ≠ Signal urgency
- ✅ **Different sources**: Curated catalog ≠ Dynamic signals
- ✅ **Different lifecycles**: Workflow severity is static, signal severity varies per incident
- ✅ **No coupling needed**: Workflow search uses signal severity as **filter input**, not stored relationship

**Current Usage** (v1.0):
- ✅ **Search Filtering**: Operators filter workflows by severity in catalog search
- ✅ **Aggregation/Analytics**: Severity distribution reporting (BR-STORAGE-033)
- ✅ **Trending**: Track workflow usage by severity level over time

**Future Possibilities** (Roadmap):

1. **🎯 Workflow Prioritization Queue** (High Value)
   - **Use Case**: When multiple workflows are pending for different resources, execute critical workflows first
   - **Current Blocker**: Single-resource-at-a-time constraint prevents queue scenarios
   - **Future Enabler**: If we support multi-resource execution or queuing, severity becomes primary sort key
   - **Industry Practice**: Most platforms use severity for execution priority (PagerDuty, ServiceNow, etc.)
   - **Example**: During incident storm, "Scale database (critical)" executes before "Restart cache (low)"

2. **⏱️ SLO/SLA Time Targets** (Compliance Value)
   - **Use Case**: Different time-to-execution and time-to-completion targets by severity
   - **Industry Practice**:
     - Critical: Start within 1 minute, complete within 5 minutes
     - High: Start within 5 minutes, complete within 15 minutes
     - Medium: Start within 15 minutes, complete within 1 hour
     - Low: Best effort, may be deferred during high load
   - **Metrics**: Track SLO compliance by severity level
   - **Example**: Alert if critical workflow hasn't started within 1 minute

3. **🔐 Approval Requirements** (Safety Value)
   - **Use Case**: Critical workflows require human approval, low workflows auto-execute
   - **Integration Point**: RemediationOrchestrator approval workflow
   - **Industry Practice**: Risk-based automation (automate low-risk, gate high-risk)
   - **Example**: "Delete PVC (critical)" → manual approval, "Clear cache (low)" → auto-approve

4. **📢 Notification Routing** (Operational Value)
   - **Use Case**: Route workflow execution notifications to different channels by severity
   - **Current**: Notifications sent uniformly
   - **Future**: Critical → PagerDuty + Slack, Medium → Slack only, Low → Log only
   - **Integration Point**: Notification controller routing logic
   - **Example**: Critical workflow failure pages on-call engineer immediately

5. **🔄 Retry/Timeout Policies** (Resilience Value)
   - **Use Case**: Higher severity workflows get more aggressive retry/timeout policies
   - **Industry Practice**:
     - Critical: 5 retries, 10-minute timeout per attempt
     - Low: 2 retries, 2-minute timeout per attempt
   - **Integration Point**: WorkflowExecution controller retry logic
   - **Example**: Critical workflows retry more aggressively to maximize success rate

6. **🚦 Rate Limiting/Throttling** (Stability Value)
   - **Use Case**: During high load, throttle low-severity workflows to preserve capacity for critical ones
   - **Industry Practice**: Kubernetes Priority Classes, AWS Service Quotas
   - **Current**: No throttling implemented
   - **Future**: Shed low-severity load when system is overloaded
   - **Example**: During incident storm, defer "low" workflows until critical backlog clears

7. **📊 Workflow Selection Scoring** (AI/ML Enhancement)
   - **Use Case**: Use workflow severity as factor in hybrid weighted scoring (DD-WORKFLOW-003)
   - **Current**: Severity not used in scoring
   - **Future Enhancement**: Boost score for workflows matching or exceeding signal severity
   - **Example**: Signal with severity="critical" → prefer workflows marked severity="critical" over "low"
   - **Integration Point**: DataStorage workflow search scoring algorithm

8. **📈 Cost/Resource Allocation** (Efficiency Value)
   - **Use Case**: Allocate more resources (CPU, memory, parallelism) to critical workflows
   - **Industry Practice**: Kubernetes QoS classes, AWS Fargate pricing tiers
   - **Example**: Critical workflows get dedicated worker nodes, low workflows share capacity

9. **🔍 Audit Detail Level** (Compliance Value)
   - **Use Case**: More detailed audit logging for critical workflows (full parameter sets, step-by-step execution)
   - **Current**: Uniform audit detail for all workflows
   - **Future**: Critical workflows generate exhaustive audit trails for compliance
   - **Example**: Critical workflow logs every parameter, low workflow logs summary only

10. **🎨 UI/UX Differentiation** (User Experience)
    - **Use Case**: Visual indicators in dashboards (red badges for critical, gray for low)
    - **Future**: Workflow catalog UI shows severity-based color coding
    - **Example**: Critical workflows highlighted in red in execution history

**Decision** (2026-01-15):
- **Current State**: Workflow severity is **underutilized** - only used for filtering/aggregation
- **High-Value Next Step**: **Workflow prioritization queue** when multi-resource execution is enabled
- **Recommendation**: Keep severity as separate domain - future use cases validate this decision
- **No Action Required**: DataStorage implementation is future-ready for these enhancements

**DD-SEVERITY-001 v1.1 Alignment**:
- Workflow severity predates DD-SEVERITY-001 and uses `critical/high/medium/low`
- Signal severity normalized by SignalProcessing uses same values: `critical/high/medium/low/unknown`
- **Alignment is coincidental but beneficial** - same enum values reduce confusion
- **No integration required** - domains remain independent

**Triage Decision (2026-01-15)**:
- **Decision**: Keep workflow severity and signal severity as separate domains
- **Rationale**: Different semantics, different sources, no business need for coupling
- **Status**: ✅ No changes required to DataStorage for DD-SEVERITY-001 v1.1
- **Reference**: [DD-SEVERITY-001 Week 4 DataStorage Triage](../../../architecture/decisions/DD-SEVERITY-001-severity-determination-refactoring.md#week-4-consumer-updates--datastorage-triage)

---

#### **BR-STORAGE-034: Incident Trend Aggregation**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Aggregate incident counts over time for trend analysis (hourly, daily, weekly)
- **Business Value**: Enable trend analysis and capacity planning
- **Test Coverage**:
  - Integration: `test/integration/datastorage/aggregation_api_test.go:17`
  - Integration: `test/integration/datastorage/aggregation_api_test.go:329`
- **Implementation**: `pkg/datastorage/server/aggregation_handlers.go`
- **Related BRs**: BR-STORAGE-030 (aggregation API)

---

### **Category 10: Workflow Catalog CRUD API (BR-STORAGE-038 to BR-STORAGE-042)**

#### **BR-STORAGE-038: Workflow Catalog Create API (Internal)**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Provide **internal** REST API endpoint for creating remediation workflows (`POST /api/v1/workflows`) with inline content validation. This endpoint is consumed exclusively by the AuthWebhook (AW) -- it is NOT a user-facing management endpoint. User-facing registration is via RemediationWorkflow CRD (BR-WORKFLOW-006).
- **Business Value**: Enable CRD-based workflow registration bridged through the AuthWebhook for the DS catalog
- **Request Payload**: `{content: string (JSON-serialized CRD spec), source: "crd", registeredBy: string}`
- **Validation**: DS parses inline content, validates schema (BR-WORKFLOW-004), action type FK (DD-WORKFLOW-016), uniqueness constraint (DD-WORKFLOW-012)
- **Duplicate Handling**: If `(workflow_name, version)` exists in disabled state, re-enables and returns `previouslyExisted: true`
- **Test Coverage**:
  - Unit: `test/unit/datastorage/workflow_crud_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
- **Implementation**: `pkg/datastorage/server/workflow_handlers.go:HandleCreateWorkflow`
- **Related BRs**: BR-WORKFLOW-001 (workflow registry), BR-WORKFLOW-006 (CRD spec), ADR-058 (webhook architecture)
- **Design Decisions**: DD-WORKFLOW-002 v3.0 (UUID primary key), DD-WORKFLOW-012 (workflow immutability)

#### **BR-STORAGE-039: Workflow Catalog Retrieval API**
- **Priority**: P0
- **Status**: ✅ Active
- **Description**: Provide REST API endpoint for retrieving a single workflow by UUID (`GET /api/v1/workflows/{workflow_id}`) returning the complete workflow object including spec, parameters, and detected labels
- **Business Value**: Enable external services (HolmesGPT-API, AIAnalysis) to validate workflow existence and retrieve full workflow spec for parameter/image validation
- **Use Cases**:
  - **HAPI Workflow Validation**: HolmesGPT-API validates `workflow_id` exists before returning to AIAnalysis (DD-HAPI-002)
  - **Parameter Schema Retrieval**: HAPI retrieves `spec.parameters[]` to validate LLM-generated parameters
  - **Image Pullspec Validation**: HAPI retrieves `spec.container_image` for OCI format validation
  - **Workflow Existence Check**: 200 OK = exists, 404 = not found
- **Response Fields**:
  - `workflow_id`: UUID primary key
  - `workflow_name`: Human-readable name
  - `version`: Semantic version
  - `spec.container_image`: OCI container image reference
  - `spec.parameters[]`: Parameter schema (name, type, required, default)
  - `spec.steps[]`: Workflow execution steps
  - `detected_labels`: Signal type, severity, resource management labels
  - `is_enabled`, `is_latest_version`: Status flags
- **Test Coverage**:
  - Unit: `test/unit/datastorage/workflow_crud_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
- **Implementation**: `pkg/datastorage/server/workflow_handlers.go:HandleGetWorkflowByID`
- **Related BRs**: BR-WORKFLOW-001 (workflow registry), BR-STORAGE-024 (RFC 7807 for 404)
- **Design Decisions**: DD-WORKFLOW-002 v3.0 (UUID primary key)
- **Cross-Service Integration**:
  - **HolmesGPT-API**: Uses this endpoint for `validate_workflow_exists` tool (Q17 in AIANALYSIS_TO_HOLMESGPT_API_TEAM.md)
  - **AIAnalysis**: May use for defense-in-depth validation

#### **BR-STORAGE-040: Workflow Catalog Search API**
- **Priority**: P0
- **Status**: ✅ Active
- **Version**: 2.0 (Multi-environment workflow capability - January 28, 2026)
- **Description**: Provide REST API endpoint for semantic search of workflows (`POST /api/v1/workflows/search`) with hybrid weighted scoring and multi-environment workflow support
- **Multi-Environment Workflow Support (v2.0)**:
  - **BREAKING CHANGE**: Workflows can declare multiple target environments (array in storage)
  - Workflow storage: `environment: ["staging", "production"]` (workflow works in BOTH)
  - Search filter: `environment: "production"` (single value from Signal Processing)
  - SQL uses PostgreSQL JSONB `?` operator for array containment check
  - Wildcard support: `environment: ["*"]` matches ALL environments
  - Validation: `minItems: 1` (explicit declaration required, no default)
  - Rationale: Single workflow can be reused across multiple environments without duplication
- **Business Value**: Enable AI-driven workflow discovery based on incident characteristics; single workflow reusable across multiple environments reduces duplication
- **Version History**:
  - v1.5 (2026-01-26): REVERTED - Implemented search-side arrays incorrectly
  - v2.0 (2026-01-28): CORRECT - Implemented storage-side arrays with search-side single value
- **Test Coverage**:
  - Unit: `test/unit/datastorage/workflow_search_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
  - Integration: `test/integration/datastorage/workflow_search_multi_env_test.go` (v2.0 - updated)
  - E2E: `test/e2e/datastorage/19_workflow_multi_environment_test.go` (v2.0 - planned)
- **Implementation**: `pkg/datastorage/server/workflow_handlers.go:HandleSearchWorkflows`
- **Related BRs**: BR-STORAGE-012 (embedding generation), BR-STORAGE-013 (query performance)
- **Design Decisions**: DD-WORKFLOW-004 v2.0 (hybrid scoring + multi-environment), DD-WORKFLOW-001 v2.5 (multi-environment schema)

#### **BR-STORAGE-041: Workflow Catalog Update API**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Provide REST API endpoint for updating mutable workflow fields (`PATCH /api/v1/workflows/{workflow_id}`) - only status and metrics are mutable per DD-WORKFLOW-012
- **Business Value**: Enable workflow status updates without creating new versions
- **Test Coverage**:
  - Unit: `test/unit/datastorage/workflow_crud_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
- **Implementation**: `pkg/datastorage/server/workflow_handlers.go:HandleUpdateWorkflow`
- **Related BRs**: BR-WORKFLOW-001 (workflow registry)
- **Design Decisions**: DD-WORKFLOW-012 (workflow immutability - only status/metrics mutable)

#### **BR-STORAGE-042: Workflow Catalog Disable API**
- **Priority**: P1
- **Status**: ✅ Active
- **Description**: Provide REST API endpoint for disabling workflows (`PATCH /api/v1/workflows/{workflow_id}/disable`) - soft delete with audit trail
- **Business Value**: Enable workflow deprecation without data loss
- **Test Coverage**:
  - Unit: `test/unit/datastorage/workflow_crud_test.go`
  - Integration: `test/integration/datastorage/workflow_catalog_test.go`
- **Implementation**: `pkg/datastorage/server/workflow_handlers.go:HandleDisableWorkflow`
- **Related BRs**: BR-WORKFLOW-002 (workflow versioning/deprecation)
- **Design Decisions**: DD-WORKFLOW-012 (disable = soft delete)

---

## 🔗 **ADR & Design Decision References**

### **Architecture Decision Records**

1. **ADR-032: Data Access Layer Isolation**
   - **Impact**: Data Storage Service is the ONLY service with direct PostgreSQL access
   - **Affected BRs**: ALL (entire service exists to implement ADR-032)
   - **Status**: ✅ Fully Implemented

2. **ADR-033: Incident-Type Aggregation**
   - **Impact**: Replaces workflow_id aggregation with incident-type aggregation
   - **Affected BRs**: BR-STORAGE-031 (success rate aggregation)
   - **Status**: ✅ Implemented (deprecated workflow_id endpoint documented)

3. **ADR-016: Podman-Based Integration Testing**
   - **Impact**: Integration tests use Podman containers for PostgreSQL and Redis
   - **Affected BRs**: All integration-tested BRs
   - **Status**: ✅ Implemented

### **Design Decisions**

1. **DD-007: Kubernetes-Aware Graceful Shutdown**
   - **Impact**: 4-step shutdown pattern for zero-downtime rolling updates
   - **Affected BRs**: BR-STORAGE-028 (graceful shutdown)
   - **Status**: ✅ Fully Implemented

---

## 📈 **Coverage Analysis**

### **Unit Test Coverage** (85%+ of Active BRs)

**Covered BRs** (35 BRs):
- Category 1 (Audit): BR-STORAGE-002, 003, 004
- Category 2 (Query): BR-STORAGE-005, 006, 007, 008
- Category 3 (Observability): BR-STORAGE-010, 018, 019
- Category 4 (Security): BR-STORAGE-011, 025, 026
- Category 4.5 (Self-Audit): BR-STORAGE-180, 181, 182
- Category 5 (Embedding): BR-STORAGE-012, 013
- Category 6 (Dual-Write): BR-STORAGE-014, 015, 016
- Category 8 (REST API): BR-STORAGE-021, 022, 023, 024, 027, 028
- Category 9 (Aggregation): BR-STORAGE-031
- Category 10 (Workflow CRUD): BR-STORAGE-038, 039, 040, 041, 042

**Not Covered by Unit Tests** (integration-only):
- BR-STORAGE-001, 009, 017, 020, 030, 032, 033, 034

**Rationale**: These BRs require real database integration (PostgreSQL + Redis) and are covered by integration tests.

### **Integration Test Coverage** (30%+ of Active BRs)

**Covered BRs** (14 BRs):
- BR-STORAGE-001, 004, 008, 017, 020, 028, 030, 031, 032, 033, 034, 038, 039, 040

**Rationale**: Integration tests validate real database operations, HTTP API endpoints, graceful shutdown, and workflow catalog behavior.

### **E2E Test Coverage** (Critical Paths)

**Covered BRs** (7 BRs):
- BR-STORAGE-008 (embedding service)
- BR-STORAGE-012 (workflow search)
- BR-STORAGE-038-042 (workflow CRUD lifecycle)

**Rationale**: E2E tests cover critical user journeys for workflow catalog operations.

---

## 🚀 **V1.1 Enhancement Business Requirements**

### **Category: Query API Enhancements (V1.1)**

#### **BR-STORAGE-035: Cursor-Based Pagination**
- **Priority**: P1
- **Status**: 📋 Planned (V1.1) — Tracked in [GitHub #288](https://github.com/jordigilh/kubernaut/issues/288)
- **Description**: Support cursor-based pagination for audit event queries to handle real-time data with high write volumes. Cursor format: `base64(event_timestamp + event_id)` for uniqueness.
- **Business Value**:
  - **Consistency**: No missed/duplicate records during pagination
  - **Performance**: Efficient for large result sets (uses index on `event_timestamp`)
  - **Real-time**: Handles concurrent writes gracefully
- **Acceptance Criteria**:
  - `GET /api/v1/audit/events?cursor={cursor}&limit={limit}` endpoint
  - Backward compatibility with offset-based pagination maintained
  - Response includes `next_cursor` for subsequent requests
  - Cursor decoding validates timestamp and event_id format
- **Test Coverage**: TBD (V1.1 implementation)
- **Implementation**: TBD (V1.1)
- **Related BRs**: BR-STORAGE-006 (pagination support), BR-STORAGE-023 (pagination validation)
- **Reference**: DD-STORAGE-010 (Query API Pagination Strategy)

---

### **Category: Performance Optimization (V1.1)**

#### **BR-STORAGE-036: Parent Event Lookup Index**
- **Priority**: P1
- **Status**: 📋 Planned (V1.1) — Tracked in [GitHub #289](https://github.com/jordigilh/kubernaut/issues/289)
- **Description**: Create database index on `(parent_event_id, parent_event_date)` to optimize child event lookups and event chain traversal.
- **Business Value**:
  - **Performance**: Faster queries for "find all children of parent X"
  - **Observability**: Efficient event chain traversal for debugging
  - **AI Analysis**: Faster causality analysis for RCA
- **Acceptance Criteria**:
  - Index created: `idx_audit_events_parent_lookup ON audit_events (parent_event_id, parent_event_date) WHERE parent_event_id IS NOT NULL`
  - Query performance for child lookups < 50ms (vs. current full table scan)
  - Index size monitored via Prometheus metrics
- **Test Coverage**: TBD (V1.1 implementation - performance benchmarks)
- **Implementation**: TBD (V1.1 - migration script)
- **Related BRs**: BR-STORAGE-032 (unified audit table), BR-STORAGE-007 (query performance tracking)
- **Reference**: FK_CONSTRAINT_IMPLEMENTATION_SUMMARY.md

---

### **Category: Data Integrity (V1.1)**

#### **BR-STORAGE-037: Historical Parent-Child Backfill**
- **Priority**: P2
- **Status**: 📋 Planned (V1.1) — Tracked in [GitHub #290](https://github.com/jordigilh/kubernaut/issues/290)
- **Description**: Backfill `parent_event_date` for existing audit events to enable historical event chain queries. Run during maintenance window.
- **Business Value**:
  - **Completeness**: Enable historical event chain queries
  - **Compliance**: Full audit trail for all events
  - **Analytics**: Complete causality data for trend analysis
- **Acceptance Criteria**:
  - Migration script backfills `parent_event_date` from parent event's `event_date`
  - Progress logging for long-running backfill (e.g., every 10,000 rows)
  - FK constraint validation after backfill completes
  - Rollback plan documented for failed backfill
- **Test Coverage**: TBD (V1.1 implementation - integration test with sample data)
- **Implementation**: TBD (V1.1 - migration script + validation)
- **Related BRs**: BR-STORAGE-032 (unified audit table)
- **Reference**: FK_CONSTRAINT_IMPLEMENTATION_SUMMARY.md
- **Considerations**:
  - Run during maintenance window (may be slow for large datasets)
  - Monitor database load during backfill
  - Consider batching for very large datasets (>1M events)

---

### **Defense-in-Depth Coverage**

**BRs with 2x+ Coverage** (Unit + Integration or Integration + E2E): 10+ BRs
- BR-STORAGE-004 (idempotent schema): Unit + Integration
- BR-STORAGE-008 (embedding generation): Unit + Integration + E2E
- BR-STORAGE-028 (graceful shutdown): Unit + Integration
- BR-STORAGE-031 (success rate): Unit + Integration
- BR-STORAGE-038-042 (workflow CRUD): Unit + Integration + E2E

**Rationale**: Critical production readiness features and workflow catalog operations have multi-tier test coverage.

---

## 🎯 **Priority Distribution**

| Priority | Count | Percentage | Description |
|----------|-------|------------|-------------|
| **P0** | 25 | 56% | Core functionality, security, production readiness |
| **P1** | 15 | 33% | Observability, aggregation, performance |
| **P2** | 1 | 2% | Data integrity enhancements (V1.1 planned) |
| **N/A** | 1 | 2% | Reserved (BR-029) |
| **Planned** | 3 | 7% | V1.1 BRs (BR-035, 036, 037) |
| **Total** | 45 | 100% | |

**Notes**:
- V1.1 BRs (BR-STORAGE-035, 036, 037) are planned enhancements and not yet implemented
- BR-STORAGE-009 (embedding cache) is deferred to V1.1
- BR-STORAGE-029 is reserved for future use

---

## ✅ **Confidence Assessment**

**Documentation Accuracy**: 100%
**Test Coverage Completeness**: 100% (all 41 active V1.0 BRs have test coverage)
**Implementation Verification**: 100%

**BR Numbering Gap Resolution** (v1.4):
- ✅ BR-STORAGE-004: Documented as "Idempotent Schema Initialization"
- ✅ BR-STORAGE-008: Documented as "Embedding Generation"
- ✅ BR-STORAGE-018: Documented as "Structured Logging"
- ✅ BR-STORAGE-029: Reserved for future use (intentionally skipped)

**V1.0 Status**: ✅ **COMPLETE** - All gaps resolved, 100% alignment with authoritative documentation.

---

## 📚 **References**

- **Implementation Plan**: `docs/services/stateless/data-storage/implementation/IMPLEMENTATION_PLAN_V4.8.md`
- **ADR-032**: Data Access Layer Isolation
- **ADR-033**: Incident-Type Aggregation
- **ADR-016**: Podman-Based Integration Testing
- **DD-005**: Observability Standards (logging, metrics)
- **DD-007**: Kubernetes-Aware Graceful Shutdown Pattern
- **DD-STORAGE-004**: Embedding Caching Strategy
- **DD-STORAGE-005**: pgvector String Format
- **Test Files**: `test/unit/datastorage/*.go`, `test/integration/datastorage/*.go`, `test/e2e/datastorage/*.go`
- **Implementation**: `pkg/datastorage/**/*.go`

---

**Last Updated**: December 6, 2025
**Next Review**: V1.1 planning (cursor pagination, parent event index)

