# DD-WORKFLOW-017: Workflow Lifecycle Component Interactions

**Date**: February 5, 2026
**Updated**: March 4, 2026
**Status**: Approved
**Decision Maker**: Kubernaut Architecture Team
**Authority**: AUTHORITATIVE - This document governs the end-to-end workflow lifecycle and component interactions
**Affects**: Data Storage Service, AuthWebhook Service, HolmesGPT API, Remediation Orchestrator, Signal Processing, Workflow Execution
**Related**: DD-WORKFLOW-016 (Action-Type Indexing), DD-WORKFLOW-012 (Immutability Constraints), DD-STORAGE-008 (Catalog Schema), DD-WORKFLOW-001 v2.6 (Label Schema), DD-AUDIT-003 (Audit Trace), ADR-058 (Webhook-Driven Registration), BR-WORKFLOW-006 (RemediationWorkflow CRD), DD-WEBHOOK-001 (Webhook Matrix), DD-004 (RFC 7807 Error Responses)
**Supersedes**: DD-WORKFLOW-005 (Schema Extraction), DD-WORKFLOW-007 (Manual Registration)
**Version**: 1.2

---

## Changelog

### Version 1.2 (2026-03-04) -- CURRENT

- **Phase 3 (Execution)**: Updated for Issue #518 -- `executionEngine` removed from WFE spec. WE controller resolves engine at runtime from DS catalog via `WorkflowQuerier.GetWorkflowSchemaMetadata` (F6: consolidated single DS call) and persists in `wfe.Status.ExecutionEngine` (immutable once set). RO no longer sets engine on WFE; notification path reads engine from WFE status. No silent "tekton" default -- WFE fails explicitly if DS has no engine.

### Version 1.1 (2026-03-04)

- **BREAKING**: Replaced OCI pullspec registration with CRD-based registration via RemediationWorkflow CRD
- Phase 1 rewritten: operators create `RemediationWorkflow` CRD; AuthWebhook intercepts and bridges to DS
- DS `POST /api/v1/workflows` is now an internal API consumed only by the AuthWebhook
- Added AuthWebhook (AW) to components table
- Updated scope: CRD-based registration is now in-scope for V1.0
- Added cross-references to ADR-058 and BR-WORKFLOW-006
- Phase 4 (Operational Management): disable via CRD DELETE in addition to REST PATCH
- Superseded docs (DD-WORKFLOW-005, DD-WORKFLOW-007) archived

### Version 1.0 (2026-02-05)

- Initial version defining end-to-end workflow lifecycle
- Consolidates workflow creation, discovery, usage, and disable/enable in one document
- Aligned with `action_type` design (DD-WORKFLOW-016) and three-step discovery protocol
- Supersedes DD-WORKFLOW-005 (registration) and DD-WORKFLOW-007 (manual registration)
- V1.0 scope only: manual REST API registration

---

## Problem Statement

Workflow lifecycle documentation is fragmented across multiple documents that predate the `action_type` paradigm shift (DD-WORKFLOW-016):

- **DD-WORKFLOW-005** covers schema extraction and registration but uses the `signal_type`-based design and mixes V1.0, V1.1, and V1.2 scopes in a single document
- **DD-WORKFLOW-007** describes CLI-based manual registration, already partially superseded by the CRD approach
- **DD-WORKFLOW-008** is a version roadmap, not a component interaction specification

No single document provides a consolidated view of how components interact across the full workflow lifecycle with the `action_type` design. This creates ambiguity for implementers who need to understand the end-to-end flow for creation, discovery, usage, and operational management of workflows.

---

## Decision

Define a single authoritative document (this DD) that describes the complete workflow lifecycle for V1.0, covering four phases:

1. **Creation** (Registration): How workflows enter the catalog
2. **Discovery** (Three-Step Protocol): How the LLM discovers and selects workflows
3. **Usage** (Execution): How selected workflows are executed
4. **Operational Management** (Disable/Enable/Deprecate): How workflows are managed post-creation

Each phase specifies the components involved, the API calls between them, the validation rules, and the audit traces emitted.

---

## Scope

**V1.0**. This document describes the current implementation scope:

- CRD-based workflow registration via `RemediationWorkflow` CRD (BR-WORKFLOW-006, ADR-058)
- AuthWebhook bridges CRD lifecycle to DS catalog (internal `POST /api/v1/workflows`)
- `action_type`-based catalog indexing (DD-WORKFLOW-016)
- Three-step LLM discovery protocol
- Disable via CRD DELETE or REST API PATCH
- Enable via REST API PATCH

**Out of scope** (deferred to future versions):
- Full reconciliation controller for CRD-DS state drift (V1.1)
- CLI registration tool (V1.2)
- Image signing and verification (V1.2)
- Workflow versioning and rollback UI (V1.2)

---

## Components

| Component | Abbreviation | Role in Workflow Lifecycle |
|-----------|-------------|---------------------------|
| **Data Storage** | DS | Workflow catalog CRUD, schema validation from inline content (internal API), audit traces |
| **AuthWebhook** | AW | Intercepts RemediationWorkflow CRD CREATE/DELETE; bridges to DS internal API; SOC2 user attribution; audit events (ADR-058) |
| **HolmesGPT API** | HAPI | LLM tool orchestration, three-step discovery, post-selection validation |
| **Signal Processing** | SP | Signal enrichment, context labels |
| **Remediation Orchestrator** | RO | End-to-end remediation lifecycle management. Creates and watches CRDs sequentially (SP -> AIAnalysis -> WorkflowExecution -> Notification). Makes all routing decisions. |
| **Workflow Execution** | WE | Pure executor (no routing logic). Reconciles WorkflowExecution CRDs, resolves execution engine from DS catalog at runtime (Issue #518), creates and monitors execution primitives (Tekton PipelineRun, Kubernetes Job, or Ansible AWX job). |
| **Operator** | -- | Human actor: creates RemediationWorkflow CRDs, manages lifecycle |
| **LLM** | -- | AI model: performs RCA, selects action type and workflow |

---

## Phase 1: Workflow Creation (Registration)

### Overview

Workflows are registered by creating a `RemediationWorkflow` CRD in the `kubernaut-system` namespace. The AuthWebhook (AW) intercepts the CREATE operation via a `ValidatingWebhookConfiguration`, extracts the CRD spec, and forwards it to the Data Storage internal API for validation and catalog population. The DS `POST /api/v1/workflows` endpoint is an **internal API** -- it is NOT a user-facing management endpoint.

**Key design principle**: The CRD is the **user-facing interface** for workflow registration. Operators use standard `kubectl apply` or GitOps tools (Flux, ArgoCD) to manage workflows. The AW bridges the Kubernetes API to the DS catalog, capturing authenticated user identity for SOC2 CC8.1 compliance.

See [ADR-058](./ADR-058-webhook-driven-workflow-registration.md) for the full architecture decision and [BR-WORKFLOW-006](../../requirements/BR-WORKFLOW-006-remediation-workflow-crd.md) for the CRD specification.

### Component Flow

```mermaid
sequenceDiagram
    participant Op as Operator
    participant K8s as K8s API Server
    participant AW as AuthWebhook
    participant DS as DataStorage
    participant DB as PostgreSQL

    Note over Op: Author RemediationWorkflow YAML<br/>(spec per BR-WORKFLOW-004)

    Op->>K8s: kubectl apply -f remediationworkflow.yaml
    K8s->>AW: AdmissionReview (CREATE)
    AW->>AW: Extract authenticated user (req.UserInfo)
    AW->>AW: Marshal CRD spec to JSON content

    AW->>DS: POST /api/v1/workflows (internal)
    Note right of AW: {content: JSON, source: "crd", registeredBy: user}

    DS->>DS: Parse and validate schema (BR-WORKFLOW-004)
    DS->>DB: SELECT action_type FROM action_type_taxonomy WHERE action_type = $1
    DB-->>DS: Validate action_type exists (FK)
    DS->>DS: Validate mandatory labels (DD-WORKFLOW-001 v2.6)
    DS->>DS: Validate immutability rules (DD-WORKFLOW-012)
    DS->>DB: Check duplicate prevention (UNIQUE constraint)

    alt Validation passes
        DS->>DB: INSERT INTO remediation_workflow_catalog (status='active')
        DB-->>DS: workflow_id (UUID)
        DS-->>AW: 201 Created {workflowId, status, previouslyExisted}
        AW->>AW: Emit audit: remediationworkflow.admitted.create
        AW-->>K8s: Allowed
        K8s-->>Op: RemediationWorkflow created
        Note over AW: Async goroutine: update CRD .status<br/>(workflowId, catalogStatus, registeredBy, registeredAt)
    else Duplicate (409)
        DS-->>AW: 200 OK (re-enabled, previouslyExisted: true)
        AW->>AW: Emit audit: remediationworkflow.admitted.create
        AW-->>K8s: Allowed
    else Validation fails
        DS-->>AW: 400 Bad Request (validation errors)
        AW->>AW: Emit audit: remediationworkflow.admitted.denied
        AW-->>K8s: Denied (DS error message forwarded)
        K8s-->>Op: Admission denied
    end
```

### Registration Interface

The operator creates a `RemediationWorkflow` CRD:

```yaml
apiVersion: kubernaut.ai/v1alpha1
kind: RemediationWorkflow
metadata:
  name: scale-conservative
  namespace: kubernaut-system
spec:
  version: "1.0.0"
  description:
    what: "Scales deployment replicas conservatively"
    whenToUse: "When pods are under CPU pressure"
    whenNotToUse: "When cluster is at capacity"
  actionType: ScaleReplicas
  labels:
    severity: ["critical", "high"]
    environment: ["production", "staging"]
    component: ["apps/v1/Deployment"]
    priority: P1
  execution:
    engine: tekton
    bundle: "quay.io/kubernaut-ai/workflow-scale:v1.0.0@sha256:abc123..."
  parameters:
    - name: replicas
      type: integer
      required: true
      description: "Target replica count"
      minimum: 1
      maximum: 100
```

The AW marshals the full CRD spec to JSON and passes it as the `content` field to the DS internal API, with `source: "crd"` and `registeredBy` set to the authenticated Kubernetes user.

### Validation Rules

DS validates the inline content from the AW. All error responses are surfaced to the operator as admission denial messages:

| # | Validation | Authority | Admission Result |
|---|-----------|-----------|-----------------|
| 1 | Schema format valid (required fields, types) | BR-WORKFLOW-004 | Denied: validation errors |
| 2 | `action_type` exists in `action_type_taxonomy` table | DD-WORKFLOW-016 (FK) | Denied: unknown action type |
| 3 | Mandatory labels present: `severity`, `component`, `environment` | DD-WORKFLOW-001 v2.6 | Denied: validation errors |
| 4 | `workflow_name` + `version` is unique (UNIQUE constraint) | DD-WORKFLOW-012 | Allowed (re-enable) |
| 5 | Description is non-empty | DD-WORKFLOW-012 | Denied: validation errors |
| 6 | At least one parameter defined | BR-WORKFLOW-004 | Denied: validation errors |

**Duplicate handling (validation #4)**: If the `(workflow_name, version)` combination already exists in a disabled state, DS re-enables the workflow and returns success with `previouslyExisted: true`. The CRD `.status.previouslyExisted` field reflects this.

### Async Status Update

After admission succeeds, the AW spawns a goroutine that updates the CRD `.status` subresource via `k8sClient.Status().Update()`. This uses a fresh context with a 10-second timeout. Status fields populated:

| Field | Value |
|-------|-------|
| `status.workflowId` | UUID from DS response |
| `status.catalogStatus` | `"active"` (or DS-returned status) |
| `status.registeredBy` | Authenticated Kubernetes user |
| `status.registeredAt` | Current timestamp |
| `status.previouslyExisted` | `true` if re-enabled from disabled |

**Trade-off**: There is a brief window where the CRD exists but `.status` is empty. The DS catalog is the source of truth; CRD status is informational. See ADR-058 for the full rationale.

### Audit Traces

Registration generates audit traces at two levels:

| Event Type | Component | Description | Authority |
|-----------|-----------|-------------|-----------|
| `remediationworkflow.admitted.create` | AW | CRD CREATE admitted, workflow registered | DD-WEBHOOK-003, ADR-058 |
| `remediationworkflow.admitted.denied` | AW | CRD CREATE denied (validation/auth failure) | DD-WEBHOOK-003, ADR-058 |
| `datastorage.workflow.created` | DS | Workflow added to catalog (internal) | DD-AUDIT-003 |

AW audit events include: authenticated user, resource name, correlation ID (admission UID), namespace.

### Authentication

Authentication is handled at the Kubernetes API server level. The AW extracts the authenticated user from `req.UserInfo` in the admission request (SOC2 CC8.1). The DS internal API receives the user identity in the `registeredBy` field.

---

## Phase 2: Workflow Discovery (Three-Step Protocol)

### Overview

When a signal is detected that requires remediation, the LLM discovers available workflows through a three-step protocol defined in DD-WORKFLOW-016. This section describes the component interactions for the full discovery flow.

### Component Flow

```mermaid
sequenceDiagram
    participant SP as SignalProcessing
    participant HAPI as HolmesGPT_API
    participant LLM as LLM
    participant DS as DataStorage

    SP->>HAPI: Signal context (severity, component, environment, priority, custom_labels, detected_labels)
    HAPI->>HAPI: Render prompt with signal context

    Note over HAPI,LLM: RCA Investigation Phase
    HAPI->>LLM: Prompt with signal + investigation tools
    LLM->>LLM: RCA investigation (kubectl, metrics, logs)

    alt Problem resolved or inconclusive
        LLM-->>HAPI: WorkflowNotNeeded / investigation_inconclusive
    else Problem identified, needs remediation
        Note over LLM,DS: Step 1 — Action Type Discovery
        LLM->>HAPI: list_available_actions(severity, component, environment, priority, ...)
        HAPI->>DS: GET /api/v1/workflows/actions?severity=...&component=...
        DS-->>HAPI: Action types with taxonomy descriptions + workflow counts
        HAPI-->>LLM: Rendered action types (clean taxonomy data)

        Note over LLM: LLM selects action_type based on RCA + descriptions
        LLM->>LLM: Select action_type

        Note over LLM,DS: Step 2 — Workflow Selection
        LLM->>HAPI: list_workflows(action_type, severity, component, environment, priority, ...)
        HAPI->>DS: GET /api/v1/workflows/actions/{action_type}?severity=...&component=...
        DS-->>HAPI: Workflows with descriptions (sorted by final_score, stripped before LLM)
        HAPI-->>LLM: Rendered workflows (all for selected action type)

        Note over LLM: LLM reads ALL workflows, selects best fit
        LLM->>LLM: Select workflow_id

        Note over LLM,DS: Step 3 — Parameter Lookup
        LLM->>HAPI: get_workflow(workflow_id, severity, component, environment, ...)
        HAPI->>DS: GET /api/v1/workflows/{workflow_id}?severity=...&component=...
        DS-->>HAPI: Full parameter schema (security gate applied)
        HAPI-->>LLM: Rendered parameter schema

        LLM->>LLM: Populate parameters from RCA context
        LLM-->>HAPI: Selected workflow_id + populated parameters
    end
```

### Step Details

Each step is fully specified in DD-WORKFLOW-016. This section provides the component interaction summary:

| Step | Tool | DS Endpoint | Input | Output | Authority |
|------|------|-------------|-------|--------|-----------|
| 1 | `list_available_actions` | `GET /api/v1/workflows/actions` | Signal context filters | Action types from taxonomy + workflow count | DD-WORKFLOW-016 |
| 2 | `list_workflows` | `GET /api/v1/workflows/actions/{action_type}` | Action type + signal context filters | Workflows with descriptions (paginated) | DD-WORKFLOW-016 |
| 3 | `get_workflow` | `GET /api/v1/workflows/{workflow_id}` | Workflow ID + signal context filters (security gate) | Full parameter schema | DD-WORKFLOW-016 |

### Filter Parity

All three DS endpoints apply the **same context filters**: severity, component, environment, priority, custom_labels, detected_labels. This is a security guarantee -- see DD-WORKFLOW-016 for details.

### Pagination

Steps 1 and 2 support pagination (default 10 items per page). Step 3 returns a single workflow or an error. For Step 1 (action type selection), the LLM may select from the first page if a clear match exists, requesting additional pages only if needed. For Step 2 (workflow selection), the LLM **must review ALL available workflows** before making a decision -- if `has_more` is true, the LLM must request subsequent pages until all workflows have been reviewed. This prevents premature selection of a less effective workflow.

### HAPI Post-Selection Validation

After the LLM returns its selected `workflow_id` and populated parameters, HAPI validates the selection by querying DS directly:

```mermaid
sequenceDiagram
    participant HAPI as HolmesGPT_API
    participant DS as DataStorage

    HAPI->>DS: GET /api/v1/workflows/{workflow_id}?severity=...&component=...
    DS-->>HAPI: Workflow with parameter schema

    HAPI->>HAPI: Validate workflow_id exists and is active
    HAPI->>HAPI: Validate action_type is in taxonomy
    HAPI->>HAPI: Validate parameter types + mandatory/optional flags
    HAPI->>HAPI: Validate all required parameters provided

    alt Validation passes
        HAPI->>HAPI: Proceed to execution
    else Validation fails
        HAPI->>HAPI: Reject LLM response, trigger error handling
    end
```

This validation uses **current data** from DS (not cached), ensuring the workflow has not been disabled or modified since the LLM's discovery queries. See DD-WORKFLOW-016 HAPI Validation section.

### Audit Traces

Discovery generates audit traces at the DS data layer for each step:

| Event Type | Component | Description | Authority |
|-----------|-----------|-------------|-----------|
| `workflow.catalog.actions_listed` | DS | Action types returned for signal context (Step 1) | DD-WORKFLOW-014 v3.0 |
| `workflow.catalog.workflows_listed` | DS | Workflows returned for selected action type (Step 2) | DD-WORKFLOW-014 v3.0 |
| `workflow.catalog.workflow_retrieved` | DS | Single workflow parameter schema retrieved (Step 3) | DD-WORKFLOW-014 v3.0 |
| `workflow.catalog.selection_validated` | DS | HAPI post-selection validation re-query result | DD-WORKFLOW-014 v3.0 |

**Note**: HAPI does **not** emit audit events. All discovery audit events are generated by DS at the data layer (per DD-AUDIT-001 and DD-WORKFLOW-014 v3.0).

---

## Phase 3: Workflow Execution

### Overview

After HAPI validates the LLM's selection, the AIAnalysis CR status is updated with the selected workflow and parameters. The Remediation Orchestrator (RO) -- which owns the end-to-end remediation lifecycle -- detects this status change and creates a WorkflowExecution CRD. The Workflow Execution (WE) controller -- a pure executor with no routing logic -- reconciles the CRD, resolves the execution engine from the DS workflow catalog at runtime (Issue #518), persists it in `wfe.Status.ExecutionEngine`, and creates the appropriate execution primitive (Tekton PipelineRun, Kubernetes Job, or Ansible AWX job).

### Component Flow

```mermaid
sequenceDiagram
    participant HAPI as HolmesGPT_API
    participant RO as RemediationOrchestrator
    participant WE as WorkflowExecution
    participant K8s as Kubernetes

    HAPI->>RO: AIAnalysis CR status updated with workflow_id + parameters
    Note right of HAPI: RO watches AIAnalysis status changes

    RO->>RO: Check cooldown period (no duplicate execution)
    RO->>RO: Routing decisions (blocking, resource busy, etc.)
    RO->>K8s: Create WorkflowExecution CRD
    Note right of RO: Includes workflow_id, parameters,<br/>targetResource (no executionEngine — Issue #518)

    WE->>K8s: Reconcile WorkflowExecution CRD
    WE->>DS: Resolve executionEngine from catalog (WorkflowQuerier)
    DS-->>WE: engine ("tekton", "job", or "ansible")
    WE->>WE: Persist engine in wfe.Status.ExecutionEngine (immutable)
    WE->>WE: Select executor via ExecutorRegistry (Strategy pattern)

    alt status.executionEngine = "tekton"
        WE->>K8s: Create Tekton PipelineRun from OCI bundle
    else status.executionEngine = "job"
        WE->>K8s: Create Kubernetes Job
    else status.executionEngine = "ansible"
        WE->>K8s: Launch AWX Job Template via REST API
    end

    K8s-->>WE: Execution status updates
    WE->>K8s: Update WorkflowExecution CRD status

    RO->>K8s: Watch WorkflowExecution status change
    RO->>RO: Update RemediationRequest status
```

### Execution Context

The WorkflowExecution CRD spec includes:

| Field | Source | Description |
|-------|--------|-------------|
| `workflowId` | LLM selection (validated by HAPI) | UUID of the workflow to execute |
| `parameters` | LLM-populated (validated by HAPI) | Workflow parameters as key-value pairs |
| `targetResource` | Signal context | Kubernetes resource to remediate |
| `signalFingerprint` | SP | Unique identifier for the triggering signal |

The WE controller resolves execution context at runtime and persists it in status:

| Status Field | Source | Description |
|-------|--------|-------------|
| `executionEngine` | Resolved from DS workflow catalog by WE controller | `"tekton"`, `"job"`, or `"ansible"` (Issue #518) |

**Note**: `executionEngine` is **not** on the WFE spec. The WE controller resolves it from the DS workflow catalog at runtime using `WorkflowQuerier.GetWorkflowSchemaMetadata(workflowId)` (F6: all catalog artifacts in a single DS call) and persists it in `wfe.Status.ExecutionEngine` (immutable once set). If the catalog entry has no engine defined, the WFE fails explicitly with `ConfigurationError` -- there is no silent default. WE uses the `ExecutorRegistry` (Strategy pattern) to select the correct executor based on the resolved engine.

### Audit Traces

Execution generates audit traces across RO and WE:

| Event Type | Component | Description | Authority |
|-----------|-----------|-------------|-----------|
| `remediation.execution.started` | RO | WorkflowExecution CRD created | DD-AUDIT-003 |
| `workflow.execution.completed` | WE | PipelineRun or Job finished (success/failure) | DD-AUDIT-003 |
| `remediation.execution.completed` | RO | Overall remediation result | DD-AUDIT-003 |

**Note**: Discovery audit events (`workflow.catalog.*`) are listed in Phase 2. LLM decision audit events are generated by the AI Analysis Controller.

---

## Phase 4: Operational Management

### 4.1 Disable Workflow

Operators can disable a workflow in two ways:

**Option A: Delete the RemediationWorkflow CRD** (preferred for GitOps):

```mermaid
sequenceDiagram
    participant Op as Operator
    participant K8s as K8s API Server
    participant AW as AuthWebhook
    participant DS as DataStorage
    participant DB as PostgreSQL

    Op->>K8s: kubectl delete remediationworkflow my-workflow
    K8s->>AW: AdmissionReview (DELETE)
    AW->>AW: Extract workflowId from CRD status
    AW->>DS: PATCH /api/v1/workflows/{id}/disable (best-effort)
    DS->>DB: UPDATE status='disabled', disabled_at=NOW()
    AW->>AW: Emit audit: remediationworkflow.admitted.delete
    AW-->>K8s: Allowed (always, even if DS fails)
    K8s-->>Op: CRD deleted
```

**Note**: CRD DELETE always succeeds regardless of DS response. This prevents GitOps drift.

**Option B: REST API PATCH** (for operational management without CRD changes):

```mermaid
sequenceDiagram
    participant Op as Operator
    participant DS as DataStorage
    participant DB as PostgreSQL

    Op->>DS: PATCH /api/v1/workflows/{workflow_id}/disable
    Note right of Op: Body: {"reason": "Causing issues in production"}

    DS->>DS: Authenticate operator (DD-AUTH-002)
    DS->>DB: UPDATE status='disabled', disabled_at=NOW(), disabled_by=$op, disabled_reason=$reason
    DB-->>DS: Updated row
    DS->>DB: INSERT audit trace: datastorage.workflow.updated
    DS-->>Op: 200 OK

    Note over DS: Disabled workflows are excluded from<br/>all three discovery endpoints<br/>(WHERE status = 'active')
```

**Effect on Discovery**: A disabled workflow is immediately excluded from:
- `GET /api/v1/workflows/actions` (Step 1) -- action types with zero active workflows are not returned
- `GET /api/v1/workflows/actions/{action_type}` (Step 2) -- disabled workflows filtered out
- `GET /api/v1/workflows/{workflow_id}` (Step 3) -- returns 404 Not Found for a disabled workflow

**Step 3 security gate — 404 Not Found (RFC 7807)**:

When Step 3 (`get_workflow`) receives a `workflow_id` that either does not exist, is not active, or does not match the signal context filters, DS returns 404. This prevents the LLM from using workflows outside the allowed context (defense-in-depth per DD-WORKFLOW-016):

```json
{
  "type": "https://kubernaut.ai/problems/workflow-not-found",
  "title": "Not Found",
  "detail": "No active workflow found for id '550e8400-...' matching the provided signal context filters",
  "status": 404,
  "instance": "/api/v1/workflows/550e8400-e29b-41d4-a716-446655440000"
}
```

**Note**: The error intentionally does not distinguish between "workflow does not exist", "workflow is disabled/deprecated", and "workflow exists but doesn't match filters". This prevents information leakage about catalog contents.

### 4.2 Enable Workflow

Operators can re-enable a previously disabled workflow.

```mermaid
sequenceDiagram
    participant Op as Operator
    participant DS as DataStorage
    participant DB as PostgreSQL

    Op->>DS: PATCH /api/v1/workflows/{workflow_id}/enable
    Note right of Op: Body: {"reason": "Issue resolved, workflow safe to use"}

    DS->>DS: Authenticate operator (DD-AUTH-002)
    DS->>DB: UPDATE status='active', disabled_at=NULL, disabled_by=NULL, disabled_reason=NULL
    DB-->>DS: Updated row
    DS->>DB: INSERT audit trace: datastorage.workflow.updated
    DS-->>Op: 200 OK

    Note over DS: Workflow is immediately available<br/>in all three discovery endpoints
```

### 4.3 Deprecate Workflow

Operators can deprecate a workflow to permanently remove it from discovery. This is a **terminal state** -- a deprecated workflow cannot be re-enabled or disabled. To provide the same remediation capability, the operator must register a new workflow version. Deprecated workflows remain visible in historical and audit queries.

```mermaid
sequenceDiagram
    participant Op as Operator
    participant DS as DataStorage
    participant DB as PostgreSQL

    Op->>DS: PATCH /api/v1/workflows/{workflow_id}/deprecate
    Note right of Op: Body: {"reason": "Replaced by v2.0.0 with improved scaling logic"}

    DS->>DS: Authenticate operator (DD-AUTH-002)
    DS->>DS: Verify current status is 'active' or 'disabled'

    alt Current status is active or disabled
        DS->>DB: UPDATE status='deprecated', reason=$reason
        DB-->>DS: Updated row
        DS->>DB: INSERT audit trace: datastorage.workflow.updated
        DS-->>Op: 200 OK
    else Current status is deprecated
        DS-->>Op: 409 Conflict (invalid-transition, RFC 7807)
    end

    Note over DS: Workflow is permanently excluded from<br/>all three discovery endpoints.<br/>Visible in historical/audit queries only.
```

**Effect on Discovery**: Identical to disable -- the workflow is immediately excluded from all three discovery endpoints. Unlike disable, this is irreversible.

### 4.4 Status Transitions

The following status transitions are allowed per DD-WORKFLOW-012:

```mermaid
stateDiagram-v2
    [*] --> active: Registration (POST)
    active --> disabled: Disable (PATCH)
    disabled --> active: Enable (PATCH)
    active --> deprecated: Deprecate
    disabled --> deprecated: Deprecate

    note right of disabled: Excluded from discovery,<br/>can be re-enabled
    note right of deprecated: TERMINAL — excluded from discovery,<br/>carries reason,<br/>visible in historical/audit queries
```

**Terminal state**: `deprecated` is a terminal state. Once a workflow is deprecated, it cannot be re-enabled or disabled. To offer the same remediation capability, the operator must register a new workflow version. Deprecated workflows remain visible in historical and audit queries (e.g., "which workflow was used in remediation #12345?") but are excluded from all three discovery endpoints.

All lifecycle PATCH endpoints require a mandatory `reason` field in the request body (e.g., `{"reason": "Causing OOM in production"}`). This reason is recorded in the audit event payload for SOC2 traceability.

| Transition | Endpoint | Mutable Fields Changed | Audit Event |
|-----------|----------|----------------------|-------------|
| `active -> disabled` | `PATCH /api/v1/workflows/{id}/disable` | `status`, `disabled_at`, `disabled_by`, `disabled_reason` | `datastorage.workflow.updated` |
| `disabled -> active` | `PATCH /api/v1/workflows/{id}/enable` | `status`, `disabled_at` (NULL), `disabled_by` (NULL), `disabled_reason` (NULL) | `datastorage.workflow.updated` |
| `active -> deprecated` | `PATCH /api/v1/workflows/{id}/deprecate` | `status`, `reason` | `datastorage.workflow.updated` |
| `disabled -> deprecated` | `PATCH /api/v1/workflows/{id}/deprecate` | `status`, `reason` | `datastorage.workflow.updated` |

**Invalid transitions** (rejected with 409 Conflict, RFC 7807 per DD-004):

| Attempted Transition | Error Type (DD-004) |
|---------------------|---------------------|
| `deprecated -> active` | `invalid-transition` |
| `deprecated -> disabled` | `invalid-transition` |

**RFC 7807 example — Invalid transition (409)**:

```json
{
  "type": "https://kubernaut.ai/problems/invalid-transition",
  "title": "Conflict",
  "detail": "Cannot enable workflow 'scale-conservative' (550e8400-...): current status 'deprecated' is terminal. Register a new workflow version instead.",
  "status": 409,
  "instance": "/api/v1/workflows/550e8400-e29b-41d4-a716-446655440000/enable"
}
```

**Missing reason field (400)**:

All lifecycle PATCH endpoints reject requests without a `reason` field:

```json
{
  "type": "https://kubernaut.ai/problems/validation-error",
  "title": "Bad Request",
  "detail": "Field 'reason' is required for all lifecycle transitions",
  "status": 400,
  "instance": "/api/v1/workflows/550e8400-e29b-41d4-a716-446655440000/disable"
}
```

### 4.5 Audit Trace Payload for Lifecycle Transitions

All lifecycle transitions emit a `datastorage.workflow.updated` audit event using the `CommonEnvelope` format (per ADR-034). The payload **must** include the previous and new status to provide full auditability:

```json
{
  "version": "1.0",
  "service": "data-storage",
  "operation": "workflow_status_changed",
  "status": "success",
  "payload": {
    "workflowId": "550e8400-e29b-41d4-a716-446655440000",
    "workflowName": "scale-conservative",
    "actionType": "ScaleReplicas",
    "previousStatus": "active",
    "newStatus": "disabled",
    "reason": "Causing issues in production",
    "updatedBy": "operator@example.com"
  }
}
```

| Payload Field | Type | Required | Description |
|--------------|------|----------|-------------|
| `workflowId` | string | Yes | UUID of the workflow |
| `workflowName` | string | Yes | Human-readable workflow name |
| `actionType` | string | Yes | The workflow's action type (for querying by action) |
| `previousStatus` | string | Yes | Status before the transition |
| `newStatus` | string | Yes | Status after the transition |
| `reason` | string | Yes | Mandatory on all transitions (disable, enable, deprecate) |
| `updatedBy` | string | Yes | Operator identity from auth context |

### 4.6 Authentication

Workflow lifecycle operations use two authentication mechanisms:

| Operation | Interface | Auth Mechanism | SOC2 Control |
|-----------|-----------|---------------|-------------|
| Registration (CREATE) | `RemediationWorkflow` CRD | K8s admission `req.UserInfo` via AW | CC8.1 (Attribution) |
| Disable (DELETE) | `RemediationWorkflow` CRD | K8s admission `req.UserInfo` via AW | CC8.1 (Attribution) |
| Disable (direct) | `PATCH /api/v1/workflows/{id}/disable` | DD-AUTH-002 middleware | CC8.1 (Attribution) |
| Enable | `PATCH /api/v1/workflows/{id}/enable` | DD-AUTH-002 middleware | CC8.1 (Attribution) |
| Deprecate | `PATCH /api/v1/workflows/{id}/deprecate` | DD-AUTH-002 middleware | CC8.1 (Attribution) |

---

## Cross-Reference: Authoritative Documents

This DD consolidates the workflow lifecycle but does **not** supersede the following authoritative documents:

| Document | Remains Authoritative For | Referenced In |
|----------|--------------------------|---------------|
| **ADR-058** | Webhook-driven registration architecture (ValidatingWebhook, async status, DS bridge) | Phase 1 (registration) |
| **BR-WORKFLOW-006** | RemediationWorkflow CRD specification (fields, lifecycle, RBAC) | Phase 1 (registration) |
| **DD-WEBHOOK-001** | CRD Webhook Requirements Matrix (all 4 handlers) | Phase 1 (registration) |
| **DD-WEBHOOK-003** | Webhook-complete audit pattern | Phase 1 (audit traces) |
| **DD-WORKFLOW-012** | Immutability constraints (mutable vs immutable fields) | Phase 1 (validation), Phase 4 (status transitions) |
| **DD-WORKFLOW-016** | Action-type indexing, three-step discovery protocol, SQL queries, pagination | Phase 2 (discovery) |
| **DD-WORKFLOW-001 v2.6** | Mandatory label schema (6 labels including `action_type`) | Phase 1 (validation) |
| **DD-STORAGE-008** | Database schema (`remediation_workflow_catalog` table) | Phase 1 (INSERT), Phase 4 (UPDATE) |
| **DD-AUDIT-003** | Audit trace event types and requirements | Phase 1, 3, 4 (audit traces) |
| **DD-004** | RFC 7807 error response standard, error type URI convention | Phase 4 (transition errors) |
| **ADR-043** | Workflow schema definition standard (`/workflow-schema.yaml`) | Phase 1 (schema format) |

This DD **supersedes**:

| Document | Reason |
|----------|--------|
| **DD-WORKFLOW-005** | Registration now uses CRD + AW bridge (ADR-058). Archived. |
| **DD-WORKFLOW-007** | Manual registration now uses CRD. Archived. CLI deferred to V1.2. |

---

## Business Requirements

### BR-WORKFLOW-017-001: CRD-Based Registration with Action Type FK Validation

- **Category**: WORKFLOW
- **Priority**: P0 (blocking for V1.0 registration)
- **Description**: MUST support workflow registration via `RemediationWorkflow` CRD. The AuthWebhook intercepts CRD CREATE, extracts the spec, and forwards it to the DS internal API as inline content. DS validates the schema (BR-WORKFLOW-004), validates that `action_type` exists in the `action_type_taxonomy` table (FK constraint), and inserts into the catalog. Workflows with unknown action types or invalid schemas are rejected at admission time. The DS `POST /api/v1/workflows` endpoint is internal-only, consumed exclusively by the AW.
- **Acceptance Criteria**:
  - `RemediationWorkflow` CRD CREATE triggers AW admission handler
  - AW marshals CRD spec to JSON and calls DS `POST /api/v1/workflows` with `content`, `source: "crd"`, `registeredBy`
  - DS validates inline schema content (BR-WORKFLOW-004 format)
  - `action_type` from schema is validated against `action_type_taxonomy` (FK constraint)
  - Registration is rejected (CRD denied) if schema is invalid or action type is unknown
  - Duplicate `(workflow_name, version)` triggers re-enable of disabled workflow (`previouslyExisted: true`)
  - CRD `.status` is populated asynchronously after successful admission
  - Audit events emitted: `remediationworkflow.admitted.create` (success), `remediationworkflow.admitted.denied` (failure)
  - Unit tests cover valid CRDs, invalid schemas, unknown action types, duplicate handling, auth failures

### BR-WORKFLOW-017-002: Disabled Workflow Exclusion from Discovery

- **Category**: WORKFLOW
- **Priority**: P0 (blocking for V1.0 operational safety)
- **Description**: MUST exclude disabled workflows from all three discovery endpoints. A disabled workflow must not appear in `list_available_actions`, `list_workflows`, or `get_workflow` responses. If disabling a workflow causes an action type to have zero active workflows, that action type must not appear in `list_available_actions`.
- **Acceptance Criteria**:
  - All three DS discovery endpoints filter by `status = 'active'`
  - Disabling the last active workflow for an action type removes that action type from Step 1 results
  - Re-enabling a workflow immediately restores it to discovery results
  - Integration tests verify end-to-end disable -> discovery exclusion -> enable -> discovery inclusion

### BR-WORKFLOW-017-003: Audit Traces for Lifecycle Transitions

- **Category**: WORKFLOW
- **Priority**: P0 (blocking for SOC2 compliance)
- **Description**: MUST emit audit traces for all workflow lifecycle transitions: creation, disable, enable, and deprecation. Each trace must include the operator identity, timestamp, and the transition details.
- **Acceptance Criteria**:
  - `datastorage.workflow.created` emitted on successful registration with `created_by`
  - `datastorage.workflow.updated` emitted on all status transitions with `updated_by` and old/new status
  - Audit traces are queryable via the audit API for compliance reporting
  - Unit tests verify audit trace emission for each lifecycle transition

### BR-WORKFLOW-017-004: RFC 7807 Error Responses for All Workflow Endpoints

- **Category**: WORKFLOW
- **Priority**: P0 (blocking for DD-004 compliance)
- **Description**: All HTTP error responses (4xx, 5xx) from workflow registration and lifecycle management endpoints MUST use RFC 7807 Problem Details format per DD-004. Each error response must include the `type` URI (`https://kubernaut.ai/problems/{error-type}`), `title`, `detail`, `status`, and `instance` fields, with `Content-Type: application/problem+json`.
- **Acceptance Criteria**:
  - All validation errors (400, 409, 422, 502) during registration use RFC 7807 format with domain-specific error types (e.g., `unknown-action-type`, `duplicate-workflow`, `schema-not-found`, `image-pull-failed`)
  - Invalid lifecycle transitions return 409 with error type `invalid-transition` and actionable `detail` text
  - Missing mandatory `reason` field on lifecycle PATCH endpoints returns 400 with error type `validation-error`
  - `Content-Type: application/problem+json` header is set on all error responses
  - Unit tests verify RFC 7807 compliance for every error path in the validation rules table

---

## End-to-End Lifecycle Summary

```mermaid
flowchart TD
    subgraph creation [Phase 1: Creation]
        A1["Operator creates RemediationWorkflow CRD"] --> A2["AW intercepts, forwards to DS"]
        A2 --> A3{DS validates}
        A3 -->|Pass| A4[Workflow active in catalog]
        A3 -->|Fail| A5["CRD admission denied"]
    end

    subgraph discovery [Phase 2: Discovery]
        B1[SP sends signal context to HAPI] --> B2[LLM performs RCA]
        B2 -->|Needs remediation| B3["Step 1: list_available_actions"]
        B3 --> B4["Step 2: list_workflows"]
        B4 --> B5["Step 3: get_workflow"]
        B5 --> B6[LLM populates parameters]
        B6 --> B7[HAPI validates selection]
    end

    subgraph execution [Phase 3: Execution]
        C1[RO detects AIAnalysis complete] --> C2[RO creates WorkflowExecution CRD]
        C2 --> C3[WE selects executor and runs workflow]
        C3 --> C4[Result reported back to RO]
    end

    subgraph management [Phase 4: Management]
        D1[Operator disables workflow] --> D2[Excluded from discovery]
        D3[Operator enables workflow] --> D4[Included in discovery]
        D5[Operator deprecates workflow] --> D6["Excluded from discovery (terminal)"]
    end

    A4 --> B1
    B7 --> C1
    A4 --> D1
    A4 --> D3
    A4 --> D5
```

---

**Document Version**: 1.2
**Last Updated**: March 4, 2026
**Status**: Approved
**Authority**: AUTHORITATIVE - Governs workflow lifecycle component interactions (V1.0)
**Confidence**: 95%

**Confidence Gap (5%)**:
- CRD-DS state drift (~2%): No reconciliation controller in V1.0. If DS and CRD state diverge, manual intervention is required. Mitigated by the fact that CRD CREATE/DELETE is the only registration path.
- Execution flow detail (~1%): RO and WE component interactions are described at summary level. Detailed execution protocol is governed by existing WE and RO design documents.
- Deprecation API (~1%): PATCH endpoint for deprecation is defined but not yet implemented. Disable/enable is the V1.0 priority.
- Goroutine status update (~1%): Async status update may fail, leaving CRD `.status` empty. DS catalog is source of truth. See ADR-058.

**Next Review**: After V1.0 deployment validates the end-to-end CRD registration flow
