# DD-PLAYBOOK-011: Tekton Pipeline OCI Bundles for Playbook Execution

**Status**: ✅ Approved  
**Version**: 1.0  
**Date**: 2025-11-15  
**Confidence**: 100%  

---

## Context

Kubernaut v1.0 requires a mechanism to execute operator-defined remediation playbooks with parameters provided by the LLM. The current approach involves maintaining a generic "playbook-executor" Pipeline in the Remediation Execution Engine that runs container images.

An alternative approach is to store playbooks as **Tekton Pipeline OCI Bundles** that can be executed directly by Tekton with parameters.

---

## Decision

**APPROVED**: Use Tekton Pipeline OCI Bundles for playbook execution in v1.0.

Playbooks will be packaged as OCI bundles containing:
1. Tekton Pipeline definition (thin wrapper around container image)
2. Parameter schema (JSON Schema)

The Remediation Execution Engine will create PipelineRuns that reference OCI bundles directly, and Tekton will pull and execute the Pipeline with provided parameters.

---

## Rationale

### 1. Simpler Remediation Execution Engine (HIGH CONFIDENCE - 100%)

**Current Approach** (Generic Pipeline):
- Remediation Execution Engine must maintain generic "playbook-executor" Pipeline
- Complex parameter marshaling to environment variables
- Generic Pipeline must handle all playbook types

**OCI Bundle Approach**:
- No generic Pipeline needed (in bundle)
- Remediation Execution Engine just creates PipelineRun with bundle reference
- Tekton handles Pipeline lifecycle

**Result**: Less code to write and maintain in Remediation Execution Engine.

### 2. Atomic Versioning (HIGH CONFIDENCE - 100%)

**Current Approach**:
- Container image versioned separately from Pipeline
- Risk of version mismatch (Pipeline v1.0 + Container v1.1)
- Manual coordination required

**OCI Bundle Approach**:
- Pipeline + container image reference versioned together in bundle
- Single OCI tag (e.g., `v1.0.0`) for complete playbook
- Impossible to have version mismatch

**Result**: Safer deployments and easier rollbacks.

### 3. Future Flexibility (HIGH CONFIDENCE - 100%)

**Current Approach**:
- Adding validation/verification steps requires Remediation Execution Engine changes
- Generic Pipeline limits operator customization
- Hard to evolve without breaking changes

**OCI Bundle Approach**:
- Operators can add validation/verification tasks to Pipeline
- No Remediation Execution Engine changes needed
- Easy evolution from single-task to multi-task Pipelines

**Result**: Future-proof architecture.

### 4. Industry Standard (HIGH CONFIDENCE - 100%)

**Verification**:
- Tekton Bundles are GA (stable) since Tekton Pipelines v0.18
- OCI bundles widely adopted (Helm, Flux, Tekton)
- Official Tekton documentation provides complete examples

**Result**: Proven, production-ready technology.

---

## Technical Verification

### Official Tekton Documentation Confirmation

**Source**: https://tekton.dev/docs/pipelines/pipelineruns/

**Confirmed Capabilities**:
1. ✅ Tekton Pipelines can be stored as OCI artifacts
2. ✅ Pipelines can be executed directly from OCI bundles
3. ✅ Parameters can be provided at execution time
4. ✅ Feature is GA (production-ready)

### Example from Official Documentation

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: example-pipeline-run
spec:
  pipelineRef:
    resolver: bundles
    params:
      - name: bundle
        value: docker.io/myrepo/mypipeline-bundle:1.0
      - name: name
        value: mypipeline
      - name: kind
        value: Pipeline
  params:
    - name: param1
      value: "value1"
    - name: param2
      value: "value2"
```

---

## Implementation

### Playbook Structure

```
quay.io/kubernaut/playbook-oomkill-scale-down:v1.0.0 (OCI Bundle)
├── pipeline.yaml (Tekton Pipeline wrapper)
└── playbook-schema.json (parameter definitions)
```

### Pipeline Wrapper (Thin - 10-20 lines)

```yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: oomkill-scale-down
  annotations:
    playbook.kubernaut.io/version: "1.0.0"
spec:
  params:
    - name: TARGET_RESOURCE_KIND
      type: string
    - name: TARGET_RESOURCE_NAME
      type: string
    - name: TARGET_NAMESPACE
      type: string
    - name: SCALE_TARGET_REPLICAS
      type: string
  
  tasks:
    - name: execute-playbook
      taskSpec:
        params:
          - name: TARGET_RESOURCE_KIND
          - name: TARGET_RESOURCE_NAME
          - name: TARGET_NAMESPACE
          - name: SCALE_TARGET_REPLICAS
        steps:
          - name: run-remediation
            image: quay.io/kubernaut/playbook-oomkill-scale-down-container:v1.0.0
            env:
              - name: TARGET_RESOURCE_KIND
                value: $(params.TARGET_RESOURCE_KIND)
              - name: TARGET_RESOURCE_NAME
                value: $(params.TARGET_RESOURCE_NAME)
              - name: TARGET_NAMESPACE
                value: $(params.TARGET_NAMESPACE)
              - name: SCALE_TARGET_REPLICAS
                value: $(params.SCALE_TARGET_REPLICAS)
```

**Note**: Container image remains unchanged - same Dockerfile, same scripts.

### Remediation Execution Engine Implementation

```go
func (w *WorkflowEngine) createPipelineRunFromBundle(
    playbook *Playbook,
    params map[string]string,
) error {
    // Convert parameters to Tekton format
    tektonParams := make([]tektonv1.Param, 0, len(params))
    for name, value := range params {
        tektonParams = append(tektonParams, tektonv1.Param{
            Name:  name,
            Value: tektonv1.ParamValue{Type: "string", StringVal: value},
        })
    }
    
    // Create PipelineRun referencing OCI bundle
    pipelineRun := &tektonv1.PipelineRun{
        ObjectMeta: metav1.ObjectMeta{
            Name:      generatePipelineRunName(playbook.ID),
            Namespace: w.namespace,
        },
        Spec: tektonv1.PipelineRunSpec{
            PipelineRef: &tektonv1.PipelineRef{
                ResolverRef: tektonv1.ResolverRef{
                    Resolver: "bundles",
                    Params: []tektonv1.Param{
                        {
                            Name:  "bundle",
                            Value: tektonv1.ParamValue{
                                Type:      "string",
                                StringVal: playbook.TektonBundle,
                            },
                        },
                        {
                            Name:  "name",
                            Value: tektonv1.ParamValue{
                                Type:      "string",
                                StringVal: playbook.PipelineName,
                            },
                        },
                        {
                            Name:  "kind",
                            Value: tektonv1.ParamValue{
                                Type:      "string",
                                StringVal: "Pipeline",
                            },
                        },
                    },
                },
            },
            Params: tektonParams,
        },
    }
    
    // Create PipelineRun in cluster
    return w.tektonClient.TektonV1beta1().
        PipelineRuns(w.namespace).
        Create(context.Background(), pipelineRun, metav1.CreateOptions{})
}
```

### Operator Workflow

```bash
# 1. Build container image (UNCHANGED)
docker build -t quay.io/kubernaut/playbook-oomkill-scale-down-container:v1.0.0 .
docker push quay.io/kubernaut/playbook-oomkill-scale-down-container:v1.0.0

# 2. Create pipeline.yaml wrapper (NEW - simple template)
cat > pipeline.yaml << 'YAML'
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: oomkill-scale-down
spec:
  params:
    - name: TARGET_RESOURCE_KIND
    - name: TARGET_RESOURCE_NAME
    - name: TARGET_NAMESPACE
    - name: SCALE_TARGET_REPLICAS
  tasks:
    - name: execute-playbook
      taskSpec:
        params:
          - name: TARGET_RESOURCE_KIND
          - name: TARGET_RESOURCE_NAME
          - name: TARGET_NAMESPACE
          - name: SCALE_TARGET_REPLICAS
        steps:
          - name: run-remediation
            image: quay.io/kubernaut/playbook-oomkill-scale-down-container:v1.0.0
            env:
              - name: TARGET_RESOURCE_KIND
                value: $(params.TARGET_RESOURCE_KIND)
              - name: TARGET_RESOURCE_NAME
                value: $(params.TARGET_RESOURCE_NAME)
              - name: TARGET_NAMESPACE
                value: $(params.TARGET_NAMESPACE)
              - name: SCALE_TARGET_REPLICAS
                value: $(params.SCALE_TARGET_REPLICAS)
YAML

# 3. Push OCI bundle (NEW - one command)
tkn bundle push quay.io/kubernaut/playbook-oomkill-scale-down:v1.0.0 \
  -f pipeline.yaml \
  -f playbook-schema.json

# 4. Register playbook (v1.0: manual SQL, v1.1: CRD)
kubectl apply -f playbook-registration.yaml
```

**Additional Effort**: ~5 minutes per playbook

---

## Prerequisites

### One-Time Cluster Setup

1. **Enable Tekton OCI Bundles feature flag**:
```bash
kubectl edit configmap feature-flags -n tekton-pipelines
```

Set:
```yaml
data:
  enable-tekton-oci-bundles: "true"
```

2. **Verify Tekton Pipelines version >= v0.18**:
```bash
kubectl get deployment tekton-pipelines-controller -n tekton-pipelines \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
```

3. **Configure OCI registry credentials** (if private registry):
```bash
kubectl create secret docker-registry regcred \
  --docker-server=quay.io \
  --docker-username=<username> \
  --docker-password=<password> \
  -n kubernaut-system
```

---

## Benefits

1. **Simpler Remediation Execution Engine** (HIGH VALUE)
   - No generic Pipeline management
   - Tekton handles Pipeline lifecycle
   - Less code to maintain

2. **Atomic Versioning** (HIGH VALUE)
   - Pipeline + container versioned together
   - No version mismatch possible
   - Easier rollbacks

3. **Better Observability** (MEDIUM VALUE)
   - Tekton UI shows Pipeline structure
   - Easier debugging
   - Clear execution flow

4. **Future Flexibility** (HIGH VALUE)
   - Easy to add validation/verification tasks
   - Operators control Pipeline complexity
   - No Remediation Execution Engine changes needed

5. **Industry Standard** (MEDIUM VALUE)
   - OCI bundles are standard in Kubernetes ecosystem
   - Tekton CLI has built-in bundle support
   - Works with any OCI registry

---

## Trade-offs

**Accepted**:
1. **Operator Learning Curve** (LOW IMPACT)
   - Must learn basic Tekton Pipeline syntax
   - Mitigation: Provide template + documentation
   - One-time learning cost

2. **Additional Build Step** (LOW IMPACT)
   - Must create pipeline.yaml + push bundle
   - Mitigation: Provide automation script
   - ~5 minutes per playbook

3. **Tekton Dependency** (MEDIUM IMPACT)
   - Playbooks tied to Tekton (not portable to other systems)
   - Mitigation: Container images are still portable (can run standalone)
   - Acceptable for Kubernetes-native platform

4. **OCI Bundle Feature Flag** (LOW IMPACT)
   - Must enable `enable-tekton-oci-bundles` in Tekton
   - Mitigation: One-time cluster configuration
   - Feature is GA (stable)

---

## Alternatives Considered

### Alternative 1: Generic Pipeline Approach (Rejected)

**Approach**: Remediation Execution Engine maintains generic "playbook-executor" Pipeline that runs container images.

**Rejection Reasons**:
- More complex Remediation Execution Engine (must manage generic Pipeline)
- No atomic versioning (Pipeline + container separate)
- Less flexible (hard to evolve without Remediation Execution Engine changes)
- More code to maintain

### Alternative 2: Hybrid Approach (Deferred to v1.1)

**Approach**: Support both container images and Tekton Pipeline OCI bundles.

**Deferral Reasons**:
- Adds complexity (two execution patterns)
- OCI bundle approach is simpler for v1.0
- Can add hybrid support in v1.1 if needed

---

## Success Metrics

1. **Remediation Execution Engine Complexity**: <50 lines of code for PipelineRun creation
2. **Operator Onboarding**: <10 minutes to create first playbook bundle
3. **Version Management**: 0% version mismatch incidents
4. **Execution Success**: >95% of PipelineRuns execute successfully
5. **Observability**: 100% of Pipeline executions visible in Tekton UI

---

## Implementation Plan

### Phase 1: Infrastructure Setup (1 day)
- Enable `enable-tekton-oci-bundles` in Tekton
- Document OCI bundle creation process
- Create pipeline.yaml template

### Phase 2: Remediation Execution Engine Implementation (2 days)
- Implement `createPipelineRunFromBundle` function
- Update playbook registration schema (add `tekton_bundle` field)
- Add Tekton client to Remediation Execution Engine

### Phase 3: Operator Tooling (2 days)
- Create automation script for bundle creation
- Update documentation with examples
- Create playbook registration templates

### Phase 4: Migration (1 day per playbook)
- Create pipeline.yaml wrapper for each existing playbook
- Push OCI bundles to registry
- Update playbook registrations

### Phase 5: Testing (2-3 days)
- Test OCI bundle execution
- Validate parameter injection
- Verify monitoring and observability
- Performance testing

**Total Effort**: 1-2 weeks for v1.0

---

## Verification Test Plan

### Test 1: Create and Push OCI Bundle
```bash
tkn bundle push quay.io/kubernaut/test-playbook:v1.0.0 -f pipeline.yaml
```
**Expected**: ✅ Bundle pushed successfully

### Test 2: Create PipelineRun Referencing Bundle
```bash
kubectl apply -f pipelinerun.yaml
```
**Expected**: ✅ PipelineRun created

### Test 3: Verify Tekton Pulls and Executes Pipeline
```bash
kubectl get pipelinerun test-pipelinerun -o yaml
```
**Expected**: ✅ Status shows Pipeline pulled from bundle and executing

### Test 4: Verify Parameters are Injected
```bash
kubectl logs -l tekton.dev/pipelineRun=test-pipelinerun
```
**Expected**: ✅ Container logs show correct environment variables

### Test 5: Verify Versioning
```bash
# Update bundle to v1.1.0
tkn bundle push quay.io/kubernaut/test-playbook:v1.1.0 -f pipeline-v1.1.yaml

# Create PipelineRun with v1.1.0
kubectl apply -f pipelinerun-v1.1.yaml
```
**Expected**: ✅ Tekton pulls v1.1.0 Pipeline (not v1.0.0)

---

## Related Documents

- [DD-PLAYBOOK-003-parameterized-actions.md](DD-PLAYBOOK-003-parameterized-actions.md) - Single remediation playbook pattern
- [DD-PLAYBOOK-008-version-roadmap.md](DD-PLAYBOOK-008-version-roadmap.md) - Feature roadmap
- [DD-PLAYBOOK-009-catalog-storage.md](DD-PLAYBOOK-009-catalog-storage.md) - Storage backend
- [KUBERNAUT_ARCHITECTURE_OVERVIEW.md](../KUBERNAUT_ARCHITECTURE_OVERVIEW.md) - System architecture

---

## References

- **Tekton Official Documentation**: https://tekton.dev/docs/pipelines/pipelineruns/
- **Tekton Bundle Resolver**: https://tekton.dev/docs/pipelines/bundle-resolver/
- **OCI Distribution Spec**: https://github.com/opencontainers/distribution-spec

---

## Changelog

### Version 1.1 (2025-11-15)

**Service Naming Corrections**: Updated "Workflow Engine" → "Remediation Execution Engine" per ADR-035.

**Changes**:
- Updated all architecture diagrams and workflow descriptions
- Aligned terminology with authoritative ADR-035
- No functional changes to OCI bundle implementation

### Version 1.0 (2025-11-15)

**Initial Release**: Approved design decision for Tekton Pipeline OCI Bundles.

---