# Apache Spark Optimizer

Skill ID: SK-DEA-001

## Purpose

Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization. Provides recommendations for partition strategies, shuffle reduction, memory tuning, and Catalyst optimizer hints.

## Key Features

- Execution plan analysis
- Partition optimization
- Shuffle reduction
- Memory and executor tuning
- Data skew mitigation
- Broadcast join optimization
- Caching strategies

## Applicable Processes

- ETL/ELT Pipeline
- Streaming Pipeline
- Feature Store Setup
- Pipeline Migration
