# AlphaFold Protein Structure Database Dataset

## Overview

AlphaFold is a database of highly accurate protein structure predictions generated by DeepMind's AlphaFold system. Provides 3D structure predictions with per-residue confidence estimates (pLDDT scores) and predicted aligned error (PAE) metrics for proteins across the proteome. Contains 214+ million structure predictions covering UniProt sequences. Essential for structural biology, protein function prediction, drug discovery, and understanding protein mechanisms.

**Source**: AlphaFold Database (EMBL-EBI FTP + Google Cloud Storage)
**Data Type**: AI-predicted 3D protein structures with confidence scores stored as attributes on UniProt entries

## Integration Architecture

### Storage Model

**Primary Entries**:
- UniProt IDs (e.g., `A0A009IHW8`) serve as primary keys
- AlphaFold data stored as **attributes on UniProt entries**

**Searchable Text Links**:
- AlphaFold model IDs (e.g., `AF-A0A009IHW8-F1`) indexed as keywords
- Enables lookup by both UniProt ID and model ID

**Attributes Stored** (protobuf AlphafoldAttr):

*pLDDT Confidence Metrics (from FTP PDB files):*
- `global_metric`: Average pLDDT score (0-100, overall structure quality)
- `fraction_plddt_very_high`: Fraction with pLDDT > 90
- `fraction_plddt_confident`: Fraction with pLDDT 70-90
- `fraction_plddt_low`: Fraction with pLDDT 50-70
- `fraction_plddt_very_low`: Fraction with pLDDT < 50
- `model_entity_id`: AlphaFold model identifier (e.g., AF-A0A009IHW8-F1)
- `gene`: Gene symbol (when available)

*PAE Metrics (from GCS proteome tars - 16 model organisms):*
- `max_pae`: Maximum PAE value in matrix (0-31.75 Angstroms)
- `mean_pae`: Mean PAE value across all residue pairs
- `fraction_pae_confident`: Fraction of residue pairs with PAE < 5 Angstroms
- `version`: AlphaFold model version (e.g., 4, 6)

*Fragment Information:*
- `fragment_number`: Fragment number (1, 2, 3...) from model ID
- `sequence_length`: Length of this fragment/structure in residues

**Cross-References**:
- **Bidirectional**: UniProt (dataset 1) ↔ AlphaFold (dataset 29)
- **Via UniProt**: PDB, InterPro, HGNC, and other databases

### Data Sources

**pLDDT Data**:
- Source: EMBL-EBI FTP (`ftp.ebi.ac.uk/pub/databases/alphafold/`)
- Format: Tar archive of gzipped PDB files
- Coverage: Swiss-Prot reviewed proteins (~568K structures)
- pLDDT extracted from B-factor column

**PAE Data**:
- Source: Google Cloud Storage (`gs://public-datasets-deepmind-alphafold-v4/proteomes/`)
- Format: Tar archives containing gzipped JSON files
- Coverage: 16 model organisms (Human, Mouse, Arabidopsis, etc.)
- Local cache: `/data/bioyoda/snapshots/raw_data/alphafold_pae/`

**Model Organisms with PAE Data**:
| TaxID | Organism | Shards |
|-------|----------|--------|
| 9606 | Human | 19 |
| 10090 | Mouse | 8 |
| 3702 | Arabidopsis thaliana | 14 |
| 10116 | Rat | 4 |
| 559292 | Saccharomyces cerevisiae | 1 |
| 284812 | Schizosaccharomyces pombe | 1 |
| 83333 | Escherichia coli K-12 | 1 |
| 6239 | Caenorhabditis elegans | 3 |
| 39947 | Oryza sativa | 15 |
| 7955 | Danio rerio (Zebrafish) | 6 |
| 7227 | Drosophila melanogaster | 4 |
| 44689 | Dictyostelium discoideum | 2 |
| 237561 | Candida albicans | 1 |
| 243232 | Methanocaldococcus jannaschii | 1 |
| 3847 | Glycine max (Soybean) | 9 |
| 4577 | Zea mays (Maize) | 20 |

### Special Features

**pLDDT Confidence Scoring**:
- Per-residue confidence estimates from AlphaFold predictions
- Four confidence categories with fractions summing to 1.0
- Global metric provides overall structure quality assessment

**PAE (Predicted Aligned Error) Metrics**:
- Measures confidence in relative positions of residue pairs
- Low PAE (< 5 Angstroms) indicates confident relative positioning
- Useful for domain identification and interface assessment
- Streaming JSON parser for memory-efficient processing

**Fragment Tracking**:
- Large proteins (>2700 residues) split into multiple fragments
- Fragment number extracted from model ID (AF-xxxx-F1, F2, etc.)
- Sequence length stored for each fragment

**Parallel Processing**:
- PAE processing uses configurable worker pool (default: 8 workers)
- Streaming JSON parsing avoids loading full N×N matrices
- Config: `paeWorkers` in application.param.json

**Dual Lookup Methods**:
- Query by UniProt ID → Returns AlphaFold attributes
- Query by model ID (e.g., AF-A0A009IHW8-F1) → Resolves to UniProt entry

## Use Cases

**1. Structure-Based Function Prediction**
```
Query: UniProt ID → AlphaFold structure + pLDDT scores → Identify functional domains
Use: Predict protein function from structural features
```

**2. Drug Target Analysis**
```
Query: Target protein → AlphaFold structure + PAE → Identify stable binding pockets
Use: Structure-based drug design (prefer low PAE regions)
```

**3. Domain Boundary Identification**
```
Query: Protein → PAE metrics → Identify domain boundaries
Use: Low PAE within domains, high PAE between domains
```

**4. Protein Engineering**
```
Query: Protein → Confidence scores → Identify flexible/rigid regions
Use: Guide mutagenesis and protein design experiments
```

**5. Structural Coverage Assessment**
```
Query: Protein family → Check AlphaFold confidence → Prioritize experimental structures
Use: Decide which proteins need experimental structure determination
```

**6. Model Quality Filtering**
```
Query: Proteins with global_metric > 70 AND mean_pae < 10 → High-confidence structures
Use: Filter for reliable structural models in large-scale studies
```

## Test Cases

**Current Tests** (6 total):
- All custom tests (no declarative tests for attribute-based data)

**Coverage**:
- pLDDT fraction validation (sum to 1.0)
- Global metric range validation (0-100)
- Model ID format validation (AF-{UniProtID}-F{N})
- Model ID keyword lookup
- UniProt-AlphaFold bidirectional cross-references
- High-confidence structure distribution (>70 pLDDT)

**Recommended Additions**:
- PAE metrics validation (max_pae, mean_pae ranges)
- Fragment number extraction tests
- Sequence length validation
- Version field validation
- Multi-fragment protein tests

## Performance

- **Test Build**: ~3s (20 pLDDT + 27 PAE entries)
- **pLDDT Processing**: Streaming tar.gz from EMBL-EBI FTP
- **PAE Processing**: 8 parallel workers, streaming JSON parsing
- **Swiss-Prot Full**: ~5-10 minutes (~568K structures)
- **PAE Full**: ~5 minutes (~1M entries from 109 tar files, 148GB)
- **Memory Usage**: Streaming architecture, no full extraction
- **Test Database Size**: ~5 MB

## Confidence Interpretation

### pLDDT (per-residue)

| pLDDT Range | Confidence Level | Interpretation |
|-------------|------------------|----------------|
| **> 90** | Very High | High accuracy, well-ordered regions |
| **70-90** | Confident | Generally good backbone prediction |
| **50-70** | Low | Low confidence, potential disorder |
| **< 50** | Very Low | Should not be interpreted, likely disordered |

### PAE (pairwise)

| PAE Range | Confidence Level | Interpretation |
|-----------|------------------|----------------|
| **< 5 Å** | High Confidence | Reliable relative positioning |
| **5-15 Å** | Moderate | Some uncertainty in relative position |
| **> 15 Å** | Low | Uncertain, possible domain flexibility |
| **31.75 Å** | Maximum | No confidence in relative position |

## Configuration

**application.param.json**:
```json
{
  "paeWorkers": "8"  // Number of parallel workers for PAE processing
}
```

**source1.dataset.json**:
```json
{
  "alphafold": {
    "id": "29",
    "proteomeTaxIds": "9606,10090,3702,...",  // Species for PAE data
    "attrs": "global_metric,fraction_plddt_very_high,...,max_pae,mean_pae,fragment_number,sequence_length"
  }
}
```

## Known Limitations

**PAE Coverage**:
- PAE data available only for 16 model organisms
- Other organisms have pLDDT but no PAE metrics
- Full PAE matrix not stored (only summary statistics)

**Fragment Handling**:
- Fragment boundaries (start/end residues) not stored
- Total fragments per protein not tracked
- Each fragment stored separately

**Isoforms**:
- AlphaFold typically models canonical sequence only
- Alternative isoforms may lack predictions

**Confidence Interpretation**:
- pLDDT measures local accuracy, not global fold accuracy
- High pLDDT regions can still have incorrect relative orientations
- PAE provides complementary information about relative positions

## Changelog

**January 2026**:
- Fixed field name typo: `pldddt` → `plddt` (3 d's, not 4)
- Added PAE metrics: max_pae, mean_pae, fraction_pae_confident
- Added version tracking field
- Added fragment_number and sequence_length fields
- Implemented parallel PAE processing (8 workers)
- Implemented streaming JSON parser for PAE matrices
- Downloaded PAE data for 16 model organisms from GCS

## Maintenance

- **Release Schedule**: Regular updates from AlphaFold DB (version increments)
- **pLDDT Data**: Tar archive of gzipped PDB files from EMBL-EBI FTP
- **PAE Data**: Local cache from GCS proteome tars
- **Test Data**: Fixed 20 UniProt IDs (Swiss-Prot reviewed entries)
- **Current Version**: v4/v6 (as of 2026)
- **License**: CC BY 4.0 - freely available with attribution

## References

- **Citation**: Jumper J et al. (2021) Highly accurate protein structure prediction with AlphaFold. Nature 596:583-589.
- **Database**: Varadi M et al. (2022) AlphaFold Protein Structure Database. Nucleic Acids Res. 50(D1):D439-D444.
- **Website**: https://alphafold.ebi.ac.uk/
- **FTP Download**: https://ftp.ebi.ac.uk/pub/databases/alphafold/
- **GCS Bucket**: gs://public-datasets-deepmind-alphafold-v4/proteomes/
- **API**: https://alphafold.ebi.ac.uk/api/docs
- **License**: CC BY 4.0
