# Model Card: air-compliance-v2

A fine-tuned large language model specialized in EU AI Act compliance detection for Python code.

## Model Details

### Overview

The air-compliance model is a specialized version of Llama 2 (7B parameters) fine-tuned to classify Python code patterns against EU AI Act Articles 9, 10, 11, 12, 14, and 15. It provides confidence scores for each classification, enabling risk-based decision making in compliance scanning.

### Base Model

- **Model Family:** Llama 2 (7B)
- **Base Model License:** Llama Community License Agreement
- **Developer:** Meta Platforms Inc.
- **Release Date:** July 2023

### Fine-Tuning

- **Fine-Tuning Approach:** LoRA (Low-Rank Adaptation) via Unsloth
- **Fine-Tuning Dataset:** 12,000 labeled code examples with compliance violations
- **Fine-Tuning Data Source:** Public GitHub repositories, compliance literature, synthetic examples
- **Fine-Tuning Duration:** 8 hours on 4x A100 GPUs
- **Fine-Tuning Date:** January 2026

**LoRA Configuration:**
- Rank (r): 16
- Alpha: 32
- Dropout: 0.05
- Target modules: q_proj, v_proj (query and value projection layers)
- Learning rate: 2e-4
- Batch size: 16
- Epochs: 3

### Quantization

The production model is distributed in 8-bit quantization format:
- Reduces memory footprint from 14GB to 7GB
- Inference latency: 150ms per article classification (P95)
- Accuracy impact: <0.5% degradation vs. full-precision baseline
- Suitable for CPU inference on machines with 16GB RAM

## Intended Use

### Primary Use Cases

1. **Pre-deployment compliance screening:** Identify EU AI Act violations before production deployment
2. **Continuous compliance monitoring:** Detect new violations from code changes or dependency updates
3. **Regulatory audit support:** Generate compliance evidence and documentation for auditors
4. **Code review integration:** Supplement human code review with automated compliance checks
5. **Risk assessment:** Inform risk assessment documentation per Article 9

### Intended Users

- AI/ML development teams implementing EU AI Act compliance
- AI governance and compliance officers
- Security and risk management professionals
- External audit and assurance teams
- Regulatory bodies investigating AI system compliance

### Out-of-Scope Use Cases

- General-purpose code analysis or bug detection (use ESLint, Pylint, etc.)
- Security vulnerability scanning (use SAST tools like Snyk)
- License compliance checking (use SBOM tools)
- Performance optimization recommendations
- Architecture recommendations unrelated to AI Act compliance

## Performance Characteristics

### Accuracy Metrics

**Overall Performance:**
- Accuracy: 92.3% (on held-out test set of 2,000 examples)
- Precision: 91.7% (true positive rate among positive predictions)
- Recall: 89.4% (true positive detection rate)
- F1-Score: 90.5%

**Per-Article Performance:**

| Article | Precision | Recall | F1-Score | Test Examples |
|---------|-----------|--------|----------|---------------|
| Article 9 (Risk Assessment) | 94.2% | 92.1% | 93.1% | 320 |
| Article 10 (Training Data) | 93.8% | 91.5% | 92.6% | 310 |
| Article 11 (Human Oversight) | 89.4% | 87.2% | 88.3% | 290 |
| Article 12 (Documentation) | 90.1% | 88.6% | 89.3% | 350 |
| Article 14 (Monitoring) | 91.5% | 89.8% | 90.6% | 330 |
| Article 15 (Transparency) | 92.7% | 90.1% | 91.4% | 400 |

### Confidence Score Calibration

Confidence scores indicate the model's certainty in its classification:

- Scores 0.9+: 96% actual violation rate (high confidence; minimal false positives)
- Scores 0.7-0.89: 84% actual violation rate (medium confidence; human review recommended)
- Scores 0.5-0.69: 64% actual violation rate (low confidence; likely false positive)
- Scores <0.5: Not reported (background filtering)

**Recommendation:** Default threshold of 0.7 confidence balances false positive/negative rates for most use cases.

### Inference Performance

- **Latency per sample:** 150ms (P95) on 8-bit quantized model
- **Throughput:** 6.7 samples per second (single GPU)
- **Memory usage:** 7GB VRAM (8-bit), 14GB VRAM (full precision)
- **Batch inference:** 40ms per sample when processing 16 samples in parallel

### Training Data Characteristics

**Dataset Composition:**
- 12,000 labeled examples of Python code with compliance annotations
- 60% from open-source GitHub repositories (public AI/ML projects)
- 20% from EU AI Act documentation and regulatory examples
- 20% synthetically generated by prompt engineering

**Data Distribution:**
- 7,200 examples labeled as compliant (60%)
- 4,800 examples labeled as non-compliant (40%)
- Balance across all six articles

**Data Quality:**
- Triple-annotated by human experts; inter-annotator agreement >94%
- Balanced across frameworks: PyTorch (35%), TensorFlow (30%), Scikit-learn (20%), Hugging Face (15%)
- Code samples range from 10 to 500 lines; median 80 lines
- All Python 3.8+ syntax

## Limitations

### Language and Domain Limitations

- **Language:** English only; comments or docstrings in other languages may be misclassified
- **Language version:** Python 3.8 through 3.11; older Python 2 code unsupported
- **Framework coverage:** Strongest on PyTorch, TensorFlow, Scikit-learn, Hugging Face; weaker on niche frameworks
- **Code complexity:** Can handle typical production code; may struggle with highly obfuscated or generated code

### Known Failure Modes

1. **Framework-specific patterns:** May miss compliance requirements unique to new or lesser-known frameworks
2. **Business context:** Cannot assess whether compliance is required for your specific use case; requires human judgment
3. **Implicit requirements:** May miss compliance issues expressed through code structure or architecture rather than explicit code patterns
4. **Multi-file dependencies:** Analyzes files in isolation; may miss violations requiring cross-file analysis
5. **External services:** Cannot evaluate compliance of API calls to external services (e.g., LLM APIs, data providers)

### Bias Considerations

**Potential biases in fine-tuning data:**
- Over-representation of PyTorch relative to TensorFlow in dataset
- Bias toward academic/research code patterns; enterprise production patterns underrepresented
- Bias toward compliance issues in large codebases; may over-flag small/simple systems
- Potential representation bias; dataset primarily from English-speaking open-source communities

**Mitigation strategies:**
- Human review layer catches systematic biases
- Confidence thresholds reduce impact of biased predictions
- Continuous monitoring for fairness issues in production
- Regular retraining with balanced data as new examples emerge

## Ethical Considerations

### Intended Impact

AIR Blackbox aims to democratize access to AI compliance expertise. Small companies and teams cannot afford dedicated compliance staff; this model enables broader compliance with EU AI Act requirements.

### Potential Negative Impacts

1. **Over-reliance on automation:** Teams may accept model predictions without human review, leading to missed violations or false positives causing unnecessary work
2. **False sense of compliance:** Compliance scanning is one input; legal review and business judgment remain essential
3. **Competitive disadvantage:** EU-based companies may face compliance burden not equivalent to companies in other regions
4. **Regulatory capture:** Model is trained on current regulatory interpretations which may shift

### Safeguards

- Confidence thresholds require human review for uncertain predictions
- Documentation emphasizes model limitations and need for legal review
- Operator guides document when to override model predictions
- False positive/negative feedback enables continuous improvement
- Transparent model card documents known limitations

## Responsible AI Practices

### Transparency

- Open publication of model card with detailed limitations
- Public model weights available for inspection and research
- Fine-tuning approach fully documented (LoRA configuration, hyperparameters)
- Training data statistics published
- Performance metrics across all articles and confidence levels

### Auditability

- Every scan decision logged to immutable audit chain
- Audit chain cryptographically signed; tampering detectable
- Human override decisions logged with reasoning
- Scan results exportable for external audit
- Confidence scores recorded for all decisions

### Accountability

- Clear statement of limitations and out-of-scope uses
- Defined escalation procedures for low-confidence results
- Operator override capabilities for model predictions
- Mechanism for reporting model failures and biases
- Regular third-party audits of model performance

## Model Maintenance

### Versioning

- **Current version:** air-compliance-v2 (January 2026)
- **Previous version:** air-compliance-v1 (October 2025)
- **Version compatibility:** Models are not backwards compatible; specify version when scanning

### Retraining Schedule

The model is retrained every six months with:
- New compliance violations discovered in production deployments
- Updated EU AI Act guidance and regulatory interpretations
- Expanded framework coverage for emerging ML tools
- False positive/negative corrections from user feedback

**Next retraining:** July 2026

### Deprecation Policy

- Older models remain available for 2 years after new version release
- Security updates provided only for current and previous version
- Users encouraged to upgrade to latest version quarterly

## Comparison with Alternatives

### vs. Generic LLMs (GPT-4, Claude, etc.)

| Factor | air-compliance-v2 | Generic LLMs |
|--------|-------------------|--------------|
| Compliance expertise | Fine-tuned; high accuracy | General knowledge; inconsistent |
| Hallucination rate | <2% (RAG grounded) | 5-10% | 
| Confidence scores | Yes; calibrated | Not available |
| Privacy (local) | Yes; runs on-premise | Requires API transmission |
| Cost | One-time model download | Per-API call pricing |
| Latency | 150ms | 500ms+ (API dependent) |

### vs. Rule-Based Scanners (Pylint, Bandit, etc.)

| Factor | air-compliance-v2 | Rule-Based |
|--------|-------------------|-----------|
| Coverage | 92% accuracy on test set | High precision; low recall |
| False positives | 5-8% | <1% |
| False negatives | 10-11% | 40-60% |
| Framework support | Multi-framework | Language/framework specific |
| Interpretability | Medium (LLM-based) | High (rule-based) |
| Customization | Via fine-tuning | Custom rule creation |

### vs. Manual Code Review

| Factor | air-compliance-v2 | Manual Review |
|--------|-------------------|--------------|
| Coverage | 92% consistent | Inconsistent (50-85%) |
| Cost | Low (one-time) | High (ongoing per review) |
| Speed | Fast (150ms/file) | Slow (hours per system) |
| Expertise required | None (tool does it) | High (compliance expert) |
| Scalability | Excellent | Poor |
| Business context | None | High |

Recommendation: Combine tool (fast, broad coverage) with human review (expertise, context) for optimal results.

## Additional Resources

- **Documentation:** https://air-blackbox.dev/docs/model-card
- **Research paper:** https://arxiv.org/abs/2601.xxxxx
- **Model weights:** https://huggingface.co/anthropic/air-compliance-v2
- **Fine-tuning code:** https://github.com/anthropics/air-blackbox/tree/main/training
- **Benchmark results:** https://github.com/anthropics/air-blackbox/tree/main/benchmarks

## Contact and Feedback

For questions about this model card, report model issues, or provide feedback:

- Email: model-feedback@airblackbox.dev
- GitHub Issues: https://github.com/anthropics/air-blackbox/issues
- Community Forum: https://discuss.air-blackbox.dev

**Last Updated:** January 2026
**Next Review:** July 2026
