# HuggingFace Tokenizers - Complete Learning Module

> **Fast, blazing-fast tokenization with the 🤗 Tokenizers library**

This module provides a complete, hands-on guide to the HuggingFace Tokenizers library - the fastest tokenization library available, with full alignment tracking and support for all major tokenization algorithms.

## 📚 What You'll Learn

- Build tokenizers from scratch
- Train custom tokenizers on your data
- Use pretrained tokenizers
- Understand BPE, WordPiece, and Unigram algorithms
- Master the tokenization pipeline
- Optimize for production use

## 🚀 Quick Start

### Installation

```bash
pip install tokenizers
```

### Your First Tokenizer (2 minutes)

```python
from tokenizers import Tokenizer

# Load pretrained BERT tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Encode text
output = tokenizer.encode("Hello, world!")
print(output.tokens)  # ['hello', ',', 'world', '!']
print(output.ids)     # [7592, 1010, 2088, 999]
```

## 📖 Learning Path

### 1. Read the Guide (1 hour)

Start with the comprehensive guide:

**[📘 10_huggingface_tokenizers_guide.md](./10_huggingface_tokenizers_guide.md)**

This guide covers:

- Introduction and installation
- Quick start examples
- Tokenization pipeline explained
- All components (Normalizers, PreTokenizers, Models, PostProcessors, Decoders)
- Training custom tokenizers
- Working with pretrained models
- Advanced features (padding, truncation, batch encoding)
- Complete examples
- Best practices

**Sections:**

1. **Introduction** - What and why
2. **Quick Start** - Get running in 5 minutes
3. **Tokenization Pipeline** - Understanding the process
4. **Components Deep Dive** - Each component explained
5. **Training Custom Tokenizers** - Build your own
6. **Pretrained Tokenizers** - Load existing models
7. **Advanced Features** - Encoding, padding, truncation
8. **Complete Examples** - Full implementations
9. **Best Practices** - Tips and tricks

### 2. Run Quick Start Examples (30 minutes)

Practice with the quick start script:

**[🎯 01_tokenizers_quickstart.py](./01_tokenizers_quickstart.py)**

**10 Interactive Examples:**

1. **Load Pretrained Tokenizer** - Use BERT tokenizer
2. **Build from Scratch** - Create a simple BPE tokenizer
3. **Understanding Encoding** - Explore the Encoding object
4. **Batch Encoding** - Process multiple sequences
5. **Padding & Truncation** - Handle variable lengths
6. **Encode & Decode** - Round-trip conversion
7. **Sentence Pairs** - Work with pairs (for NLI, QA)
8. **Vocabulary Inspection** - Explore token mappings
9. **Special Tokens** - Add custom tokens
10. **Performance Comparison** - Batch vs single encoding

```bash
# Run all examples
python 01_tokenizers_quickstart.py

# Or run individual examples in Python
from tokenizers_quickstart import example_1_pretrained_tokenizer
example_1_pretrained_tokenizer()
```

### 3. Train Your Own Tokenizers (45 minutes)

Learn to train custom tokenizers:

**[🏋️ 02_tokenizers_training.py](./02_tokenizers_training.py)**

**7 Training Examples:**

1. **BPE Tokenizer (GPT-2 Style)** - Byte-level BPE
2. **WordPiece (BERT Style)** - Classic BERT tokenizer
3. **Unigram (Multilingual)** - SentencePiece-style for multiple languages
4. **Code Tokenizer** - Domain-specific for programming languages
5. **Train from Files** - Use actual text files
6. **Compare Tokenizers** - See differences between models
7. **Fine-tune Tokenizer** - Add tokens to existing models

```bash
# Run all training examples
python 02_tokenizers_training.py
```

**Outputs:**

- Trained tokenizers saved in `./tokenizers/` directory
- Ready to use in your projects
- Comparison reports

### 4. Advanced Training Methods (45 minutes)

Master different training patterns:

**[🎓 03_advanced_training_methods.py](./03_advanced_training_methods.py)**

**7 Advanced Patterns:**

1. **Train from List** - Simple Python lists/tuples
2. **Train from Iterables** - Tuples, generators, any iterable
3. **🤗 Datasets Library** - Batch iterators for efficiency
4. **Gzip Files** - Read compressed files directly
5. **Batch Efficiency** - Compare single vs batch performance
6. **Custom Iterators** - Filter, transform, multi-source patterns
7. **Progress Tracking** - Monitor training with length parameter

```bash
# Run all advanced training examples
python 03_advanced_training_methods.py
```

**Key Learnings:**

- Batch iterators are 10-20x faster
- Use generators for memory efficiency
- Progress tracking with `length` parameter
- Train from any Python iterator

### 5. Production Guide (30 minutes)

Learn production-level considerations:

**[🏭 06_production_guide.md](./06_production_guide.md)**

**Critical Topics:**

- Performance optimization (batch processing, parallelization)
- Memory management (streaming, lazy loading)
- Error handling & edge cases
- Security considerations (input sanitization, rate limiting)
- Monitoring & debugging
- Common production issues & solutions

### 6. Tokenizer Comparison (20 minutes)

Understand different algorithms and choose the right one:

**[📊 07_tokenizer_comparison.md](./07_tokenizer_comparison.md)**

**Comparisons:**

- BPE vs WordPiece vs Unigram vs WordLevel
- GPT vs BERT vs T5 vs LLaMA tokenizers
- Performance benchmarks (speed, memory)
- Language support comparison
- Use case recommendations

**Includes decision tree** to help you choose!

### 7. Integration Guide (30 minutes)

Connect tokenizers to your ML workflow:

**[🔌 08_integration_guide.md](./08_integration_guide.md)**

**Integrations:**

- 🤗 Transformers (AutoTokenizer, models)
- PyTorch & TensorFlow (custom datasets)
- FastAPI / Flask (REST APIs)
- Database storage (SQLite, PostgreSQL)
- Streaming applications
- Complete working examples

## 🎯 Learning Objectives

By the end of this module, you will be able to:

- ✅ Load and use pretrained tokenizers
- ✅ Build custom tokenizers from scratch
- ✅ Choose the right tokenization algorithm for your task
- ✅ Train tokenizers on your own data
- ✅ Understand the full tokenization pipeline
- ✅ Use advanced features (padding, truncation, batching)
- ✅ Optimize tokenization for production
- ✅ Debug tokenization issues
- ✅ Compare different tokenization approaches

## 📊 File Structure

```text
1-token/
├── README_TOKENIZERS.md                   # This file - Complete guide
├── 10_huggingface_tokenizers_guide.md        # Detailed reference (1 hour)
├── 01_tokenizers_quickstart.py            # Quick start (30 min)
├── 02_tokenizers_training.py              # Training examples (45 min)
├── 03_advanced_training_methods.py        # Advanced patterns (45 min)
│
├── 02_intro.md                               # Tokenization basics
├── tiktoken_example.py                    # tiktoken examples
├── token_exploration.py                   # Token analysis
├── token_exercises.py                     # Practice exercises
│
└── tokenizers/                            # Output directory
    ├── bpe_gpt2_style.json               # Trained BPE tokenizer
    ├── wordpiece_bert_style.json         # Trained WordPiece
    ├── unigram_multilingual.json         # Trained Unigram
    ├── code_tokenizer.json               # Code-specific tokenizer
    ├── finetuned_tokenizer.json          # Fine-tuned model
    └── list_trained.json                 # List-trained example
```

## 🔑 Key Concepts

### Tokenization Algorithms

| Algorithm | Use Case | Examples |
|-----------|----------|----------|
| **BPE** | General purpose, English | GPT-2, GPT-3, RoBERTa |
| **WordPiece** | BERT-style models | BERT, DistilBERT, ELECTRA |
| **Unigram** | Multilingual, probabilistic | T5, ALBERT, XLNet |
| **WordLevel** | Simple baseline | Basic models |

### Tokenization Pipeline

```mermaid
flowchart TD
    A[Input Text] --> B[Normalization]
    B --> C[Pre-Tokenization]
    C --> D["Model (BPE / WordPiece / Unigram)"]
    D --> E[Post-Processing]
    E --> F["Output (Encoding)"]
```

### Why HuggingFace Tokenizers?

1. **Speed** ⚡
   - 10-20x faster than pure Python implementations
   - Optimized Rust core with Python bindings
   - Can tokenize 1GB of text in seconds

2. **Full Alignment** 🎯
   - Track exact character positions
   - Map tokens back to original text
   - Essential for span-based tasks (NER, QA)

3. **All Algorithms** 🧰
   - BPE (GPT-2 style)
   - WordPiece (BERT style)
   - Unigram (SentencePiece)
   - WordLevel (baseline)

4. **Production Ready** 🚀
   - Used by Transformers library
   - Battle-tested in production
   - Easy to serialize/deserialize

## 💡 Quick Reference

### Load Pretrained

```python
from tokenizers import Tokenizer

# From Hugging Face Hub
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# From local file
tokenizer = Tokenizer.from_file("my-tokenizer.json")
```

### Train Custom

```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]"])

# Train from iterator
tokenizer.train_from_iterator(texts, trainer=trainer)

# Or train from files
tokenizer.train(["data.txt"], trainer)
```

### Encode & Decode

```python
# Encode
output = tokenizer.encode("Hello, world!")
print(output.tokens)  # Token strings
print(output.ids)     # Token IDs
print(output.offsets) # Character positions

# Decode
text = tokenizer.decode([7592, 1010, 2088])
```

### Batch Processing

```python
# Encode batch
outputs = tokenizer.encode_batch(["Text 1", "Text 2", "Text 3"])

# With padding
tokenizer.enable_padding(pad_id=0, pad_token="[PAD]")
outputs = tokenizer.encode_batch(texts)

# With truncation
tokenizer.enable_truncation(max_length=512)
outputs = tokenizer.encode_batch(texts)
```

## 🎓 Exercises

### Beginner

1. Load BERT tokenizer and encode 5 sentences
2. Count tokens for different models on same text
3. Find which tokens correspond to unknown words
4. Compare BPE vs WordPiece on your text

### Intermediate

1. Train a BPE tokenizer on your domain data
2. Add 50 domain-specific tokens to BERT tokenizer
3. Build a tokenizer pipeline with custom normalizer
4. Implement padding and truncation strategy

### Advanced

1. Train multilingual tokenizer (3+ languages)
2. Build code-specific tokenizer for your language
3. Optimize vocab size for your task
4. Compare tokenizer performance on 1GB corpus

## 📈 Performance Tips

1. **Use Batch Encoding** - 10-20x faster than loops
2. **Enable Padding Efficiently** - Pad to multiple of 8 for GPU
3. **Choose Right Vocab Size** - Larger = better coverage, slower
4. **Reuse Tokenizers** - Don't reload every time
5. **Save in JSON** - Fast serialization/deserialization

## 🔍 Debugging Tips

```python
# Check vocabulary
vocab = tokenizer.get_vocab()
print(f"Vocab size: {len(vocab)}")

# Inspect tokens
for token, id in list(vocab.items())[:10]:
    print(f"{token} -> {id}")

# Track alignment
output = tokenizer.encode(text)
for i, token in enumerate(output.tokens):
    start, end = output.offsets[i]
    print(f"{token} came from: {text[start:end]}")

# Test special tokens
print(tokenizer.token_to_id("[MASK]"))
print(tokenizer.id_to_token(0))
```

## 🌟 Next Steps

After completing this module:

1. **Integrate with Transformers** - Use with 🤗 Transformers models
2. **Build NLP Pipeline** - Tokenize → Model → Decode
3. **Production Deployment** - Optimize for speed and memory
4. **Custom Algorithms** - Implement your own tokenizer
5. **Multilingual Systems** - Build language-agnostic pipelines

## 📚 Additional Resources

### Official Documentation

- [Tokenizers Docs](https://huggingface.co/docs/tokenizers)
- [Tokenizers GitHub](https://github.com/huggingface/tokenizers)
- [Transformers Integration](https://huggingface.co/docs/transformers)

### Related Topics

- **Phase 5**: [Embeddings Module](../05-embeddings/) - Next in learning path
- **tiktoken**: [OpenAI's tokenizer](./02_intro.md) - Alternative approach
- **SentencePiece**: Google's tokenizer - Another option

### Community

- [Hugging Face Forum](https://discuss.huggingface.co)
- [GitHub Discussions](https://github.com/zero-to-ai/discussions)
- [GitHub Issues](https://github.com/huggingface/tokenizers/issues)

## ⏱️ Time Estimates

| Activity | Time | Difficulty |
|----------|------|------------|
| Read guide | 1 hour | Beginner |
| Quick start examples | 30 min | Beginner |
| Training examples | 45 min | Intermediate |
| Advanced training methods | 45 min | Intermediate |
| **Production guide** | **30 min** | **Advanced** |
| **Comparison guide** | **20 min** | **Intermediate** |
| **Integration guide** | **30 min** | **Advanced** |
| Practice exercises | 2 hours | Mixed |
| **Total** | **~7-8 hours** | **Beginner-Advanced** |

## 🎯 Success Criteria

You've mastered this module when you can:

- [ ] Explain the tokenization pipeline
- [ ] Choose appropriate algorithm for your task
- [ ] Train tokenizer on your data (>95% success rate)
- [ ] Use all encoding features (padding, truncation, batching)
- [ ] Debug tokenization issues independently
- [ ] Optimize for production deployment
- [ ] Integrate with ML models

## 🤝 Contributing

Found an issue or want to add examples?

1. Fork the repository
2. Add your improvements
3. Submit a pull request

## 📝 Notes

- All examples use Python 3.7+
- Requires `tokenizers` library (pip install tokenizers)
- Some examples download pretrained models (requires internet)
- Output files saved in `./tokenizers/` directory

## ❓ FAQ

**Q: Which tokenizer algorithm should I use?**
A: Use BPE for general English, WordPiece for BERT-style, Unigram for multilingual.

**Q: How much data do I need to train?**
A: Minimum 1MB text for basic vocab, 10MB+ for production quality.

**Q: Can I use with other frameworks?**
A: Yes! Works standalone or with Transformers, FastAI, etc.

**Q: Is it faster than tiktoken?**
A: Yes, generally 2-5x faster due to Rust core.

**Q: How do I handle unknown words?**
A: All tokenizers have `unk_token` that represents unknown tokens.

**Q: Can I add new tokens later?**
A: Yes! Use `add_tokens()` or `add_special_tokens()`.

---

## Happy Tokenizing! 🚀

Built with ❤️ by the AI/ML learning community
