# Tokenization

> How text becomes numbers — tokenizers, vocabularies, training, and production tokenization pipelines.

![Status: complete](https://img.shields.io/badge/status-complete-brightgreen)

Open [`04-token.ipynb`](./04-token.ipynb) for learning objectives, prerequisites, and recommended reading order.

## Topics Covered

- BPE, WordPiece, and SentencePiece tokenizers
- HuggingFace `tokenizers` library and tiktoken
- Vocabulary training and specialised domain tokenizers
- Production integration and pipeline components

## Prerequisites

[02-data-science/README.md](../02-data-science/README.md)

## What Comes Next

[05-embeddings/README.md](../05-embeddings/README.md)
