# Understanding Tokens: The Foundation of Language Models

## What Are Tokens?
**
**Tokens** are the fundamental building blocks that language models use to process text. Think of them as the "words" that AI models understand-though they're not exactly words in the traditional sense.

When you input text like `"Hello, world!"` into an LLM, the model doesn't process it as a continuous string. Instead, it breaks it down into **tokens**-discrete units that can be words, parts of words (subwords), or even individual characters.

## Why Tokens Matter

Language models can't directly understand human text. They need to convert text into numbers (tokens) that can be mathematically processed. This process is called **tokenization**.

**Example:**
```text
Input text: "hello how are you ?"
Tokens: [15339, 1268, 527, 499, 949]
Number of tokens: 5
```

Each number represents a specific piece of text from a pre-defined vocabulary. The model processes these numeric tokens, not the original text.

### Rules of Thumb for English

Understanding token counts helps estimate costs and context limits:

- **1 token ≈ 4 characters**
- **1 token ≈ ¾ of a word**
- **100 tokens ≈ 75 words**
- **1-2 sentences ≈ 30 tokens**
- **1 paragraph ≈ 100 tokens**
- **~1,500 words ≈ 2,048 tokens**

**Real-world examples:**
- "You miss 100% of the shots you don't take" (Wayne Gretzky) = 11 tokens
- US Declaration of Independence = 1,695 tokens
- Average email = 100-300 tokens

### Context Matters: Same Word, Different Tokens

The same word can become **different tokens** based on context:

```python
# Example: The word "red" tokenizes differently based on position
"The apple is red."     # ' red' (with leading space) → token: 2266
"Red is my favorite."   # 'Red' (capitalized, no space) → token: 7738  
"I like Red apples."    # ' Red' (with space, capital) → token: 2297
```

**Why?** The tokenizer learned from real text where:
- Words mid-sentence often have a leading space
- Words at sentence start are capitalized and have no leading space
- Capitalization changes meaning/usage

This context-awareness makes tokenization more efficient than simple word splitting.

## How Tokenization Works

Modern tokenization uses algorithms like **Byte-Pair Encoding (BPE)** or **WordPiece** to intelligently split text:

1. **Common words** → Single token
   - "hello" → 1 token
   - "world" → 1 token

2. **Uncommon words** → Multiple tokens (subwords)
   - "tokenization" → might split into ["token", "ization"]
   - "supercalifragilisticexpialidocious" → many smaller pieces

3. **Numbers and special characters** → Separate tokens
   - "GPT-4" → ["G", "PT", "-", "4"] or ["GPT", "-", "4"]

## Why Not Use Words?

You might wonder: "Why not just split text by spaces into words?" Here's why tokens are better:

### Problem with Word-Based Splitting:
- **Vocabulary explosion**: English has 170,000+ words. Adding other languages would make vocabulary enormous.
- **Out-of-vocabulary words**: New words (like "ChatGPT") wouldn't be recognized.
- **No handling of typos**: "helllo" would be completely different from "hello".
- **Different forms**: "run", "running", "ran" would be treated as completely separate entities.

### Token-Based Benefits:
- **Fixed vocabulary size**: Typically 50,000-100,000 tokens covers all languages.
- **Handles any word**: Unknown words are split into known subword tokens.
- **Related words share tokens**: "happy", "happiness", "unhappy" share the "happy" token.
- **Efficient**: Balances vocabulary size with text representation accuracy.

## Tokenization in Action

Let's see how different text gets tokenized:

```python
import tiktoken

# GPT-4 uses cl100k_base encoding
encoding = tiktoken.get_encoding("cl100k_base")

# Simple sentence
text1 = "hello how are you ?"
tokens1 = encoding.encode(text1)
print(f"'{text1}' → {tokens1}")
# Output: [15339, 1268, 527, 499, 949]  (5 tokens)

# Complex word
text2 = "supercalifragilisticexpialidocious"
tokens2 = encoding.encode(text2)
print(f"'{text2}' → {len(tokens2)} tokens")
# Output: Many tokens (word is split into subwords)

# Code example
text3 = "def hello_world():"
tokens3 = encoding.encode(text3)
print(f"'{text3}' → {tokens3}")
# Output: Each code element becomes tokens
```

## Key Concepts

### 1. Token vs Character vs Word

- **Character**: Single letter/symbol (`'h'`, `'e'`, `'l'`, `'l'`, `'o'`)
- **Token**: Subword unit (`'hello'` = 1 token, or split into smaller pieces)
- **Word**: Space-separated text (`'hello'`, `'world'`)

**Token is the middle ground** between characters (too granular) and words (too many unique units).

### 2. Context Window

LLMs have a **context window** measured in tokens, not characters:
- GPT-3.5: 4,096 tokens (~3,000 words)
- GPT-4: 8,192 or 32,768 tokens (~6,000-25,000 words)
- Claude 3: 200,000 tokens (~150,000 words)

This is why token counting matters-it determines how much text the model can process at once.

### 3. Token IDs

Each token has a unique **token ID** (integer):
- "hello" → 15339
- " world" → 1917 (note the space is part of the token)
- "!" → 0

The model's vocabulary is a mapping of token IDs to text pieces.

## Why Different Models Use Different Tokenizers

Different models use different tokenization schemes:

- **GPT-4, GPT-3.5**: `cl100k_base` (~100,000 tokens)
- **GPT-3**: `p50k_base` (~50,000 tokens)
- **BERT**: WordPiece tokenization
- **LLaMA**: SentencePiece BPE

Each tokenizer was trained on different data, optimizing for:
- Language coverage (English-only vs multilingual)
- Vocabulary size (memory constraints)
- Compression efficiency (fewer tokens = faster processing)

### Important: Tiktoken is OpenAI-Specific

**`tiktoken` only works with OpenAI models.** Each company's models use different tokenizers:

```
OpenAI (GPT-4, GPT-3.5)    → tiktoken (cl100k_base, p50k_base)
Meta (LLaMA, LLaMA 2)      → SentencePiece BPE
Google (BERT, PaLM)        → WordPiece / SentencePiece
Anthropic (Claude)         → Custom tokenizer
Mistral AI                 → SentencePiece
Cohere                     → Custom tokenizer
```

**Why different tokenizers?**

1. **Different Vocabularies**: Each model has its own vocabulary
   - GPT-4: ~100,000 tokens
   - LLaMA: 32,000 tokens
   - BERT: 30,000 tokens

2. **Different Token IDs**: Same text = different token IDs across models
   ```python
   # "hello" tokenizes to different IDs:
   # GPT-4 (tiktoken):  15339
   # LLaMA:             22172 (different!)
   # BERT:              7592 (different!)
   ```

3. **Not Interchangeable**: Cannot use tiktoken for non-OpenAI models

### Using Tokenizers for Other Models

For **any model** (OpenAI, LLaMA, BERT, Mistral, etc.), use **Hugging Face Transformers**:

```python
from transformers import AutoTokenizer

# Automatically loads correct tokenizer for any model
tokenizer = AutoTokenizer.from_pretrained("model-name")

# Examples:
# LLaMA
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
tokens = tokenizer.encode("Hello world")

# BERT
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.encode("Hello world")

# Mistral
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
tokens = tokenizer.encode("Hello world")

# Works the same way for ANY model on Hugging Face
decoded = tokenizer.decode(tokens)
```

### Quick Reference: Which Tool to Use?

| Tool | Works With | Use Case |
|------|-----------|----------|
| **tiktoken** | OpenAI only (GPT-3, GPT-4, GPT-3.5) | OpenAI API token counting |
| **transformers.AutoTokenizer** | Any model on Hugging Face | Universal tokenization |
| **sentencepiece** | LLaMA, T5, XLM, Mistral | Direct use of SentencePiece models |

**Best Practice:** 
- Using OpenAI API? → Use `tiktoken`
- Using any other model? → Use `AutoTokenizer` from Hugging Face
- Building with multiple models? → Use `AutoTokenizer` (works for all)

## Practical Implications

### 1. Cost

Many AI APIs charge per token:

- OpenAI GPT-4: $0.03 per 1K tokens (input)
- Knowing token count helps estimate costs

**Token categories for billing:**

- **Input tokens** – Your prompt/request
- **Output tokens** – Model's response (usually more expensive)
- **Cached tokens** – Reused conversation history (often discounted)
- **Reasoning tokens** – Internal "thinking" steps (some advanced models)

### 2. Performance

- Fewer tokens = Faster processing
- More tokens = Higher API costs
- Context window limits based on tokens

### 3. Prompt Engineering

Understanding tokenization helps craft better prompts:

```python
# Inefficient: "Please please please help me"
# tokens: ["Please", " please", " please", " help", " me"] = 5 tokens

# Efficient: "Please help me"
# tokens: ["Please", " help", " me"] = 3 tokens
```

### 4. Non-English Languages

Tokenization is less efficient for non-English text:

```python
# English: "How are you" = 3 tokens
# Spanish: "Cómo estás" = 5 tokens (for 10 characters)
```

Non-English text often has a **higher token-to-character ratio**, which affects:
- API costs (more tokens for same content)
- Context limits (fewer words fit in the same token budget)

## From Tokens to Embeddings

Once text is tokenized, each token is converted into an **embedding** (dense vector). This is the next step in how LLMs process language:

```
Text → Tokens → Embeddings → Model Processing → Output
```

- **Tokenization** (current topic): Text → Token IDs
- **Embeddings** (next topic): Token IDs → Dense vectors with semantic meaning

## Summary

| Aspect | Description |
|--------|-------------|
| **What** | Discrete units of text (subwords) |
| **Why** | Convert text to numbers for models |
| **How** | BPE, WordPiece algorithms |
| **Encoding** | Text → Token IDs (integers) |
| **Decoding** | Token IDs → Text |
| **Typical Size** | 50K-100K vocabulary |
| **Example** | "hello" → [15339] |

## Try It Yourself

Check out `tiktoken_example.py` to see tokenization in action with real examples.

---

**Next Step:** Learn about [Embeddings](../05-embeddings/) - how tokens are converted into meaningful vector representations that capture semantic relationships.

**See Also:**
- [Glossary](../23-glossary/01_GLOSSARY.md) - Definitions of AI/ML terms