AI Engineering & LLMs (50 Qs)•Part 2: LLM Inference, Serving & Runtime Optimization
Tokenization Mechanics: Byte-Pair Encoding (BPE), Token Healing & Multi-Lingual Compression
1. Byte-Pair Encoding (BPE) Algorithm
Large Language Models do not process raw strings; they process numeric tokens. BPE builds vocabulary bottom-up from individual bytes, iteratively merging the most frequently adjacent byte pairs:
sql
Initial Bytes: [ 'l', 'o', 'w', 'e', 's', 't' ]
Pass 1: Merge ('l', 'o') ──► 'lo'
Pass 2: Merge ('e', 's') ──► 'es'
Pass 3: Merge ('es', 't') ──► 'est'
Final Tokenization: ['lowest'] or ['low', 'est']
2. Token Healing: Fixing the Trailing Whitespace Bug
Modern BPE tokenizers treat leading spaces as part of the token (e.g., " apple" $\neq$ "apple").
When prompts end abruptly with partial words or punctuation (e.g., "The URL is http:"), standard greedy tokenizers commit to sub-optimal token boundaries:
sql
Without Token Healing:
User Prompt: "http:" ──► Encoded as Token("http") + Token(":")
Model generates: Token("/") ──► Output: "http: /" (Incorrect space inserted!)
With Token Healing:
Removes the last token, backs up the pointer, and lets the model regenerate from valid prefixes:
Prompt boundary healed ──► Generates Token("://") seamlessly!
3. Multi-Lingual & Code Compression Ratios
| Language / Domain | Characters per Token (GPT-2, 50k vocab) | Characters per Token (Llama 3 / GPT-4o, 128k–200k vocab) | Impact |
|---|---|---|---|
| English Prose | ~4.0 chars / token | ~4.5 chars / token | High efficiency baseline |
| Python Code | ~3.2 chars / token | ~4.1 chars / token | Efficient whitespace & keyword encoding |
| Hindi / Arabic / East Asian | ~1.1 chars / token (Heavy byte fragmentation) | ~2.8 chars / token | 2.5x cheaper and faster due to expanded multi-lingual vocabulary |