How Things Work

Tokenization & Context Windows

How text becomes the tokens LLMs actually process — BPE/SentencePiece vocabularies, why token counts surprise you, and managing the context window as a token budget for cost and memory.

How It Works

LLMs don't read characters or words — they read tokens, subword pieces from a fixed vocabulary built by BPE or SentencePiece. Token count drives both cost (you're billed per token) and memory (the context window is a token budget). Understanding tokenization explains surprising bills, context-overflow errors, and why some languages and content types are far more expensive than they look.

Tokens, Not Words

LLMs operate on tokens — subword units from a fixed vocabulary — not raw characters or whole words. The tokenizer converts text to a sequence of integer token IDs the model embeds, and converts the model's output IDs back to text. Everything the model reads and writes is tokens.

Byte-Pair Encoding (BPE)

Vocabularies are built by algorithms like BPE or SentencePiece: starting from characters, they repeatedly merge the most frequent adjacent pairs into new tokens. The result balances vocabulary size against sequence length — common words become single tokens, while rare or novel words break into reusable pieces.

Why Token Counts Surprise You

Common English words are ~1 token; rare words, names, and made-up words split into several. Whitespace, punctuation, code, JSON, and non-Latin scripts (Chinese, emoji) consume more tokens per character. A rough rule for English: ~4 characters ≈ 1 token ≈ ¾ of a word — but always measure for accuracy.

Context Windows

A model's context window is a hard limit on total tokens — prompt plus retrieved context plus the generated response must all fit. Modern windows are large (128K to 1M+), but they're still a budget: you pay per token and must trim, chunk, or summarize input to fit. Tokens are simultaneously the unit of pricing and of memory.

Key Concepts

🧩Token

A subword unit from the model's vocabulary — the atomic input/output the model actually processes.

🔤BPE / SentencePiece

Algorithms that build the vocabulary by merging frequent character/byte pairs into reusable tokens.

📚Vocabulary

The fixed set of tokens a model knows (often ~50K–250K). Each maps to an embedding vector.

🪟Context Window

The maximum total tokens (prompt + context + response) a model can handle at once — a token budget.

📏~4 chars ≈ 1 token

A rough English heuristic; code, JSON, and non-Latin text use more tokens per character.

💰Token Pricing

APIs bill per input and output token, so token count directly drives cost (and latency).

Counting tokens & budgeting context

tsx

1// Count tokens BEFORE you send a request — it's what you're billed and limited on.
2import { encoding_for_model } from "tiktoken";   // OpenAI BPE; others use SentencePiece
3const enc = encoding_for_model("gpt-4o");
4
5const tokens = enc.encode("tokenization isn't free");
6tokens.length;     // → fewer tokens than characters; rare words split into pieces
7
8// Budget the context window: prompt + retrieved context + response must all fit.
9const BUDGET = 128_000;                       // model's context window (tokens)
10const used   = enc.encode(systemPrompt).length
11             + retrievedChunks.reduce((s, c) => s + enc.encode(c).length, 0);
12const left   = BUDGET - used - maxResponseTokens;
13if (left < 0) throw new Error("context overflow — trim or summarize input");
14
15// Rule of thumb (English): ~4 chars ≈ 1 token ≈ ¾ of a word.
16// Code, JSON, whitespace, and non-Latin scripts use MORE tokens per character.

💡

Why This Matters

Tokens are the currency of LLMs — they determine cost, latency, and the hard context limit. Practical AI engineering requires budgeting prompts and RAG context in tokens, and tokenization quirks (rare words, code, non-Latin scripts) explain real production issues. It's also a common AI-interview question once you discuss context windows.

Common Pitfalls

⚠Estimating cost/limits by character or word count instead of actual tokens — off by a lot for code and non-Latin text.

⚠Forgetting the response also consumes the context budget — reserve tokens for the output.

⚠Assuming token counts are stable across models — different tokenizers split text differently.

⚠Ignoring that whitespace, formatting, and JSON structure all cost tokens.

⚠Truncating mid-token or mid-context naively, corrupting meaning instead of summarizing/reranking.

Real-World Use Cases

1Context Overflow in a RAG Pipeline

Scenario

A RAG app stuffs many retrieved document chunks plus a long system prompt into each request. It works in testing but starts throwing 'context length exceeded' errors on longer documents in production.

Problem

The team budgeted by characters or chunk count, not tokens. Prompt + retrieved context + the reserved response space silently exceeded the model's token window, and the failures only appeared with larger inputs.

Solution

Count tokens explicitly (tiktoken / the model's tokenizer), set an explicit budget — context window minus reserved response tokens — and trim or rerank retrieved chunks to fit. Summarize or paginate when input is too large. The pipeline now stays safely within the window.

💡

Takeaway: Always budget in tokens, not characters or chunk counts. Measure prompt + context + reserved response against the context window so you never overflow at the worst moment.

2Surprise Bill From a Non-English Workload

Scenario

A product translated to several languages saw its LLM costs spike far more than the text length suggested, especially for Chinese, Japanese, and emoji-heavy content.

Problem

Token cost was estimated from English character counts, but non-Latin scripts and emoji tokenize into many more tokens per character. The same visible text consumed multiples of the expected tokens, inflating cost and latency.

Solution

Measure real token counts per language with the actual tokenizer, factor that into pricing and rate limits, and consider models/tokenizers with better multilingual efficiency. Set per-request token caps so cost is bounded.

💡

Takeaway: Tokenization efficiency varies wildly by language and content type. Estimate cost from real token counts per workload — never assume English character ratios hold for code, JSON, or other scripts.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.