Transformer Architecture
The architecture behind every modern LLM — self-attention with Query/Key/Value, multi-head attention, positional encoding, and why processing the whole sequence in parallel changed everything.
The transformer replaced sequential RNNs with self-attention: a mechanism where every token directly attends to every other token, weighting their relevance via Query/Key/Value projections. Run in parallel across many heads and stacked into deep layers (with positional encoding for order), this architecture captures long-range context efficiently — and is the foundation of GPT, Claude, Gemini, Llama, and essentially all modern LLMs.
Earlier sequence models (RNNs, LSTMs) processed tokens one at a time, making them slow and forgetful over long distances. The transformer (2017's 'Attention Is All You Need') processes the whole sequence in parallel and lets any token directly attend to any other — solving both speed and long-range dependencies.
Each token is projected into three learned vectors: a Query ('what am I looking for?'), a Key ('what do I offer?'), and a Value ('what I contribute'). Attention scores every token's Query against every token's Key (dot product, scaled), softmaxes them into weights, and produces a weighted blend of Values.
One attention pattern isn't enough, so transformers run many attention 'heads' in parallel, each with its own Q/K/V projections. Different heads specialize — some track syntax, some resolve pronouns, some link related entities — and their outputs are concatenated, giving a rich, multi-faceted view of context.
Attention alone is order-blind (it's a set operation), so position information is added to the token embeddings — modern LLMs use rotary embeddings (RoPE). Stacking dozens of attention + feed-forward layers (with residual connections and normalization) builds the deep representations that make large language models so capable.
Key Concepts
Each token attends to every other token, weighting their relevance — the transformer's core operation.
Three learned projections per token; attention = softmax(Q·Kᵀ/√d)·V.
Many parallel attention heads with separate projections, each learning different relationships.
Injects token order (often RoPE) since attention itself is permutation-invariant.
The whole sequence is processed at once (unlike RNNs), enabling efficient training on GPUs.
Skip connections and normalization that make stacking dozens of layers trainable and stable.
1# Scaled dot-product self-attention — the core of the transformer.2import numpy as np34def attention(Q, K, V):5 d_k = Q.shape[-1]6 scores = Q @ K.T / np.sqrt(d_k) # how much each token relates to each other7 weights = softmax(scores, axis=-1) # turn scores into a probability distribution8 return weights @ V # weighted blend of value vectors910# Each token is projected into THREE vectors via learned weight matrices:11# Q (query) — "what am I looking for?"12# K (key) — "what do I offer?"13# V (value) — "what do I contribute if attended to?"14# Multi-head attention runs this in parallel with different projections,15# so different heads learn different relationships (syntax, coreference, ...).1617# Order is injected separately, since attention is permutation-invariant:18x = token_embedding + positional_encoding # RoPE in most modern LLMs
The transformer is the single most important architecture in modern AI — understanding self-attention demystifies how LLMs work, why they have context limits, and why long inputs are expensive (O(n²) attention). It's essential background for prompt design, RAG, fine-tuning, and inference optimization, and a guaranteed AI-interview topic.
Common Pitfalls
1Resolving Ambiguity With Context
In 'The animal didn't cross the street because it was tired,' the word 'it' is ambiguous — does it refer to the animal or the street? A model must use the whole sentence to decide.
Word-by-word models struggle to connect 'it' back to 'animal' across several intervening words, and bag-of-words approaches ignore relationships entirely. Resolving such long-range dependencies is exactly where older architectures faltered.
Self-attention lets the 'it' token directly attend to every other token and put high weight on 'animal'. Because attention is global and parallel, the link is captured regardless of distance. Different heads can simultaneously track this coreference and other relations.
Takeaway: Self-attention's power is direct, global, weighted connections between tokens. That's how transformers handle long-range context and ambiguity that defeated RNNs — and why they scale so well to long documents.
2Understanding LLM Cost and Context Limits
An engineer notices that doubling the prompt length more than doubles latency and cost, and that models have hard context-window limits. They want to understand why.
Without knowing the architecture, context limits and the super-linear cost of long inputs feel arbitrary, making it hard to design efficient prompts, RAG chunking, or batching strategies.
It traces back to attention: every token attends to every other, so naive self-attention is O(n²) in sequence length — quadratic compute and memory. That's why context windows are bounded and long prompts get expensive, and why techniques like FlashAttention, sliding-window, and KV-caching exist to mitigate it.
Takeaway: The transformer's all-to-all attention explains LLM context limits and the quadratic cost of long inputs. Understanding it lets you reason about prompt length, RAG design, and inference optimizations rather than guessing.
Join the discussion
Comments
Loading comments...
Cookie preferences
We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.