How Attention Mechanisms Rewired the Way Machines Read

Before attention, neural networks processed language the way a telegraph operator transcribes a message: one symbol at a time, carrying forward whatever summary of past context fits into a fixed-size state. After attention, networks could process language more like a skilled reader: scanning the entire document simultaneously, weighting each word's relevance to every other word in context. This architectural shift — from sequential hidden states to global contextual representations — is the foundation of every modern language model.

The Problem Attention Solved

Recurrent neural networks (RNNs) and their improved variants (LSTMs, GRUs) maintained a hidden state — a fixed-size vector that accumulated information from all preceding tokens. In theory, this state could encode arbitrarily long context. In practice, two problems undermined this:

Vanishing gradients: during backpropagation, gradients decay exponentially as they travel through many time steps. By the time they reach early layers, the signal is negligible. The model cannot effectively learn dependencies between tokens far apart in a sequence.

Sequential processing bottleneck: each token must wait for the previous token's hidden state to be computed. This prevents parallelization across time steps, making training extremely slow on long sequences.

The result: pre-attention models effectively had a working memory of roughly 10-30 tokens, regardless of theoretical capacity.

The Additive Attention Breakthrough (2015)

Bahdanau et al. (2015) introduced attention for neural machine translation. Instead of encoding the entire source sentence into a single fixed vector, they allowed the decoder to look back at all source tokens at each generation step, computing a weighted sum of source representations.

The weights — the attention scores — were computed dynamically based on the compatibility between the decoder's current state and each source token's representation. High-scoring tokens received more weight in the sum; the model could selectively focus on the relevant part of the source at each decoding step.

This is called cross-attention: the query comes from one sequence (the decoder), and the keys and values come from another (the encoder).

Self-Attention: The Key Innovation

The transformer (Vaswani et al., 2017) extended attention in a crucial way: instead of attending from one sequence to another, every token attends to every other token in the same sequence. This is self-attention.

For each token, we compute three vectors:

• Query (Q): "what am I looking for?"
• Key (K): "what do I represent?"
• Value (V): "what information do I carry?"

Attention scores are computed as:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

The dot product QK^T measures the compatibility between every query-key pair; dividing by √d_k prevents the dot products from growing too large in high dimensions; softmax normalizes the scores to a probability distribution; the weighted sum over values produces the output representation.

The result: every token's representation is updated based on a weighted mixture of all other tokens' representations. "Bank" in "river bank" gets a representation shaped by the tokens "river" and "shore" nearby; "bank" in "central bank" gets a different representation shaped by "central" and "financial."

Multi-Head Attention

A single attention head computes one set of attention weights over the entire sequence. Multi-head attention runs several attention mechanisms in parallel, each initialized with different weight matrices, then concatenates and projects their outputs.

Different heads specialize in different types of relationships:

• Some heads learn syntactic dependencies (subject-verb agreement)
• Others learn semantic co-occurrence (entities that co-occur together)
• Some track positional proximity; others learn long-range discourse structure

This specialization is not designed — it emerges from training. Visualization tools like BertViz show that different heads in BERT attend to qualitatively different patterns.

Positional Encodings: Solving the Order Problem

Self-attention, as described, is permutation-invariant: it does not inherently know whether token A appeared before or after token B. This is solved by positional encodings — vectors added to each token's input embedding that encode position information. The original transformer used sinusoidal positional encodings; modern models use learned positional embeddings or Rotary Position Embedding (RoPE), which improves performance on long sequences.

The Quadratic Complexity Challenge

Self-attention's computational cost is O(n²) in sequence length — computing the full attention matrix requires n² dot products. For a 512-token sequence, this is manageable. For a 100,000-token document, it becomes prohibitive.

This has motivated research into efficient attention variants:

• Sparse attention (Longformer, BigBird): compute attention only between nearby tokens and a small set of global tokens
• Linear attention: approximate full attention in O(n) time via kernel methods
• Flash Attention: a hardware-aware implementation that dramatically reduces memory usage by computing attention in tiles without materializing the full attention matrix

Why Attention Changed Everything

Attention solved the information bottleneck of fixed hidden states, enabled full parallelization during training, and produced contextual representations that captured semantic relationships across arbitrary distances. These properties, combined with the scale made possible by parallelization, enabled pretraining on internet-scale corpora — and the resulting emergent capabilities of modern large language models.

Every major model since 2017 — BERT, GPT, T5, LLaMA, Gemini — is built on the transformer's attention mechanism. Understanding attention is understanding the foundation of modern NLP.

Keywords: attention mechanism, self-attention, transformer architecture, multi-head attention, NLP, BERT, GPT, transformer explained, deep learning NLP