Attention Mechanism

An attention mechanism allows a neural network to focus on the most relevant parts of its input when computing each piece of output. Instead of treating all input positions equally, attention assigns weights that reflect how much each position should influence the result, enabling models to capture relationships across a sequence.

How It Works

In the dominant form, scaled dot-product attention, each token produces a query, a key, and a value vector. The model compares a query against all keys to score relevance, normalizes the scores into weights, and uses them to take a weighted sum of the values. Self-attention applies this within a single sequence so every token can reference every other. Multi-head attention runs several attention computations in parallel, letting the model attend to different kinds of relationships at once.

Why It Matters

Attention is the core innovation behind transformers and modern language models. It captures long-range dependencies that earlier architectures struggled with, and it does so in a parallelizable way that scales to large data. Its cost grows with sequence length, which is a key reason context windows are limited and why efficient attention variants are an active area of research.

Related Terms

The attention mechanism is the heart of the transformer, enables the large-language-model to relate token positions, and drives the cost of the context-window.