Transformer

The transformer is a neural network architecture introduced in 2017 that relies on self-attention to model relationships between elements of a sequence. Unlike earlier recurrent networks that processed tokens one at a time, the transformer handles a whole sequence in parallel, which made training on very large datasets practical and led directly to today's large language models.

How It Works

A transformer turns input tokens into embedding vectors and adds positional information so order is preserved. Stacked layers each apply multi-head self-attention, which lets every token weigh and gather information from every other token, followed by a feed-forward network, with residual connections and normalization for stable training. Architectures vary: encoder-only models suit understanding tasks, decoder-only models suit generation, and encoder-decoder models suit translation-style tasks. Most modern LLMs are decoder-only.

Why It Matters

The transformer's parallelism and ability to capture long-range dependencies enabled the scaling that produced capable foundation models. Its design is now the default for language, and adapted for vision, audio, and multimodal tasks. Understanding it clarifies why context length, attention cost, and model size dominate performance and cost discussions.

Related Terms

The transformer is built on the attention-mechanism, consumes token and embedding inputs, and is the backbone of the large-language-model and broader foundation-model.