Chat Overview

Transformers: Attention Is All You Need (2017)

How the 2017 Transformer replaces recurrence with self-attention, including its encoder, decoder, training advantages, and limits.

By Chat Overview Published Updated
A Transformer block combines attention, a feed-forward network, residual paths, and normalization.
A Transformer block combines attention, a feed-forward network, residual paths, and normalization.

The Transformer is a neural-network architecture for processing sequences with attention instead of recurrence. It was introduced by Ashish Vaswani and colleagues in the 2017 paper Attention Is All You Need for machine translation.

The original model was an encoder-decoder system. The encoder read a source sentence; the decoder generated its translation. The paper's decisive change was to remove recurrent and convolutional sequence operations from the main architecture. Attention connected tokens directly, while matrix operations allowed many positions to be processed together during training.

This design later became the basis for encoder-only models such as BERT, decoder-only models such as GPT , and systems for images, audio, proteins, and multimodal data.

Historical Context

Recurrent networks process a sequence step by step. An LSTM improves the flow of information across those steps, but the state for token 20 still depends on computation for token 19. This sequential dependency limits training parallelism.

Attention had already improved recurrent encoder-decoder translation. A decoder could select relevant source states instead of relying on one fixed vector. Other researchers explored self-attention and memory-based models. The Transformer combined these directions into a complete architecture built around attention.

The paper reported better translation quality than previous systems on two WMT tasks while using less training compute. Just as important, it supplied a reusable block that scaled well on accelerator hardware.

Tokens and Position

Text is first converted into tokens and then embeddings. Attention by itself does not know whether one token came before another: reordering the inputs would reorder the outputs in the same way. The original Transformer therefore added sinusoidal positional encodings to token embeddings.

Later models use learned position embeddings, relative position biases, rotary position embeddings, or other schemes. The implementation changes, but the requirement remains. A sequence model needs some representation of order or relative position.

Scaled Dot-Product Attention

An attention layer derives three vectors from each input representation: a query, a key, and a value. A query is compared with keys to produce scores. Softmax turns those scores into weights, and the weighted values are summed.

In compact form, attention is softmax(QK^T / sqrt(d_k))V. The division by the square root of key dimension keeps dot products from growing so large that softmax becomes difficult to optimize.

For self-attention, queries, keys, and values come from the same sequence. For cross-attention, decoder queries attend to keys and values produced by the encoder.

Multiple Heads

The model does not calculate only one attention pattern. It projects the inputs into several lower-dimensional spaces, applies attention in parallel, concatenates the results, and projects them again. These parallel paths are called heads.

Different heads can learn different relations, but it is unsafe to assign a stable human meaning to every head. Some heads are redundant, and observed attention patterns do not by themselves explain a prediction.

The Encoder Block

An original Transformer encoder layer has two main sublayers:

  1. multi-head self-attention;
  2. a position-wise feed-forward network.

The feed-forward network applies the same small multilayer network at every position. Residual connections pass each sublayer's input around it, and layer normalization stabilizes the representation. Six encoder layers were stacked in the base model.

Because every source position can attend to every other source position in one layer, information can travel between distant tokens without crossing every intermediate recurrent step.

The Decoder Block

The decoder adds a third sublayer. It contains masked self-attention over the generated target prefix, cross-attention over encoder output, and a feed-forward network.

The causal mask prevents a target position from seeing later target tokens during training. Without it, the model could copy information from the answer it was meant to predict. During inference, the decoder generates one token at a time, so output generation remains sequential even though training can process target positions in parallel.

Decoder-only language models retain masked self-attention and the feed-forward network but omit the encoder and cross-attention. Encoder-only models omit causal masking because every input position may read both left and right context.

Why It Trains Efficiently

For a recurrent network, later states cannot be calculated until earlier states exist. Transformer attention for all training positions can be expressed as large matrix operations. GPUs and similar accelerators handle these operations well.

This does not mean attention is computationally cheap. Standard full attention compares every position with every other position. Its time and memory use grow quadratically with sequence length. At long context lengths, attention matrices can dominate cost.

Researchers address this with optimized exact kernels, sparse patterns, sliding windows, recurrence-like memory, compression, and approximations. These methods change the cost profile, and sometimes the information that can flow, so "long context" is not one uniform capability.

Training

The original translation model used supervised sentence pairs and token-level cross-entropy. Modern Transformer language models usually begin with self-supervised pretraining over large corpora. They are still trained with backpropagation ; removing recurrence did not remove gradient-based learning.

Residual paths, normalization, learning-rate schedules, initialization, data mixture, and numerical precision all affect whether large models train successfully. The architecture diagram is only one part of the system.

Impact and Applications

Transformers became the standard architecture for large-scale language pretraining because they combined flexible context modeling with efficient training. BERT used an encoder stack and masked-token objectives. GPT used a causal decoder stack. T5 cast many tasks into a text-to-text encoder-decoder format.

The same block can operate on other tokenized inputs. Vision Transformers divide an image into patches. Audio systems use frames or learned acoustic tokens. Protein models process amino-acid sequences. Multimodal systems combine or connect representations from several media.

This generality is a strength, not proof that one architecture is optimal for every task. Convolution remains useful when local structure and limited compute matter. Recurrence remains useful for some streaming and state-space designs.

Limits and Misconceptions

Self-attention provides a short path between tokens, but it does not guarantee that a model will use distant evidence correctly. Context-window size is a capacity limit, not a reliability measure.

Attention weights are also not a complete explanation. They show one set of intermediate coefficients among many layers, heads, residual paths, and nonlinear transformations.

Finally, the Transformer did not invent attention, encoder-decoder learning, or language modeling. Its contribution was a particularly effective arrangement of existing and new components, supported by strong translation results and a design that fit modern hardware.

Further Reading