Chat Overview

Attention Mechanisms: Bahdanau, Cho, and Bengio (2014)

How neural attention scores, weights, and combines input representations, from Bahdanau attention to Transformer self-attention.

By Chat Overview Published Updated
Attention scores candidate values against a query, normalizes the scores, and returns a weighted combination.
Attention scores candidate values against a query, normalizes the scores, and returns a weighted combination.

Attention lets a neural network build a context-dependent combination of available representations. Given a query and a set of candidates, the mechanism scores the candidates, normalizes the scores into weights, and returns a weighted sum of their values.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced a widely influential neural version in their 2014 machine-translation paper. Their decoder did not depend on one fixed representation of the whole source sentence. At each output step, it selected a different weighted combination of encoder states.

That change improved translation of longer sentences and produced a learned soft alignment between source and target words. Later work simplified the scoring functions, introduced self-attention, and made attention the central operation of the Transformer .

Historical Context

The first recurrent encoder-decoder models compressed a source sentence into a fixed-size state. The decoder generated the target from that state. As the source grew, preserving every useful detail in one vector became difficult.

Traditional statistical translation systems had explicit alignment components. Bahdanau and colleagues wanted a neural system to learn alignment jointly with translation. A bidirectional recurrent encoder produced an annotation for each source word. The decoder calculated a new context vector from those annotations before generating each target word.

Because the weights were continuous and differentiable, the complete model could be trained with backpropagation . No separate alignment labels were required.

Score, Normalize, Combine

Most attention mechanisms can be understood as three operations.

First, score each candidate. In Bahdanau attention, a small feed-forward network compared the decoder's previous state with each encoder annotation. A larger score meant that the source position was more relevant to the next target word.

Second, normalize the scores. Softmax converts them into positive weights that sum to one. If one candidate has a much larger score, most of the weight falls there. If scores are similar, the result mixes several candidates.

Third, multiply each value by its weight and add the results. The context vector is therefore a weighted average, not a hard selection of one position.

In compact pseudocode:

  • scores[i] = score(query, key[i])
  • weights = softmax(scores)
  • context = sum(weights[i] * value[i])

The terms query, key, and value became standard later. A database analogy is helpful but imperfect. The query represents what is needed now. Keys describe what each candidate can match. Values contain the information to retrieve. Unlike a database lookup, attention usually mixes multiple values and learns the projections used for every part.

Additive and Dot-Product Attention

Bahdanau attention uses a learned feed-forward scoring network. It is often called additive attention because the transformed query and key are added before a nonlinearity and final score projection.

Luong and colleagues compared several alternatives in 2015, including dot-product scoring. Dot-product attention is computationally convenient because all query-key comparisons can be calculated as a matrix multiplication.

The Transformer uses scaled dot-product attention. If query and key vectors have dimension d_k, it divides their dot products by sqrt(d_k) before softmax. Without scaling, large dot products can push softmax toward extremely sharp outputs and weak gradients.

Cross-Attention

The original translation mechanism is cross-attention. The query comes from the target-side decoder, while keys and values come from source-side encoder states. It connects two different sequences.

At one decoding step, the weights may emphasize the source word corresponding to the next translated word. At another step, they may cover several words needed for an idiom or grammatical decision. The alignment is soft and changes with the decoder state.

Cross-attention now links many kinds of representations. A text decoder may attend to image features. An image generator may attend to text embeddings. A speech decoder may attend to acoustic frames.

Self-Attention

In self-attention, queries, keys, and values are derived from the same sequence. Each position builds a representation using other positions in that sequence.

An encoder commonly allows every position to attend to every other position. A causal language-model decoder masks later positions, so a token can only use the prefix. This causal mask is what allows GPT to train on full sequences without reading future answers.

Self-attention provides a short computational path between distant positions. In bidirectional self-attention, the first token can exchange information with the last in one layer. With causal masking, the last token can use information from the first, but the first cannot attend to later tokens. A recurrent network would pass information through each intermediate step.

Multi-Head Attention

Multi-head attention calculates several attention operations in parallel using different learned projections. The head outputs are concatenated and projected back to the model dimension.

Multiple heads increase the ways a layer can combine information. One head may place weight on nearby syntax while another responds to a repeated name, but such descriptions are examples rather than guaranteed roles. Heads can overlap or become inactive.

Attention Is Not Explanation

Attention weights are visible and tempting to interpret. In an early translation model, a matrix of weights often resembles a word alignment. That makes it a useful diagnostic.

It does not follow that the weights give a complete causal explanation of the output. A modern model contains many heads and layers, value projections, feed-forward networks, and residual paths. Changing a visible weight may not have the expected effect, and other components can carry the same information.

Use attention maps to inspect behavior, not as sole proof of why a model made a decision.

Evolution and Extensions

Attention developed along several lines:

  • global attention compares against every candidate;
  • local and sparse attention restrict comparisons to reduce cost;
  • multi-query and grouped-query attention share key-value projections across query heads;
  • linear and kernelized variants approximate or rearrange the operation;
  • optimized kernels calculate exact attention with less memory traffic.

The tradeoff is usually between flexible access, computation, memory, and the structure imposed on information flow.

Impact and Applications

Attention is used in translation, language modeling, search ranking, vision, speech, recommender systems, protein modeling, and generative media. Its broad usefulness comes from content-based selection. The model can decide which available representations matter for the current input instead of relying on a fixed neighborhood.

The main limitation of full self-attention is quadratic growth. A sequence twice as long contains roughly four times as many pairwise scores. Long-context systems need substantial memory and computation or a modified attention pattern.

Attention also does not enforce factuality, consistency, or complete use of context. It provides a route through which information can flow. Training and inference determine whether the route is used well.

Further Reading