Chat Overview

GPT Models: Architecture and Training

How GPT combines causal Transformers, next-token prediction, scale, prompting, and alignment, from GPT-1 through GPT-4.

By Chat Overview Published Updated
A GPT model repeatedly predicts the next token, appends it to the context, and predicts again.
A GPT model repeatedly predicts the next token, appends it to the context, and predicts again.

GPT is a family of decoder-only Transformer models trained to predict the next token in a sequence. The architecture is comparatively simple. The consequences of training it across large and varied datasets are not.

The first GPT paper, Improving Language Understanding by Generative Pre-Training, appeared in 2018. It showed that a language model could learn general representations from unlabeled text and then be adapted to several supervised tasks. GPT-2 emphasized zero-shot transfer. GPT-3 showed that tasks could be specified with instructions and examples inside the prompt. Later systems added instruction tuning, preference-based alignment, tools, and multimodal input.

The series is historically important, but the name can cause confusion. "GPT" refers both to a general architectural family and to particular OpenAI products. Public papers provide substantial detail for GPT-1, GPT-2, and GPT-3. The GPT-4 technical report describes capabilities and evaluation but withholds architecture size, training compute, dataset construction, and other implementation details. A sound technical account must keep those evidence levels separate.

Historical Context

Language models long predate GPT. N-gram models estimated a word from a short preceding context. Recurrent networks and LSTMs learned distributed state across longer sequences. The Transformer made it possible to train attention-based models efficiently on large batches.

GPT took the Transformer's decoder and used language modeling as general pretraining. This choice avoided the need to label every example by hand. Every text sequence contains its own training targets: after seeing a prefix, predict what follows.

GPT-1 paired that pretraining with supervised fine-tuning. GPT-2, released in stages during 2019, studied whether a larger language model could perform tasks without parameter updates. GPT-3 expanded this idea to in-context learning: a prompt could contain a task description or a few demonstrations, and the model could continue in the requested pattern.

Decoder-Only Architecture

A GPT model first converts text into tokens. Tokens are not necessarily words. A tokenizer may represent a common word as one token and split an unfamiliar word into several pieces. Each token is mapped to a vector, combined with positional information, and passed through a stack of Transformer blocks.

Each block contains causal self-attention and a feed-forward network, with residual connections and normalization. "Causal" means that the attention mask prevents a position from reading future tokens during training. The representation at a position can use the prefix but not the answer that follows it.

The final representation is converted into scores for every token in the vocabulary. A softmax operation turns those scores into a probability distribution. Generation selects a token, appends it to the context, and repeats the process.

Greedy decoding always selects the highest-probability token. Sampling can produce more varied text. Temperature, top-k, and top-p settings alter how much of the distribution is considered; they do not add knowledge or reasoning to the model.

The Pretraining Objective

For tokens x1 through xn, causal language modeling maximizes the probability of each token given the tokens before it. In compact form, the model learns P(xt | x1, ..., x(t-1)) for every position.

This objective forces the network to use many kinds of regularity. Predicting text can require grammar, genre, factual association, code structure, and the relationships established earlier in the context. The model learns these regularities as numerical parameters, not as an indexed copy of the training corpus.

The objective also explains an important limitation. The model is trained to produce likely continuations, not to verify truth. A fluent false statement can score well when it resembles text patterns in the data. Grounding, tools, retrieval, and explicit verification procedures address this problem only partially.

From Fine-Tuning to Prompting

GPT-1 used the same pretrained model for several tasks but added task-specific fine-tuning. GPT-3 showed that a sufficiently capable model could infer some tasks from the context alone. Three terms are useful:

  • Zero-shot: the prompt gives an instruction but no worked example.
  • One-shot: the prompt includes one example.
  • Few-shot: the prompt includes several examples.

This is called in-context learning because the model's weights do not change. The prompt temporarily defines the pattern to follow. It differs from fine-tuning, which changes model parameters through additional training.

In-context learning made natural-language interfaces practical, but it did not make wording irrelevant. A model sees a stream of tokens. Clear task definitions, relevant context, explicit output constraints, and representative examples reduce ambiguity. The Prompting 101 guide covers this operational side.

Alignment and Instruction Following

A base language model predicts continuations. A useful assistant must also follow instructions, refuse some requests, admit uncertainty, and keep to a conversational role. Developers add post-training stages for these behaviors.

Supervised instruction tuning trains on examples of prompts and preferred responses. Preference optimization uses human or model judgments to favor one response over another. Reinforcement learning from human feedback, often shortened to RLHF, is one method. Direct preference optimization and related methods are other options.

Post-training changes behavior, but it does not replace pretraining. Nor does it guarantee that every response reflects a stable internal rule. Models can remain sensitive to prompt wording, conflicting instructions, and unfamiliar contexts.

Milestones and What Is Public

GPT-1, 2018

GPT-1 had 117 million parameters and used a 12-layer decoder-only Transformer. It pretrained on BookCorpus, then fine-tuned on classification, similarity, and question-answering tasks. The central result was transfer: one generative pretraining procedure improved several discriminative tasks.

GPT-2, 2019

GPT-2 scaled the model to 1.5 billion parameters and trained on WebText. The report framed language modeling as unsupervised multitask learning and presented zero-shot results. OpenAI initially released smaller versions and later released the full model.

GPT-3, 2020

GPT-3 had 175 billion parameters. Its paper systematically compared zero-shot, one-shot, and few-shot prompting across many tasks. Performance varied widely, and the paper documented limitations including bias, inefficient sample use, and unreliable synthesis. Its durable contribution was making prompting a serious mode of task specification.

GPT-4, 2023

GPT-4 accepted image and text inputs in the evaluations described by OpenAI and improved performance on many professional and academic benchmarks. The technical report did not disclose enough detail to reconstruct the model. Treat claims about its parameter count or exact internal architecture as speculation, not fact.

Impact and Capabilities

GPT models helped establish the foundation-model workflow: expensive general pretraining followed by adaptation to many downstream uses. One model can draft text, transform formats, classify records, explain code, or call tools because each task can be represented as a conditional sequence.

The same flexibility creates evaluation problems. A benchmark score covers a defined test, not every use of the system. Contamination can occur when test material appears in training data. Prompts can change results. A model can solve a familiar-looking item and fail a slightly altered one.

Treat output as unverified when accuracy matters. Citation requests do not guarantee real citations. Code generation does not guarantee secure code. A long context does not guarantee that every supplied fact will be used correctly.

Limits of the Family Label

Not every modern language model is GPT, and not every GPT-branded system has the same published design. Other decoder-only model families use similar broad principles with different tokenizers, attention variants, training mixtures, context mechanisms, and post-training methods.

Architecture alone also does not explain capability. Data quality, compute, optimization, evaluation, system prompts, retrieval, tools, and serving constraints all matter. The diagram at the top of this page is the stable core: causal context goes through a decoder stack to produce a next-token distribution.

Further Reading