From the archive
EleutherAI introduces Pythia for studying model training
Sixteen models and saved training checkpoints make it possible to compare how language models learn at different sizes.
EleutherAI introduced Pythia, a suite of 16 language models trained on public data in the same order. The models range from 70 million to 12 billion parameters, with 154 saved training checkpoints for each model.
Follow the training process
Most model releases provide the finished weights. Pythia also makes earlier versions available, so experiments can examine when a behavior appears during training and how it changes with model size. The release includes tools for reconstructing the data-loading process used in training.
That makes it useful for work on memorization, bias, and the relationship between training data and model behavior.
A research toolkit rather than a hosted assistant
Pythia is a collection for downloading, running, and comparing models. It does not provide a subscription chat service or a managed inference endpoint. The EleutherAI profile covers the organization's model research and evaluation tools.