architectures

GPT Architecture

A decoder-only transformer architecture that generates text autoregressively, predicting one token at a time from left to right. The GPT architecture uses masked self-attention to prevent the model from looking ahead, and scales effectively to billions of parameters.

In practice

The GPT architecture has been scaled from GPT-1 (117M params) to GPT-4 (estimated trillions of params).

GPT
Generative Pre-trained Transformer, a family of autoregressive language models developed by OpenAI. GPT models are pre-trained on large corpora of text using unsupervised learning, then fine-tuned for specific tasks. The architecture uses decoder-only transformers.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Autoregressive Model
A model that generates output one element at a time, conditioning each prediction on all previously generated elements. Most language models are autoregressive, predicting the next token based on the sequence of tokens already produced.