core concepts

Parameter Count

The total number of learnable values (weights and biases) in a neural network. Parameter count is often used as a rough proxy for model capability, though architecture and training data quality also matter significantly.

In practice

GPT-3 has 175 billion parameters, while Llama 3.1's largest variant has 405 billion parameters.

Weights
The numerical values in a neural network that are adjusted during training to minimize prediction errors. Weights determine how strongly each neuron connection influences the next layer, and a trained model's weights encode all its learned knowledge.
Scaling Laws
Empirical relationships showing that model performance improves predictably with increases in model size, dataset size, and compute. Scaling laws help researchers predict the capabilities of larger models before training them, guiding resource allocation.
Large Language Model
A neural network trained on massive text datasets to understand and generate human language. LLMs use billions of parameters to capture statistical patterns in language, enabling them to perform tasks like translation, summarization, and code generation.