training
Scaling Laws
Empirical relationships showing that model performance improves predictably with increases in model size, dataset size, and compute. Scaling laws help researchers predict the capabilities of larger models before training them, guiding resource allocation.
In practice
Chinchilla scaling laws showed that training a smaller model on more data is often more efficient than training a larger model on less data.