training

Layer Normalization

A normalization technique that normalizes the activations across features within a single training example, independent of other examples in the batch. Layer norm is the standard normalization in transformers, providing stable training dynamics.

In practice

Pre-layer normalization (normalizing before attention) has become the standard in modern transformers for better training stability.

Batch Normalization
A technique that normalizes the inputs to each layer of a neural network across a mini-batch during training. Batch normalization stabilizes and accelerates training by reducing internal covariate shift, allowing higher learning rates.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Normalization
Techniques for scaling and shifting data or intermediate representations to improve training stability and speed. Common forms include batch normalization, layer normalization, and RMS normalization, each operating across different dimensions.