Skip to content

training

Normalization

Techniques for scaling and shifting data or intermediate representations to improve training stability and speed. Common forms include batch normalization, layer normalization, and RMS normalization, each operating across different dimensions.

In practice

Layer normalization, used in transformers, normalizes across feature dimensions within each example independently.