Skip to content
AI’s

training

Weight Decay

A regularization technique that adds a penalty proportional to the magnitude of model weights to the loss function, encouraging smaller weight values. Weight decay helps prevent overfitting by limiting the model's capacity to memorize training data.

In practice

Using a weight decay of 0.01 means each weight is slightly reduced toward zero at every training step.