Skip to content
AI’s

training

Cross-Entropy

A loss function commonly used for classification tasks that measures the difference between predicted probability distributions and actual labels. Cross-entropy loss penalizes confident wrong predictions more heavily than uncertain ones.

In practice

In language modeling, cross-entropy loss measures how well the model's predicted token probabilities match the actual next token.