Skip to content
AI’s

deployment

Distillation

A technique where a smaller student model is trained to replicate the behavior of a larger teacher model. Distillation transfers the teacher's knowledge into a compact model that is faster and cheaper to run while retaining much of the performance.

In practice

DistilBERT is a distilled version of BERT that is 60% smaller and 60% faster while retaining 97% of BERT's performance.