architectures

Mixture of Depths

An architecture optimization where not all tokens are processed by every transformer layer. Mixture of Depths routes tokens through different numbers of layers based on their complexity, reducing compute for simpler tokens.

In practice

Function words like 'the' and 'is' might skip several layers, while complex terms get processed by all layers.

Mixture of Experts
An architecture where multiple specialized sub-networks (experts) exist within a model, with a routing mechanism that activates only a subset of experts for each input. This allows models to have enormous total parameter counts while only using a fraction of compute per inference.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Inference
The process of using a trained model to make predictions or generate outputs on new, unseen data. Inference is the production use of a model, as opposed to training, and its speed and cost are critical factors for deployment.