architectures

Mixture of Experts

An architecture where multiple specialized sub-networks (experts) exist within a model, with a routing mechanism that activates only a subset of experts for each input. This allows models to have enormous total parameter counts while only using a fraction of compute per inference.

In practice

Mixtral 8x7B has 46.7B total parameters but only activates 12.9B per token, making it fast despite its size.

Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Scaling Laws
Empirical relationships showing that model performance improves predictably with increases in model size, dataset size, and compute. Scaling laws help researchers predict the capabilities of larger models before training them, guiding resource allocation.
Parameter Count
The total number of learnable values (weights and biases) in a neural network. Parameter count is often used as a rough proxy for model capability, though architecture and training data quality also matter significantly.

Briefings

Mixture of Experts

Further reading