architectures

Multi-Head Attention

An extension of attention where multiple attention operations run in parallel, each with different learned projections. This allows the model to simultaneously capture different types of relationships (syntactic, semantic, positional) from the input.

In practice

One attention head might focus on syntactic structure while another captures semantic meaning, together building a richer representation.

Self-Attention
A mechanism where each element in a sequence attends to all other elements in the same sequence to compute a representation. Self-attention allows the model to capture relationships between words regardless of their distance in the text.
Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.