architectures

Attention Mechanism

A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.

In practice

When translating 'The cat sat on the mat,' attention helps the model focus on 'cat' when generating the corresponding word in French.

Self-Attention
A mechanism where each element in a sequence attends to all other elements in the same sequence to compute a representation. Self-attention allows the model to capture relationships between words regardless of their distance in the text.
Multi-Head Attention
An extension of attention where multiple attention operations run in parallel, each with different learned projections. This allows the model to simultaneously capture different types of relationships (syntactic, semantic, positional) from the input.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.

Briefings

Attention Mechanism

Further reading