architectures

Self-Attention

A mechanism where each element in a sequence attends to all other elements in the same sequence to compute a representation. Self-attention allows the model to capture relationships between words regardless of their distance in the text.

In practice

In the sentence 'The animal didn't cross the street because it was too tired,' self-attention helps the model understand 'it' refers to 'animal.'

Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.
Multi-Head Attention
An extension of attention where multiple attention operations run in parallel, each with different learned projections. This allows the model to simultaneously capture different types of relationships (syntactic, semantic, positional) from the input.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.