architectures

Positional Encoding

A method for injecting information about token position into transformer models, which otherwise have no inherent sense of order. Positional encodings can be fixed sinusoidal functions, learned embeddings, or relative position representations like RoPE.

In practice

Without positional encoding, the transformer would process 'dog bites man' and 'man bites dog' identically.

Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
RoPE
Rotary Position Embedding, a method for encoding token position information in transformers using rotation matrices. RoPE naturally encodes relative positions, enabling better length generalization and is used in most modern LLMs including Llama and Mistral.
Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.