deployment

RoPE

Rotary Position Embedding, a method for encoding token position information in transformers using rotation matrices. RoPE naturally encodes relative positions, enabling better length generalization and is used in most modern LLMs including Llama and Mistral.

In practice

RoPE allows models to extrapolate to longer sequences than seen during training by encoding positions as rotations.

Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Context Window
The maximum number of tokens a language model can process in a single input-output exchange. A larger context window allows the model to consider more information when generating responses, enabling tasks like analyzing long documents.
Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.