architectures

Transformer XL

An extension of the transformer architecture that introduces recurrence to capture longer-range dependencies. Transformer XL uses segment-level recurrence with a novel positional encoding scheme, allowing it to learn dependencies beyond a fixed context length.

In practice

Transformer XL can model dependencies 80% longer than standard transformers while being faster during evaluation.

Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Context Window
The maximum number of tokens a language model can process in a single input-output exchange. A larger context window allows the model to consider more information when generating responses, enabling tasks like analyzing long documents.
Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.