architectures

Sparse Attention

An attention variant where each token attends to only a subset of other tokens rather than all tokens in the sequence. Sparse attention reduces the quadratic computational cost of standard attention, enabling processing of much longer sequences.

In practice

Longformer uses sparse attention patterns (local + global) to process documents up to 16K tokens efficiently.

Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.
Flash Attention
An optimized attention computation algorithm that reduces memory usage from quadratic to near-linear by processing attention in blocks. Flash Attention uses tiling and kernel fusion to minimize GPU memory reads and writes, significantly speeding up training and inference.
Context Window
The maximum number of tokens a language model can process in a single input-output exchange. A larger context window allows the model to consider more information when generating responses, enabling tasks like analyzing long documents.

Briefings

Sparse Attention

Further reading