architectures

Seq2Seq

Sequence-to-Sequence, an architecture that maps an input sequence to an output sequence of potentially different length. Seq2Seq models typically use an encoder to process the input and a decoder to generate the output.

In practice

Machine translation systems use seq2seq models to convert a sentence in English into a sentence in Spanish.

Encoder-Decoder
A neural network architecture with two main components: an encoder that processes and compresses the input into a representation, and a decoder that generates output from that representation. This pattern is fundamental to translation, summarization, and other transformation tasks.
Attention Mechanism
A component in neural networks that allows the model to dynamically focus on the most relevant parts of the input when producing each part of the output. Attention computes weighted sums of input representations, where weights reflect relevance.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.