architectures

Encoder-Decoder

A neural network architecture with two main components: an encoder that processes and compresses the input into a representation, and a decoder that generates output from that representation. This pattern is fundamental to translation, summarization, and other transformation tasks.

In practice

T5 uses an encoder-decoder architecture where the encoder processes the input text and the decoder generates the output.

Seq2Seq
Sequence-to-Sequence, an architecture that maps an input sequence to an output sequence of potentially different length. Seq2Seq models typically use an encoder to process the input and a decoder to generate the output.
Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
BERT
Bidirectional Encoder Representations from Transformers, a model developed by Google that reads text in both directions simultaneously. BERT revolutionized NLP by introducing effective pre-training for understanding tasks like classification, question answering, and named entity recognition.