architectures

Residual Connection

A skip connection that adds the input of a layer directly to its output, allowing gradients to flow through the network without degradation. Residual connections enable training of very deep networks and are a key component of transformers.

In practice

In a transformer block, the output is layer_norm(x + attention(x)), where the '+x' is the residual connection.

Transformer
A neural network architecture introduced in the 2017 paper 'Attention Is All You Need' that processes sequential data using self-attention mechanisms instead of recurrence. Transformers enable massive parallelization during training, making them the foundation of modern LLMs and vision models.
Backpropagation
The algorithm used to compute gradients of the loss function with respect to each parameter in a neural network by propagating errors backward through the layers. Backpropagation enables efficient training of deep networks using gradient descent.
Deep Learning
A subset of machine learning that uses neural networks with many layers (deep architectures) to learn hierarchical representations of data. Deep learning has driven breakthroughs in computer vision, NLP, and speech recognition.