---
title: Linear Layers and Activation Functions in Transformer Models
url: https://www.dataloco.com/en/linear-layers-and-activation-functions-in-transformer-models
published: 2026-09-14T02:10:49+00:00
language: en
section: Developers
source: https://machinelearningmastery.com/linear-layers-and-activation-functions-in-transformer-models/
publisher: Dataloco
---

# Linear Layers and Activation Functions in Transformer Models

Attention operations serve as the signature of transformer models, yet linear layers and activation functions are equally vital components. This article explores the necessity of these elements in transformer architecture.

The attention layer is fundamental to transformer models, aligning various elements within a sequence and transforming the input into an output sequence. It performs an affine transformation, resulting in a weighted sum of the input at every sequence element. While linear layers are integral, activation functions introduce non-linearity, crucial for learning complex patterns. Following the attention layer, a feed-forward network (FFN) or multi-layer perception network (MLP) is typically added to enhance learning capabilities.

In a standard transformer block, the input initially passes through the attention layer and subsequently through the feed-forward network, typically implemented as nn.Linear in PyTorch. An activation function is then applied within this network to introduce non-linearity into the transformation. The feed-forward network usually comprises multiple linear layers, where the first expands the dimension to explore various representations, while the last contracts it back to the original dimension.

The MLP sublayer in the BERT model is structured with two linear modules. Upon receiving an input sequence, the first linear module expands the dimension and applies the GELU activation function. The output then passes through a second linear module to revert to the original dimension. This design often employs an intermediate dimension that is four times larger than the original dimension, a common practice within transformer models.

Activation functions such as GELU and SwiGLU are favored in transformer models due to their ability to introduce non-linearity, enabling the learning of intricate patterns. Traditional activation functions like hyperbolic tangent (tanh), sigmoid, and rectified linear unit (ReLU) are also prevalent. Recent innovations like SwiGLU combine the Swish function with a linear function, allowing models to learn complex patterns without necessitating additional layers.