The idea: same formula, different axis

A layer in a neural network sees a batch: a table with one row per example and one column per feature. A normalization layer shifts and scales those numbers so they have a steady size, which keeps training stable. BatchNorm, LayerNorm and RMSNorm all use almost the same formula. The difference is which numbers they group together to compute the mean μ and spread σ. BatchNorm groups each column: one feature across all the examples in the batch. LayerNorm groups each row: one example across its own features. RMSNorm also works per row but only rescales.

Three 4 by 4 grids of examples e1 to e4 and features f1 to f4. BatchNorm stripes the columns and keeps one mu per feature under the grid. LayerNorm stripes the rows and keeps one mu per example beside the grid. RMSNorm stripes the rows and keeps one rms per example.
BatchNorm normalizes down the columns (features); LayerNorm and RMSNorm normalize along the rows (examples).

The three formulas

For a group of numbers x, all three compute a normalized x̂ and then apply a learned scale γ and shift β: y = γ·x̂ + β. The page sets γ = 1 and β = 0 so you can see x̂ itself.

groupx̂statistics kept
BatchNormone feature, all B examples of the batch(x − μ) / √(σ² + ε)running μ and σ² per feature
LayerNormone example, all its features(x − μ) / √(σ² + ε)none
RMSNormone example, all its featuresx / √(mean(x²) + ε)none

ε is a tiny number (10⁻⁵) that stops a division by zero. In a real network a feature is one of hundreds or thousands of channels; the page uses 4 features and up to 4 examples so every number fits on screen.

One batch, three answers

Batch A holds e1 = (2, 6, −1, 10), e2 = (4, 2, 1, 14), e3 = (9, 10, 11, 10) and e4 = (1, 2, −3, 6). BatchNorm's column means are μ = (4, 5, 2, 10) and σ = (3.08, 3.32, 5.39, 2.83), so e1 becomes (−0.65, 0.30, −0.56, 0.00): each value says how e1 compares with the other examples on that feature. LayerNorm uses e1's own μ = 4.25 and σ = 4.15 and gives (−0.54, 0.42, −1.27, 1.39): how each feature compares with e1's other features. RMSNorm divides by e1's rms = 5.94 and gives (0.34, 1.01, −0.17, 1.68). (Demo: one batch, three norms side by side)

e3 = (9, 10, 11, 10) shows the difference between LayerNorm and RMSNorm. LayerNorm removes its mean of 10 and gives (−1.41, 0, 1.41, 0). RMSNorm keeps the offset and gives (0.90, 1.00, 1.10, 1.00).

Your batchmates change your output

Batch B keeps e1 but replaces the other three with bigger numbers. BatchNorm's μ rises to (9.5, 12, 5, 17.5), and e1's output drops from (−0.65, 0.30, −0.56, 0.00) to (−1.65, −1.60, −1.60, −1.65): the same input, a different answer, only because of who else was in the batch. LayerNorm and RMSNorm give e1 exactly the same numbers as before. In training this noise acts a little like a regularizer; for anything else it is a problem. (Demo: same e1, new batchmates — only BatchNorm changes)

Small batches

BatchNorm needs enough examples to estimate each feature's mean and spread. With 2 examples every column comes out as exactly −1 and +1, whatever the values were. With 1 example each mean is the value itself, σ = 0 and every output is 0. PyTorch stops with “Expected more than 1 value per channel when training”. LayerNorm does not care: e1's row is the same for a batch of 4, 2 or 1. (Demo: small batches — 2 examples, then 1) This matters for big models, where one GPU may only fit a few long examples.

Training vs inference: running statistics

At inference a model often sees one example at a time, so BatchNorm cannot use batch statistics. During training it keeps a running mean and running variance per feature, moved 10 % toward each batch (momentum 0.1, starting at μ = 0, σ² = 1). model.eval() switches it to those saved values. LayerNorm and RMSNorm keep nothing and compute the same thing in both modes.

Training: BatchNorm normalizes with batch mu and sigma and updates running mu and sigma squared as 0.9 times running plus 0.1 times batch; at inference it uses the saved running values. LayerNorm and RMSNorm compute each example's own statistics in both modes.
Only BatchNorm behaves differently in training and inference; LayerNorm and RMSNorm do the same thing in both.

Before any training the running statistics are still (0, 1), so at inference e1 comes out unchanged as (2, 6, −1, 10). After 30 training batches the running μ is (3.62, 4.77, 0.01, 9.69), near the data's real means (4, 5, 2, 10), and e1 comes out as (−0.63, 0.44, −0.19, 0.10). Forgetting model.eval(), or evaluating on data whose statistics differ from training, are classic BatchNorm bugs. (Demo: inference — running statistics before and after training)

Why transformers use LayerNorm or RMSNorm

A transformer's activations have the shape batch × tokens × features. Sentences have different lengths, so batches are padded, and at generation time the model produces one token at a time for one user. Statistics over the batch would mix sentences, count padding and change between training and generation. LayerNorm normalizes each token's feature vector on its own, so none of that matters. It sits around every self-attention and MLP block (before them in “pre-LN” models such as GPT-2 onward). Llama, Mistral and many newer models use RMSNorm: it skips the mean and the β shift, saves a little compute, and trains just as well. CNNs for images still often use BatchNorm, where batches are large and every image has the same shape.

BatchNormLayerNormRMSNorm
depends on other examplesyes (training)nono
batch of 1✗ fails in training✓✓
train ≠ inferenceyes, running statsnono
subtracts the meanyesyesno
typical useCNNs (ResNet)transformers (BERT, GPT-2)LLMs (Llama, Mistral)

BatchNorm's training-time behaviour and its pairing with dropout are shown step by step on Dropout and Batch Normalization; the layers being normalized are the ones on Neural Network: the Forward Pass.

Before any normalization layer, the size of the random starting weights already decides whether the signal survives; see Weight Initialization: Xavier and He.

What the page leaves out

Real layers learn γ and β per feature; the page fixes them at 1 and 0. BatchNorm in a CNN takes its statistics over the batch and every pixel position of a channel, not just the batch. The page also leaves out GroupNorm and InstanceNorm (statistics over groups of channels or one image), SyncBatchNorm across GPUs, folding BatchNorm into the previous layer's weights for fast inference, and the exact placement debates (pre-LN vs post-LN, QK-norm).

On the mind map

Open the Machine Learning mind mapMachine LearningLearning setup: what feedback, how to scoreLearning setupLinear models: weighted sum w·x + bLinear modelsOther classic models: trees, margins, neighboursOther classic modelsUnsupervised: find structure without labelsUnsupervisedTraining: loss, gradient, small stepTrainingNeural networks: layers of σ(W·a + b)Neural networksArchitectures: how layers share informationArchitecturesLanguage models: predict the next tokenLanguage modelsLLM speed and tuning: cheaper to run, safer to useLLM speed and tuningRetrieval and agents: give the model data and toolsRetrieval and agentsGradient descentGradient descentstep against the gradientSGD vs Momentum vs AdamSGD vs Momentum vs AdamAdam scales each step per weightSoftmax and cross-entropySoftmax and cross-entropylogits to probabilities; gradient p − yBackpropagationBackpropagationchain rule from output back to inputBackprop intuitionBackprop intuitioneach output wants a nudge, passed backDropout and batch normDropout and batch normdiffer in train mode vs eval modeBatchNorm vs LayerNormstats over batch vs over one exampleWeight initializationWeight initializationXavier/He keep activation variance steadyLearning rate schedulesLearning rate schedulesWarm up, then cosine-decay the step sizeOpen the Machine Learning mind mapMachine Learningloss, gradient, small stepTrainingGradient descentGradient descentstep against the gradientSGD vs Momentum vs AdamSGD vs Momentum vs AdamAdam scales each step per weightSoftmax and cross-entropySoftmax and cross-entropylogits to probabilities; gradient p − yBackpropagationBackpropagationchain rule from output back to inputBackprop intuitionBackprop intuitioneach output wants a nudge, passed backDropout and batch normDropout and batch normdiffer in train mode vs eval modeBatchNorm vs LayerNormstats over batch vs over one exampleWeight initializationWeight initializationXavier/He keep activation variance steadyLearning rate schedulesLearning rate schedulesWarm up, then cosine-decay the step size
This page on the Machine Learning mind map · 62 pages in 10 branches · open it to see the whole map and test yourself in recall mode.