AI Grounds
Open AI Grounds on a desktop
These interactive lessons need a larger screen. Please continue on a desktop or laptop computer.
AI Grounds
These interactive lessons need a larger screen. Please continue on a desktop or laptop computer.
Guided discovery
Change a feature. Follow its row. Keep the axes straight.
Each row represents one token—a position in this toy sequence. Its features are illustrative numeric signals, not generated word embeddings. LayerNorm here uses all four features in the selected row.
| Token | Feature 1 | Feature 2 | Feature 3 | Feature 4 |
|---|---|---|---|---|
| The | 0.20 | -0.80 | 0.40 | 0.10 |
| cat · selected | -1.00 | -1.00 | 1.00 | 1.00 |
| sat | 0.30 | 0.50 | -0.70 | -0.10 |
Edit cat, feature 1, only. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.
Gamma multiplies this feature’s z in all tokens. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.
Beta adds to this feature’s output in all tokens. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.
Make row constant sets the selected row’s four inputs to zero. Scale and Shift are per-feature parameters shared across all three tokens; you edit them manually here.
Fixed numeric axes −5 to 5 keep spread changes visible. Each numbered lane follows one feature; the outlined dot marks your selection. x and y are numeric activations; z is dimensionless. Exact values are below.
z = (-1.000000 − (0.000000)) / √(1.000000 + 0.00001) = -0.999995
y = 1.20 × (-0.999995) + (0.10) = -1.099994
Substitutions are rounded; calculations use full precision. Epsilon (0.00001) is a positive term protecting the denominator. z’s sign follows x−mean; Scale and Shift act afterward.
Average of this token’s four raw features.
0.0000
mean = Σx / 4
Normalized mean 0.000000; output mean 0.075000.
Average squared distance from the raw row mean.
1.0000
variance = Σ(x−mean)² / 4
Raw std √variance = 1.000000. Denominator √(variance+epsilon) = 1.000005.
Population standard deviation around y’s actual mean.
1.1344
yᵢ = Scaleᵢ × zᵢ + Shiftᵢ
var(z) = variance/(variance+epsilon) = 0.999990. Per-feature parameters need not preserve zero mean or unit spread.
| Feature | Input x | Normalized z | Output y |
|---|---|---|---|
| 1 · selected | -1.000000 | -0.999995 | -1.099994 |
| 2 | -1.000000 | -0.999995 | -0.999996 |
| 3 | 1.000000 | 0.999995 | 0.999995 |
| 4 | 1.000000 | 0.999995 | 1.399995 |
Declare the layout first. LayerNorm input [1, 3, 4] has one sequence, three token positions, four features; normalized_shape=4 uses four features per token. For BatchNorm1d, transpose the same data to [N=1, C=4, L=3]: one sequence, four feature channels, three positions. Its training forward calculation uses each channel across those three positions, as this column shows. The positions are not three independent batch examples.
| Token | F1 | F2 | F3 | F4 | Column z |
|---|---|---|---|---|---|
| The | 0.20 | -0.80 | 0.40 | 0.10 | 0.620757 |
| cat | -1.00 | -1.00 | 1.00 | 1.00 | -1.410811 |
| sat | 0.30 | 0.50 | -0.70 | -0.10 | 0.790054 |
Selected column mean -0.166667, population variance 0.348889; z=(x−column mean)/√(column variance+0.00001). This comparison omits affine parameters to isolate axes. LayerNorm selected row mean 0.000000, variance 1.000000.
Population variance divides by the number of contributing values. LayerNorm normalizes the declared trailing dimensions; this demo uses four features. It computes statistics from the current input in both training and evaluation. Per-feature gamma and beta are shared across leading token/sequence dimensions. The lab permits positive scales 0.4–1.8; real parameters can also be zero or negative.
The three words label illustrative rows; features have no assigned linguistic meaning. No optimizer, running-statistic update or training-quality comparison runs. The BatchNorm example describes the training forward grouping with explicit N,C,L axes; evaluation behavior can use saved running statistics.
cat, feature 1. Row mean 0.000000, variance 1.000000, denominator 1.000005. Selected x -1.000000, z -0.999995, y -1.099994. Scale 1.20, Shift 0.10 are shared with this feature in other tokens.