One unmasked toy attention head. Q, K and V are hand-authored: no model training or learned word meanings. Every modeled token can receive weight, including the query itself and later positions.

Your modeled tokens

Original context sentence: The duck sat by the river bank

Six simplified modeled tokens: The · duck · sat · by · river · bank. The second the is omitted from this toy; this is not a complete tokenizer.

Query token

Attention sharpness

Multiply scaled scores before softmax. Higher means less uniform here; it preserves their rank. Equivalent temperature = 1 / sharpness. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.

Weights from query bank

Q = [0.95000, 0.08000, 0.18000]
scoreᵢ = Q · Kᵢ / √3 · weightᵢ = softmax(1.9 × score)ᵢ

Softmax normalizes exponentiated scores into positive weights that add to 1. Each lane uses the same 0–100% scale, with no minimum bar width.

The11.28%

duck16.86%

sat12.44%

by14.57%

river26.29%

bank · Q18.56%

bank compares with every key. All 6 values contribute to the weighted output.

Largest weight

Highest-weight token: river.

26.29%

All attention weights sum to 100%

Largest is below half. It is not a majority or a causal-importance proof.

Entropy

Spread of the weight distribution, in nats (natural-log units).

1.75002

H = −Σ weight × ln(weight)

Maximum at uniform weights: ln(6) = 1.79176 nats. Relative spread H/ln(6) = 0.97670.

Mixed value output

output = Σ weightᵢ × Vᵢ. Each feature has arbitrary toy units; these three numbers are not class probabilities and need not sum to 1. Values here lie on a common 0–1 scale.

Shore feature0.47781

Finance feature0.04336

Glue feature0.31161

Weighted value contributions

Multiply each V by its attention weight, then add all six rows. Keys choose weights; values supply content. Products are rounded for display; sums use full precision.

Value coordinates in order: Shore, Finance, Glue. All six tokens contribute.
Token	Weight	Value V	Weight × V
The	11.28%	[0.04000, 0.02000, 0.74000]	[0.00451, 0.00226, 0.08346]
duck	16.86%	[0.38000, 0.02000, 0.12000]	[0.06406, 0.00337, 0.02023]
sat	12.44%	[0.10000, 0.02000, 0.52000]	[0.01244, 0.00249, 0.06470]
by	14.57%	[0.18000, 0.02000, 0.62000]	[0.02622, 0.00291, 0.09032]
river	26.29%	[1.00000, 0.01000, 0.06000]	[0.26292, 0.00263, 0.01577]
bank	18.56%	[0.58000, 0.16000, 0.20000]	[0.10765, 0.02970, 0.03712]
Sum	100%	—	[0.47781, 0.04336, 0.31161]

Q, K and V arithmetic

Q uses the selected token’s query vector. Multiply matching Q/K coordinates and add, then divide by √3. Softmax uses exp(focused score − maximum) divided by the sum of these shifted exponentials; subtracting the same maximum is numerically stable and preserves weights.

All source vectors and score stages; values are shown in the contribution table.
Token	Stored Q	Key K	Q(selected) · K	Score / √3	Sharpness × score	Weight
The	[0.05000, 0.03000, 0.92000]	[0.04000, 0.02000, 0.90000]	0.20160	0.11639	0.22115	11.28%
duck	[0.52000, 0.04000, 0.22000]	[0.55000, 0.03000, 0.24000]	0.56810	0.32799	0.62319	16.86%
sat	[0.16000, 0.02000, 0.72000]	[0.16000, 0.03000, 0.76000]	0.29120	0.16812	0.31944	12.44%
by	[0.30000, 0.02000, 0.80000]	[0.30000, 0.03000, 0.82000]	0.43500	0.25115	0.47718	14.57%
river	[0.98000, 0.02000, 0.16000]	[1.00000, 0.02000, 0.12000]	0.97320	0.56188	1.06757	26.29%
bank	[0.95000, 0.08000, 0.18000]	[0.62000, 0.25000, 0.26000]	0.65580	0.37863	0.71939	18.56%

Compare the two fixtures

Both rows select bank at the current sharpness. This switch changes bank’s Q and the fixture’s K/V arrays together. It cannot isolate a context-only effect or prove learned language understanding.

Fixture	bank Q	Sharpness	Highest-weight token	Weight	Mixed output
River Bank	[0.95000, 0.08000, 0.18000]	1.9	river	26.29%	[0.47781, 0.04336, 0.31161]
Money Bank	[0.08000, 0.95000, 0.20000]	1.9	cash	27.16%	[0.05537, 0.45400, 0.40756]

This shows one attention-head output. Learned projections, multiple heads, output projection, residual connections, feedforward layers and causal masks are outside this lab. Attention weights alone do not explain a full model’s decisions.

River Bank, query bank, sharpness 1.9. Largest weight 26.29% for river. Entropy 1.75002 nats. Mixed values [0.47781, 0.04336, 0.31161].

Open AI Grounds on a desktop

Transformer Attention