A token is one symbol or learned piece. This is character-based, within-word BPE. The fixed alphabet has a–z, = and </w>: 28 symbols, covering every example. </w> is one word-end symbol. Whitespace separates words and is not encoded. This is not a byte-level or production tokenizer.

Your training text

Repeated stems and suffixes make the merge choices easy to see. Each repeated word occurrence contributes to frequency.

low lower lowest
new newer low

Merge steps

Select a prefix of the learned rules, from 0 through 8. Exact edits commit with Enter or on blur. Inspection text never trains new rules. Use arrow keys on the slider. Press Enter or leave the number field to apply an exact edit.

Highest-frequency next pairs

Counts of adjacent symbol pairs across current training words, including repeated occurrences.
Pair	Count	Frequency status
l + o	4	Tied maximum
o + w	4	Tied maximum

Ties use a fixed preset preference, then code-unit order as a fallback. Frequency alone does not pick a unique winner. These counts come from training, not the inspection words.

Last training replacement

No rule applied yet. Training words begin as characters plus one </w> each.

Inspection: lowest newer

Apply the same ordered rules to these words. Learned pieces have a pale fill; each outlined piece counts as one token.

lowest

lowest</w>

newer

newer</w>

Inspection tokens

Number of current inspection pieces, including word-end symbols.

7 + 6 = 13

At 0 merges: 13. 0% fewer tokens.

Vocabulary symbols

Fixed base symbols plus unique learned pieces; original symbols remain available.

28 + 0 = 28

Token count is not a measurement of bytes, memory use or semantic understanding.

Compare original examples

Apply the current rules without retraining. New whole words can still use known base characters; learned pieces need not have linguistic meaning.

All original comparison strings, with exact before/after token counts including end symbols.
Text	Current pieces	Tokens at 0 → now	Reduction
lowest newer	lowest</w>newer</w>	13 → 13	0%
glow tower	glow</w>tower</w>	11 → 11	0%
xyz	xyz</w>	4 → 4	0%

All pair frequencies and learned rules

Counts of adjacent symbol pairs across current training words, including repeated occurrences.
Pair	Count	Frequency status
l + o	4	Tied maximum
o + w	4	Tied maximum
w + </w>	3	Below maximum
w + e	3	Below maximum
e + r	2	Below maximum
e + w	2	Below maximum
n + e	2	Below maximum
r + </w>	2	Below maximum
e + s	1	Below maximum
s + t	1	Below maximum
t + </w>	1	Below maximum

Only already-applied rules appear below; future selections are not shown. Preset preferences break ties only among maximum-frequency pairs.

Step	Learned rule	Frequency before merge

Counts across merge prefixes

Both quantities use counts, with no independent rescaling. For these preset corpora every rule adds a distinct vocabulary symbol, but some inspection counts stay unchanged.

Merge prefix	Inspection tokens	Vocabulary symbols
0 · Current	13	28
1	12	29
2	11	30
3	10	31
4	9	32
5	8	33
6	8	34
7	7	35
8	7	36

The alphabet covers the listed lowercase examples and =; arbitrary Unicode or unknown characters are outside this lab. Fewer tokens do not guarantee a better model, smaller stored vocabulary or lower total memory cost.

Repetition, 0 merges. Inspection 13 tokens; vocabulary 28 symbols. No rule applied.

Open AI Grounds on a desktop

Byte Pair Encoding Lab