Multilingual BPE Tokenizer
One shared 10,000-token vocabulary over the Wikipedia India page in English, Hindi, Telugu, and Kannada, evaluated on the page's HTML → Markdown rendition (links preserved). Every number on this page is computed live in your browser by running the shipped tokenizer over the shipped page texts. Nothing is hardcoded.
Fertility ratios & token statistics
| language | page | words | tokens | X = tokens / words | ≤ 1.2 |
|---|
How the numbers are computed
The eval text is each India page's HTML converted
to Markdown with links preserved (html2text), NFC-normalized, whitespace
canonicalized to single spaces, and no other cleaning: numbers, punctuation,
and link syntax stay attached to words, and case is preserved.
Words = whitespace-delimited chunks of that markdown.
Tokens = BPE encoding of each word (alphabet of
characters + 256 byte-fallback tokens +
ordered merges, applied greedily by rank,
standard Sennrich BPE). There is no <unk>: characters
outside the alphabet encode as UTF-8 byte tokens and decode losslessly.
The tokenizer was built from these same four pages. Train = eval
is deliberate: it measures the balance ceiling, not generalization (the
write-up covers the held-out cost).
Because "word" is a convention, the same tokenizer is also reported
under two others: counting words as \w+ regex matches on the
markdown gives ; on the plain-text
extract (no markup) with a whitespace split it gives
.
Download the tokenizer
tokens.txt lists every token as
id → token. combined.json is the full tokenizer
(base alphabet and ordered merges), enough to reproduce every number on
this page. Direct file: tokenizer.json.
Try it
Source: India on en/hi/te/kn.wikipedia.org, rendered HTML → Markdown via html2text. Tokenizer, pipeline, and a held-out generalization analysis are in the write-up.