One Vocabulary, Four Languages: A BPE Tokenizer From Scratch
Commercial tokenizers quietly charge some languages more than others. I wanted to measure that myself. Then I wanted to see how far from-scratch engineering can push against it. So, an experiment. Take the Wikipedia India page in English, Hindi, Telugu, and one more language. Build a BPE tokenizer from scratch with a single 10,000-token vocabulary shared across all four. Optimize one number: 1000 / (X_max − X_min), where each X is fertility, the tokens a language needs per word. The metric only rewards balance. A tokenizer that’s great at English and bad at Telugu scores worse than one that’s mediocre at both. ...