https://therahulbhati.github.io/tags/multilingual/ Recent content in Multilingual on Rahul Bhati Hugo -- 0.151.0 en-us Fri, 10 Jul 2026 21:30:00 +0530 https://therahulbhati.github.io/posts/multilingual-bpe-tokenizer/ Fri, 10 Jul 2026 21:30:00 +0530 https://therahulbhati.github.io/posts/multilingual-bpe-tokenizer/ <p>Commercial tokenizers quietly charge some languages more than others. I wanted to measure that myself. Then I wanted to see how far from-scratch engineering can push against it. So, an experiment. Take the Wikipedia <strong>India</strong> page in English, Hindi, Telugu, and one more language. Build a BPE tokenizer from scratch with a single 10,000-token vocabulary shared across all four. Optimize one number: <code>1000 / (X_max − X_min)</code>, where each <code>X</code> is fertility, the tokens a language needs per word. The metric only rewards <em>balance</em>. A tokenizer that&rsquo;s great at English and bad at Telugu scores worse than one that&rsquo;s mediocre at both.</p>