Blog

Same Meaning, Different Price: Measuring the Cost of Uzbek in LLM

Language models charge the same price per token for Uzbek text as for English, but they count more tokens for it. We measured what this costs across 15 tokenizers, from the largest providers to open models, examined what drives the difference, and calculated what the new alphabet will change.
Key fingings
  1. Uzbek text in Latin script takes 1,55–2,17 times as many tokens as the same text in English with all five providers: OpenAI, Anthropic, Google, Qwen and DeepSeek. Texts originally written in Uzbek fall closer to the lower end of the range, and translations from English closer to the upper end
  2. Compared with Russian, Uzbek costs 1,24–1,54 times more, depending on the model. Genre has little effect on this ratio: for each model it varies by no more than 8%
  3. With most providers, Uzbek text in Latin script uses roughly 6–19% fewer tokens than in Cyrillic, while with Claude both scripts cost the same
  4. Responses become more expensive along with prompts: a visible response in Uzbek costs 1.7–1.8 times more than in English. Claude Haiku 5.5, with default settings, used hidden reasoning, and with it the billed output cost 2.1–2.3 times more
  5. Claude Opus 5.5 and GPT-5.6 Sol have the same list price, yet processing the same Uzbek text costs 1.7 times more on Claude because its tokenizer splits the text into more tokens. Models should therefore be compared by the total cost of your own text, not by the price per token
  6. Switching to the new alphabet will barely affect token counts with the five largest providers: the change stays within 4% either way, so the transition will not raise AI costs
  7. The premium for Uzbek depends not on how many people speak it, but on how much Uzbek text the models have seen. That makes open Uzbek datasets a form of digital infrastructure: the more of them there are, the cheaper, and likely better, AI services in Uzbek become for everyone, from startups to government agencies
  8. The new alphabet needs data in place ahead of time. Until there is enough text in the new spelling, models will handle it less well than the old one. Parallel corpora in both spellings, open conversion tools and shared typing standards are best ready before April 2027, not after.
  9. No single organization can do this alone. Building corpora, morphological dictionaries, tokenizers and quality benchmarks for Uzbek will take companies, universities and government working together, in dialogue with model developers
How we measured, which texts and models we used, and the results for each model are described in detail in the full report.

Why the Same Meaning Costs Different Amounts

Language models read and write text not in words but in tokens, the fragments into which a tokenizer splits it. Providers charge for each token at the same rate for every language, but the number of tokens needed for the same meaning varies from language to language. An English word usually takes a single token, because English makes up the largest share of the data the models were trained on. A word in a language that appeared there less often breaks into several fragments, and it costs more.
Researchers noticed that languages are split unevenly as early as 2019–2021, but at the time it was treated as a question of quality. It became a question of money in 2023, when language models turned into paid services billed by the token. That is when the first names for it appeared: first tokenization premium, and later token tax. In this report, we call it the language premium.

Where the Premium Comes From

The gap between languages comes from three groups of causes, and they combine differently for each language.
The first concerns data. A tokenizer’s vocabulary is built from the texts it was trained on, and words and word parts that appeared there often become whole tokens. What matters is not how many people speak a language, but how much of it is written online. Hindi, Bengali and Urdu are among the most widely spoken languages in the world, yet the tokenizers of major models use 3–4 times as many tokens for them as for English. And Chinese and Burmese, both written in non-Latin scripts, cost the GPT-4 tokenizer 1.9 and 11.7 times as much as English in 2023: there is plenty of Chinese text in the data and almost no Burmese.
The second concerns how the tokenizer is built: which algorithm it uses, how many slots its vocabulary has, and how it splits text before training. When researchers trained tokenizers for 11 Southeast Asian languages on the same data, changing the algorithm alone narrowed the gap between languages by about a third. Providers’ decisions have a similar effect: when OpenAI moved from the GPT-4 tokenizer to the GPT-4o tokenizer, the number of Cyrillic tokens in its vocabulary grew almost twentyfold, and Ukrainian text became 41% cheaper.
The third concerns the properties of the language itself: long words with chains of affixes, the script, and different ways of writing the same letters. In Swahili and Zulu, where words are built from chains of affixes, models use 2.6–2.8 tokens per word. Script matters too: when texts in Turkic languages are converted from Cyrillic and Arabic script into a common Latin alphabet, the number of tokens drops by 22–41%. But the advantage lies not in Latin script as such, but in whichever script is more common online: Ukrainian written in Latin script, by contrast, becomes 2–19% more expensive. These features make a language expensive only when the model’s vocabulary knows it poorly: Turkish also builds words from chains of affixes, yet it costs models noticeably less than Uzbek.

Uzbek Through the Eyes of a Tokenizer

Uzbek is spoken by about 40 million people, making it the second most widely spoken Turkic language after Turkish. But for language models, it is awkward in several ways at once.
Let’s start with how Uzbek words are built. Uzbek is agglutinative: a word is assembled from a root and a chain of affixes, each adding one meaning. For example, kitoblarimizdan (“from our books”) consists of the root kitob (“book”) and affixes for plural, possession and direction, and this single word carries the meaning of three English ones. If a model’s vocabulary does not know these parts, the tokenizer splits the word by frequency rather than meaning. In a study at Urgench State University, one of the tokenizers split the verb kulimsira (“to smile”) into ku, lim, si and ra, leaving nothing of the root kul (“to laugh”).
Then the alphabet comes into play, and there are two of them. From 1940, Uzbek was written in Cyrillic; in 1993 the country officially switched to Latin script, but Cyrillic is still widely used. This matters for models: in our measurement, the same text in Latin script took roughly 6–19% fewer tokens than in Cyrillic with every provider except Anthropic. And the Latin alphabet itself has the letters oʻ and gʻ with an apostrophe that is written with four different characters, each of which is a separate symbol to a tokenizer. Standard tokenizers also treat the apostrophe as punctuation and break the word on it, so oʻzbek turns into “o”, an apostrophe and “zbek”. In our calculations, the choice of character alone changed the token count by up to 5%.
Still, grammar and the alphabet do not explain the gap. Turkish is just as agglutinative and is also written in Latin script, yet with GPT-4o the same 30 sentences took 609 tokens in Turkish and 763 in Uzbek, a quarter more. The difference comes from the vocabulary: models whose vocabularies explicitly include Uzbek use almost as many tokens for Uzbek text as for English.

What the Alphabet Reform Will Change

On 21 October 2026, the president signed the law on the new Uzbek alphabet, and from 22 April 2027 oʻ, gʻ, sh and ch will be replaced by the letters ö, ğ, ş and ç (Law of the Republic of Uzbekistan No. O‘RQ-1184 of 21 October 2026). One of the stated goals of the reform is to develop the language in the digital environment, so it is natural to ask whether this will make Uzbek cheaper for AI.
We tested this by replacing the letters in all our texts. Text in the new spelling does become shorter, by 4% in characters, but this barely carries over into tokens. With the largest providers, the number of tokens changes by no more than 4%: it falls slightly with Claude, Gemini and Qwen, and rises slightly with OpenAI and DeepSeek models.
The reform also has a welcome side effect: along with oʻ and gʻ, the confusion between four variants of the apostrophe will disappear, which currently costs up to 5% in extra tokens. In the longer term, much depends on how quickly text in the new spelling builds up online. The experience of Kazakhstan, which is also moving to Latin script, shows that a change of script can at first even make working with models more expensive, until tokenizers are retrained on new texts. That is why preparing open data in the new alphabet early matters.

Why Averages Aren’t Enough

A shortage of text in a language’s training data explains only part of the gap. The rest is shaped by technical decisions: the tokenizers of the largest models were built on text dominated by English and around its structure. That is why conclusions drawn for one language cannot simply be carried over to another. Each language has to be studied on its own, with all its particular features, from how its words are built to which character is used for the apostrophe.
How a language is used in practice matters just as much. Uzbek is written in both Latin and Cyrillic script, and the apostrophe in oʻ and gʻ is typed however people happen to type it, which alone shifts the token count by several percent. The tasks a company uses a model for change the picture even more. Two models with the same list price can differ by a factor of 1.7 in what they charge for the same text, a system prompt in English takes one and a half to two times fewer tokens than in Uzbek, and a model that decides on its own to reason before answering can make an Uzbek response even more expensive. General figures from studies are useful as a benchmark, but real costs are best calculated on your own texts and your own tasks.

Related articles

Articles