← How LLMs Work, for PMs
Foundational

Multilingual AI Products: Token Costs by Language, Evaluation Beyond English and Code-Switching

Language changes an AI product's cost, quality and speech accuracy. Older tokenizers needed several times more tokens for Indian languages than for English; newer ones narrow the gap. Benchmarks built from translated English questions miss cultural knowledge and code-switching. Learn to measure tokens and quality per language, and the public infrastructure India and the EU are building.

CONCEPT / HOW LLMS WORK, FOR PMS

TL;DR: A multilingual AI product is several products sharing a model. Cost and latency vary by language because tokenizers split some scripts into far more tokens: on GPT-4's tokenizer, the same text took about 4.8 times as many tokens in Hindi and 7.7 times in Tamil as in English, per the appendix of Petrov and colleagues' paper. Newer tokenizers narrow the gap, so measure your own. Quality varies too, and benchmarks translated from English hide it, because they miss local knowledge and the code-switched speech people actually use. Speech recognition is weakest where training audio is scarce. Budget, evaluate and staff per language.

The token premium

A tokenizer is trained mostly on text from some languages, so it learns efficient pieces for those and falls back to small fragments for others. Petrov and colleagues (NeurIPS 2023) measured the same sentences translated into many languages and found tokenized lengths differing "up to 15 times". For GPT-4's tokenizer, Italian needed about 1.6 times the tokens of English and Arabic about 3 times; the paper's appendix gives Indian languages far higher multiples.

DIAGRAM
Tokens for the same text, relative to English English1.0 Italian1.6 Arabic3.0 Hindi4.8 Marathi5.1 Bengali5.8 Tamil7.7 Telugu8.3 Odia12.5 GPT-4 tokenizer (cl100k), FLORES-200 sentences. Petrov et al., NeurIPS 2023; Indian languages from the paper's appendix table. Newer tokenizers reduce these multiples.
Read diagram description

Tokens needed for the same text relative to English on GPT-4's cl100k tokenizer, from Petrov and colleagues, 2023: English 1, Italian about 1.6, Arabic about 3.0, Hindi 4.8, Marathi 5.1, Bengali 5.8, Tamil 7.7, Telugu 8.3 and Odia 12.5.

Newer tokenizers narrow the gap. OpenAI's GPT-4o launch post showed one sample sentence needing 2.9 times fewer tokens in Hindi and 3.3 times fewer in Tamil than before, a company illustration on a single sentence rather than a benchmark. Sarvam AI reported that existing multilingual models needed 4 to 8 tokens per word for Indic scripts against 1.4 for English, and that its own tokenizer needed 1.4 to 2.1 (company-reported).

Why a PM cares. Tokens are the unit of price, latency and context. Illustratively: if an English support conversation uses 2,000 tokens, the same conversation at a 4.8 times premium uses 9,600, costing nearly five times as much, generating more slowly and filling the context window sooner. The premium your users pay depends on your model's tokenizer, so measure it: log tokens per message by language on real traffic, and price plans and rate limits with that number, as tokens and context windows describes.

Quality beyond English

Benchmarks report quality gaps that average scores hide.

  • IndicGenBench (ACL 2024), 29 Indian languages: "a significant performance gap in all languages compared to English".
  • MILU, 11 Indian languages and over 42 models: the best, GPT-4o, averaged 74%, and models did worse in culturally specific areas such as arts, law and governance than in STEM.
  • Global MMLU, 42 languages: 28% of the widely used MMLU questions need culturally sensitive knowledge, and 84.9% of its geography questions concern North America or Europe. Translating such a test does not make it local.

Code-switching is a gap of its own. Many Indian users write and speak Hinglish, Tanglish or other mixes. A 2023 EMNLP study found multilingual LLMs underperformed much smaller models fine-tuned on code-switched text, concluding that multilingualism "does not inherently imply proficiency with code-switching texts".

What to do. Build a golden set per language from real user messages, including code-switched ones and local references, labelled by native speakers. Report pass rates per language, not one average, and set a minimum bar for each language you launch. Decide per language whether to answer natively or translate through English; translate through English or answer natively works through that trade.

Speech

Speech recognition accuracy tracks training data. The Whisper paper found word error rate halving for every 16 times more training audio in a language, and named Telugu among the languages doing worse than expected for their data, possibly because the tokenizer suited them poorly. For a voice product, evaluate the recogniser per language and accent on your own call audio, and consider a chained design that lets you pick the best recogniser per language, as in realtime voice agents.

Public infrastructure

Governments are funding language coverage directly. India's BHASHINI platform reported in March 2026 that it powers more than 500 government websites, handles over 15 million inferences a day and covers 36 text and 23 voice languages. Under the IndiaAI Mission, Sarvam AI was selected in April 2025 to build an open-source 120-billion-parameter model, alongside other selections including Soket AI, Gnani AI and Gan AI. In Europe, the OpenEuroLLM project, announced in February 2025 with a budget of €37.4 million, is building open-source models covering all official EU languages. For a PM these are sourcing options, especially for government work and data that must stay in-country.

In the interview

Practise prompts such as "launch our assistant in Hindi, Tamil and Bengali", "why is our Tamil support bot so expensive?" or "design a voice assistant for rural users". A weak answer translates the English product. A Hire answer evaluates in each language. A Strong Hire answer measures the token premium per language and prices for it, builds native golden sets with code-switched and local cases, reports quality per language with a launch bar each, chooses native or translate-through per language, evaluates speech per accent, and considers local models and public infrastructure where they fit.

Next: tokens and context windows and golden sets.

RELATED CONCEPTS
PRACTISE THIS CONCEPT