Multilingual TTS Model Accuracy Across Non-English Languages
Standard benchmarks mask real problems with pitch, code-switching, and complex text.

- Written by
- The Voice Layer Editors
- Updated
- November 6, 2026
- Reading time
- 9 min read
- Sources cited
- 6 sources ↓
The way people measure TTS accuracy in different languages, and why those metrics stay imperfect
Multilingual text-to-speech still leaves coverage gaps that sheer size alone doesn't fix. Models trained across many languages can be fluent, even eerily lifelike, for Mandarin, English, or Japanese, yet stumble over Thai pauses, a misplaced tonal contour, or a Hindi sentence where an English noun drops mid-clause. Moving vocabulary between languages was never where the real trouble lay. Sound sets vary, speech patterns vary, writing rules vary, and a system shaped by one language’s timing brings it into other languages, where it feels out of place.
A pair of training philosophies dominate this space, and neither is free. Per-language models specialize and deliver solid accuracy, but adding any language forces extra training rounds and more compute and data, and the expense grows with coverage rather than shrinking against it. Multilingual systems take the opposite path: one setup, common settings, lower costs for wider use, but each language’s small style patterns get mixed together. Both methods still struggle to balance breadth with detail, and whoever says they don't is pushing something.
Prosody is the core reason this remains difficult, and it's not cosmetic. Stress placement, rhythm, the shape of intonation, even the basic notion of a "stressed syllable," vary by language in ways that don't map cleanly onto each other. A system that generates English-shaped contour speech and slips in another tongue's vocabulary yields output that's still intelligible yet melodically off, like a traveler reciting phrases memorized off a guidebook. In Mandarin, a bad pitch contour alters the word's sense outright, not just how it sounds. One that comes across as foreign is just a nuisance. When a system says something the speaker never meant, that's an entirely separate problem, yet researchers still score both mistakes alike.
Word Error Rate serves as a workhorse metric across speech research: feed synthesized audio to ASR, check substitutions, deletions, and insertions versus the right text, then give it as a percent. Computing it costs little, contrasting systems is straightforward, and that practicality, not any inherent superiority, is why it wins out. Error Rate works at a finer level, which helps with logographic scripts such as Chinese, where picking out word boundaries is still an open modeling question. Speaker similarity, typically cosine similarity between embeddings of speakers, tells if a cloned voice remains itself when it crosses into a new language. Mean Opinion Score brings in the human side, with listeners scoring naturalness, but running it costs a lot and the numbers shift between languages and cultures in ways experts can't lock down.
More benchmark tests and metrics have appeared, but they don't necessarily fix the core blind spot. TTS Arena has an ELO leaderboard built from user choices, with a 1,339 score for Fish Speech V1.5 there. Fish Audio S2 treats Seed-TTS-Eval as a key benchmark for English and Chinese. MiniMax's Multilingual Testset covers 24 major tongues, testing speech generation and voice mimicry from samples. Another company's test data covers 9 language groups and evaluates error rates in real-world audio, not studio audio. MINT-Bench tests TTS that follows prompts in multiple languages and shows that tone and structure, not just smooth speech, divide a great voice system from an average one. MINT-Bench sees them as a key industry-wide bottleneck.
People still run standbys such as Common Voice and FLEURS, but they depend on scripted speech or varieties of standardized language, plus their data splits shift across papers, which hurts how closely they match real-world situations. EmergentTTS-Eval targets that issue, with challenging TTS scenarios with more complex language than the simple, brief lines common in older tests.
What's broken is exactly that flatness. MINT-Bench says testing here is still rough: tests skip too much, they don't look close enough, and multilingual handling falls way short of the hype. Most benchmarks test simple short sentences and go no further. They never test how a system handles a contact number, a noun transliterated into a different script, a 40-word sentence packing multiple embedded clauses, or text switching languages partway through. A system may score highly on a clean benchmark, then break when people are typing actual sentences into it, while the industry's reliance upon WER for its main metric flatters the models instead of probing them.
Where multilingual TTS still falls short: pitch patterns, switching languages mid-sentence, and hard written material
Tonal languages put the most phonological pressure on TTS, since their pitch functions as grammar, not decoration. One comparative evaluation covering English, German, Spanish, Japanese, French, and Mandarin produced intelligible speech in all but one of those languages. Mandarin stood apart: mostly intelligible, yet marked by pronunciation issues the others didn't display. That single case shows where industry's blind spot lies, and it’s no coincidence that the tonal language stands out.
Thai puts its own twist on this same issue. Thai’s tonal system calls for careful modeling, and because Thai script doesn't show word boundaries explicitly, segmentation and prediction get harder before the system handles the tone. Current Thai TTS models often struggle with rhythm and intonation, and the standard answer of adding more examples hits a dead end when good Thai collections remain hard to find. More data can't fix this if there's none to be had, and acting like it can only puts off the reckoning.
Recording shortages alone don't account for it structurally. Some tonal languages drop lexical tone because the discrete inventories TTS systems use to produce speech rely on a framework that was never built to store tone. This representational gap appears to extend beyond any single tonal language, though its precise causes remain under investigation. The gap may stem from how systems encode pitch, affecting performance across tonal languages. Gathering extra Mandarin audio won't fix a codebook never built to encode pitch.
Code-switching, where one utterance pulls in a pair of languages, is arguably the toughest of the three challenges and the furthest from cracked. Take a sentence like "मुझे one laptop चाहिए," Hindi and English blended mid-sentence the way bilingual speakers actually talk. Getting that right takes word-level language or token-level language detection, distinct phoneme inventories for every language used, plus prosody transitions that never lurch or seem robotic at the boundary. One paper (arXiv:2609.11545) on Thai, Vietnamese, Indonesian, and Swahili found systematic failures on complex text, even where standard sentences work smoothly.
The same paper stress-tests complex text directly, rather than relying on tidy clean benchmark sentences, making it a rare case that bothers. They ran OmniVoice and VoxCPM2 alongside MMS-TTS in those multilingual settings, getting the same result each time: hard stuff like figures, calendar entries, proper names, big blocks of text, or weird marks shows problems that standard checks miss. Every model failed in its own way: failures on normalization, named entities, or long-text output. No single villain stands out, and that's the point. A standard benchmark's clean WER score says almost nothing about how a system handles messy text in real use.
The same pattern appears in complex text, tones, and code-switching. Models build their main representations from the language that dominates the training data, in most cases a high-resource option such as English and Mandarin, then transfer those habits to languages whose structure is unlike it or that show up less in training. The resulting problems slip past standard evaluation, which was never built to catch them.
What today's top multilingual TTS models support and how they're built
OmniVoice has the broadest coverage, trained using 581,000 hours worth of open-source multilingual audio across over 600 languages, a setup that dwarfs most other options. Architecturally, it's a diffusion LLM-style discrete non-autoregressive model that maps text directly to multi-codebook acoustic tokens, skipping the two-stage cascaded pipeline older TTS systems relied on. Qwen3-0.6B weights are where its backbone initializes, while models trained lacking language-model initialization like that consistently have worse WER. OmniVoice was tested on multilingual benchmarks covering Chinese and English, plus up to 102 languages, for intelligibility, naturalness, and speaker similarity, while inference hits a real-time ratio of 0.025, about 40 times the pace of playback in real-time.
But breadth carries a hidden price. The same research on Thai, Vietnamese, Indonesian, and Swahili (2609.11545) found OmniVoice consistently failing on difficult material in those very tongues. However, covering 600 languages does not guarantee consistent performance across all of them. Breadth alone is a figure on a spreadsheet, not real coverage.
Fish Speech V1.5 goes the other way: narrower and more focused. It runs on a DualAR architecture, two autoregressive transformers, trained on more than 300,000 hours in English and Chinese audio plus more than 100,000 hours in Japanese. Its payoff is clear in the results: TTS Arena ELO of 1,339, English WER of 3.5% with CER of 1.2%, Chinese CER of 1.3%. It handles 13 languages, but its training data focuses on English and Chinese, with additional Japanese data, and the accuracy results reflect that focus.
Fish Audio S2 carries that foundation forward and goes past it. On MiniMax's 24-language Multilingual Testset, it hits the smallest WER in 11 out of the 24 languages and tops speaker similarity for 17. It records the lowest error rate across every language in CV3-Eval, moving the error to 3.01 from 3.96 versus predecessor Fish Audio S1 for a 23.9% relative drop. It does best consistently across Chinese, Japanese, Korean, English, and the main European languages, but MiniMax-Speech plus ElevenLabs still win on some low-resource languages, generally ones with a relatively small amount of training data. The Fish Audio S2 Pro variant comes first in EmergentTTS-Eval, where it leads evaluations and It also achieves a Chinese WER of 0.54% and English WER of 0.99% on Seed-TTS Eval, with cross-lingual voice cloning across over 80 languages.
CosyVoice2-0.5B picks a different path, optimizing to handle real-time playback and dialect coverage over sheer breadth. It handles Chinese, covering Cantonese, Sichuan, Tianjin, Shanghainese, and Wuhan dialects; English, Japanese, and Korean; plus cross-lingual scenarios, at latency fit for live audio. LibriTTS test-clean gives it a WER of 2.89% (±0.18), with speaker similarity of 74.30, which its evaluators call human-level for English. Pronunciation error rate falls 30% to 50% compared with its predecessor CosyVoice v1.0, and MOS goes from 5.4 up to 5.53. Few multilingual benchmarks measure local speech differences in a single language, while CosyVoice2's focus fills a gap the larger models leave.
Head-to-head WER by language: what the Qwen3.5-Omni benchmark comparison reveals about which systems lead where
For the best multi-system comparison, the Qwen3.5-Omni technical writeup evaluated custom-voice speech across languages using data gathered from APIs during March 2026. The systems were Qwen3.5-Omni-Plus, ElevenLabs Multilingual v2, Gemini-2.5 Pro-Preview-TTS (voice Achernar), MiniMax-Speech-2.8-HD, and GPT-Audio-2025-08-28 using voice Alloy.
Chinese showed the widest gap of any language, and it wasn't even close. Qwen3.5-Omni-Plus landed at 0.785 and MiniMax at 0.786 WER, practically even, while GPT-Audio trailed with 0.829. Gemini-2.5 Pro trailed with 1.890, while ElevenLabs was furthest behind on 3.801. From 0.785 up to 3.801, that gap is the widest spread across any language tested here, and it says something clear about training data: when tone and character-level pronunciation matter, systems not built around deep corpora in a common tongue struggle. Mandarin isn't a language where broad multilingual competence carries over by itself. It takes serious commitment, period, and these WER results prove that point more strongly than any marketing ever could.
English narrowed the gap significantly, as Qwen3.5-Omni-Plus achieved 0.839, near Gemini-2.5 Pro's score in this comparison. The tighter clustering shows this: for most of these systems, English has the most training data of any language, so gaps visible in other languages shrink here to almost nothing. A single score means less than how much each system's spot changes from one language to the next. A model that scores high in English can rank far lower in Chinese; that instability is the main takeaway for someone selecting a platform for one deployment.
Coverage numbers only go so far, and vendors are well aware. Marketing pays for Coverage claims, but WER results by language end up in an appendix. That detailed table is where the real choice happens, not the big language number.
Methodology & sources
- Ultimate Guide - The Best Open Source Models for Multilingual Speech Recognition in 2026 - SiliconFlow
Provided Fish Speech V1.5 benchmark figures including TTS Arena ELO score, WER, CER, and training data hours.
- Chatterbox Multilingual v3: TTS with embedded watermarking for 21 languages and 4 dialects
Referenced in relation to CosyVoice2-0.5B coverage details.
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
Described EmergentTTS-Eval as targeting complex, realistic TTS scenarios beyond simple short sentences.
- OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
Provided details on OmniVoice's architecture, training data scale, language coverage, and inference speed.
- https://arxiv.org/pdf/2609.11545
Provided findings on systematic failures of OmniVoice and other models on complex text in Thai, Vietnamese, Indonesian, and Swahili.
- https://arxiv.org/html/2604.15804v2
Provided the Qwen3.5-Omni multi-system WER comparison data across Chinese, English, and other languages.