Multi-Language Switching Detection in Bilingual Voice Assistants
Researchers identify why voice assistants fail when speakers switch languages mid-sentence.

- Written by
- The Voice Layer Editors
- Published
- October 5, 2026
- Reading time
- 9 min read
- Sources cited
- 5 sources ↓
What this covers
Detecting when bilingual voice assistants switch between languages is still an unsolved engineering problem, and the reasons are structural. Two problems strike together: the monolingual acoustic model hits its phoneme limit while the language model loses its predictive context, precisely when the speaker switches languages.
Why code-switching breaks voice assistants at the phoneme boundary
Code-switching means a speaker moves among two or more languages inside one utterance, and for multilingual speakers it shows up all the time in real speech, not as a rare edge case. Someone might weave Hindi into English, or Spanish into English, in the middle of a sentence or even a thought, with no cue that software is built to expect. One review of code-switching-aware NLP covered five research tracks, more than 15 tasks, over 30 datasets, and over 80 languages, yet found large language models still usually falter on mixed-language input. That matters because it shows the breakdown is central to multilingual tech, not out at the margins.
The breakdown occurs at a precise, identifiable instant: the moment a speaker switches tongues, the sound stream jumps between distinct phoneme systems, while monolingual training leaves the model with no way to cross that divide. Before that frame, everything aligned with what the model had been trained to recognize. Beyond that point, the sounds may come from a phonetic repertoire entirely unfamiliar to the model.
Both systems falter together when the switch happens. The acoustic model encounters unfamiliar phonemes, then either marks them as unknown or offers guesses it trusts only weakly. At the same time, the language model’s expectations collapse: tuned for what should come next in Language A, it is confronted instead with evidence from Language B, a string it had treated as virtually impossible. A further snag deepens the mismatch: words that sound alike across languages but mean different things can steer the acoustic model to a mistaken reading before the language model can intervene.
Systems that rely on spotting the language first, routing segments to monolingual models, and reassembling the results fail to resolve this. Such architectures merely introduce fresh vulnerabilities. Each transition from identifying to directing to merging delays the output, while the segmentation frequently bisects a word at the precise boundary of the language change. That boundary is precisely where the acoustic signal grows most ambiguous. So the classifier performs worst exactly when its judgment matters most.
Seeing both breakdowns stack up, with the acoustics and the language model failing together at the switch point, gives you the test for evaluating a supplier’s claims about code-switching. Software may deal smoothly with one-off foreign terms yet break at true intra-sentential code-switches; that contrast is what matters.
Resource imbalance between languages and the failure at switching points
Switch points do not break down symmetrically. Because the model collapses much faster toward the language with fewer training examples, this asymmetry steers the breakdown.
If a pair has one language with much more data than the other, the system works well on that first language and has problems with the second even in normal speech that uses only one language. Language switching widens that gap instead of closing it, since the weakest language model must handle the toughest acoustic call right at the switch point.
This disparity worsens if one tongue features a speech sound entirely absent from the other. The acoustic model typically faces its least familiar speech sounds immediately following a code switch, precisely when the system can least afford such novelty.
Beyond uneven resources, accent creates another obstacle. In a second language, a speaker’s accent reflects the pull of the first language, producing pronunciations that fall outside the comfort range of both monolingual recognizers. Unlike the boundary collapse discussed earlier, this harms recognition even for correctly routed mid-switch audio, since the failure lies not in routing but in speech that lands between the two learned distributions.
Mistakes in a mixed-language sentence are not distributed uniformly. They pile up where the language changes, then linger in the side with the thinner training base. A bilingual system should therefore be judged by where its mistakes point, since strong Spanish-English switch handling can mask consistent, quantifiable breakdowns in pairings with scarcer data.
Scarce, narrow training data as the structural cause behind both failure modes
Studies on ASR for mixed-language utterances trace both failures to one limit: very little cross-lingual audio of alternating languages exists. Lacking authentic, well-proportioned audio of actual language alternation means that tweaking network designs alone cannot train a system to navigate these shifts properly.
As language coverage expands, scarcity compounds: a system designed for N languages must handle quadratically many pairings, making it rapidly impractical to gather authentic mixed-language utterances across them all. The literature instead clusters around a handful of pairings with ample resources, especially Chinese-English, leaving most bilingual combinations used in the real world largely untouched. Recent generalization studies find near-total gaps in public code-switching evaluation sets for pairings like Korean-Japanese and Korean-German.
The field relies on two main fixes, each introducing its own distortions. Text-to-speech tools stitch together single-language clips to create synthetic code-switching audio, yet the joins sound unnatural since no generative process captures how actual switching flows. Models built from such recordings end up memorizing the splicing artifacts rather than capturing how genuine switches actually sound. Tuning for a specific pair benefits only that combination, and creating individual models per pairing encounters the identical combinatorial explosion rendering complete data gathering unfeasible.
Even the biggest new datasets still contain this distortion. The 2025 SwitchLingua corpus draws its material from LinguaMaster outputs built to be both organized and artificial, alongside audio recorded by 174 bilingual people. In CS-FLEURS, some audio is read by people, but most recorded time is produced by speech synthesis systems. Training on such material may teach a model to key on splice-like acoustic traces while still earning strong synthetic-test scores, then fail when it encounters production audio captured from real conversation.
Fine-tuning one pair at a time only works for a few combos, and synthetic samples teach a bad acoustic habit, so which architecture could actually tackle the language change?
End-to-end multilingual architectures at the switch point
Multilingual end-to-end systems process mid-utterance switches all at once without a separate routing stage, cutting out the delay and segmentation mistakes cascade pipelines cause right when it hurts most.
In practice, single-pass architectures hold a unified representation of sound and language spanning both languages simultaneously, eliminating any need for routing decisions on thin evidence. Removing that identification stage also removes an extra failure point compounding the acoustic uncertainty already present at the boundary.
Fully integrated pipelines still face a genuine tradeoff between streaming and offline modes. Spotting a transition instantly means relying on brief audio segments, which leaves fewer acoustic clues to identify where languages shift and raises error rates beyond what offline methods produce. Offline, non-synchronous decoding consistently yields superior language-switch precision since the network reviews the entire recording prior to finalizing its output. Choosing streaming over offline operation represents a foundational architectural commitment for implementers, far exceeding a trivial setup choice.
A separate research direction explores whether code-switching skills learned on familiar language pairs can transfer to combinations the model hasn't encountered. Studies of weight-combining and cross-domain robustness find that bilingual code-mixing ASR systems built this way do extend to unfamiliar language combinations, though only to a limited degree, so there are real improvements but real constraints too. Current merging and generalization methods weren't designed for the specific challenges of mixed-language speech recognition, and applying them unchanged still leaves unseen pairs out of reach.
A single language pair with strong support makes clear what can be achieved in favorable conditions. At the 2022 ISCSLP event for Chinese-English mixed-speech ASR, over 40 teams participated, with the winners’ system reducing Mixture Error Rate far beyond the baseline. With sufficient data, tailoring a robust multilingual foundation model to one language pair remains effective: CALCS 2025 included work that tuned a single-language ASR system for Yoruba-English mixed speech, and this focused update delivered clear improvements even in a lower-resource setting.
None of this clears the generalization barrier. Even the best neural setup, well-matched to the language pairs it has been trained on, runs into the same wall the moment it is asked on a pair it has never seen during training.
Dialogue-act awareness in detecting the switch
Viewing switches between languages as an acoustic event alone misses the reason speakers switch. The conversational purpose drives the change, rather than mere proximity between two phoneme spaces.
Researchers at Columbia University published a study at 2025 EMNLP where they trained a model to predict code-switching on a corpus of spontaneous Spanish-English speech. They found that both discourse and communicative function shape switching, but communicative function has the bigger effect overall. Language alternation tends to happen precisely where turns change, topics pivot, or the intended listener changes. The organization of the dialogue, rather than raw sound alone, defines these transition points. Relying solely on phonetic shifts causes such systems to misinterpret language changes occurring within unclear audio segments. Such an approach also overlooks the discourse cues that would have signaled an upcoming alternation before the sound became unclear.
That shows a real gap in the way current systems are put together. Whether the system uses a language-ID cascade or a fully integrated design, it usually treats the signal in isolated frame-by-frame or brief-segment pieces, apart from the wider purpose of the conversation. And in either setup, the system still lacks a sense of the speaker’s immediate move in the exchange, just a cue to the language the current frame seems to be in.
The biggest practical pushback against adding that representation is that dialogue-act modeling brings extra delay and structural overhead, while the classifiers used to recognize such acts need labeled examples even rarer than recordings of speakers alternating between languages. Yet that concern hardly demands bolting on a distinct act-classification component to whatever pipeline already exists. A sufficiently scaled polyglot dialogue system could absorb this skill implicitly, provided its training corpus features authentic unrehearsed conversation. This requirement circles directly to the previously noted shortage of data: demanding conversational context makes authentic training material a stricter bottleneck, not a more forgiving one.
What standard word error rate cannot measure at a switch point
Word error rate treats each error the same, regardless of where it falls within an utterance or which switch caused it, and so misses the specific failures above even when they happen constantly.
Three separate evaluation problems build on this blind spot. One issue is ambiguous script choice: when a language pair mixes scripts, a correctly recognized word may have multiple valid written forms, yet standard WER still marks a valid rendering of the inserted language in the host language’s script as an error. Kadaoui's team introduced PolyWER at the 2024 EMNLP meeting so transliterated or translated matches would count as correct.
Mistakes at the boundary act differently than mistakes in other parts of the utterance. An overall WER score for the whole sentence obscures whether mistakes pile up precisely where languages change or distribute uniformly throughout, with such concentration indicating the structural flaw discussed here and uniform distribution suggesting a separate vulnerability. PIER, or Point-of-Interest Error Rate, comes from a speech research lab and narrows WER down to specific words of interest. When evaluating code-switching, targeting those embedded-language terms yields a far sharper view of whether a model copes with transitions inside a word versus between words.
Both flat WER plus PIER collapse every switching category into one error tally, a third and structural issue. HiKE, unveiled at the 2026 EACL conference as a tiered assessment framework designed for Korean-English code-switching, captures this architecture across diverse switching categories, unlike the other pair of metrics.
Vicomtech’s assessment of a Basque-Spanish ASR system shows in practice that a model may post the best possible WER score yet still fail on the code-switches users care about most, since lowering a single aggregate metric does not reveal where leftover errors occur. Using several metrics at the same time takes more time and engineering work, and those metrics will not always line up, which makes choosing a model harder than checking one score. But that mismatch is useful information in its own right. Strong WER paired with weak PIER shows that errors pile up around switch boundaries, giving teams the evidence they need to forecast its behavior on real production audio.
Methodology & sources
- Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs
Provided findings on code-switching ASR generalization to unseen language pairs, including the weight-combining approach and its limited transfer to unfamiliar language combinations.
- Beyond Monolingual Assumptions: A Survey on Code-Switched NLP in the Era of Large Language Models across Modalities
Cited as the source for the survey covering five research tracks, more than 15 tasks, over 30 datasets, and over 80 languages on code-switching NLP.
- Closing the Gap in Low-Resource ASR: Leveraging Multilingual Models for Code-Switched Yoruba-English Speech
Provided the example of CALCS 2025 work fine-tuning an ASR system for Yoruba-English mixed speech with clear improvements in a lower-resource setting.
- Discourse-Driven Code-Switching: Analyzing the Role of ...
Provided the Columbia University study finding that communicative function and discourse structure shape code-switching more than acoustic proximity alone.
- Do modern speech LLMs and re-scoring techniques improve bilingual ASR performance for Basque and Spanish in domain-specific contexts? - ScienceDirect
Provided the Vicomtech Basque-Spanish ASR evaluation showing a model can achieve strong WER yet still fail on the code-switches users care about most.