Real-Time ASR Adaptation for Domain-Specific Terminology
Specialized speech systems need runtime vocabulary injection and fine-tuning to match domain audio.

- Written by
- The Voice Layer Editors
- Published
- October 6, 2026
- Reading time
- 10 min read
- Sources cited
- 6 sources ↓
What this covers
Speech recognition systems meant for general use stumble on domain terminology, and the reason is not missing training data. The root cause is that their internalized expectations diverge too much from how specialized fields actually sound and phrase things. Bridging that gap demands multiple strategies working together: runtime vocabulary injection, fine-tuning, and architectural redesign. Every approach targets a distinct aspect of the challenge.
Why General-Purpose ASR Fails on Domain Terminology
Google Health AI's model card for MedASR puts a number on the gap that motivates this entire discussion. On the RAD-DICT radiology dictation dataset, a domain-tuned model paired with a 6-gram language model (MedASR + 6-gram LM) reaches a 4.6% word error rate, while Whisper v3 Large, a strong general-purpose model, reaches 25.3% on the same data. A base greedy version of MedASR, without the language model, still scores 6.6%. The five-fold gap between the domain-tuned system and the general one is not a rounding error. It reflects a structural fact about how these models learn: a model trained end-to-end on broad speech memorizes the distribution it saw, and its accuracy falls off sharply the moment the input departs from that distribution.
The United-MedASR team describes adapting to the clinical domain as presenting "unexplored challenges," despite those very general-purpose systems handling everyday conversational transcription well. Such a characterization is valuable since it pinpoints the issue accurately. These models struggle with healthcare recordings because their predictive distributions derive from corpora that look nothing like a radiologist dictating, legal testimony, or medication verification by a pharmacist. Studies of English ASR datasets for clinical use published in PMC verify this: systems boasting strong aggregate accuracy still err much more frequently on the healthcare vocabulary that matters most, such as drug names alongside dosages, diagnoses, plus negations. Omitting one negation from medical documentation, such as the "not" within "patient does not have a history of," threatens patient safety regardless of leaderboard WER scores.
Audio quality introduces its own gap alongside vocabulary. In Botnoi Group's fine-tuning of NVIDIA's Canary models, contact-center sound, downsampled into narrowband at 8 kHz, codec-damaged, and exposed to packet loss, adds a second mismatch layer beyond vocabulary. Typical packaged ASR systems learn from wideband 16 kHz recordings, pristine studio capture, or reasonably clear video. Use it in a contact-center setting, and errors climb steeply even before vocabulary problems are considered. Because vocabulary and acoustics compound instead of trading off, the answer to "can't a large enough general model just cover most domains through scale?" is simply no: although LLM-native ASR has made general systems much stronger, RAD-DICT's 4.6% versus 25.3% WER gap shows that scale by itself cannot bridge dense, specialized vocabulary. What is needed is a distribution shift remedy, not just a larger version of the same method.
How contextual biasing and hotword injection work at runtime
The quickest and most workable way to handle this mismatch is keeping the network's parameters exactly as they are. Known as contextual biasing or hotword injection, this approach feeds specific vocabulary to the system with the audio during inference, letting the decoder raise the likelihood of those items whenever they match the sound. Think of a voice helper absorbing an unfamiliar person's name, a clinic tool ingesting someone's active prescriptions, or a courtroom app surfacing references for a particular matter. Each of these uses the same setup: the vocabulary gets supplied for each utterance or session, and no retraining is required.
With ASR moving toward large language models, this method has been forced to adapt. NIM4-ASR, released in April 2026 by NIO's Advanced Intelligent Systems Group, brings hotword biasing to LLM-based ASR through retrieval-enhanced generation. Its million-entry hotword index delivers, at inference, the pertinent subset for each utterance in under a millisecond, quick enough to support live production streaming.
Alibaba's Tongyi Lab released a comparable two-phase approach in December 2025, dividing the task into retrieval and injection stages. Their model, GLCLAP, initially extracts a small set of the top-k candidate hotwords from a broad vocabulary. The LLM-based ASR model then receives those options as textual prompts, having been separately fine-tuned via Generative Rejection-Based Policy Optimization (GRPO). During that tuning phase, the reward function simultaneously targets correct hotword identification alongside general transcription fidelity. This dual focus enables the system to sharply lower keyword error rates without sacrificing overall sentence accuracy across standard evaluations.
Both systems tackle the same core difficulty: conventional hotword biasing typically assumes compact, static term sets, often only tens or hundreds of entries, while expanding to unrestricted lexicons at hundred-thousand to million-item scale creates serious coverage and pronunciation-variant challenges. NIM4-ASR-style retrieval offers a way to handle that scale, but it also adds machinery: maintaining an index, making lookup fast, and keeping the setup aligned with the evolving vocabulary.
Contextual biasing stays confined to the lexical half of the problem: it raises the probability of words and phrases the system already knows, but leaves the model's perception of sound untouched. Sound that is band-limited, squeezed through a codec, and full of noise still yields acoustic representations that go uncorrected. Fixing those demands adapting the network's parameters directly.
Fine-Tuning With Real Domain Audio
Training the system with labeled in-domain recordings reshapes how it hears and predicts language simultaneously, enabling fixes beyond the reach of runtime biasing. The approach is pricier too: teams must gather labeled real-world recordings, and they may weaken the model's broad-domain accuracy, a compromise addressed below.
Botnoi Group demonstrated the practical value of this approach by adapting NVIDIA's Canary 180M Flash alongside the 1B Flash variant to Thai telephony audio. Using prompted speech and recordings from live voicebots, the team built a training corpus, layered in audio augmentation tailored to phone networks, and conducted four trials focused on latency, adapting the models to Thai, handling noisy calls, and recognizing specialized terms like names and addresses. Testing against Botnoi's proprietary evaluation set showed the character error rate plunging from a baseline of 23.31% down to 9.04%. Such improvement resolves the core signal challenges of phone calls, including limited bandwidth, codec distortion, and typical call-center background noise that word lists alone cannot overcome.
Built on the first, a further adaptation pass focused on the names and addresses that matter most to the business. It brought CER down to 3.78% from 16.98%. Which stage comes first matters here. Applying telephony-acoustic adaptation before domain-jargon adaptation means each stage corrects its own layer of the mismatch, instead of asking one adjustment to repair both sound and terminology together.
Fine-tuning ran end to end on one NVIDIA A100 GPU carrying 40GB of VRAM. After that step, the leaner Canary 180M Flash variant landed close to 1B Flash on accuracy while slashing the cost to run inference, which pushes parameter efficiency from a side question to a real production one. Botnoi's testing also followed real-time factor, written as RTFx, because even a model polished through training that writes great transcripts is still useless if it can't keep pace with a live call inside a voicebot setup. Speed and accuracy need to be judged together, not balanced against each other afterward.
Obtaining authentic in-domain recordings in such volume carries a steep price tag. Sourcing voices spanning varied devices, environments, and ambient sounds, then annotating the results, demands substantial effort and funding. This limitation drives the pair of adaptation strategies discussed next, namely augmenting with synthetic data and adapting decoders using text alone.
What synthetic data augmentation contributes when real audio is scarce
If real domain audio is scarce, synthetic speech generated using curated domain terminology can cover vocabulary coverage gaps, if synthetic audio stays grounded in the domain’s acoustic and lexical realities.
At the 2026 PROPOR conference, researchers introduced GARAGEM, a dataset focused on Brazilian Portuguese speech for car-repair settings, showing the principle in practice. It pairs real recordings taken from the web with artificial utterances built around vetted automotive-repair terminology. The pipeline follows four reproducible phases: real-speech gathering, domain-aware artificial-speech creation, merging the dataset, and tuning the ASR models. When evaluated on Whisper, Wav2vec 2.0, plus Conformer, every model achieved lower word and character error rates after synthetic samples were paired with real recordings. Synthetic data especially helped the systems recognize repair-specific vocabulary that the recorded training audio did not contain.
This approach only works because of domain grounding. Ordinary synthesis engines inject distinctive sonic flaws while missing the cadence, tempo, and ambient sound typical of actual workplaces. By matching word choice, delivery patterns, and acoustic diversity to authentic paired recordings, GARAGEM achieves this. As a result, the generated speech mirrors real-world input rather than offering an artificially polished substitute disconnected from deployment realities. Garage maintenance audio specifically features ambient interference, amateur voices, and improvised wording. A pristine artificial dataset leaves systems wholly unequipped for such challenges.
This same logic carries over into the medical field as well. By drawing on ICD-10 (the International Classification of Diseases), MIMS (the Monthly Index of Medical Specialties), and FDA records, United-MedASR assembles a domain-specific medical lexicon and then fine-tunes Whisper against that enriched corpus. Results from automotive repair work and from clinical dictation indicate the method carries over to any domain where reliable, well-organized terminology references can be drawn upon.
The benefit or harm of synthetic data hinges on if it substitutes for authentic recordings or supplements them. GARAGEM shows the value of supplementing rather than substituting: authentic audio remains essential for preserving acoustic quality, while synthetic data's job is to push lexical coverage past whatever those genuine samples managed to capture. Such generated material inevitably grows stale. Because specialized vocabularies across medicine, tech, and similarly dynamic domains evolve rapidly, any generated dataset anchored to a static lexicon inevitably falls out of step with how people actually speak. This constraint makes pairing synthetic augmentation with injecting vocabulary at runtime and adapting text-only decoders a practical necessity.
Text-only decoder adaptation: reshaping language priors without any audio
Fine-tuning just the model's decoder on domain text, without any paired audio, can produce big gains for recognizing domain vocabulary, since unfamiliar terminology mostly trips up the language priors of the decoder rather than the acoustic representations inside the encoder.
What makes the method appealing is the supply imbalance. Creating matched audio-and-transcript examples costs money because it requires recorded speech and a careful written version. In contrast, large stores of domain writing may already exist, from clinical guidelines and repair manuals to legal briefs and technical documentation, and can be used here without making recordings or adding annotations.
Harman International's Whisper variant, shown at MMLoSo during EMNLP 2025, strips out audio input entirely and retrains just the decoder, keeping the encoder's acoustic features frozen. The approach introduces learnable bias vectors in the cross-attention layers, augmented by a gating network that routes inputs across multiple experts, allowing the model to internalize specialized vocabulary patterns from text alone. The MoE router is what sets this apart from a basic decoder adaptation: it can send distinct slices of a domain's vocabulary, such as drug names, anatomical terms, or procedural phrasing in clinical notes, to separate experts, instead of making a single adapter handle the full breadth of a specialty. Reported gains reach as much as 56% lower word error rate than the baseline, measured against the pretrained starting model rather than a fixed floor applying to all domains and datasets.
Text-only adaptation leaves the mismatch in the audio signal untouched. Issues like codec distortion, narrowband audio, unfamiliar accents, and noise-induced hallucinations persist because altering the decoder's linguistic priors cannot fix flaws rooted in how the encoder encodes sound rather than in the likelihood it gives a specific token. If the model never encountered a spoken form during training, adjusting text alone offers no remedy. Such shortcomings demand encoder-level refinement using authentic recordings or artificially generated speech tied to the target domain. When domain recordings are entirely absent, begin by tuning the decoder on text alone, then introduce synthetic speech to fill acoustic gaps or bias toward hotwords as evolving terms require, guided by available resources.
The Catastrophic Forgetting Tradeoff
Any route that alters weights, including full real-audio tuning, synthetic-audio tuning, plus decoder adaptation from text alone, brings the same tradeoff: better handling of specialized terms often weakens broad speech recognition. It comes with retuning one shared parameter space for a new data mix: domain terms gain resources that had supported general transcription, and the system's shift toward clinical wording or legal-reference patterns makes it less suited to everyday conversation.
Research tracked under arXiv:2608.30158 characterizes this tradeoff across a range of adaptation methods, showing that the forgetting effect is consistent rather than an occasional side effect tied to one particular fine-tuning recipe. The implication for anyone adapting ASR to a specialized domain is that the question isn't whether a given method causes some degree of forgetting, since most of them do to varying degrees, but how to structure a system so that improving domain accuracy doesn't require accepting a fixed cost against general accuracy.
The approach now drawing interest is to split the architecture itself: domain-specific capacity sits in one component and general capability in another, rather than forcing a single weight pool to serve both. Contextual biasing leaves the weights untouched, while full fine-tuning revises the whole set at once; in a decoupled design, the domain shift is confined to its own component while the shared core stays stable. Harman International's text-only decoder work takes a similar step with mixture-of-experts routing: separate expert modules handle distinct sub-vocabulary slices, so the system avoids revising the same decoder across the board and keeps domain adaptation contained. The implication is that domain generality should be engineered as an up-front division, not balanced by regularization, letting a system built to accurately handle radiology dictation preserve its performance on ordinary phone calls.
Methodology & sources
- High-Precision Medical Speech Recognition Through Synthetic Data and Semantic Correction: United-MedASR
Provided details on United-MedASR's use of ICD-10, MIMS, and FDA records to build a medical lexicon, and the WER comparison between MedASR and Whisper v3 Large on the RAD-DICT dataset.
- Combining Real and Synthetic Speech for ASR Adaptation ...
Provided the GARAGEM dataset example of combining real and synthetic speech for automotive-repair ASR adaptation in Brazilian Portuguese.
- Domain-Specific Adaptation for ASR through Text-Only ...
Provided details on Harman International's text-only decoder adaptation approach with MoE routing and the reported up to 56% WER reduction.
- NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
Provided details on NIM4-ASR's retrieval-enhanced hotword biasing with a million-entry index delivering results in under a millisecond for live streaming.
- Domain-Adaptive ASR for Telephony AI Agents: Fine-tuning Canary Flash Models for Enterprise Contact Center Applications
Provided the Botnoi Group fine-tuning results for NVIDIA Canary models on Thai telephony audio, including the two-stage CER reductions from 23.31% to 9.04% to 3.78%.
- [2512.21828] Contextual Biasing for LLM-Based ASR with Hotword Retrieval and Reinforcement Learning
Provided details on Alibaba's GLCLAP two-phase hotword retrieval and injection approach using GRPO fine-tuning.