End-of-Utterance Detection Accuracy in Noisy Environments

Noise won't stay politely in a track's background. It harms the very markers end-of-utterance detection reads. Formants, contours, how long pauses run, the little silences between "thinking" and "finished": all take damage." Once they blur, the detector no longer detects. Now it's just guessing.

Most of this drop is predicted by Signal-to-noise. When SNR goes under 10 dB, the rate climbs quickly, while Deepgram's noise-robust ASR data has rates roughly doubling when SNR moves between 15 dB to 5 dB in large deployments. You see this all the time, not just rarely. Production environments routinely exceed 90 dBA of background noise, and spikes can hit 120 dB, levels that drive SNR far beyond what clean-room accuracy figures tell you.

That benchmark versus production gap is worth your attention. Deepgram notes systems reaching 95 percent accuracy in Aurora 4 benchmark tests drop under 70 percent on actual shop-floor recordings with clanging machinery plus shouted directions. Fora Soft's 2026 playbook shows the same thing for real apps: any voice setup demoing at a WER of 6 percent inside a silent space might drift over 30 percent in a storage building. Other people talking make this the worst case, because they use the same band of sound as whoever you want to hear, and nothing basic can separate one from the rest cleanly.

Every end-of-utterance method using just plain audio inherits the full harm before picking anything. This is the limit the following layer faces. Once harm is baked inside a waveform, no smart thresholding fixes it.

What traditional VAD actually does and where its ceiling is

Voice activity detection asks one simple thing: is anyone talking in the audio, or not? When the "not speech" answer holds long enough, usually 500 milliseconds to a second, the system declares the turn over. The process is simple to explain bluntly, yet it's worth the effort, since many groups overrate it.

At its simplest, Energy-based VAD just sets thresholds for loudness. It's cheap, yet since anything loud enough counts as talking, background noise keeps setting off false positives. Probabilistic models such as Silero VAD improve by judging signal statistically instead of only using thresholding, while production systems using this method can reach 85 to 100 milliseconds of latency with high accuracy, when properly tuned.

When things get tough, Picovoice's open-source VAD benchmark reveals how far apart the results spread. For 0 dB SNR, roughly by the noisiest real-world test, Cobra’s Voice Activity Detection reached 98.9 percent accuracy, with around 5 false activations across 100 non-speech samples. Silero VAD managed 87.7 percent accuracy. Even though it's cheap and still used in production, WebRTC VAD fell to 50 percent. Pure chance.

That ceiling stays put. However well-tuned a VAD is, it doesn't know language. It has no way to tell "I want to pay my," a sentence still building, from "I want to pay my bill," a sentence done with a trailing beat. Both are the same in audio: gap, followed by silence. That is why voice-agent deployments so often hear complaints about 2 to 3 seconds of delay before the agent gets back. A well-tuned VAD engine still waits 500 milliseconds, up to double, in silence-wait before committing to a reply, and pressing the threshold further can't cure a shortcoming the threshold wasn't meant to handle. VAD tuning alone means solving a different layer entirely. VAD checks audio. Judging if the conversational turn is done falls to endpointing, a different task entirely.

The endpointing layer: what sits between VAD and a committed end-of-turn decision

VAD lets it know the waveform has activity. Endpointing figures out what that event means for the talk, pulling from more than raw acoustic cues: how the sound moves, the transcript's finalized punctuation and how complete it is, whether what's said so far forms a complete idea, or, when nothing else works, a plain silence clock as the final option.

You figure out Endpointing delay by counting from when the caller stops talking until the software commits on an end-of-turn decision. A lower value here is not automatically any win, since committing fast but wrong only means this agent answers while the caller's still going. In a production setting, four numbers count. What matters: delay, false cutoff rate (stopping while someone still talks), interruption rate (barge-ins triggered through noise instead of purpose), plus task success, whether a caller got what they needed.

One clear-cut failure illustrates the problem. Callers saying numbers aloud one digit at a time were stopped during mid-sequence, since the brief pauses between digits hit the same silence threshold as a sentence-ending break. It finalized its transcript prematurely and continued. Here, The failure came from endpointing logic, not from detecting speech. Changing the threshold never fixes this, since the program had to recognize that digits were still being spoken rather than just counting those gaps.

How semantic and LLM-based approaches compensate for what acoustic VAD cannot see

What drives all these approaches is asking something different. Semantic end-of-utterance detection asks whether what's been transcribed by now makes a finished point, not whether any sound has stopped. Because conversational completion and acoustic silence differ, our signal shifts from waveform into language, and viewing them interchangeable remains a mistake that acoustic-only systems usually make.

Reframing matters exactly in the places where acoustic VAD falls short: crosstalk, background talk, phone-line compression. When a semantic model reads the words "I want to pay my", it predicts more will follow; after reading "I want to pay my bill", it predicts the idea is done, so it uses that prediction before any silence gap opens instead of sitting through it. Semantic VAD remains robust amid noise because it relies on transcribed speech rather than the distorted waveform.

Multiple projects following this approach launched or appeared in writing throughout 2024 plus 2025. For judging semantic completeness within streaming transcripts, LiveKit's turn-detector model used a language model with open-weights, but LiveKit's Agents SDK later shifted to an audio-based turn detector, a reversal worth scrutiny. Using a small-model approach, Smart Turn v2 from Pipecat AI's lineup plus that lightweight Turnsense release address this issue, with the latter fine-tuning SmolLM2-135M specifically for that task. Other projects take a similar approach. Speechmatics offers end-of-turn detection as a tool, plus its force-end-of-utterance API so teams can finalize transcription manually, bringing answers below 250 milliseconds if applied properly. For full-duplex, real-time setups, Phoenix-VAD applies an LLM built through sliding-window methods. FastTurn runs CTC results through a compact language model called Qwen3-0.6B to guess when a turn ends, but Its performance degrades when CTC transcription drops, especially during overlapping speech and background noise.

Your instinct might push you toward the largest model, but that tradeoff against speed means it isn't always the best move. Thai end-of-turn detection research from KMUTT with SCBX showed compact fine-tuned models giving near-instant results that fit agents running on-device, though Larger models often achieve better accuracy but at the cost of increased latency per call. A practical guardrail in noisy environments is to require several consecutive high-confidence readings before committing to end-of-turn, rather than acting on a single frame or misread word. By combining both acoustic plus semantic signal, you gain on two fronts: background noise matters less since a transcript decides whenever that waveform breaks down, and odd accents matter less because text decides once prosody misleads us.

The three-layer production stack that closes the noisy WER gap

Fora Soft's 2026 playbook points to a lesson worth noting: combining three targeted pieces in sequence drops noisy-condition WER from a 25 to 40 percent band down to roughly 8 to 12 percent, hitting levels clean-audio systems were hitting just years before. Any single layer on its own falls far short of that mark. If you jump right to using a fancier acoustic model while passing over your front-end, you miss most of the improvement.

One layer serves as neural noise suppression at the front-end. On everyday machines, Systems including Krisp plus NVIDIA Maxine alongside DeepFilterNet process audio instantly, and any suppressor matching Krisp can drop noisy WER by 20 to 40 percent before the model starts. With Older-style spectral subtraction preprocessing, SNR can look better in tests while hurting WER, since it strips harmonics plus noise, and identical sound ranges with noise also hold prosodic cues for a model. Picking the right front-end matters equally to whether you have one.

Next comes the noise-hardened acoustic model layer. Using audio with hum, HVAC noise, overlapping speech, and compression while teaching the model, instead of using plain clips and expecting it still generalizes, brings WER down as much as 7.5 percent versus clean-trained baselines; newer contrastive approaches have gone above 20 percent in rough noisy benchmarks, according to Deepgram. The 2026 lineup names Deepgram's Nova-3 (its in-house evaluation shows 6.84 percent WER, though a vendor-reported WER doesn't substitute for checking against your own audio), plus Whisper Large v3 Turbo, Conformer-RNNT architectures, and NVIDIA Parakeet. Deepgram says CVS deals with over 140,000 pharmacy conversations each hour, hitting 92 percent accuracy through noise-trained models that skip preprocessing entirely. A healthcare startup, per the same source, moved from 60 to 92 percent accuracy while cutting latency in half, simply by swapping a generic API for a domain-trained model.

The final layer uses domain fine-tuning along with biasing for keyterms. APIs can take as many as 1,000 added entries during inference, giving accuracy for drug labels, product numbers, and nouns that a basic model might mangle. A setup showing 15 percent WER yet catching each drug name right holds more worth in operationally useful ways than another at 10 percent WER which garbles those terms, since aggregate WER hides exactly those failures hurting you downstream. Specifically for end-of-utterance tasks, such fine-tuning shapes how well the semantic completion model matches the vocabulary and sentence patterns of its actual setting: a pharmacy line sounds nothing like a warehouse dispatch line, and a model ought to sound like it truly belongs in its domain.

One setup worth naming beyond those levels: draft-first speculative or staged tiered answer creation, where it begins drafting once likely semantic completion appears, before any silence gets checked. If handled right, the agent won't interrupt a caller yet answers almost immediately once their turn wraps up. In 2026, Fora Soft's usual hosted setup uses Krisp with Nova-3 plus keyterm biasing; for self-hosted, it uses DeepFilterNet alongside Whisper Large v3 Turbo with a LoRA fine-tune.

What to measure to know whether your EOU detection actually holds up in noise

Vendor benchmarks, taken from English audio in which frontier models score 5 to 7 percent WER, say little about noisy lines. Vendor WER isn't your WER. When evaluating these systems, groups most often slip up by relying on it like a proxy for real production, and naming that mistake is worth doing.

WER should get segmented by medium-noise, heavy-noise, accented, and clean speech. One blended score still hides exactly which failure appears after it’s rolled out widely. When tracking end-of-utterance detection specifically, key metrics include endpointing delay (high latency ruins any voice agent during real use, not just average values), false cutoff rate, interruption rate, and task success. For any voice agent using LLM, whether a caller's task was finished matters beyond transcription accuracy by itself.

You should track Keyterm accuracy apart from total WER. This asks whether it transcribes key terms used in the domain, like any drug name, ID, or SKU. End-of-utterance detection on a transcript left semantically incomplete when a keyterm gets mangled fails downstream in a way which traces to an audio issue rather than to turn-detection.

It's worth saying why Validating all this across large volumes is so hard. Research on turn-taking still improves modeling, but the area hits a clear limitation: there’s little open audio from the noise that production systems meet, including machines, compressed calls, mid-thought callers and crosstalk. Since standard benchmarks may not reflect real-world conditions, teams often create their own test data.