Voice Activity Detection Libraries for Streaming Voice AI

Models assessed: energy thresholds, neural approaches, and GMMs

Voice detection has a single task: many times per second, it tags a brief audio chunk as non-speech or talking, then pushes that result downstream. It won’t remove noise, tell who is speaking, transcribe speech, or spot a wake word. Those are distinct tasks that are bundled together in one tool. VAD only checks one thing: is someone talking in this frame?

It gets there by audio capture, framing, extraction of features, classifier scoring, plus a probability output from 0 to 1. A popular open-source tool, Silero VAD uses 512 samples per chunk, which is 32 milliseconds with a 16kHz sample setting. Downstream, reaction speed, misfire rate, and how natural or robotic the chat feels all hinge on what occurs within that 32-ms slice and the value the model emits.

Three types of detector handle this task, and each breaks differently. These systems check volume, frequency changes, and sound patterns against fixed settings. Low CPU, but bad for call systems: a loud bang or gear change gets treated as talking because it checks sound level, not words.

Probability models of speech and non-speech distributions let these detectors do more than just threshold raw energy. WebRTC VAD falls into this category. It's more disciplined compared to an energy threshold, yet hand-engineered rules still break down when the audio gets messy.

Neural detectors pick up where speech ends and noise begins using labeled audio, holding context between frames rather than judging each one on its own. They become far more resistant to changing noise, though they require a runtime: PyTorch, a vendor's SDK, or ONNX.

The idea that neural models can't handle real-time tasks is proven wrong by the data. Silero VAD, running in Python on an AMD CPU, processes a full hour of audio in about 15.4 seconds (a real-time factor of 0.004, around 0.43% CPU usage). Cobra VAD runs at an RTF of 0.005 using that same setup. They differ in speed, but not by much here. What sets them apart is how each deals with noise and the range of devices it supports.

Neural models diverge from earlier detectors here too, in what they return. GMMs and energy thresholds give a straight yes-or-no answer. Neural detectors give a float, a probability, and hand off threshold-setting to the team that builds the final system. Most of the tuning sits in that handoff, and many teams underestimate how much it takes.

WebRTC VAD: the lightweight baseline and its accuracy ceiling

Google's WebRTC project is where WebRTC VAD originated. It runs GMMs and accepts only 10, 20, or 30 millisecond frames of 16‑bit mono PCM at 8000, 16000, 32000, or 48000 Hz. These are hard constraints. Audio that doesn't fit the spec must be resampled, no exceptions.

It's extremely lightweight, it's proven across enormous number of WebRTC deployments in browsers and software, and drops into any project free as open-source, with minimal setup friction. It’s extremely lightweight, it’s already been proven across a huge number of WebRTC deployments out there in browsers, and it slips into a project for free, open-source, with hardly any setup friction.

Across the models, accuracy drops most sharply in noise. In a held-out 100 files slice of the UrbanSound8K dataset with road noise, jackhammers, sirens, and building-site audio, WebRTC VAD gets the weakest mark among these models: 58.0% accuracy. Works in a calm room. It won't cut it at drive-thru windows or inside a call center full of chatter, and no downstream tuning rescues any classifier misreading raw input.

Silero VAD: the open-source neural standard for CPU-first deployments

Silero VAD, a neural model, comes via ONNX and PyTorch, is MIT-licensed, and needs no telemetry or API key. The model weighs roughly 2 megabytes, yet it handles far more than its size suggests.

It's taught on speech spanning 6,000 languages and more, and the payoff comes as generalization: it performs across accents and in domains it wasn't fine-tuned on, with no retraining. It works with 512 samples, which is 32 milliseconds at a 16kHz rate, and processes each chunk on one CPU core in less than a millisecond. It works with 16kHz and 8kHz audio, supporting both modern recordings and legacy telephony sample rates.

Many real-time voice agents on CPU use Silero, with little reason to pick anything else for lightweight deployments. It's free, runs light in the stack, and generalizes without Cobra's required licensing step. If you jump to a heavier tool before Silero truly falls short, you're solving an issue that isn't there yet.

Cobra VAD and pyannote: the cases for proprietary accuracy and offline precision

Cobra VAD from Picovoice goes the other way than Silero: closed and proprietary, developed on many hours of audio in different settings. It also outputs a 0 to 1 float, with threshold logic up to the user. Picovoice reports 99% accuracy and ships SDKs for phones, desktop, and embedded platforms. With 5% false positives, Picovoice's benchmark shows Cobra has 12 times fewer mistakes than Silero, and versus WebRTC, 50 times fewer. It also works on the Raspberry Pi Zero using around 5% CPU, a genuine argument for embedded viability, not just a footnote.

Picovoice's threshold-setting notes show the point: its probability output works like a dial, not one locked value. A fast-food drive-through should set a more sensitive threshold, because losing bits of an order is worse than the odd false positive. A robocall detector should run a higher one, since triggering before the line is even picked up is worse than missing the first "hello." Same model, same float output, two completely different decisions depending on what an error costs in that specific context.

Cobra needs a paid option before going live, but a no-cost AccessKey from Picovoice's Console handles non-commercial and testing. It doesn't work out of the box like Silero does. That trade-off means accuracy and broad support, but with a licensing step. Unless Cobra's noise handling is a must, the fee doesn't pay off.

Pyannote Audio works differently. Built on PyanNet, pyannote/segmentation-3.0 handles audio in sliding 10-second chunks advancing by 1 second apiece, then joins the overlapping predictions via hysteresis smoothing. The model stays small: a modest count of weights and 5.7 megabytes stored. On Hugging Face, where around 200 models carry a voice-activity-detection tag, pyannote's segmentation model alone pulls more than 6 million downloads a month.

pyannote reaches 62.0% on UrbanSound8K noise benchmark, above WebRTC but behind Cobra and Silero. Starting with release 3.0, its diarization setup adds 150 to 300 ms of after-the-call work, so it can't run as the main VAD for a live voice agent, period. Its popularity is driven by a different use case: batch jobs, call transcripts, pipelines for diarization, offline tasks where latency of a quarter-second is no big deal.

A newer entrant called NOVA-VAD hits 93.0% accuracy running lightweight and CPU-only on the UrbanSound8K benchmark, outperforming WebRTC's 58.0%, pyannote's 62.0%, plus Silero's 87.0% for noisy-audio clips. As tests change, keep an eye on the 93.0% accuracy NOVA-VAD reports for UrbanSound8K. It's not yet ready to build on, since it's a community project lacking the field history the other four hold, and a single benchmark isn't the same as a model proven in an actual call center.

The production controls that determine whether VAD feels natural: hysteresis, hangover, and pre-roll

Frame-level accuracy matters, but it can't stand alone. The classifier's output is just one input. The caller's real feel comes from what the buffering and surrounding state-transition layer does with that number, but most groups brush right past it on their way to launch.

Pre-roll comes before everything else. It's a ring buffer holding the audio captured just before the system declared "speech," and without it, the first syllable of every utterance gets clipped clean off. Callers pick it up right away, even if they can't say why.

Hysteresis fixes another problem: flickering. One threshold to start a segment and a second, lower one to end it stops an uncertain frame from flipping it on and off by itself. Lacking that asymmetry, it stutters between voice and silence many times in one utterance.

Hangover is a trailing buffer that stays open after the probability drops, protecting trailing consonants and soft sounds as someone stops talking. If any one of these settings is stripped out, the model's main accuracy number quits predicting how the app feels in conversation. A benchmark leader can lose out to one of those mid-tier models with tighter buffering logic. Accuracy in tests and accuracy in real conversations are two different things.

Why frame-accurate VAD alone cannot end a conversational turn

When the speaker goes quiet for a 200-millisecond pause, they might be finished, taking air, looking up a number, or picking their words. Acoustic VAD can only hear silence, so it can't tell those situations apart. Counting on VAD alone for a conversational system, as a number of voice-agent builds did, is a mistake, and this section explains why.

VAD, timing, and meaning-based turn detection each work as their own layer rather than rival answers to one problem. VAD asks whether sound fills this frame, acoustically and continuously. It decides if the pause has lasted enough to mark the close of a spoken turn, using timing over VAD's event feed. Semantic finish-of-turn checks whether the person has wrapped up a thought, using grammar, tone, and how the talk has gone so far.

Without that final stage, it treats silence as the end of a turn, and the dialogue falls apart. A VAD-only system that barges in on a caller who just said "I need to think about that for a moment" has broken the conversation, and no amount of frame-level accuracy fixes it, because the model was never built to answer that particular question.

But the ledger does show a real latency benefit. When VAD picks up silence, some pipelines drop the audio buffer on the spot rather than riding out a set timeout, cutting latency from about 500 milliseconds to around 125. The model itself stays exactly the same. The buffering logic around the model handles everything.

How Pipecat, Inworld, LiveKit, and Deepgram Flux handle turn-detection stacks in 2026

By default, Pipecat uses its Smart Turn model to evaluate whether someone is done talking, combining VAD flagging with additional logic. Silero VAD comes built in for on-device use, while Krisp VIVA VAD can be used when apps require higher sample rates.

LiveKit's TurnDetector reads audio ahead of any completed transcript, combining semantic content with acoustic cues such as intonation, tone, and timing. LiveKit calls it state-of-the-art for accuracy at end-of-turn. Skipping transcript finality is the key choice: holding for a transcript to finalize adds lag that acoustic and prosodic cues do not require.

Inworld runs Semantic VAD over Silero VAD for audio input, pairs it with Smart Turn to control reaction speed, and includes a basic VAD for straightforward setups. Deepgram Flux targets conversational agents and works across 10 languages with auto-detect.

Across these stacks, the same call: VAD is not the end for any of them. It's just input for a judgment acoustic methods can't handle alone, and tools still pushing VAD as the complete answer will go on barging in while people are mid-sentence.