Duplex Audio Handling in WebRTC-Based Voice AI Applications

WebRTC handles simultaneous two-way voice better than alternatives like WebSocket.

Cover illustration for “Duplex Audio Handling in WebRTC-Based Voice AI Applications”
Written by
The Voice Layer Editors
Updated
November 6, 2026
Reading time
5 min read
Sources cited
8 sources ↓

Most demos skip the toughest part of voice AI: Duplex audio. Understanding how those parts connect is what makes a voice agent feel like a real conversation instead of one that talks over itself.

WebRTC's Default Status as the Transport for Voice AI

Voice AI requires audio flowing both ways simultaneously, live. That cuts out HTTP’s wait-for-a-reply request pattern, and strict TCP too, because packet order is not what to optimize when a stray packet shows up after it should.

The browser offers three transport choices, and choosing one is an architectural call, not a stylistic one. WebSocket runs over TCP, which guarantees delivery in sequence, but not for free: if a packet drops, TCP halts every later packet until that missing one is retransmitted, while PCM/base64 encoding runs near 10x the bitrate of Opus. In one line: TCP stops everything to retransmit a dropped packet, and voice has no time for that.

Picking WebSocket drags that flaw along for good. One WebSocket dropped packet stalls both audio directions at once, a failure mode duplex cannot tolerate. WebRTC started as a transport for real-time audio, with echo cancellation, voice activity detection, and Opus already included, not separately bolted on. It won because the packet-loss work was already done for you, not because it was popular.

Diagram: Why WebSocket Fails Duplex Voice (and WebRTC Doesn't). Visualizes: Show a side-by-side contrast between WebSocket/TCP and WebRTC as transports for duplex voice AI.

Full-duplex, and why most voice AI systems don't have it yet

Full-duplex lets both sides speak and hear audio at once, like a real phone call. Callers barge mid-thought, talk over each other, and slip in an "mm-hmm" with no formal signal. Half-duplex works the other way: one side speaks, then the next, rotating in a strict, mechanical fashion, and that's the mode almost every shipped voice agent runs in at present.

They chose it deliberately, no oversight: most companies still ship half-duplex systems and label them conversational. The usual setup restricts people to turn-based back-and-forth, holding for an explicit prompt or a pause before it will respond. It's the pipeline. Voice activity detection must trigger first, followed by ASR, LLM, and TTS in strict sequence. When turns can only happen one after another, the architecture just won't let them cross.

The latency numbers that define whether duplex feels natural or broken

That gap is the ballgame: when a pause feels like someone listening, it beats one that feels like waiting on a phone call to customer service.

Cascaded pipelines slow things down even more. Optimistically, turnaround runs 1.5 seconds, but typical network conditions mean it stretches to 2 or even 3 seconds.

How Opus and the jitter buffer keep audio coherent across both directions simultaneously

WebRTC requires Opus for voice, and it covers a broad spread: 6 kbit/s to 510 kbit/s bitrate, 2.5 to 60 millisecond frames, 8 kHz to 48 kHz sampling. Voice AI rarely touches much of that.

Audio goes over SRTP as one steady Opus feed, never needing codec transcoding or a carrier handover the way a call routed on the regular telephone network does. Network conditions still wobble, though NetEQ manages them: the jitter buffer Opus leans on as its framework to adapt when packets arrive too soon, too slowly, or out of sequence, smoothing timing before the ear hears it. Without it, duplex audio will degrade into misalignment and stutter even if the codec runs perfectly.

Acoustic echo cancellation's role, limits, and necessity for duplex

All duplex voice agents face the same basic constraint. The agent's voice plays out of the speakers on the user's end, the microphone grabs it again, and without a fix, the app answers its own audio like it was a real question.

To fix this, acoustic echo cancellation keeps a copy of the speaker’s audio, called the reference signal, and starts subtracting the predicted echo from the microphone captures. The browser needs access to each half of the loop for this to work. If the agent's audio is rendered through a path the browser's audio pipeline can't see, the canceller loses its reference signal to subtract against and the mechanism breaks.

If AEC breaks, the problem spreads. The agent's voice starts setting off the voice activity detector, so it cuts off mid-sentence as if someone had spoken. AEC is upstream from most of the duplex chain, meaning its failure carries downstream, not some glitch you can patch later on.

Voice activity detection and barge-in: how the system decides whose turn it is

The gatekeeper here, voice activity detection, picks out actual talking from silence or room sound. This is what controls STT access and picks the moments when someone can jump in mid-sentence. It runs on audio after echo cancellation, so if AEC upstream corrupts the signal, downstream fails: false input gets in when the agent's voice slips by.

The filtering runs across a pair of stages. A voice classifier decides if that candidate audio is truly spoken words, not a cough, a shutting sound, or background hiss.

Each classifier carries trade-offs; no available option is simply the stronger one. Built for streaming, Silero VAD is a multilingual neural network that runs 5 to 8% false-positive hits when fed noisy phone audio. The older GMM-based classifier in WebRTC, WebRTC VAD, runs faster but loses accuracy for that speed, mainly with songs or the background hum common in a call centre. Picking one means betting on the failure mode a deployment can tolerate.

Differences between the pipeline approach and speech-to-speech approaches in handling duplex

Diagram: Cascaded Pipeline vs. Speech-to-Speech: The Duplex Gap. Visualizes: Illustrate the cascaded pipeline as a strict left-to-right sequence — VAD → STT → LLM → TTS — where each stage must complete before the next begins, producing an…

Rival approaches define how a voice loop works, with duplex treated unlike in each.

The cascaded method strings speech-to-text, an LLM, and text-to-speech together in a well-understood way using swappable parts you can swap out on their own. VAD must precede STT entirely, the LLM receives a complete transcript to process, and TTS begins streaming audio before the LLM completes generating. It remains debuggable and modular, with each part improving at its own pace, but the sequential design caps how much it can approach duplex. Tuning can't change that the three stages have to play out one after the other.

Models built for speech-to-speech drop that pipeline, running encoded audio through the loop without converting it to words. Truly closing that gap likely requires letting go of the cascaded pipeline's modularity, instead of just trimming milliseconds from each part. Can cascaded systems shrink their three stages and compete without a rebuild? That's the bet for the coming period.

Methodology & sources

  1. WebRTC for Voice AI: how the transport layer works

    Cited as the basis for describing WebRTC's origins as a transport built for real-time audio.

  2. Switching my AI voice agent from WebSocket to WebRTC — what broke and what I learned

    Cited directly for the point that TCP stops everything to retransmit a dropped packet, illustrating why WebSocket fails for duplex voice.

  3. How Does WebRTC Power Bi-Directional Voice and Video in AI Agents?

    Cited for the description of NetEQ as the jitter buffer framework that manages network timing variations for Opus audio.

  4. https://futureagi.com/blog/voice-ai-barge-in-turn-taking-2026/

    Cited for the Silero VAD false-positive rate figure of 5 to 8% on noisy phone audio.

  5. Why WebRTC Is the Best Transport for Real-Time Voice AI Architectures

    Cited for the argument that WebRTC won adoption because its packet-loss handling was already built in.

  6. https://dev.to/orca_forge/browser-voice-interaction-ai-pitfall-guide-2026-16-common-traps-with-aec-getusermedia-and-40hd

    Cited for the failure mode where AEC loses its reference signal when agent audio is rendered outside the browser's audio pipeline.

  7. https://futureagi.com/glossary/opus-codec/

    Cited for the Opus codec's technical specifications including bitrate range and frame size parameters.

  8. https://softcery.com/lab/ai-voice-agents-real-time-vs-turn-based-tts-stt-architecture

    Cited for the description of the cascaded pipeline architecture stringing STT, LLM, and TTS together as swappable modular parts.

The Voice Layer Editors