Latency Budgets for Natural-Feeling Voice AI Responses

An AI voice reply must fit within the gap between human speakers, typically under 300 milliseconds, or it fails. Neurology, not choice, sets that gap, so any extra millisecond a system runs past it has to be pulled from this four-stage pipeline: turning speech into text, running language model inference, converting text back into speech, and the network in between. We walk through that budget one stage at a time, drawing on production data covering 2025 and 2026 to trace where those milliseconds end up. Nearly all production setups miss that baseline by a wide margin, and the cause isn't obscure technical work. Developers are polishing the incorrect stage, and the costly one sits untouched.

Torreira's 2015 research with Levinson showed that such quick turnaround comes through predicting the moment someone's about to stop talking, not through reacting once a pause lands. The following speaker's response gets ready before whoever is talking stops. This leaves most current voice AI out of the running since it still holds off until speech has clearly stopped before responding. These perceptual thresholds stay blunt: below 300 milliseconds, people miss the gap. In the 300-to-400 range, you can tell something feels off. Once 500 passes, they ask if anyone listened. After 1,000 milliseconds, they think the line went dead. ITU-T G.114 agrees on telecom, with one-way delay kept below 150 milliseconds for clear interactivity, and stricter G.1051 calls two-way delay past 250 milliseconds too much for conversation.

How far production voice AI actually sits from that human baseline

The industry median for production voice agents runs roughly 1,400 to 1,700 milliseconds, according to industry data from 2025 to 2026. Past 300 milliseconds, a gap stops registering entirely, so this is roughly 5 times slower. Say it straight: it's broken, not some rounding error, or a fair price for operating widely.

Things deteriorate at higher percentile levels. At P90, response times are around 3 seconds, bringing clear frustration plus repeated talk-overs. Hitting P95 means a wait near 5 seconds, so bad that most people cut in halfway through, from routine, not rudeness. At the P99 percentile, production voice AI response time is 8 to 15 seconds. This response is no longer just delayed. The conversation has failed, full stop.

None of this shows up in user complaints as "latency," though. Users report that the voice system seems sluggish. They report the bot keeps interrupting, or fails to understand once they finish talking. People point at a symptom; the reason sits just under it, across milliseconds only the tech staff sees. Hamming's research found that 68% of callers quit when the system seems sluggish. In 2025, Retell benchmark data showed call teams had 40% extra hang-ups after a voice system needed over one second to reply. This isn't a footnote for devs, it's churn driven by a stopwatch.

What the pipeline actually looks like and where the total budget comes from

A cascaded voice pipeline runs its stages one after another. Audio first becomes words via STT, a language model then writes the reply, TTS turns that reply into sound, and it moves through the network toward the caller. With naive setup, each stage waits until the earlier one is done before doing its part, and that one choice kills the biggest share of latency budgets.

By 2026, a few firm figures have become the standard. To sound conversational, a baseline TTFA (first audio) must stay under 500 millisecond. High-end builds shoot for under 300 milliseconds. Reading benchmark notes from Retell AI's team, engineering guides on dev.to, plus research by Hamming AI's lab, 800 milliseconds keeps coming up as where conversational talk falls apart in business deployments.

In 2026, A single metric, TTFAB, is sometimes used to measure the gap from when a caller stops speaking to when the reply's first audio byte begins. Watching P95 alongside P50, rather than relying on averages alone, helps catch the slow moments users feel most. A system can hit a solid middle yet still seem unusable when the P95 keeps generating extended silent pauses. The mean leaves out the slow moments people feel most, so a tool built on the mean alone can't tell you if the product works in real life.

Once you work through the math, it doesn't cut you any slack. The total grows as each stage's worst case stacks up in a sequential pipeline. With an 800-millisecond budget, no single stage can afford a slip. One common budget allocation gives STT 350 ms, the LLM 375 ms to start producing output, and TTS 100 ms, adding up before network delay is counted. It already passes 800 milliseconds, even before any network delay is counted. One proposed sub-500-millisecond target allocates network delay at 30 to 50 ms, STT’s initial response at 100 to 150 ms, LLM response with prefix caching at 200 to 300 ms, TTS output at 80 to 150 ms, and system overhead at 50 to 100 ms. This gap in those budgets is about more than faster machines. Careful design, sound architecture, plus one stage which dwarfs all others: the LLM's.

The LLM stage: the dominant bottleneck and where the most budget is at stake

Studies from 2025 to 2026 by Hamming AI's team show the LLM accounts for about 70% of all delay across a standard voice setup. That figure alone ought to reshape where engineers direct their focus. Chasing STT or TTS shavings while the LLM eats seven-tenths of the budget is polishing the doorknob while the door is off its hinges, and any team that hasn't looked at TTFT first has its priorities backwards.

What matters in voice isn't total response speed but TTFT, the wait for a first token, since streaming allows TTS to begin voicing before a model's full reply lands. For a sub-500-millisecond pipeline, TTFT has to sit at 200 to 300 milliseconds across. Using leaderboard benchmark numbers to proxy readiness for voice is an error, since they're basically irrelevant. Research shows that a model may perform well on benchmarks but still fail in live calls if it takes too long to produce its first token.

No other lever changes this figure as much as Prompt prefix caching, so skipping it throws away your biggest gain for nothing. Prefix caching can reduce TTFT to 200 to 300 milliseconds, compared to longer delays without it. That delay roughly triples or doubles just for re-processing input the model already handled. A fix is simple: leave your system prompt stable, still byte-identical, first in every call, and put all changing details, like timestamps and user IDs, last.

Systems built for deep thought don't belong in real-time voice, period, regardless of what else they do well. Some systems designed for deeper reasoning may take several seconds to respond after a user finishes speaking. Prompting can't fix something this deep. No prompt trick closes that 4-second gap when your real-time conversation budget is 800 milliseconds, so the mismatch stays. Proactive inference, running the model before the user finishes, can help close the gap to human-like response times.

STT and TTS ceilings: the stages that bracket the LLM and where provider choice is decisive

The LLM has STT and TTS flanking it, yet people overlook both despite their unfinished state. A 2025 dev guide sets STT latency around 200 to 350 milliseconds per full transcript, while streaming STT gives first partial transcripts in 100 to 200 milliseconds. Non-streaming STT waits until the speaker finishes, adding noticeable delay before the rest of the pipeline begins. Streaming STT takes audio live, bringing that overhead down to 100 to 200 milliseconds on most calls. In 2026, no justification exists for releasing a non-streaming setup.

One simple fix matters more than it ought: a "flush." Flushing the entire audio buffer the moment the system catches no voice, instead of letting a timeout run, drops STT latency to roughly 125 from around 500 milliseconds. That small change produces outsized results, which surface over and over in production guidance exactly since rolling it out costs little yet stays overlooked.

FutureAGI's 2026 provider review lists STT tools with first partials below 200 milliseconds: Deepgram Nova-3, AssemblyAI's Universal-1, plus Speechmatics Ursa. Being fast, though, isn't all that matters here. For serious deployments, staying under 5% error is standard, and 5 to 10% works when the audio is noisy or comes from telephony. Robustness matters most in call-center and on-the-go audio, while punctuation plus capitalization count for more than you'd expect: any language model parsing some unpunctuated transcript starts from degraded input before producing even one reply.

For TTS, the stated goal is 75 to 150 ms before speech begins, with 150 ms as a common upper target. The companies differ a lot in this area. In FutureAGI's 2026 figures, Cartesia Sonic is among the faster TTS providers in some benchmarks; ElevenLabs Turbo v2.5 has been measured with higher TTS latency in some setups. A swing exceeding 200 milliseconds separates providers doing an identical job, and you can't claw those lost milliseconds anywhere else in your pipeline. Whatever TTS provider gets selected, it eats up nearly the whole budget, fixed in place when the deal closes, not when the system goes live.

Network latency: the part of the budget determined by geography, not code

Network delay remains the one piece in this budget that smart code never fixes, because it stays bounded by geography and the routing used to reach its end. Hamming AI's 2025 to 2026 numbers show the map: US East over to US West runs 60 to 80 ms there and back, US over to Europe runs 80 to 150, while Asia from the US runs 150 to 250.

Hamming's research shows that any cross-continental setup uses 20 to 30 percent of the allowance just for network, out of a total budget of 800 milliseconds, before STT plus the LLM and TTS spend one millisecond. A large share of the budget, up to 30 percent, is consumed by physics before the system has done any work at all.

Don't spend time weighing alternatives, since Co-location alone works. Putting the STT stack, LLM model, and TTS onto one network with the voice traffic removes added hops caused by stitching vendors across different locations. Co-located setups can help reduce total latency, with some architectures targeting tighter budgets. SIP signaling adds overhead while the connection opens, separate from mid-conversation round-trip delay, which later shows itself. Checking code on your own machine hides the bulk of network latency, notoriously tough to spot early. Production percentile checks across P95 thresholds plus P99 remain the only practical way to catch it, so anyone leaning on dev benchmarks alone to size up network behaviour is looking in the wrong place.

How streaming overlap changes the math from additive to concurrent

Put STT, LLM, plus TTS with internet delay one after another: sub-500-millisecond answers become impossible, per FutureAGI's 2026 report. Any naive pipeline hits that wall, however quick one stage becomes. Sequential architecture doesn't work in low-latency settings. That is disqualifying.

With Streaming, the steps overlap instead of adding up. As STT keeps turning the caller's words into partial transcripts, the LLM will begin generating its reply. TTS begins synthesizing and speaking the reply's first words as the LLM keeps working on later ones. Wired well, caller-perceived gap comes from the slowest route within the system, rather than each stage's processing added together.

In real terms, the gap is a big one. In 2026 tests by Parloa, non-streaming cascades usually take 800 to 2,000 ms, while the same parts, set up for side-by-side step-by-step work, can get below 500 ms. Same components, revised wiring, over a full moment.

Another option is a speech-to-speech architecture that takes audio in and sends it back out, skipping STT along with TTS. This makes the pipeline shorter, but you give up a lot in exchange: knowing what went on at every stage, once a failure hits. Most groups won't accept that deal. The 2026 research from Coval's, along with work by Infrabase ai's, says observability is why most 2026 production deployments still use three-stage cascaded systems where streaming covers everything. With a three-stage pipeline, every stage leaves a record, so debugging can show where a conversation turned sideways rather than guessing later. To make that overlap happen, streaming needs to be active in each stage. One stage left non-streaming kills concurrency, so the pipeline reverts back to additive math.

What the 2025 benchmark data shows when these budgets are tested against real platforms

Retell AI's July 2025 benchmark tested Synthflow, Twilio Voice, and itself on matching scripts for voice-to-voice latency. Synthflow's averaged latency was 420 milliseconds, from a 380 low case to a 480 worst case, and its consistency score was 9.2 on a 10-point scale. Retell averaged 780 milliseconds, with results ranging between 720 and 840, earning a consistency score rated at 8.8. Twilio Voice came in at 950 ms, between 880 and 1,100, and got 7.5 for consistency.

A Retell AI latency face-off from July 2025 put Retell AI, Google Dialogflow CX, Twilio Voice alongside PolyAI through WebSocket timestamp checks in the East US, West US, plus EU areas. Retell got 9.2 points from 10, taking 180 ms for the first sound, 620 ms total delay, 140 ms to handle cut-ins, 45 ms of timing variation, plus 99.7% unbroken audio. Google Dialogflow CX got a 7.1/10 rating, though the data won't show separate TTFT figures individually or end-to-end numbers. Twilio Voice showed up in both sets of benchmarks, sitting steady between 800 and 1,100 milliseconds in each.

These trade-offs carry as much weight as the figures. That 420-millisecond average comes paired to lower feature depth and reduced customization: speed gets traded directly for configurability, so each team should know what side it is choosing first. Twilio's extra latency comes from a system built around carrier-grade reliability and coverage everywhere, treating speed minimization as secondary, defensible in certain deployments, indefensible wherever latency gets checked against a human baseline set at 200 milliseconds. On July 7, 2025, Retell AI's Warm Transfer 2.0 dropped handoff latency 40% through pre-loading and pre-established pools, a fix built for one weak spot in that handoff.

Overall, the benchmark data leaves one conclusion beyond dispute: 800 milliseconds is only the entry point, one that satisfies no one, and nothing to applaud. That just marks where things begin to go wrong. Every tested service that went past it displayed the problems Hamming's production data had previously flagged: extra interruptions, extra hang-ups, and callers asking whether anyone's still on the line.