Turn-Taking Protocols in Multiparty Voice Conversations

Most voice agents sold today still follow dyadic dialogue patterns, even though multiparty conversations make Turn-taking work completely differently. They pause until quiet falls, then speak. That holds up with one caller alone, then collapses when a third voice joins. Research done in 2024 through 2025 lays out how far this fracture goes, and makes clear what companies have avoided: bolting on the third-party detector to any dyadic turn-taking setup isn't temporary, it's a basic mistake, one no tuning fixes.

The structural features of multiparty conversation that have no dyadic equivalent

A May 2025 review by Sapkota et al. posted on arXiv, lays out modeling challenges that any setup built for multiparty conversation must meet at once. One demand is modeling, figuring out everyone's thoughts and emotions as they shift, rather than only what the previous speaker intended. Then, semantic tracking: whom each person is talking to in a group chat, since a sentence there doesn't automatically land on "the other party" as it would in a two-person exchange. Third, agent modeling means guessing the conversation’s path, even when to say nothing.

You won't see how much the third one needs those earlier two until you look closer. Addressee choice is hard: when two people talk, each utterance is aimed at the other person. Once a third voice joins, the agent needs to decide whether it's the one being addressed, if another person is, or if that remark got tossed to everyone. Bring in one or two more voices, and you get bystanders: participants who are part of the conversation but not directly addressed, requiring focus tracking since they could jump in at any moment.

Bids for the Floor compete as well. Two people speaking at the same time over audio resolves: one serves as agent, the other is the person, and one waits. No default fallback speaker exists among multiple participants. Conversation analysts long treated Adjacency as a tidy question-then-answer pattern, but these turn into webs: speaker A queries, speaker B replies, then speaker C jumps in competing before B finishes. Sequentiality, dialogue modeling's backbone, doesn't survive once that occurs. Add variance in feelings and ways people act, and coordination no longer has any dyadic equivalent to use.

Transition Relevance Places and next-speaker prediction as the foundation of any multiparty system

In 1974, conversation analysts observed that speakers signal a turn's coming close long before it ends, through cues like tone, thought completion, and gaze direction. We name these spots Places of Transition Relevance, acting as the hinge where a new speaker might take over, though nothing is certain.

New VAP work, in literature (Saga with Pelachaud, Sorbonne University, arXiv 2506.03980), cuts acoustic cues apart. Acoustic cues like pitch changes may signal turn transitions, while flat or incomplete intonation can indicate a speaker intends to continue., a cue that whoever is speaking intends to continue. A quick intake of air or lifting your arm during face-to-face talk are Turn-initial cues showing you want the floor. With two-person conversation, noticing a TRP gives you nearly all that matters, because only one other speaker could get the floor. With several people talking, that TRP simply leaves the floor up for grabs. The task hasn't gotten harder, really, it's just got another question stacked above it: who gets it?

Next-speaker prediction now stands as its own line of research, split off from work on end-of-turn. Early work in 2024 explored sentence-based prediction in multiparty transcripts. Wu and colleagues (2023) built AutoGen, a framework for multi-agent setups, which handles this by having one LLM agent estimate the upcoming speaker's role from conversation history. In Frontiers (June 2025), Nonomura and Mori show the limit: this approach limits autonomy for individual agents. Going the opposite direction, where agents pick their own moments to speak through self-selection only reproduces that floor-competition trouble within a multi-agent setup. A workable architecture may favor structured turn allocation over self-selection to maintain conversation flow.

Voice Activity Projection and the move toward multimodal, multi-speaker acoustic modeling

Voice Activity Projection views turn-taking like a prediction, not a snapshot. Instead of asking "has this person stopped talking," VAP asks "what's each person's voice activity likely to look like over the next second or two," predicted continuously from audio alone. It's a framing that shifts things meaningfully, less an on-off switch than a forecast.

You can't just plug in a third feed to stretch VAP past two-person recordings and cover extra speakers. That means reworking the training data: two-speaker VAP architecture has no way to show a third party competing over the floor. In July 2025, a paper (arXiv 2507.07518) made the initial real attempt here by training VAP models using three-way conversations from a Japanese dataset about various subjects, pushing VAP into truly triadic ground instead of noise plus dyadic.

Pelachaud's and Saga's Sorbonne paper turns multimodal, using visual and pre-trained audio encoders, so it reads visual signals with acoustic cues, and shows competitive performance in turn-taking benchmarks. MuVAP (2026, arXiv 2606.16731) pushes practicality harder: it bases predictions on visual tracking with one lens and one microphone, and that matters in human-robot settings, since lugging extra microphones is not an option. MuVAP further introduces Role-Relative Projection by collapsing the N-speaker conversation to one fixed two-slot frame (whoever holds the floor now against whoever takes it after) so the system doesn't need to compute each speaker combination in one pass, combinatorial overload that scales poorly beyond a handful of people.

Production systems like Moshi (Défossez et al., 2024), Gemini's Live voice mode, and OpenAI's Voice Mode all get cited in this literature as reference points for where the technology is headed commercially. But that constraint isn't gone: plenty of the models still depend on multi-camera rigs or microphone arrays, and few survive beyond the bench, so a gap between research benchmarks and real-world deployment persists.

The backchannel-versus-interruption classification problem and why getting it wrong feels worse than silence

Detecting when a person speaks is not the difficult part now. The hard part is deciding, in the space of a few hundred milliseconds, whether that sound was a real interruption asking the agent to stop, or just a backchannel, a "yeah," an "mm-hm," a quiet "okay" that means keep going, the listener is listening. Get Miscall it wrong, and a caller thinks something's not right, though they don't know why. This is what makes conversational timing uncanny: the words fit, but the beat is off.

Retell AI's documentation lays out a live setup that checks 3 things at once during the agent’s speech. Even the slightest voiced audio trips the streaming voice activity detector. The streaming transcriber returns an early partial transcript in around a tenth of one moment. Then a semantic classifier reads that partial transcript and decides whether it's carrying real intent (something like "wait, go back") or just acknowledgment noise. The barge-in fires only on real intent. Retell's documentation flags another metric worth watching closely, the false-barge rate, meaning how often a backchannel cuts off the agent when no response was needed. It's more difficult to show than a basic interruption total, but it follows caller frustration far better, and any group optimizing only for latency alone without checking it will release a twitchy agent.

You shouldn't apply a single fixed threshold to every situation. Analysis of production calls across deployed voice agents in 2025–2026 shows that a legal disclosure or payment confirmation step calls for a much stricter barge-in threshold than an open-ended support chat, where a caller interrupting mid-sentence is often just how people talk. Using one threshold, tuned just once for all cases, is what this runbook tries to fix.

A framework from Microsoft Research called RESPOND (arXiv, March 2026) formalizes this as three-way options each instant: silence, backchannel, or turn claim. Backchannels convey engagement while leaving the floor untouched. A Cooperative turn means jumping in while that speaker is still talking, offering helpful overlap instead of a harsh cutoff. In its 2026 preprint, RESPOND notes that the Backchannel Controller picks timing plus wording with a degree of built-in randomness, since backchannels hold little semantic meaning alone; an "mm-hm" that shifts sounds more human than something metronomic.

Controllable turn-taking: the RESPOND framework's two-dial approach to matching conversational context

RESPOND (Responsive Engagement Strategy for Predictive Orchestration and Dialogue, by Lee, Andrist, Panay, Bohus, Wilson, Hernandez, plus Churikov from Microsoft Research) builds on streaming audio processing and incremental semantic parsing, continuously working out both when to jump in and how. Structurally, the design simulates duplex by talking while it listens rather than ping-ponging across those two.

This framework exposes two dials for whoever's designing an agent, with neither tied to the other. Backchannel Intensity controls how frequently the agent acknowledges whoever's speaking, running from barely-there up to fully present. Turn Claim Aggressiveness controls whether that agent will jump while a speaker is still talking, ranging from holding back until an explicit handoff happens to speaking up mid-thought. Since its two dials are separate, with one counseling-style agent, strong backchannel intensity (regular, kind acknowledgments) can pair with weak turn claim aggressiveness, not interrupting, while for any fast-paced brainstorming agent, the opposite can work.

RESPOND's literature draws on user work, especially with older participants, finding agents feel more natural when letting people cut in and giving well-placed acknowledgment. It also finds that low latency goes with greater enjoyment in a conversation, even when an agent sometimes overlaps a speaker. Overlap, put differently, doesn't automatically seem bad. For developers of such tools, one rule applied across each deployment will seem off somewhere, and no latency optimization fixes a rule tuned for a mismatched tone of voice. Those dials are there since counseling software and brainstorming platforms share no ground here, and the whole two-dial approach was made to stop people from handling both alike.

Floor management architecture for multi-agent conversations: the Convener model

So far, we've covered a single agent with one person, or an agent guessing about a group. When several agents speak with people and one another together, a new issue arises: who controls the floor?

A paper from November 2024 (arXiv 2411.05828) takes Open Voice Interoperability Initiative's interoperability specifications (OVON was its old handle under the Open Voice Network) and extends them to tackle that exact issue. The system's architecture puts one Convener Agent in charge of the common conversational space, where it relays turns among participants and decides who may talk. Beneath it are the Floor-Shared Conversational Space, shared ground all parties work from, a Floor Manager that enforces Convener's speaking-turn choices, room for over two simultaneous participants, plus explicit treatment of interruptions and of agents arriving uninvited.

This paper shows why it matters for safety and conversation alike: if an agent moves to hijack control of the floor using a sales pitch it had no request to make, then Convener can block its turn, or kick it out of the conversation, with other participants not told about the intervention. Today's multi-agent frameworks let floor access emerge out of an agent's own individual choices about timing, which means no one spot can stop that hijack. Under a Convener architecture, the choice is explicit, auditable, and logged, not implicit or undetectable.

Instead of focusing on architecture, the June 2025 Frontiers paper by Nonomura and Mori's team tackles this issue through conversation. Their method builds adjacency-pair norms alongside self-selection rules accounting for every agent's inner condition right within next-speaker choice. In trials with multi-agent Murder Mystery play, it reduced dialogue breakdowns while helping knowledge transfer and sound thinking. The Murder Mystery framework and OVON tackle overlapping issues from opposite sides: one handles interoperability, the other how conversational turn taking works. Both reach the same point: for floor management across multi-agent settings, planned architecture is required. Implicitly, both reject the idea that it emerges when individual agents simply guess right.

What training data and benchmarks for multiparty turn-taking actually exist in 2025

This modeling still matters little without usable data for building and checking systems, while multiparty turn-taking remains behind there.

So far, the fullest offering is When2Speak, released in 2025 by Duke University researchers. This synthetic dataset was made using a four-stage pipeline, holding over 215,000 individual items across 16,000 conversations, each featuring between two and 6 speakers in every conversation. Its coverage spans conversational tones of many kinds, and treats each turn as binary: SPEAK versus SILENT. Other datasets fall short: none match it for open access, reinforcement learning built into the training pipeline, or full coverage of all five testing dimensions the writers targeted.

These numbers deserve a closer look. Supervised fine-tuning on When2Speak pushed average Macro F1 scores up 60% across models with 4 billion or more parameters, and the single largest jump hit 120%. That's a real gain, but it comes with a catch: models trained this way stayed stubbornly cautious, missing roughly half of all warranted interventions (a Missed Intervention Rate averaging 0.50), even at the larger model sizes. Supervised fine-tuning teaches networks to seem plausible, not to actually do anything. Asymmetric reward shaping in reinforcement learning, where the setup penalizes it more for keeping quiet at the wrong times compared with occasionally speaking out of turn, cut that rate to somewhere in a 0.186 to 0.218 range, and lifted detection from 0.479 up to somewhere around 0.78 to 0.81. That gap between the supervised fine-tuning numbers and the RL numbers is the whole argument for why reward shaping belongs in the training pipeline, not as an optional add-on.

DiscussLLM works along similar yet different lines: silent-token supervision is paired with synthetic conversation creation, showing a system when it should jump into a talk. It now defines intervention types suited to training: correction of factual errors, defining an idea the participants appear to lack, supplying data, pointing to a reference, or synthesizing the discussion up to far. It's narrower, working like an information-quality checklist rather than a broad framework for conversational timing, compared to When2Speak.

With both of these in place, real datasets exist alongside real benchmarks, something lacking as recently as two or three years ago. What's still missing is an agreed-upon test each new multiparty model runs through, and that gap is what most holds the field back, more than any modeling issue. Until the field has one benchmark, checking each paper's claims against another's remains imprecise, however strong one individual system's results seem alone.