How Voice Agents Persist Session State Across Multi-Turn Conversations
Voice agents must store conversation memory outside the LLM to handle multi-turn dialogue.

- Written by
- The Voice Layer Editors
- Published
- October 19, 2026
- Reading time
- 9 min read
- Sources cited
- 2 sources ↓
What this covers
Voice agents persist session state by keeping it outside the LLM and supplying the right parts on each turn. For multi-turn dialogue to make sense, agents need layered memory, structured dialogue state, and a way to keep speech in step with ongoing tasks. A larger context window cannot replace that architecture, because continuity is a property of the running system.
Why LLM inference resets on every call, and what that means for voice
LLM APIs treat each inference call as independent and retain nothing from prior turns unless the application supplies it again. Model weights are static snapshots of training data. A large context window increases what an application can provide at once, but it offers no recollection of a conversation that ended yesterday. The human experience is continuous while the inference engine repeatedly starts without that history.
Voice makes this mismatch harder to manage because context reconstruction competes with speech processing for a narrow latency budget. After speech-to-text and text-to-speech take their share, little time remains to assemble context and generate a response. Rasa's production evaluation notes that latency above one second feels like incompetence to callers, and that voice punishes mistakes chat forgives. The industry's answer is to externalize state into layered stores that outlive any single LLM call.
The three-tier memory architecture that compensates for statelessness
The dominant memory pattern separates information by how soon it is needed and how long it must survive. The first tier is the in-context window, which contains the current dialogue, recent tool outputs, and active slot values assembled into the prompt. This information is immediately available to the model, but the window is finite and provides no persistence after the inference call ends.
The second tier is a session store, such as Redis, that holds conversation context, active tasks, and recent tool outputs throughout a session. Fast retrieval lets the application reconstruct the next prompt without treating each inference call as a new conversation. The third tier is a long-term store, such as a vector database, that holds summaries of past interactions and what the agent has learned about user preferences. Semantic similarity can find relevant information from the past without an exact key lookup. Across these tiers, the tradeoff runs from the fastest, shortest-lived context to slower retrieval from durable memory.
The architecture therefore needs a selection policy as well as storage. Too little context breaks coherence, while too much consumes time that the voice interaction cannot spare. Loading all long-term records at session start can undermine the latency benefit of the voice architecture itself. The engineering discipline lies in deciding what to inject for this turn, what to promote into longer-lived memory, and when to evict stale state.
How dialogue state tracking structures what is said within a call
Within a session, storing the conversation is not the same as understanding its current meaning. Dialogue State Tracking, or DST, builds and updates structured slot-value pairs that track what the user wants and the constraints they have set across turns. If a caller says “actually make that aisle” three turns later, the useful operation is a slot update. The structured state records the current request even when the wording that established it is scattered across the conversation.
DST methods include discriminative, generative, and schema-driven models, using attention, memory, and transfer learning to keep values current. However, the choice of method does not remove the need for a precise contract between agents. An agent writing customer_id as the string "12345" and another reading customerId as the number 12345 disagree on both name and type. That contract bug can force the customer to repeat information already provided. A shared schema definition in code gives every agent the same fields and types, so compatibility no longer rests on an informal agreement about the design.
Multi-agent hand-offs also need the full transcript, structured slots, and a hand-off context note, each with its own role. These preserve the original exchange, its current constraints, and the context needed to continue, rather than asking one representation to serve every purpose. τ-Voice research attributes the majority of voice agent failures to agent behavior, with ASR and the environment accounting for fewer failures. That finding makes careful use of DST part of designing the agent as well as building the infrastructure. Even when storage works well, it cannot fix an agent's mistaken reading of what was said.
Voice-specific state: interruptions, barge-in, and in-flight tool calls
Voice adds events that a text-only conversation does not have to represent. The agent may be speaking while the caller starts a new utterance, with speech output and LLM generation still in progress. The state envelope must track whether the agent was speaking, its last complete or partial utterance, tool calls that are still unresolved, and how many interrupts the session has had. These fields distinguish what the agent intended to say from what it actually delivered.
When a caller interrupts, the in-progress utterance needs a flag showing that it was cut short. The next LLM turn can then decide whether to resume, repeat, or start fresh. Similarly, a tool call that has fired but not resolved remains part of the state even though no result is yet available.
Barge-in handling must stop TTS output, cancel or stash in-flight LLM generation, and preserve what was said within a tight latency budget. Production implementations allocate only tens of milliseconds to voice activity detection, stopping speech, and cancelling generation. Yet stopping output is only the visible part of the operation. Naive implementations often omit preservation of the interrupted utterance and unresolved tool calls, forcing callers to repeat context the system already held.
Foreground–background coordination as a solution to concurrent speech and task execution
When work takes longer than a conversational turn, the system needs more than event tracking. The architecture must separate the real-time conversation from durable task execution so that speech can remain responsive while work continues. Tiered memory supplies context, DST represents the current request, and interrupt state records what happened during delivery. A coordination runtime connects those records to tasks whose lifetimes do not match conversational turns.
Qwen-Audio-Agent implements this separation through a Frontend Agent, a Backend Agent, and an Orchestration Runtime. The Frontend Agent handles the dialogue and chooses whether to make direct tool calls or use backend delegation. The Backend Agent performs delegated work in its own context, while the runtime maintains task state, coordinates requests for user input and authorization, and schedules result delivery. The Qwen-Audio-Agent Technical Report reports a task success rate of 91.04% for mixed direct-plus-delegated execution on an in-house cockpit benchmark, outperforming direct-only and all-delegated configurations.
When a caller interrupts the agent, a database query already sent by the backend keeps running. If a result arrives while the caller is speaking, the runtime retains it until an appropriate conversational moment. Execution completion and conversational delivery are separate state transitions.
The Talker-Reasoner split addresses the same problem from the response-latency side. ConvFill pairs a small on-device Talker with a frontier Reasoner, allowing grounded filler responses while retrieval and reasoning proceed. The Talker integrates streamed Reasoner output into its ongoing response, which requires a representation of the background work. ConvFill's live user study found that participants ranked it on par with frontier models overall, preferred it for retrieval-heavy tasks, and rated it significantly more responsive.
Cross-call persistence: carrying state beyond a single session
Even with coordination within a session, the LLM starts with no history when a caller returns. Cross-call persistence must retrieve useful history before the first turn and preserve new information after the conversation ends. In the webhook pattern, a pre-call initiation webhook lets a service look up a returning caller in a memory store. The service injects the retrieved context as a dynamic variable into the system prompt, while a post-call webhook receives the full transcript for storage.
This retrieve-on-entry, persist-on-exit round trip is the minimal cross-call memory loop for teams wiring their own storage. However, retaining a transcript and reconstructing useful context are different tasks. Summarization, the most common long-term memory approach, reduces the amount to retrieve at the cost of specificity. The precise preference “aisle seat near the front” may become only “user discussed seat preferences”. That summary records the existence of a preference without retaining enough information to honor it.
Richer summaries and larger windows therefore do not by themselves establish identity continuity. Rasa's Memory layer addresses continuity across both channels and sessions, allowing a billing issue raised in chat to resume with a voice agent the next day without repetition. That capability extends the memory problem beyond reconnecting two telephone calls to preserving context across different forms of interaction. Across platforms, though, agents still have no standard way to share state or hand a live session to another vendor's agent.
How platform architecture shapes what state a team can actually manage
A platform determines which parts of this architecture a team can control. Rasa Voice runs in the customer's own environment, so teams can control where voice data flows. Its Memory layer and Orchestrator keep context across voice and chat, so cross-channel continuity is built in without the team having to put together a webhook integration. For enterprises in financial services, healthcare, and government that cannot place call data in a vendor-hosted environment, self-hosted deployment is a hard requirement. In those settings, control over deployment belongs in the memory architecture as well as in procurement.
Other platforms address different constraints in the voice system. Cognigy, now NiCE Cognigy, offers a built-in voice gateway, support for a wide range of languages, and cloud and on-premises deployment for contact center voice. Parloa began as a purpose-built contact center voice platform rooted in the European and DACH market and is expanding globally as an AI Agent Management Platform. SoundHound provides a proprietary speech-to-meaning engine designed for speed, relevant when ASR and intent recognition latency are the binding constraint. These capabilities address different parts of the system and do not, by themselves, establish identical access to memory or task state.
Foreground–background coordination specifically depends on exposed runtime primitives that manage tasks separately from conversational turns. Qwen-Audio-Agent releases its orchestration runtime as open source, making that coordination layer inspectable and adaptable, unlike an opaque hosted runtime. Platform evaluators should ask whether speech interruption is decoupled from task cancellation and whether tool-call state survives barge-in. They should also ask whether the memory layer can work with custom storage backends.
What cascaded pipelines lose that speech-to-speech architectures preserve, and the gap neither solves
Even with full control of memory and task state, a system cannot bring back signals the pipeline has already thrown away. In a cascaded ASR→LLM→TTS pipeline, transcription removes paralinguistic information before the LLM receives the caller's words. The text alone does not tell the model whether the caller sounded angry, hesitant, or relieved. Semantic state mechanisms can retain what was said while leaving tone and environmental conditions implicit in raw audio that most state stores never receive. The ConvFill paper also identifies compounded latency across separate speech recognition, generation, and synthesis stages as a source of gaps in conversational flow.
This loss matters across turns because people match each other's speech rate, pitch, and rhythm as they talk, a process known as conversational entrainment. A reply much slower than the caller's speech can disrupt that rhythm and reduce engagement. Yet current production platforms do not systematically track or persist entrainment, or expose a paralinguistic state field for the agent to use.
Speech-to-speech models reduce this loss by operating directly on audio, but real-time constraints on a single LLM backbone limit flexibility for tool use and retrieval. ConvFill's Talker-Reasoner design responds to this tradeoff between latency, capability, and paralinguistic fidelity. Qwen-Audio-3.0-Realtime Plus with the Qwen-Audio-Agent architecture points toward another combination, pairing full-duplex voice with an asynchronous backend. Neither direction makes every relevant signal persistent simply by preserving audio during inference. Builders should know which of those signals their pipeline captures, while engineers still need to work out how to track paralinguistic state across turns in a systematic way.
Methodology & sources
- Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
Provided details on the ConvFill Talker-Reasoner architecture, its live user study results, and the compounded latency problem in cascaded pipelines.
- Qwen-Audio-Agent Technical Report
Provided the Frontend Agent/Backend Agent/Orchestration Runtime architecture and the 91.04% task success rate benchmark result.