Acoustic Echo Cancellation in Full-Duplex Voice AI Systems
How voice AI stops hearing itself, from DSP filters to neural models.

- Written by
- The Voice Layer Editors
- Published
- November 11, 2026
- Reading time
- 8 min read
- Sources cited
- 3 sources ↓
What this covers
- Full-duplex defined: both parties (or human and AI agent) listen and speak simultaneously, with no explicit turn boundaries
- Contrast with turn-based systems: user speaks, system processes, system responds — each step cleanly separated, echo not a real problem
- In full-duplex, the AI's own speaker output bleeds back into its microphone — that returned signal is acoustic echo
- Echo is not just annoying noise; it actively corrupts the microphone signal the system depends on to hear the user
- Double-talk: the hardest sub-case — both sides speaking at the same time — is precisely when natural conversation is most alive (interruptions, backchannels, affirmations)
- Barge-in as a concrete stakes example: if a user interrupts the AI mid-sentence and the echo canceller fails, the system either mishears the interruption or treats its own voice as the user speaking
- Sub-100ms response time is now a standard expectation — latency budget leaves almost no room for heavy processing before a decision must be made
- Frame the rest of the piece: AEC is the foundational problem that must be solved before anything else (turn-taking, ASR, VAD, barge-in) can work reliably
What acoustic echo is, physically and mathematically
- Basic signal path: AI plays audio through a loudspeaker → sound travels through room → some reaches the microphone → microphone captures it alongside the user's voice
- The microphone signal y(n) = near-end speech s(n) + additive noise n + echo component d(n) — where e is the far-end reference signal convolved with the acoustic echo path response h
- Echo path h is the room's impulse response: it encodes reflections, absorption, distance, speaker/mic placement — unique to every physical environment
- Classic linear assumption: if h were fixed and linear, a matched filter could subtract e cleanly
- Why that assumption fails in practice:
- Room acoustics change as people move, doors open, temperature shifts
- Loudspeaker and amplifier electronics introduce nonlinear distortions — compressed spectrum, clipping
- Mobile hardware adds device-specific nonlinear distortions not present in the training environment
- Time-varying latency between reference signal and microphone capture: can range from milliseconds to hundreds of milliseconds depending on hardware
- Nonlinearity matters: a linear filter cannot subtract a nonlinearly distorted echo — it leaves residual artifacts that downstream ASR and VAD treat as speech
- PESQ and ERLE as the field's primary quality metrics: PESQ captures perceptual speech quality during double-talk; ERLE measures echo suppression during single-talk — two different failure modes requiring separate evaluation
How classical DSP-based AEC works and where it breaks down
- Adaptive filter approach: estimate the echo path h using the known far-end reference signal, then subtract the estimated echo from the microphone signal
- Core mechanism: the filter updates its coefficients continuously using algorithms (e.g., LMS, RLS variants) to track changes in the acoustic environment
- Works reasonably well when: room is stable, loudspeaker is linear, latency between reference and mic is fixed and known
- Documented failure modes:
- Fast time-varying conditions — the filter cannot converge before the echo path has already changed
- Nonlinear distortion — the echo is not a linear function of the reference, so the linear filter cannot remove it
- Double-talk — near-end speech correlates with nothing in the reference, causing the filter to misadapt and suppress the user's voice instead
- Hardware-induced latency variation — unknown or shifting delay between reference and mic signal misaligns the filter tap
- PSTN/telephony context: traditional telephony networks assume and rely on endpoint or network-level AEC; at 8 kHz PSTN audio quality, both the problem and the solution are degraded versions of what modern voice AI needs
- The gap DSP-based methods leave is precisely the conditions that define real-world deployment: mobile hardware, varied rooms, simultaneous speech
How neural AEC learns to cancel what filters cannot
- Core insight: instead of explicitly modeling h, a neural network learns to map the corrupted microphone signal + far-end reference → clean near-end speech, absorbing nonlinearity implicitly
- Operating in the complex spectrum domain: STFT-based feature extraction captures both magnitude and phase; compressed spectrum features (real and imaginary parts of compressed spectrum) have proven helpful for AEC tasks
- What the network must learn simultaneously:
- Linear echo: the component a DSP filter could in principle remove
- Nonlinear residual: distortion introduced by electronics and room physics
- Noise suppression: background noise that is neither echo nor near-end speech
- Speaker separation: in multi-speaker scenarios, which voice is the target
- Parallel / multi-task structure: separately estimating noise and echo using sequentially connected networks, then removing both integrally — avoids letting one suppression task interfere with the other
- Deep learning resolves the nonlinear echo path from low-dimensional space into linear high-dimensional space — a conceptual reframe of what the model is doing geometrically
- Hybrid DSP + neural pipeline: many production systems run a linear AEC (LAEC) module first using adaptive filter algorithms to suppress the bulk of linear echo, then pass the LAEC output, microphone signal, and reference signal jointly into a neural post-filter — the neural stage only has to handle what DSP leaves behind
- Sequence modeling matters: architectures using dilated convolutions can utilize many frames of historical context; research (GTCNN work in S2) shows this outperforms LSTM alternatives for this task
Speaker embeddings and the personalization layer in AEC
- Core idea from the PAEC system (S2): enroll speech snippets from one or both speakers (near-end and/or far-end), with enrolling both being a special case, extract speaker embeddings, and condition the AEC network on those embeddings
- Speaker encoder: a lightweight speaker recognition network (fast ResNet) extracts utterance-level embeddings from enrolled snippets; embeddings are projected to compact representations and concatenated into the network's sequence modeling blocks
- What speaker conditioning enables:
- Distinguish target near-end speech from echo of the far-end speaker — two voices that may have similar timbral properties
- Suppress interfering speech (a third speaker in the room) separately from both echo and background noise
- Avoid over-suppression of the target speaker during double-talk, which is the failure mode that makes conversations feel clipped or robotic
- "Knowing who you are talking to" framing from S2: speaker identity information improves DNN-based AEC performance, and both near-end and far-end identity contribute — knowing who is speaking now matters more than knowing the other party, but both help
- Practical caveat: requires enrollment — a design constraint for consumer voice AI systems where users cannot be assumed to have registered voice profiles
- Enrollment-free configurations: the same architecture can run without speaker embeddings (unconditional mode), trading some suppression precision for deployment simplicity
The alignment problem that hardware makes worse
- Alignment defined: for echo cancellation to work, the network must compare the microphone signal and the far-end reference signal at the same moment in time — but hardware introduces unpredictable delays between them
- Time-varying latency: in real-world full-duplex spoken dialogue systems, latency between reference and mic ranges from milliseconds to hundreds of milliseconds, and it shifts during a session
- Energy disparity: different hardware stages (amplifiers, codecs, room acoustics) change the amplitude relationship between the reference signal and the echo that actually arrives at the mic
- Why this breaks neural networks: implicit alignment — learned during training — cannot compensate for persistent shifts that differ from what the model has seen; misalignment distorts feature fusion and amplifies residual artifacts
- Heterogeneous hardware context: consumer and enterprise voice AI runs on a wide range of devices — phones, smart speakers, USB peripherals, telephony endpoints — each with its own latency profile and nonlinear characteristics
- The three streams that must be aligned: the raw microphone signal, the far-end reference signal, and the LAEC output — each carrying different information, each subject to its own hardware-induced distortion
LMPAN: what a lightweight on-device AEC system actually looks like in 2026
- From Alibaba's Qwen Business Unit of Alibaba; accepted to INTERSPEECH 2026 (arXiv:2607.02062, submitted July 2026)
- Target deployment context: on-device inference in full-duplex spoken dialogue systems — phones and edge hardware, not server-side processing
- Size: 480K parameters and 126 MACs — achieves performance comparable to the state-of-the-art lightweight model DeepVQE-S while ensuring real-time inference capability
- Four synergistic innovations:
- Multi-path alignment stage: explicitly corrects temporal and energy mismatches across all three input streams (reference, LAEC output, microphone) before feature fusion — addresses the hardware-induced problem that implicit learning cannot solve
- Attention-based adaptive fusion: dynamically integrates enhanced LAEC features and microphone features without requiring auxiliary VAD or double-talk detectors — adapts to changing acoustic conditions without extra modules
- Post-filtering with dynamic target generation: preserves speech integrity and prevents over-suppression, keeping the output useful for downstream ASR and VAD — not just clean audio, but audio that downstream components can act on correctly
- SSL-guided two-stage training: uses WavLM self-supervised learning representations to train for perceptual fidelity, not just signal metrics — the model learns what sounds good to a listener and to a speech recognizer
- Why the multi-path alignment is architecturally novel: prior attention-based frameworks performed implicit alignment; LMPAN makes alignment an explicit, dedicated stage with its own learnable parameters
- Downstream task orientation: the dynamic target generation strategy is designed so that what comes out of LMPAN is optimized specifically for ASR and VAD accuracy — AEC is not the end goal, it is a precondition for the voice AI tasks that matter
The mobile-specific challenge and how small-footprint models address it
- Paper: Jiang & Tian, "A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions," accepted ICASSP 2025 (arXiv:2508.07561, submitted August 2025)
- Mobile-specific challenges the paper targets:
- Varying hardware across device manufacturers and models
- Nonlinear distortions from mobile speaker/mic components
- Long and variable latency between reference and microphone streams
- Three methodological contributions:
- Diverse data augmentation: simulates the range of hardware and acoustic conditions so the model generalizes rather than overfitting to a single device profile
- Progressive learning: incrementally improves AEC effectiveness across training stages, with demonstrated improvement in speech quality metrics
- Tailored post-processing for downstream tasks: different parameter configurations for VAD versus ASR — recognizing that the optimal AEC output for detecting speech presence differs from the optimal output for recognizing speech content
- Evaluation confirms improvement in both ERLE (single-talk echo suppression) and PESQ (perceptual quality in double-talk), plus measurable gains in VAD and ASR accuracy downstream
- Streaming inference design: model processes audio frame-by-frame without buffering a full utterance — necessary for real-time interaction at mobile latency budgets
- Common thread with LMPAN: both papers treat ASR and VAD performance as the real evaluation target, not just audio quality scores — reflects the field's understanding that AEC is infrastructure for voice AI, not an end in itself
How the field measures progress: the ICASSP AEC Challenge benchmarks
- The ICASSP Acoustic Echo Cancellation Challenge series, led primarily by Microsoft Research, is the field's central public benchmark for comparing AEC approaches
- Dataset scale: recordings from more than roughly ten thousand real audio devices and human speakers in real environments, plus a synthetic dataset — the scale matters because real devices are the source of the hardware variation that breaks classical methods
- Evolution across challenge editions:
- Added mobile scenarios as a distinct track — recognizing that phone deployments have different distortion profiles than desktop/conferencing setups
- Added word accuracy rate (ASR metric) alongside audio quality metrics — same philosophical shift as LMPAN and Jiang et al.: downstream task performance is what counts
- Audio upgraded to 48 kHz as an important requirement for many scenarios
- ICASSP 2023 challenge reduced algorithmic + buffering latency budget from 40ms to 20ms — directly tied to the real-time collaboration requirement
- The two largest persistent improvement areas across challenge editions:
- Single-talk near-end quality: background noise, reverberation,
Methodology & sources
- https://arxiv.org/pdf/2205.15195
Provided the personalized AEC design with speaker embeddings, enrollment options and the GTCNN-style sequence modeling discussed in the article.
- LMPAN: A Lightweight Multi-Path Alignment Network for Joint Full-Duplex Acoustic Echo Cancellation and Noise Suppression
Supplied the LMPAN details: parameter count, multi-path alignment, attention fusion, post-filtering and SSL-guided training.
- A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions
Described the small-footprint mobile AEC approach, including data augmentation, progressive learning and task-specific post-processing for VAD and ASR.