Phoneme-Level Timestamp Alignment in Speech Recognition APIs
Phoneme-level timestamps require a separate alignment model, not just better transcription.

- Written by
- The Voice Layer Editors
- Published
- October 9, 2026
- Reading time
- 10 min read
- Sources cited
- 8 sources ↓
What this covers
- The two architectural lineages that approach the alignment problem differently
- What the comparative evidence shows about which approaches achieve tighter boundaries
- Where the standard pipeline breaks: disfluency, translation, and LLM-generated timestamps
- The emerging frontier: phoneme-level neural alignment and what it changes
Aligning timestamps to individual sounds is its own engineering challenge, distinct from transcription, though marketing for speech APIs constantly blurs this line. Pinning down precisely where each sound begins and stops in a recording takes a specialized model, its own training approach, and typically a separate tool in today's production pipelines, rather than the system that handled the speech-to-text conversion.
Transcription is just asking a model to figure out which words someone said. Alignment, on the other hand, asks for the exact millisecond the audio switched between consecutive phonemes. A system can drive word-error rate down to nearly zero and still miss those timestamps, since the optimization target never incentivizes pinpointing where sounds change. Fully integrated ASR pipelines typically yield timing markers merely as a decoding artifact, never because their optimization objective demanded accurate temporal boundaries.
The challenge lies partly in the speech signal, not only in how it is modeled. In theory, a phoneme boundary marks the changeover between articulatory actions: a tongue gesture, voicing switching on or off, and a reshaped airstream. In actual speech, those changeovers smear together. Because of co-articulation, adjacent phonemes color each other acoustically, so hand-labelers working on the same clip often place the boundary in different spots. Even the label the system is learning against has no sharp contour.
Precision needs change with the application. In subtitle syncing, boundary errors of a hundred milliseconds or even more often go unnoticed. For prosody studies, detailed phonetics, and lip-sync-based dubbing, even boundary mistakes of just tens of milliseconds can be significant. Whether alignment is "Good enough" depends on the downstream task, which also explains why forced alignment remains its own specialty alongside mainstream ASR.
The two architectural lineages that approach the alignment problem differently
Two bets tackle this problem from opposite sides, and that divide explains most of what developers notice when choosing a tool.
A more traditional branch is found in forced-alignment tools whose acoustic machinery uses Gaussian Mixture Model and Hidden Markov Model methods, with Montreal Forced Aligner as the best-known case. Their GMM-HMM sound models rely on Viterbi search to treat speech as ordered phoneme-state progressions over time. Here, both the loss function and the way decoding is done are aimed at alignment, because the design was made to put boundaries in the right place, not to work out which words were said. They also start from a different point: the transcript is already known to be right, and the tool lines it up with the audio. That task is more limited than transcription, and this tighter focus lets the systems specialize.
This newer lineage stems from end-to-end neural ASR systems, resolving the tension between transcription and timing objectives through a dual-model approach where one generates the transcript and another manages alignment. WhisperX shows this most clearly: Whisper generates the transcript, which then passes to a distinct wav2vec2.0 phoneme classifier that determines the timing. Splitting the task so one model handles transcription and another handles alignment resolves the objective mismatch head-on. WhisperX outputs stop at word-level granularity rather than phoneme-level detail, so those requiring sub-word accuracy must account for this limitation upfront.
Alongside these two, a third approach has appeared, and it brings a failure mode of its own. When LLM-based ASR systems emit timestamps while predicting the next token, nothing built-in forces those timestamps to arrive in sequence. Because the model generates each token sequentially rather than exploring a constrained space of legal alignments, a later word may end up with a timestamp that precedes an earlier one's. Researchers dub the issue temporal non-monotonic hallucination: a flaw only this architecture suffers, since neither classical aligners nor a Whisper-plus-wav2vec2.0 pipeline face it in the same manner.
The three lineages differ in the priority each is built to serve. The classical lineage buys boundary precision but needs a verified transcript first. Split-model neural methods are more adaptable but still hit a ceiling at word level. LLM-native methods make the setup simpler, but still cannot reliably preserve sequence. None of the three wins on its own, and each choice shows its cost once the systems are tested directly against each other.
What the comparative evidence shows about which approaches achieve tighter boundaries
Direct comparative evidence confirms the architectural split's prediction, with the classical lineage performing better than neural ASR marketing suggests. At Interspeech 2024, Rousso's team compared HMM-based alignment with word-level timestamps from WhisperX and MMS, finding MFA superior across all tested tolerance thresholds. Version 3.0 of MFA advances this performance even more. Its evaluation spanned three languages, where it surpassed both aligner types in the majority of comparisons while keeping mean boundary errors below 15 milliseconds.
Those figures belong with the setup that produced them, not in a universal leaderboard. Alignment accuracy can only be judged against a reference set, with humans supplying the ground-truth labels used to check the system's output. Large annotated corpora of this sort are usually available only for clean English read speech, the very setting classical aligners were designed around. Because the test data sit so squarely inside MFA’s comfort zone, topping that evaluation alone says little about its behavior, or any system’s, on messy unscripted speech across languages.
Measurement limitations are beginning to narrow. Because both PCMI alongside WACS were confirmed across 85 tongues within FLEURS, gauging alignment merit no longer demands manually labeled ground truth. This enables evaluation of informal dialogue and under-resourced audio scenarios far beyond the reach of manual labeling. Whether traditional methods retain their advantage under such expanded testing is a question researchers are finally equipped to resolve.
Where the standard pipeline breaks: disfluency, translation, and LLM-generated timestamps
Performance may look strong for pristine scripted English speech, but it weakens in the conditions production teams see all the time. The benchmark results above rest on three assumptions, each one broken by a different condition, and production teams are least ready for the failure mode specific to LLM-native systems.
The presence of disfluencies undermines any expectation of an exact word-for-word transcript. Conventional alignment tools expect a perfect match between spoken sounds and written text, yet everyday talk constantly introduces stutters, restarts, and filler sounds that immediately violate this requirement. At Interspeech 2023, Kouzelis with coauthors tackled this mismatch by designing a weakly supervised phoneme-level alignment method to handle situations in which spoken audio diverges from its written record. Other studies frame documentation guidelines as variables requiring explicit modeling instead of predetermined rules, since choosing between literal and cleaned-up records alters the input available to subsequent alignment systems, making such decisions deliberate.
The idea that timing error remains uniform from one task to another collapses once translation enters the pipeline. Studies of word-level timing that used NVIDIA's Canary ASR platform found a clear pattern: moving from single-language transcription into automatic speech translation yields substantially greater timestamp inaccuracy, while measurable declines appear in both COMET and BLEU results. Timestamps produced after translation are less accurate than those from single-language ASR, a decline with practical weight in dubbing work that depends on exact phoneme-level alignment to drive lip synchronization and viseme mapping. Errors in translation and errors in alignment compound one another.
When LLMs generate timestamps natively, the pipeline can no longer presume chronological output; in production, that failure follows from the design flaw noted above. At ACL 2026, LLM-ForcedAligner identifies the central problem directly: when speech-language LLMs process timestamp tokens like text, later outputs can be stamped with times that jump backward ahead of earlier ones. Its remedy switches to a nonsequential decoding scheme, cutting average timing shift sharply versus autoregressive baselines while accommodating long-form audio runs of as much as 300 seconds. For teams whose production systems depend on ordered timestamps for correctness rather than polish, backend architecture should be treated as safety-relevant, not merely a performance choice.
The emerging frontier: phoneme-level neural alignment and what it changes
Two systems released in 2026 push the frontier beyond the ceiling on word-level precision that has long defined neural alignment output, yet neither has faced a comparative benchmark as broad as Rousso's analysis or MFA 3.0's evaluation.
First comes Fully Differentiable Neural Forced Alignment using Soft Dynamic Programming. MMS, along with WhisperX plus NVIDIA's Canary, still stops at words when producing its finest alignment detail. The method instead tackles alignment below the word, with the authors saying they know of no earlier neural forced-aligner built to operate on phonemes. A learnable decoder driven by soft differentiable dynamic programming lets the system optimize the whole pipeline with contrastive training that separates stable phoneme spans from their boundary shifts. In practice, the design tackles the opening mismatch head-on by making boundary precision, not word accuracy, part of the model’s training objective from the outset.
LLM-ForcedAligner is the other one, already covered above for how it handles monotonicity. What it brings to the frontier is as much about how it's built as what it achieves: a decoding scheme that lets LLM-based alignment approach the dependability classical methods already enjoy, while preserving the adaptability an LLM-based pipeline offers.
Until either approach faces the same sort of head-to-head evaluation that yielded the findings of Rousso and colleagues and of MFA 3.0, assertions of general-condition superiority over classical alignment stay unproven. The reference-free measures introduced above, among them PCMI together with WACS, were validated across 85 FLEURS languages and thereby eliminate the need for gold annotations, the obstacle that has long made multilingual alignment evaluation impractical. The significance is not limited to one system: for the first time, alignment quality can be gauged across many languages at once, which should speed work on under-resourced tongues and reveal weaknesses that English test sets built from clean corpora were never equipped to detect. Projects such as OpenBibleTTS, having built large speech datasets and TTS models for 37 languages with little representation, suggest the scale of this frontier beyond the well-resourced pairs that have so far absorbed most alignment research.
What commercial ASR APIs currently offer and where the phoneme gap sits
Commercial speech APIs now make per-word timing the default offering. Today, none of the leading providers includes supported phoneme-level output.
AssemblyAI's Universal-2 model handles transcription across many languages and returns word-level timestamps in its output. In April 2025, the firm launched its speech-language model Slam-1 as a public beta, adding protective content filters, an LLM Gateway, and streaming across several languages. Rivals like Deepgram plus major cloud platforms (Amazon, Azure, and Google) include word timing per token by default in their transcription offerings without extra setup. None of these providers natively output timestamps down to the phoneme.
Today, phone-by-phone timing still requires leaving commercial API products behind. Instead, teams turn to aligners such as MFA, NVIDIA's NeMo Forced Aligner, or WhisperX, noting that WhisperX natively reports only word-level timing unless it is extended. For developers, phoneme timing therefore means either post-transcription alignment in a separate step or wiring MFA into the workflow, with either route adding real complexity that commercial APIs have not absorbed for anyone's use.
Researchers are working on that limit by changing the architecture itself. The 2026 release of In-Sync adapts LLMs attuned to spoken input for ASR by embedding timestamp forecasting at the word level directly within the model. This is one of several signs that the field wants to bake tighter alignment directly into neural architectures. Relying solely on post-hoc fixes for alignment is no longer the only approach. Commercial offerings still lack phoneme-level precision from that research, meaning practitioners continue relying on the previously mentioned tools and their associated intricacies.
How to choose and implement the right alignment approach for a given use case
A sound choice starts with three checks, taken in sequence: the required output granularity, the nature of the audio conditions, and the acceptable amount of pipeline complexity. If convenience leads a team to choose a commercial API before they verify the need for phoneme-level detail, the mismatch may emerge only after voiceover localization or speech-sound analysis is in progress.
Granularity is the first question to settle. If timing at the word level suffices, the commercial APIs with the least friction are AssemblyAI, Deepgram, Amazon, Azure, and Google, while WhisperX steps in if the timestamps those APIs deliver lack the needed precision and running forced alignment on top is worthwhile. When phoneme-level timing is what is needed, MFA remains the go-to, still posting the best boundary accuracy in published benchmarks; otherwise one of the newer neural options covered earlier works, giving up a little precision at boundaries in return for greater pipeline flexibility.
Audio quality is the next factor. Traditional tools such as MFA, built on HMM-GMM architecture, are most thoroughly validated and perform strongest on clear, scripted recordings. A weakly supervised method from Kouzelis et al. along with other approaches built to handle mismatches between text and sound, better suits unscripted or disfluent input. Applying conventional alignment to disfluent speech fails silently. It returns believable but misplaced boundaries, a failure mode more dangerous than an obvious crash. In multilingual and low-resource scenarios, current WACS and PCMI methods can estimate quality without requiring gold-standard labels. MFA 3.0 also matters here because it reaches more languages and dialects through standardized IPA lexicons and phone mapping across languages. When translation is part of the pipeline, the error budget has to be larger, since it produces measurable losses in later timestamp precision.
Third comes pipeline complexity, the point at which the trade-offs turn concrete. MFA wants an accurate transcript prepared ahead of time, a lexicon of pronunciations, and its own separate alignment stage, three prerequisites that commercial APIs do without. WhisperX merges its Whisper and wav2vec2.0 pairing into what feels like one step, though results never go below word level, and since wav2vec2.0's alignment model lacks universal language support, verify each language before adopting it in production.
Each approach fits a specific mix of the precision required, the state of the recording, and the processing burden a team is prepared to take on. The mistake the evidence here points away from is treating either as a universal fallback, instead of picking the tool to match the three questions above.
Methodology & sources
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Described WhisperX's architecture combining Whisper transcription with wav2vec2.0 phoneme alignment, and its word-level output limitation.
- Montreal Forced Aligner and the state of speech-to-text alignment in 2026
Provided details on MFA 3.0's performance across languages, mean boundary errors below 15 ms, and its use of standardized IPA lexicons.
- Word Level Timestamp Generation for Automatic Speech Recognition and Translation
Supplied findings on timestamp accuracy degradation when moving from single-language ASR to speech translation, including COMET and BLEU metric declines using NVIDIA's Canary platform.
- OpenBibleTTS: Large-Scale Speech Resources and TTS Models for Low-Resource Languages
Cited as evidence of large-scale speech dataset construction for low-resource languages, illustrating the scale of the alignment frontier beyond well-resourced language pairs.
- In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions
Described the In-Sync system that adapts speech-aware LLMs for ASR by embedding word-level timestamp forecasting directly within the model architecture.
- LLM-ForcedAligner: A Non-Autoregressive and Accurate ...
Identified the temporal non-monotonic hallucination problem in LLM-based ASR and described LLM-ForcedAligner's non-autoregressive decoding solution, including support for 300-second audio runs.
- A Comparison of Modern ASR Methods for Forced Alignment
Provided comparative benchmark data showing MFA outperforming WhisperX and MMS across all tested tolerance thresholds at Interspeech 2024.
- Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming
Described the Fully Differentiable Neural Forced Alignment method using soft dynamic programming, which targets phoneme-level boundary precision as a training objective.