Sentence Boundary Detection for Streaming TTS Chunk Splitting
How to split streaming text for speech without breaking prosody.

- Written by
- The Voice Layer Editors
- Published
- November 11, 2026
- Reading time
- 8 min read
- Sources cited
- 8 sources ↓
What this covers
- The streaming TTS promise: audio begins before the full input is known, eliminating the wait-for-completion bottleneck
- But synthesis is not phoneme-by-phoneme — it is prosody-aware, and prosody requires context
- A question ends differently from a statement; emphasis in a list depends on the whole list
- Committing to the first word with no lookahead leaves intonation wrong by the time the sentence resolves
- Per Soniox's technical wiki: streaming would be free if a voice could begin from the first phoneme with no context — it cannot, because of prosody
- The fundamental curve every streaming TTS team must choose a point on: lower time-to-first-audio vs. better prosody
- Chunking too small (word or two-word level): synthesis engine lacks context, voice sounds choppy
- Waiting for a full natural boundary: prosody improves, but first-audio is delayed
- The usual compromise: chunk at sentence or phrase boundaries — gather enough text to read one unit well, start speaking it, generate the next unit while the first plays
- This tradeoff is not incidental — it is the central design decision in every streaming TTS pipeline, and every subsequent engineering choice flows from it
The latency budget that makes split strategy urgent
- Users begin to perceive conversational AI as unnatural at response delays beyond 500–700 ms, per AssemblyAI's voice agent architecture guidance
- What a well-budgeted pipeline looks like end-to-end
- Twilio's engineering allocation: 350 ms for STT, 375 ms for LLM first token, 100 ms for TTS first byte — targeting a 1,115 ms median mouth-to-ear gap
- Retell's production stack runs around 600 ms end-to-end by streaming every stage into the next rather than waiting for completion
- A polargrid.ai sub-400 ms pipeline on Ada Lovelace hardware: 364 ms p50 total — STT 148 ms, LLM 96 ms, TTS 117 ms
- Deepgram's streaming TTS runtime: p95 TTFB under 200 ms at high concurrency
- The threshold matters: above roughly 700 ms callers get awkward, repeat themselves, or hang up; below it, they forget they're talking to AI (per Retell)
- Implication for chunk splitting: TTS first-byte latency is the one stage the chunking strategy directly controls — and it is also the stage most sensitive to how early a flush is triggered
- This budget frames all subsequent design choices: the split strategy must hit the TTS-first-byte target without sacrificing prosody below the perceptibility threshold
The sentence buffer pattern and how production frameworks implement it
- The sentence buffer as the critical bridge between streaming LLM output and TTS input
- Accumulates tokens; yields complete sentences by detecting sentence-ending punctuation (.,!,?) followed by whitespace
- Excludes false positives from abbreviations (Dr., Mr., PM.) and decimal numbers
- Enforces a minimum sentence length (10 characters) to avoid sending fragments to TTS
- Flushes any remaining text when the LLM stream ends
- Per the arXiv paper (2603.05413): this is the "sentence aggregation" pattern used by Pipecat's SentenceAggregator and LiveKit's text processing pipeline
- The overlap strategy this enables: sentence one is being spoken while the LLM generates sentence two — the key latency reduction technique
- Per AssemblyAI: streaming LLM tokens into TTS at sentence boundaries dramatically reduces perceived latency because the user hears the first sentence while generation of the second continues
- Per ElevenLabs latency optimization guidance: starting LLM work on stable STT partials and streaming tokens into TTS at sentence boundaries are among the highest-leverage optimizations (listed first and second, roughly in order of impact, out of a list of five)
- How specific frameworks expose this pattern
- Pipecat's CartesiaTTSService: consumes TextFrame with text_aggregation_mode=TextAggregationMode.SENTENCE (default), flushing to TTS WebSocket at sentence boundaries
- Deepgram Aura (/v1/speak): recommends splitting at sentence boundaries as the simplest and most effective approach for preserving natural speech while enabling faster time-to-first-byte
- Deepgram Flux (/v2/speak): a different model — server places flush boundaries internally; client should stream text as produced and not chunk client-side
- Chunk size guidance for Deepgram Aura by use case: voice assistants aim for 50–100 character chunks; call center bots use complete sentences; long-form content uses larger chunks (200–400 characters) to preserve intonation
- Note the divergence: Flux's server-side flushing is a design direction that shifts responsibility away from the client, which has implications for teams building portable chunking logic
Where rule-based sentence splitting breaks down
- Punctuation-based SBD carries a well-documented assumption: clean, well-punctuated input — per the ACM WWW '21 paper (Donabauer et al.): SBD typically relies on the presence of punctuation, and even with punctuation it can be difficult; without punctuation and casing, even humans struggle to determine sentence boundaries
- Failure mode 1 — decimal and localization bugs (real production issue, hass-edge-tts GitHub issue #76, opened 2026-09-11)
- The streaming path splits on a separator list that includes a comma
- In locales using comma as decimal separator (German, French, Italian, Spanish), every decimal value loses its spoken word: "22,1 Grad" is spoken as "zweiundzwanzig [pause] eins"
- Verified by Whisper transcription of the returned MP3; direct edge_tts call produces correct output, streaming path does not
- Intermittent appearance: which decimal value gets split depends on where the incoming token chunks land
- Workaround in use: instruct the conversation agent to spell out decimal numbers — a prompt-level patch that does not help users who cannot edit the system prompt
- Suggested fix: exclude the pattern (?<=\d),(?=\d) from the separator match; never emit a chunk whose last character is a comma
- Failure mode 2 — numeric false splits (DemaConsulting/Speech PR #27, 2026-09-16)
- Applying clause-level splitting (,/;/:) unconditionally treats punctuation inside numeric or token-like text as a clause boundary
- "The time is 12:30." becomes "The time is 12:" + "30." — two synthesis calls for a single time expression
- "1,000" is split into two calls even though neither punctuation is a clause separator
- Recommended fix: restrict secondary splitting to actual clause separators; at minimum preserve punctuation between digits and URI-like tokens
- Failure mode 3 — oversized sentence overflow and word-boundary fallback (FluidAudio/FluidInference PR #935)
- Kyutai's original PocketTTS implementation cuts an oversized sentence only at,;: and never between words; a piece still over the limit is left whole
- FluidAudio cannot leave such a piece whole because its KV cache is fixed at 512 positions
- FluidAudio PR #935 keeps and improves the existing word-boundary fallback: makes it rarer by recognizing more punctuation, less audible by balancing the parts
- Failure mode 4 — orphan fragment artifacts (FluidAudio PR #935)
- The single-word orphan fix from the PocketTTS text chunker produces audible artifacts on long French text (smart apostrophe + clause-cascade)
- Failure mode 5 — degenerate punctuation-only pieces (DemaConsulting/Speech PR #27)
- Spurious repeated periods ("Sentence one..... Sentence two") produce punctuation-only chunks that become standalone, pointless synthesis calls
- Fix: merge these degenerate pieces into the preceding chunk
- Common thread across all five: rule-based splitting treats punctuation as a proxy for linguistic structure — and that proxy fails systematically at the edges of natural language and internationalization
Secondary splitting strategies and the tradeoffs each introduces
- Clause-based splitting: split long sentences at commas and semicolons
- Deepgram Aura documentation names this explicitly as an advanced chunking strategy
- DemaConsulting PR #27 applies clause-level splitting (,/;/:) unconditionally rather than only past the 200-character length budget — proactively bounding long sentences before they become one large, slow synthesis call
- Tradeoff: introduces the decimal/URI false-split failure modes catalogued above; must be paired with digit-aware exclusion rules
- DemaConsulting measured: VITS-Piper synthesizes at RTF roughly 0.16–0.18 (no audible pauses); Kokoro int8 at RTF roughly 1.5 — confirming remaining Kokoro pauses are a model/compute throughput limit, not a chunking defect
- Ellipsis-aware silence injection: treat genuine ellipsis (... or spaced... variants) differently from normal chunk boundaries
- DemaConsulting PR #27 adds a documented EllipsisPauseMilliseconds constant applied as PostSilenceMs only to chunks ending in a genuine ellipsis; every other chunk boundary keeps zero added silence
- Rationale: ellipsis signals a deliberate narrative pause; treating it the same as a mid-sentence comma removes expressive intent
- Look-ahead buffering: increase pending segment capacity to give the synthesis pipeline more real-time margin
- DemaConsulting PR #27 raises PendingSegmentCapacity from 5 to 8 for extra real-time look-ahead margin
- True concurrent synthesis was investigated and deliberately rejected: ISynthesisEngine is not safe for concurrent Generate() calls (enforced by existing regression test)
- Fixed-context KV cache constraints impose hard upper bounds that interact with splitting strategy
- FluidAudio's KV cache is fixed at 512 positions; the PocketTTS chunk size of 50 tokens means sentences over that size must be cut
- FluidAudio PR #938 (2026-09-19): treats the 50-token PocketTTS chunk size as a grouping preference rather than a hard limit; admits longer sentences using the selected voice's real KV offset plus a speech-duration cache budget; bounds IO and stateful generation loops at the 512-position cache boundary; closes related issue #933 (69-token sentence forcing cuts)
- Voice-dependent splitting: different voices have different KV offsets, so the correct maximum chunk size is voice-specific, not a single constant
- NLP-based and adaptive chunking: mentioned by Deepgram as advanced options — use natural language processing to find semantic boundaries, or adjust chunk size based on content complexity
- In the GitHub ecosystem, tools like yasbd-lib (high-accuracy, rule-based SBD with drop-in adapter for pysbd) offer faster and more accurate segmentation than naive punctuation scanning
- No single off-the-shelf NLP segmenter eliminates the localization and numeric edge cases without domain-specific tuning
- First-chunk optimization: specially optimize the first chunk for minimal latency while allowing subsequent chunks to be larger for prosody — Deepgram documentation names this as a distinct strategy worth considering
The lookahead problem: how much future text the model actually needs
- The core issue: a synthesis model committing to audio before knowing how the sentence ends can produce prosodic decisions that are wrong in context — stress, pitch contour, pause placement
- MambaVoiceCloning (2026) look-ahead window findings
- Look-ahead L provides the next L seconds of mel frames, conditioning the SSM update and preventing premature prosodic decisions when punctuation occurs near the boundary
- Chunk boundaries remain perceptually smooth for L ≥ 0.5 s; only L = 0.25 s shows occasional discontinuities or shortened pauses
- WER for L = 0.50 s: 9.4% vs. 7.3% for L = 2.00 s (Table 5 of that paper) — a meaningful quality gap, though L = 0.50 s is likely acceptable for most conversational applications
- The prosodic-boundary-aware post-training approach (Liu et al., arXiv:2603.06444, 2026)
- Proposes adapting a pretrained LLM-based TTS model using weakly time-aligned data so it learns early stopping at specified content boundaries when given limited future text
- During inference: a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation
- Performance at chunk size k=5 words, lookahead f=2 words on Seed-TTS-Eval:
Methodology & sources
- Streaming TTS splits at commas: decimal commas lose the spoken word ("22,1" is spoken as "22 1") · Issue #76 · hasscc/hass-edge-tts
Supplied the decimal-comma failure case, where '22,1 Grad' is split and a spoken word is lost in streaming TTS.
- Text Chunking for TTS | Deepgram's Docs
Provided Deepgram Aura's chunking guidance, including sentence-boundary splitting, chunk sizes by use case, and the advanced strategies.
- Improve TTS chunking and reduce mid-dictation playback pauses by Malcolmnixon · Pull Request #27 · demaconsulting/Speech
Supplied the numeric false-split examples, punctuation-only chunks, ellipsis pauses, look-ahead capacity, and RTF measurements.
- Making Sense of Subtitles: Sentence Boundary Detection and Speaker Change Detection in Unpunctuated Texts
Supported the point that sentence boundary detection depends on punctuation and is hard even for humans without it.
- fix(tts/pocket): keep cache-safe long sentences whole by Alex-Wengg · Pull Request #938 · FluidInference/FluidAudio
Described how a fixed 512-position KV cache and voice-specific offsets shape the maximum chunk size.
- Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input
Provided the prosodic-boundary-aware post-training approach with limited lookahead and sliding-window inference.
- feat(tts/pocket): cut oversized sentences at more kinds of punctuation and balance word splits by arvindvenkataramani · Pull Request #935 · FluidInference/FluidAudio
Supplied the oversized-sentence cutting, word-boundary fallback, and orphan-fragment failure modes.
- https://arxiv.org/pdf/2603.05413
Described the sentence aggregation pattern in Pipecat and LiveKit.