Sentence Boundary Detection for Streaming TTS Chunk Splitting

How to split streaming text for speech without breaking prosody.

Cover illustration for “Sentence Boundary Detection for Streaming TTS Chunk Splitting”
Written by
The Voice Layer Editors
Published
November 11, 2026
Reading time
8 min read
Sources cited
8 sources ↓
  • The streaming TTS promise: audio begins before the full input is known, eliminating the wait-for-completion bottleneck
  • But synthesis is not phoneme-by-phoneme — it is prosody-aware, and prosody requires context
    • A question ends differently from a statement; emphasis in a list depends on the whole list
    • Committing to the first word with no lookahead leaves intonation wrong by the time the sentence resolves
    • Per Soniox's technical wiki: streaming would be free if a voice could begin from the first phoneme with no context — it cannot, because of prosody
  • The fundamental curve every streaming TTS team must choose a point on: lower time-to-first-audio vs. better prosody
    • Chunking too small (word or two-word level): synthesis engine lacks context, voice sounds choppy
    • Waiting for a full natural boundary: prosody improves, but first-audio is delayed
    • The usual compromise: chunk at sentence or phrase boundaries — gather enough text to read one unit well, start speaking it, generate the next unit while the first plays
  • This tradeoff is not incidental — it is the central design decision in every streaming TTS pipeline, and every subsequent engineering choice flows from it

The latency budget that makes split strategy urgent

  • Users begin to perceive conversational AI as unnatural at response delays beyond 500–700 ms, per AssemblyAI's voice agent architecture guidance
  • What a well-budgeted pipeline looks like end-to-end
    • Twilio's engineering allocation: 350 ms for STT, 375 ms for LLM first token, 100 ms for TTS first byte — targeting a 1,115 ms median mouth-to-ear gap
    • Retell's production stack runs around 600 ms end-to-end by streaming every stage into the next rather than waiting for completion
    • A polargrid.ai sub-400 ms pipeline on Ada Lovelace hardware: 364 ms p50 total — STT 148 ms, LLM 96 ms, TTS 117 ms
    • Deepgram's streaming TTS runtime: p95 TTFB under 200 ms at high concurrency
  • The threshold matters: above roughly 700 ms callers get awkward, repeat themselves, or hang up; below it, they forget they're talking to AI (per Retell)
  • Implication for chunk splitting: TTS first-byte latency is the one stage the chunking strategy directly controls — and it is also the stage most sensitive to how early a flush is triggered
  • This budget frames all subsequent design choices: the split strategy must hit the TTS-first-byte target without sacrificing prosody below the perceptibility threshold

The sentence buffer pattern and how production frameworks implement it

  • The sentence buffer as the critical bridge between streaming LLM output and TTS input
    • Accumulates tokens; yields complete sentences by detecting sentence-ending punctuation (.,!,?) followed by whitespace
    • Excludes false positives from abbreviations (Dr., Mr., PM.) and decimal numbers
    • Enforces a minimum sentence length (10 characters) to avoid sending fragments to TTS
    • Flushes any remaining text when the LLM stream ends
    • Per the arXiv paper (2603.05413): this is the "sentence aggregation" pattern used by Pipecat's SentenceAggregator and LiveKit's text processing pipeline
  • The overlap strategy this enables: sentence one is being spoken while the LLM generates sentence two — the key latency reduction technique
    • Per AssemblyAI: streaming LLM tokens into TTS at sentence boundaries dramatically reduces perceived latency because the user hears the first sentence while generation of the second continues
    • Per ElevenLabs latency optimization guidance: starting LLM work on stable STT partials and streaming tokens into TTS at sentence boundaries are among the highest-leverage optimizations (listed first and second, roughly in order of impact, out of a list of five)
  • How specific frameworks expose this pattern
    • Pipecat's CartesiaTTSService: consumes TextFrame with text_aggregation_mode=TextAggregationMode.SENTENCE (default), flushing to TTS WebSocket at sentence boundaries
    • Deepgram Aura (/v1/speak): recommends splitting at sentence boundaries as the simplest and most effective approach for preserving natural speech while enabling faster time-to-first-byte
    • Deepgram Flux (/v2/speak): a different model — server places flush boundaries internally; client should stream text as produced and not chunk client-side
  • Chunk size guidance for Deepgram Aura by use case: voice assistants aim for 50–100 character chunks; call center bots use complete sentences; long-form content uses larger chunks (200–400 characters) to preserve intonation
  • Note the divergence: Flux's server-side flushing is a design direction that shifts responsibility away from the client, which has implications for teams building portable chunking logic

Where rule-based sentence splitting breaks down

  • Punctuation-based SBD carries a well-documented assumption: clean, well-punctuated input — per the ACM WWW '21 paper (Donabauer et al.): SBD typically relies on the presence of punctuation, and even with punctuation it can be difficult; without punctuation and casing, even humans struggle to determine sentence boundaries
  • Failure mode 1 — decimal and localization bugs (real production issue, hass-edge-tts GitHub issue #76, opened 2026-09-11)
    • The streaming path splits on a separator list that includes a comma
    • In locales using comma as decimal separator (German, French, Italian, Spanish), every decimal value loses its spoken word: "22,1 Grad" is spoken as "zweiundzwanzig [pause] eins"
    • Verified by Whisper transcription of the returned MP3; direct edge_tts call produces correct output, streaming path does not
    • Intermittent appearance: which decimal value gets split depends on where the incoming token chunks land
    • Workaround in use: instruct the conversation agent to spell out decimal numbers — a prompt-level patch that does not help users who cannot edit the system prompt
    • Suggested fix: exclude the pattern (?<=\d),(?=\d) from the separator match; never emit a chunk whose last character is a comma
  • Failure mode 2 — numeric false splits (DemaConsulting/Speech PR #27, 2026-09-16)
    • Applying clause-level splitting (,/;/:) unconditionally treats punctuation inside numeric or token-like text as a clause boundary
    • "The time is 12:30." becomes "The time is 12:" + "30." — two synthesis calls for a single time expression
    • "1,000" is split into two calls even though neither punctuation is a clause separator
    • Recommended fix: restrict secondary splitting to actual clause separators; at minimum preserve punctuation between digits and URI-like tokens
  • Failure mode 3 — oversized sentence overflow and word-boundary fallback (FluidAudio/FluidInference PR #935)
    • Kyutai's original PocketTTS implementation cuts an oversized sentence only at,;: and never between words; a piece still over the limit is left whole
    • FluidAudio cannot leave such a piece whole because its KV cache is fixed at 512 positions
    • FluidAudio PR #935 keeps and improves the existing word-boundary fallback: makes it rarer by recognizing more punctuation, less audible by balancing the parts
  • Failure mode 4 — orphan fragment artifacts (FluidAudio PR #935)
    • The single-word orphan fix from the PocketTTS text chunker produces audible artifacts on long French text (smart apostrophe + clause-cascade)
  • Failure mode 5 — degenerate punctuation-only pieces (DemaConsulting/Speech PR #27)
    • Spurious repeated periods ("Sentence one..... Sentence two") produce punctuation-only chunks that become standalone, pointless synthesis calls
    • Fix: merge these degenerate pieces into the preceding chunk
  • Common thread across all five: rule-based splitting treats punctuation as a proxy for linguistic structure — and that proxy fails systematically at the edges of natural language and internationalization

Secondary splitting strategies and the tradeoffs each introduces

  • Clause-based splitting: split long sentences at commas and semicolons
    • Deepgram Aura documentation names this explicitly as an advanced chunking strategy
    • DemaConsulting PR #27 applies clause-level splitting (,/;/:) unconditionally rather than only past the 200-character length budget — proactively bounding long sentences before they become one large, slow synthesis call
    • Tradeoff: introduces the decimal/URI false-split failure modes catalogued above; must be paired with digit-aware exclusion rules
    • DemaConsulting measured: VITS-Piper synthesizes at RTF roughly 0.16–0.18 (no audible pauses); Kokoro int8 at RTF roughly 1.5 — confirming remaining Kokoro pauses are a model/compute throughput limit, not a chunking defect
  • Ellipsis-aware silence injection: treat genuine ellipsis (... or spaced... variants) differently from normal chunk boundaries
    • DemaConsulting PR #27 adds a documented EllipsisPauseMilliseconds constant applied as PostSilenceMs only to chunks ending in a genuine ellipsis; every other chunk boundary keeps zero added silence
    • Rationale: ellipsis signals a deliberate narrative pause; treating it the same as a mid-sentence comma removes expressive intent
  • Look-ahead buffering: increase pending segment capacity to give the synthesis pipeline more real-time margin
    • DemaConsulting PR #27 raises PendingSegmentCapacity from 5 to 8 for extra real-time look-ahead margin
    • True concurrent synthesis was investigated and deliberately rejected: ISynthesisEngine is not safe for concurrent Generate() calls (enforced by existing regression test)
  • Fixed-context KV cache constraints impose hard upper bounds that interact with splitting strategy
    • FluidAudio's KV cache is fixed at 512 positions; the PocketTTS chunk size of 50 tokens means sentences over that size must be cut
    • FluidAudio PR #938 (2026-09-19): treats the 50-token PocketTTS chunk size as a grouping preference rather than a hard limit; admits longer sentences using the selected voice's real KV offset plus a speech-duration cache budget; bounds IO and stateful generation loops at the 512-position cache boundary; closes related issue #933 (69-token sentence forcing cuts)
    • Voice-dependent splitting: different voices have different KV offsets, so the correct maximum chunk size is voice-specific, not a single constant
  • NLP-based and adaptive chunking: mentioned by Deepgram as advanced options — use natural language processing to find semantic boundaries, or adjust chunk size based on content complexity
    • In the GitHub ecosystem, tools like yasbd-lib (high-accuracy, rule-based SBD with drop-in adapter for pysbd) offer faster and more accurate segmentation than naive punctuation scanning
    • No single off-the-shelf NLP segmenter eliminates the localization and numeric edge cases without domain-specific tuning
  • First-chunk optimization: specially optimize the first chunk for minimal latency while allowing subsequent chunks to be larger for prosody — Deepgram documentation names this as a distinct strategy worth considering

The lookahead problem: how much future text the model actually needs

  • The core issue: a synthesis model committing to audio before knowing how the sentence ends can produce prosodic decisions that are wrong in context — stress, pitch contour, pause placement
  • MambaVoiceCloning (2026) look-ahead window findings
    • Look-ahead L provides the next L seconds of mel frames, conditioning the SSM update and preventing premature prosodic decisions when punctuation occurs near the boundary
    • Chunk boundaries remain perceptually smooth for L ≥ 0.5 s; only L = 0.25 s shows occasional discontinuities or shortened pauses
    • WER for L = 0.50 s: 9.4% vs. 7.3% for L = 2.00 s (Table 5 of that paper) — a meaningful quality gap, though L = 0.50 s is likely acceptable for most conversational applications
  • The prosodic-boundary-aware post-training approach (Liu et al., arXiv:2603.06444, 2026)
    • Proposes adapting a pretrained LLM-based TTS model using weakly time-aligned data so it learns early stopping at specified content boundaries when given limited future text
    • During inference: a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation
    • Performance at chunk size k=5 words, lookahead f=2 words on Seed-TTS-Eval:

Methodology & sources

  1. Streaming TTS splits at commas: decimal commas lose the spoken word ("22,1" is spoken as "22 1") · Issue #76 · hasscc/hass-edge-tts

    Supplied the decimal-comma failure case, where '22,1 Grad' is split and a spoken word is lost in streaming TTS.

  2. Text Chunking for TTS | Deepgram's Docs

    Provided Deepgram Aura's chunking guidance, including sentence-boundary splitting, chunk sizes by use case, and the advanced strategies.

  3. Improve TTS chunking and reduce mid-dictation playback pauses by Malcolmnixon · Pull Request #27 · demaconsulting/Speech

    Supplied the numeric false-split examples, punctuation-only chunks, ellipsis pauses, look-ahead capacity, and RTF measurements.

  4. Making Sense of Subtitles: Sentence Boundary Detection and Speaker Change Detection in Unpunctuated Texts

    Supported the point that sentence boundary detection depends on punctuation and is hard even for humans without it.

  5. fix(tts/pocket): keep cache-safe long sentences whole by Alex-Wengg · Pull Request #938 · FluidInference/FluidAudio

    Described how a fixed 512-position KV cache and voice-specific offsets shape the maximum chunk size.

  6. Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

    Provided the prosodic-boundary-aware post-training approach with limited lookahead and sliding-window inference.

  7. feat(tts/pocket): cut oversized sentences at more kinds of punctuation and balance word splits by arvindvenkataramani · Pull Request #935 · FluidInference/FluidAudio

    Supplied the oversized-sentence cutting, word-boundary fallback, and orphan-fragment failure modes.

  8. https://arxiv.org/pdf/2603.05413

    Described the sentence aggregation pattern in Pipecat and LiveKit.

The Voice Layer Editors