Chatterbox-Flash

Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS

Chatterbox-Flash is a zero-shot streaming TTS model that converts an autoregressive decoder into a block-diffusion decoder. It uses prior-calibrated scoring and early decoding for efficient, low-latency synthesis without changing the model architecture, enabling high-quality speech with streaming support.

• tts
• voice-cloning
• streaming
• low-latency
• autoregressive

Authors: Deokjin Seo, Gangin Park, Kihyun Nam

Categories: cs.SD, cs.AI, eess.AS

Comment: 8 pages, 4 figures, 9 tables

Published 2026-05-29 · Updated 2026-06-01

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub25,266 stars148 watchers3,359 forksMIT
• Hugging Face2,243,054 downloads1,659 likes

Impact

• Semantic Scholar0 citations

Abstract

We present Chatterbox-Flash, a zero-shot text-to-speech model obtained by fine-tuning a pretrained autoregressive TTS decoder into a block-diffusion decoder, enabling parallel token generation within each block while retaining block-by-block streaming. We find that naively transferring mainstream block-diffusion decoding to discrete speech tokens degrades quality, as a long-tail token distribution biases parallel position selection toward a few high-frequency tokens. To mitigate this without architectural modification, we introduce two inference-time techniques: prior-calibrated scoring, which subtracts the block-level marginal token distribution, and an early-decoding schedule, which adaptively terminates iteration based on calibrated confidence. On standard zero-shot TTS benchmarks, Chatterbox-Flash attains high-fidelity synthesis comparable to strong autoregressive and non-autoregressive baselines, while supporting streaming inference with time-to-first-packet on par with streaming AR systems and substantially lower real-time factor. Code and audio samples are available at https://github.com/resemble-ai/chatterbox-flash.

---

Introduction

Chatterbox-Flash is a zero-shot text-to-speech system that converts a pretrained autoregressive decoder into a block-diffusion decoder while keeping the original model architecture intact. The core goal is to combine three properties that are usually in tension: high speech quality, native streaming, and substantially lower decoding latency than sequential autoregressive generation.

The paper starts from the observation that modern zero-shot TTS is often dominated by autoregressive models because they synthesize high-quality speech and can stream token-by-token, but they are inherently sequential. Diffusion-style decoders can generate multiple tokens in parallel, yet most mainstream diffusion decoding methods do not transfer cleanly to discrete speech codecs. In codec space, the token distribution is highly skewed toward a few dominant tokens, especially silence- or low-energy-like codes, and that skew can corrupt the position-ranking logic used by block diffusion during parallel unmasking.

Chatterbox-Flash addresses this with two inference-time additions rather than architectural changes: prior-calibrated scoring, which subtracts a token prior from the per-position model confidence, and an early-decoding schedule, which stops denoising early when calibrated confidence is already high enough. Together, these preserve block-wise streaming while sharply reducing the compute required per generated block.

The model is evaluated on standard English zero-shot TTS benchmarks and is reported to match or exceed strong autoregressive and non-autoregressive baselines on quality while being the only comparison system with native streaming support in the authors’ setup. The main empirical message is that block diffusion can work well for speech, but only if the decoding rule is calibrated to the non-uniform prior structure of discrete audio tokens.

Method

The system is built on top of Chatterbox-TTS, an open-source two-stage zero-shot TTS pipeline. Stage 1 is the decoder that maps text and prompt speech into discrete speech tokens at 25 Hz. Stage 2 is a flow-matching vocoder that converts those tokens to waveform chunks in a streaming-friendly way. Chatterbox-Flash keeps the codec, speaker encoder, and vocoder pipeline, and replaces the original autoregressive decoder with a block-diffusion decoder trained by masked denoising.

Architecture and conditioning

The decoder conditions on a composite input $$ \mathbf{c} = [\mathbf{e}_s, \mathbf{x}_{\text{text}}, \mathbf{x}_{\text{speech}}], $$ where $\mathbf{e}_s$ is a global speaker embedding from a GE2E-trained voice encoder, $\mathbf{x}_{\text{text}}$ is the input text token sequence, and $\mathbf{x}_{\text{speech}}$ is the prompt speech token sequence extracted from the reference audio. The speech output is a sequence of codec tokens $\mathbf{y} = (y_1, \ldots, y_T)$.

In the original autoregressive factorization, the speech sequence is modeled as $$ p(\mathbf{y} \mid \mathbf{c}) = \prod_{t=1}^{T} p(y_t \mid y_{block factorization. A length-$T$ sequence is partitioned into $B = \lceil T/D \rceil$ non-overlapping blocks of size $D$, and the sequence probability becomes $$ p(\mathbf{x}) = \prod_{b=1}^{B} p\!\left(\mathbf{x}^{(b)} \mid \mathbf{x}^{(

Hybrid attention mask

Training uses a hybrid attention pattern: the conditioning prefix is causal, speech tokens in the current block attend bidirectionally within the block, and attention is causal across blocks. This means the current block can use the clean prefix and all previous committed blocks, but it cannot see future blocks. The paper emphasizes that the conditioning prefix remains causal to preserve the pretrained backbone’s embedding space and maintain monotonic text-to-speech alignment.

Training objective

Training uses complementary masking over the speech tokens. At each step, the model samples a noise time $t \sim \mathcal{U}(\epsilon, 1-\epsilon)$, derives a per-token mask probability from a fixed noise schedule, and replaces a subset of speech positions with a mask token. A complementary view is added in the same batch so that every position is supervised under both masked and unmasked contexts.

The denoising loss uses a token-shift parameterization: a masked position is predicted from the hidden state at the previous position rather than from the mask token itself. For a masked token position $i$, the per-token loss is $$ \ell_i = -\log p_\theta\!\left(y_i \mid \mathbf{c}, \mathbf{x}^{(

This formulation keeps the backbone’s autoregressive interface intact, but exposes each block to bidirectional context during training. The practical effect is that the model can still run block by block at inference time, yet learn a parallel denoising objective inside the block.

Inference: block-autoregressive decoding

At inference, blocks are decoded left to right. Once a block is committed, it becomes clean context for the next block, and key-value caches are appended sequentially. The conditioning prefix is encoded once at the start and reused across all later blocks because it never attends to speech tokens. Within each block, the forward pass predicts a token distribution at every masked position, and the decoder must decide both which positions to unmask and how many positions to commit per step.

The paper argues that naive block-diffusion decoding fails on discrete speech because the model’s confidence is often dominated by the marginal frequency of common tokens instead of by local contextual fit. This is especially harmful near block boundaries, where incorrect early commits can cascade into boundary-induced context truncation and corrupt all later blocks.

Prior-calibrated scoring

To correct this bias, Chatterbox-Flash uses prior-calibrated scoring, which is a pointwise mutual information-style score: $$ s_i^{(k)} = \log p_i^{(k)}(\hat{x}_i^{(k)}) - \log \bar{p}(\hat{x}_i^{(k)}). $$ The first term is the conditional log-confidence at position $i$ in decoding step $k$. The second term subtracts the marginal likelihood of the same token under a reference prior $\bar{p}$, so the score rewards tokens that are specifically supported by local context rather than simply common in the codec vocabulary.

The reference prior is not estimated from the current block’s context, because that would be self-referential. Instead, the paper defines an unconditional block prior computed once by running a single forward pass on an all-masked sequence with conditioning zeroed out: $$ \bar{p}(v) = \frac{1}{D} \sum_{j=1}^{D} p_\theta\!\left(v \mid [\mathrm{MASK}]^D,\, \mathbf{c} = \mathbf{0}\right)_j. $$ This cached prior depends only on the model and block size, and is reused throughout inference.

Unmasking schedule and early decoding

For how many positions to commit at each step, the paper builds on a time-shifted schedule. The cumulative fraction of unmasked tokens by step $k$ is $$ r_k = \frac{\tau \cdot (k/K)}{1 + (\tau - 1) \cdot (k/K)}, $$ with total denoising budget $K$ and shift parameter $\tau$. The new fraction to unmask at step $k$ is the difference between successive $r_k$ values.

On top of that, the authors introduce early decoding. They compute a per-step threshold from the calibrated scores, $$ \theta_k = \operatorname{Quantile}\!\left(\{s_i^{(k)}\}_{i \in \mathcal{M}}, q_k\right), $$ where $$ q_k = \max\!\left(0,\, 1 - \alpha \cdot \frac{k+1}{K}\right). $$ At early steps, only the most confident positions are released; later steps relax the threshold and commit more positions. The system combines the time-shift schedule and the confidence threshold, and stops once all positions are unmasked or the step budget $K$ is reached. This is the mechanism that turns calibrated confidence into a compute-saving policy.

Classifier-free guidance

The decoder also supports classifier-free guidance. The conditional and unconditional logits are combined as $$ \ell_i = (1+w)\,\ell_i^c - w\,\ell_i^u, $$ and the token to commit is chosen from the guided distribution. Importantly, the prior-calibrated ranking score is computed on the conditional branch alone, so the position ordering is not distorted by the guidance scale $w$. The default CFG scale is $w = 1.0$.

The implementation uses FlashInfer-based attention kernels, paged key-value cache management, a frozen prefix cache, a growing block cache, cache snapshots for CFG, and CUDA graph replay for fixed-size block execution. These are serving optimizations rather than algorithmic changes, but they are important for the reported latency and throughput numbers.

Training data and experimental setup

The model is trained on approximately $70$k hours of English speech, corresponding to $43.8$M utterances from $528$k speakers. The training mix spans large-scale read speech, expressive and anechoic speech, accented English, and a sizable amount of privately collected audiobook, conversational, IVR, and short-form data. The paper explicitly notes that the private portion broadens coverage for conversational, expressive, and short-form utterances beyond standard read-speech corpora.

Evaluation is performed on two public English zero-shot TTS benchmarks: LibriSpeech-PC test-clean and Seed-TTS test-en. The reported objective metrics are SIM-o (cosine similarity between speaker embeddings of generated and reference speech), WER against the input text, and UTMOS for naturalness. For LibriSpeech-PC, WER is computed using HuBERT-based ASR; for Seed-TTS, the paper uses Whisper-large-v3. Baseline numbers are taken from OmniVoice’s report.

The canonical inference configuration is block size $D = 16$, denoising budget $K = 8$, time-shift parameter $\tau = 0.5$, CFG scale $w = 1.0$, sampling temperature $T = 0.2$, and position temperature $\beta = 5$. The paper reports two primary operating points: a quality-oriented one with $\alpha = 0$ and an efficiency-oriented one with $\alpha = 0.5$.

Training starts from a pretrained Chatterbox-TTS checkpoint and continues with AdamW, cosine learning-rate decay, a peak learning rate of $10^{-5}$, $10\%$ warmup, effective batch size $440$, and bf16 precision. The model is trained with block size $D = 32$, then evaluated at smaller or equal inference block sizes. All experiments are run on NVIDIA H100 GPUs.

Main results

The headline result is that the block-diffusion conversion preserves or improves quality relative to the AR backbone while adding streaming-friendly parallelism. Compared with the original Chatterbox AR model, Chatterbox-Flash improves both speaker similarity and WER on the two benchmarks, while keeping UTMOS essentially unchanged.

Relative to the AR backbone, Chatterbox-Flash improves on LibriSpeech-PC from SIM-o $0.707$ to $0.717$ and WER from $1.99$ to $1.67$, while matching UTMOS at $4.29$. On Seed-TTS test-en, SIM-o improves from $0.685$ to $0.704$ and WER from $2.20$ to $1.96$, with UTMOS staying effectively unchanged at about $4.09$ to $4.10$.

The most important decoding finding is that Fast-dLLM v2-style threshold decoding transfers poorly to discrete speech tokens: WER exceeds $14$ on both benchmarks. By contrast, the time-shift schedule and the prior-calibrated scoring rule produce nearly identical objective quality on the saturated read-speech benchmarks, which the authors interpret as evidence that PMI’s main value is not direct quality gain at high compute, but a better-calibrated confidence signal for early stopping.

With early decoding enabled, the average number of denoising steps drops from $8$ to $6.47$ on LibriSpeech-PC and to $6.10$ on Seed-TTS, while WER remains essentially unchanged on LibriSpeech-PC and increases only slightly on Seed-TTS. This is the paper’s main quality–compute trade-off result.

Streaming efficiency

The paper also reports streaming-oriented latency metrics against Qwen3-TTS, an autoregressive streaming baseline. The two metrics are time to first packet (TTFP) and real-time factor (RTF). Chatterbox-Flash is competitive on TTFP and substantially better on RTF, especially as block size grows and early decoding becomes more aggressive.

The best reported streaming configuration, $D = 32$ with $\alpha = 0.75$, reaches TTFP $103$ ms and RTF $0.076$, corresponding to roughly $13\times$ real-time synthesis. The authors argue that this is where block diffusion becomes especially compelling for deployment: TTFP stays close to streaming autoregressive systems, but steady-state throughput is far lower in wall-clock cost.

Human evaluation

A side-by-side human study against ElevenLabs v3, a frontier commercial zero-shot TTS system, uses independent 5-point Likert ratings for naturalness and speaker similarity. Chatterbox-Flash achieves a slightly lower mean naturalness score but a cleaner low tail, and substantially higher speaker similarity.

The paper interprets this as evidence that the model preserves reference-speaker characteristics very well despite being trained on a smaller corpus than many of the larger baselines, while still remaining competitive on perceived naturalness.

Hard-sample evaluation

To test whether prior calibration matters beyond saturated read speech, the authors also evaluate on EmergentTTS-Eval, which contains more prosodically and linguistically difficult utterances. Here, PMI helps more clearly: overall WER drops by $10.6\%$ relative, while the judge-based MOS stays essentially unchanged.

Ablations and analysis

The most informative ablations are about block size, denoising budget, and the two inference-time calibration parameters $\alpha$ and $\beta$. The paper’s overall conclusion is that decoding quality saturates quickly on the easy benchmarks, but compute can still be reduced substantially by using calibrated early termination.

Block size. With inference block size swept over $D \in \{8, 16, 24, 32\}$, SIM-o and UTMOS remain almost flat, but WER is stable only up to $D = 16$ and then degrades sharply at larger blocks. This indicates that large blocks make parallel unmasking too hard: the model must commit more positions simultaneously than it can confidently rank, even after prior calibration.

Step budget. With fixed-step decoding and $D = 16$, quality becomes usable only once $K \geq 3$, and it plateaus by about $K \geq 6$. The authors therefore choose $K = 8$ as the default, which offers some headroom for adaptive early termination without entering the unstable low-step regime.

Early decoding. At $D = 16$ and $K = 10$, increasing $\alpha$ progressively saves more compute: about $20\%$ at $\alpha = 0.5$, $32\%$ at $\alpha = 0.75$, and $41\%$ at $\alpha = 1.0$. WER rises only mildly across this range. The canonical efficiency setting is therefore $\alpha = 0.5$, which gives a meaningful compute reduction with negligible quality loss.

Sampling temperature. The speech-token sampling temperature is swept over $T \in \{0.2, 0.4, 0.6, 0.8, 1.0\}$. The default $T = 0.2$ is the best read-speech operating point, minimizing mean WER while leaving SIM-o and UTMOS nearly unchanged. Higher temperatures gradually degrade WER.

Position temperature and decoding method. The position-level Gumbel temperature $\beta$ is compared at $\beta = 0$ versus $\beta = 5$ across both the TS schedule and PMI, and at step budgets $K \in \{2, 5, 8\}$. The key finding is that $\beta = 5$ consistently improves WER over deterministic top-$n_k$ selection, especially at low $K$. Comparing PMI and the TS schedule at $\beta = 5$, the two methods track each other closely on the saturated benchmarks, reinforcing the claim that PMI is mainly valuable as a calibrated confidence signal rather than as a direct quality improvement when compute is plentiful.

The appendix also reports that the streaming server version of the model incurs a modest quality drop relative to offline decoding. At the default streaming setting $D = 16$, $\alpha = 0.5$, the server scores SIM-o $0.688$, WER $2.03$, and UTMOS $4.07$ on LibriSpeech-PC, compared with offline $0.713$, $1.67$, and $4.28$. Larger blocks increase the quality gap, especially in WER, which is consistent with the main block-size ablation.

Two additional scaling explorations are important for understanding the method’s limits. A fully causal CARD-style block formulation works only at very small block sizes and starts to show prosodic collapse from $D = 5$ onward. A block-size-annealing, self-distillation recipe can grow the block up to roughly $D = 8$, but beyond that confidence collapses and sampling becomes highly sensitive to temperature and CFG scale. These experiments suggest that the current bottleneck is not simply engineering, but the difficulty of preserving both prosody and confidence calibration as block size increases.

Limitations

The paper is unusually explicit about its limitations. First, the authors prioritize training stability in a fixed mixed-data regime and do not isolate the contribution of individual data sources. Second, the model becomes unstable when fine-tuned with much larger blocks, specifically $D \geq 128$, even with prior-calibrated decoding; this indicates that the method does not yet scale arbitrarily in parallelism. Third, on saturated read-speech benchmarks, prior-calibrated scoring and the time-shift baseline are statistically equivalent in objective quality, so the main advantage of PMI in the reported setup is its usefulness as a calibrated confidence signal for early stopping rather than a universal quality improvement.

The authors also note that whether prior calibration helps more under tighter compute budgets or on out-of-domain references remains open. Likewise, full-sequence diffusion systems such as OmniVoice may still have advantages when block-size scaling becomes unstable, and the paper views pseudo-streaming adaptations of such systems as a useful future comparison point.

Conclusion

Chatterbox-Flash shows that a pretrained autoregressive zero-shot TTS decoder can be converted into a practical block-diffusion model for streaming synthesis without changing the architecture. The two key ideas are prior calibration to neutralize dominant-token bias in discrete speech codecs and early decoding to turn that calibrated confidence into compute savings. On standard zero-shot benchmarks, the model matches strong baselines in quality, improves over its AR predecessor, and delivers much lower real-time factor with competitive time-to-first-packet. The result is a compelling design point for production-oriented, streaming, zero-shot TTS.