dots.tts

dots.tts Technical Report

dots.tts is a 2B-parameter continuous autoregressive text-to-speech model that generates speech in a semantically structured continuous latent space. Innovations include full-history conditioning and self-corrective post-training for robust, expressive, and low-latency multilingual speech.

• tts
• autoregressive
• voice-cloning
• emotion
• realtime
• streaming
• low-latency

Demos

The demos highlight dots.tts's state-of-the-art text-to-speech ability with very low word error rates and high speaker similarity across multiple languages. Watch the architecture overview for its unique fully continuous pipeline and the evaluation charts demonstrating top benchmark performance and voice cloning quality. These visuals illustrate dots.tts's synthesis accuracy, stability, and expressiveness.

Authors: Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, Kai Yu

Categories: cs.SD, cs.AI, eess.AS

Published 2026-06-05 · Updated 2026-06-05

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub789 stars9 watchers84 forksApache-2.0
• Hugging Face817 downloads46 likes

Impact

• Semantic Scholar0 citations

Abstract

We present dots.tts, a 2B-parameter continuous autoregressive text-to-speech (TTS) foundation model that models speech in a continuous latent space. Compared with existing continuous autoregressive models, our key innovations are threefold. First, we train an AudioVAE with multiple objectives to build a semantically structured and prediction-friendly continuous speech space. Second, we use full-history conditioning in the flow-matching head to preserve long-range consistency and reduce drift during generation. Third, we apply reward-free self-corrective post-training to the flow-matching head to further improve robustness and acoustic quality. After being trained on a large-scale multilingual corpus, dots.tts achieves the best average performance on Seed-TTS-Eval, with WERs of 0.94%/1.30%/6.60% and SIM scores of 81.0/77.1/79.5 on the zh/en/zh-hard test sets, respectively. Across other benchmarks, dots.tts also consistently demonstrates open-source state-of-the-art performance, exhibiting strong generation stability, voice cloning ability, and emotional expressiveness. For efficient inference, we further apply CFG-aware MeanFlow distillation, enabling low-latency speech generation with first-packet latencies of 85/54 ms in output streaming and dual-streaming modes, respectively. To facilitate reproducible research and practical deployment, we release the training and inference code, together with the pretrained, post-trained, and MeanFlow-distilled checkpoints, under the Apache 2.0 license.

---

Overview

dots.tts is a 2B-parameter continuous autoregressive text-to-speech foundation model. The paper frames its goal as closing two bottlenecks in continuous-AR TTS: long-range error accumulation during autoregressive rollouts, and an immature post-training stack compared with discrete-token cascade systems. The main design choice is to model speech in a continuous latent space instead of discrete acoustic tokens, while keeping the system fully end-to-end and streamable.

The reported design combines three ideas:

• a high-fidelity AudioVAE trained with reconstruction, WavLM alignment, and multitask supervision so the continuous latent space is both expressive and easy for the downstream generator to predict;
• a fully history-conditioned AR flow-matching head that preserves long-range consistency by conditioning on the full generated history rather than a truncated context;
• reward-free self-corrective post-training plus CFG-aware MeanFlow distillation to improve robustness and reduce inference cost.

The paper reports strong performance on Seed-TTS-Eval, MiniMax-Speech multilingual evaluation, CV3-Eval, and EmergentTTS-Eval, together with low-latency streaming behavior on a single H800 GPU.

Problem setting and motivation

The introduction argues that modern TTS expectations go beyond intelligibility: users want expressive output, controllability, real-time synthesis, and coverage of neutral reading, emotional dialogue, paralinguistic events, singing, and general audio. The paper contrasts three TTS routes:

• NAR flow-matching / diffusion systems, which are good for offline synthesis but less natural for interactive dialogue;
• discrete-token autoregressive systems, which reuse LLM infrastructure and dominate deployment, but are limited by tokenizer bottlenecks;
• continuous-latent autoregressive systems, which remove quantization bottlenecks and can better cover speech plus richer audio phenomena, but suffer from error accumulation.

The paper’s thesis is that continuous AR becomes practical once the latent space is structured for prediction, the rollout is made long-range stable, and post-training is adapted to the continuous setting.

Model architecture

dots.tts is decomposed into two decoupled parts: a frozen AudioVAE that defines the continuous representation, and an autoregressive backbone that predicts that representation patch by patch. The AudioVAE encodes 48 kHz mono speech into a 128-dimensional latent stream at 25 Hz, corresponding to about 1920x temporal downsampling, and decodes with a causal BigVGAN-style decoder.

The autoregressive backbone has three modules:

• a semantic encoder that compresses each newly generated audio patch into a compact semantic summary;
LLM initialized from Qwen2.5-1.5B Base, which consumes BPE text tokens together with the audio-semantic stream;
• a flow-matching head that generates each next 4-frame latent patch.

The audio-side interface to the LLM is intentionally reduced to a 6.25 Hz semantic summary of the generated history, not the raw VAE latent. The paper emphasizes that this decoupling is important for long-rollout stability because the LLM reasons over compact semantic history while the acoustic detail is handled by the flow-matching head.

The core streaming cycle is:

1. LLM consumes text and the latest 6.25 Hz audio-semantic token.
2. The AR flow-matching head predicts the next 4-frame latent patch at 25 Hz.
3. The semantic encoder converts that patch back into one 6.25 Hz embedding.
4. The embedding is fed back into the LLM for the next step.

AudioVAE

The AudioVAE follows a HoliTok-style recipe with a causal BigVGAN-v2 decoder and fully causal convolutions on both encoder and decoder sides. The encoder uses strided residual blocks with downsample strides $[2, 2, 2, 4, 6, 10]$, ending in a posterior projection that outputs per-frame mean and log-variance. In addition to the standard KL prior, the model adds flow regularization to shape a smooth latent space.

The paper trains the AudioVAE in two stages:

• Stage 1 optimizes reconstruction: multi-period plus sub-band CQT adversarial losses, multi-scale mel-spectral reconstruction, feature matching, and KL plus flow regularization.
• Stage 2 adds learnability supervision: a frame-level alignment loss against frozen WavLM features and a multitask downstream block with ASR, emotion, and speaker classification objectives. The retained encoder becomes the semantic frontend used by the backbone.

The reported motivation is that a reconstruction-only continuous latent can still be too difficult for an LLM to predict reliably, so the second stage makes it semantically structured without sacrificing fidelity.

Semantic encoder

The semantic encoder is reused from the AudioVAE training pipeline. It first halves the frame rate with a strided causal-convolution projector, then applies a 24-layer causal Transformer with hidden size $1024$ and FFN size $4096$, and finally groups outputs in pairs to yield an end-to-end $4\times$ downsampling from 25 Hz latent frames to 6.25 Hz LLM tokens. Its outputs are pretrained to align with semantic structure rather than low-level acoustics.

Autoregressive flow-matching head

The acoustic generator is a Diffusion Transformer with 18 layers, hidden size $1024$, FFN size $4096$, RoPE in every layer, RMSNorm with QK-norm, and adaLN-zero modulation conditioned on diffusion time and a speaker embedding. It predicts the rectified-flow vector field and is solved with a small number of Euler steps at inference.

Each step synthesizes a 4-frame latent patch at 25 Hz. The head conditions on three streams in a shared hidden space: the current LLM hidden state, the clean patches from all earlier audio positions, and the noisy patch under denoising. A frozen CAM++ x-vector is added as a global speaker condition. Classifier-free guidance is enabled by independently dropping the LLM hidden stream and the speaker stream with probability $0.5$ during training.

The paper stresses that the head is trained with a block-causal attention mask so that parallel training exactly matches per-step inference. In the concatenated training sequence, a cause half $C$ contains the clean history and a generation half $Z$ contains noisy patches. The mask ensures that $Z_n$ can attend only to $H_{\le n}$ and $P_{

Objectives and training recipe

The paper gives explicit objectives for the AudioVAE, backbone pretraining, self-corrective alignment, and MeanFlow distillation. A compact summary is below.

AudioVAE objective

The AudioVAE loss is:

$$ \mathcal{L}_{\mathrm{vae}} = \mathcal{L}_{\mathrm{mel}} + \mathcal{L}_{\mathrm{adv}} + \mathcal{L}_{\mathrm{fm}} + \beta_{\mathrm{kl}}\mathcal{L}_{\mathrm{kl}} + \lambda_{\mathrm{wavlm}}\mathcal{L}_{\mathrm{wavlm}} + \lambda_{\mathrm{sup}}\mathcal{L}_{\mathrm{sup}}. $$

Here the first group is reconstruction-oriented, while the second group adds WavLM alignment and multitask supervision for downstream learnability.

Backbone pretraining

Backbone pretraining uses two losses:

• Flow matching on each 4-frame latent patch, where the DiT predicts the velocity field between Gaussian noise and the clean patch.
• EOS prediction, implemented as a small stop head on a detached LLM hidden state so that stop prediction does not interfere with rollout dynamics.

The key point is that the flow-matching loss is the only loss that backpropagates through the LLM and semantic encoder, so the LLM is optimized end-to-end through the acoustic objective rather than via a separate text-only loss.

In simplified form, the flow objective is:

$$ \mathcal{L}_{\mathrm{fm}} = \mathbb{E}\left[\left\|v_\theta(x_t, t, H_{\le n}, P_{

The total backbone pretraining loss is the sum of flow matching and EOS loss.

Self-corrective alignment

After pretraining, the paper updates only the DiT acoustic generator inside the AR-FM head. The LLM, semantic encoder, AudioVAE, and speaker encoder are frozen. This stage is explicitly reward-free: it does not use a reward model, human preference model, or external acoustic teacher. Instead, the model rolls itself off the data manifold by one step, samples auxiliary off-trajectory states, and learns to steer those states back toward the clean endpoint.

The training setup reported in the paper is:

• global batch: about 4 hours of audio;
• 50K optimization steps;
• 5K-step linear warmup to peak learning rate $3 \times 10^{-5}$, then cosine decay to $2 \times 10^{-6}$;
• auxiliary correction weight $\lambda_{\mathrm{aux}} = 1.0$;
• one-step rollout uses CFG scale $\gamma_{\mathrm{soar}} = 1.2$;
• each main sample generates $N = 6$ auxiliary correction states.

The authors describe this stage as directly addressing the mismatch between pretraining and inference, where small velocity errors accumulate over multiple autoregressive patches.

CFG-aware MeanFlow distillation

The final stage freezes the self-corrected DiT as a teacher and trains a student DiT with the same backbone plus a duration embedder. The teacher trajectory is generated with a 16-step Euler solver, and classifier-free guidance is fused into the teacher target so that the student directly learns the CFG-guided mean velocity over a time interval $[t_a, t_b]$.

The student is trained for 50K steps with the same 8-hour global audio batch used in pretraining, 5K-step warmup, cosine decay to $2 \times 10^{-6}$, peak learning rate $1 \times 10^{-4}$, and interval sampling from a log-normal distribution with mean $-0.4$ and standard deviation $1.0$. Anchor samples are mixed with probability $0.5$ for stability.

At inference, the MeanFlow student predicts interval updates in a single conditional pass per step, avoiding separate conditional and unconditional evaluations because CFG has already been distilled into the target.

Reported training schedule

• AudioVAE: stage 1 for 500K steps on 9.6-second crops; stage 2 for another 200K steps with WavLM alignment.
• Backbone pretraining stage 1: 100K steps on Emilia only, freezing the LLM and updating only the semantic encoder and AR-FM head.
• Backbone pretraining stage 2: 700K steps on the full data mixture, with all modules unfrozen.
• Backbone pretraining stage 3: 100K steps on a higher-quality filtered subset, with learning-rate annealing from $2 \times 10^{-4}$ to $3 \times 10^{-5}$.
• Self-corrective alignment: 50K steps.
• MeanFlow distillation: 50K steps.

Data

The backbone is trained on 1.5M hours of speech, captioned speech, and a small fraction of general audio from three sources.

• In-house speech pool: about 1.2M hours after filtering. The pipeline includes vocal enhancement, source separation, speaker-aware diarization, and ASR routing, with Whisper-Large-v3 used for English and most other languages, and Paraformer used for Mandarin Chinese. Clips are filtered by cross-ASR consistency, effective bandwidth, UTMOS, and intra-clip x-vector variance.
• Open-source corpora: about 300K hours from Emilia, LibriTTS-R, HiFi-TTS, HiFi-TTS-2, WenetSpeech4TTS, AISHELL-3, Magicdata, MLS, MSR-86K, IndicVoices-R, EuroSpeech, WaxalNLP-TTS, and FLEURS. This mix provides most non-CJK language coverage.
• Caption-style data: about 7K hours combining AutoACD and an in-house subset augmented with Gemini-generated descriptions of speaker traits, emotion, delivery, and acoustic environment. The paper uses this to bootstrap style control and general-audio generation.

The paper also notes that the AudioVAE is trained on 48 kHz audio and includes a small general-audio share so the latent can cover non-speech sounds, even though the released backbone is still speech-heavy.

Evaluation protocol

Content fidelity is evaluated using ASR-based error rate: WER for English and other non-Chinese languages, and CER reported as WER for Chinese for table compatibility. Speaker similarity is measured as cosine similarity between WavLM-SV embeddings of generated speech and the reference prompt. The paper uses the benchmark default ASR settings for multilingual evaluation: Whisper-Large-v3 for English and multilingual benchmarks, Paraformer for Mandarin Chinese, and the benchmark default ASR for the remaining MiniMax languages.

For baseline comparison, the paper cites original-paper numbers or official default inference configurations and does not retune baselines. The compared systems include CosyVoice 2/3, Qwen3-TTS, IndexTTS 2, FireRedTTS-2, Seed-TTS, MiniMax-Speech, Fish-Audio S2, F5-TTS, MegaTTS3, DiTAR, VibeVoice, and VoxCPM 2.

Experimental results

AudioVAE reconstruction quality

The first quantitative check is the VAE reconstruction task on LibriSpeech test-other. The main takeaway is that the learned 48 kHz continuous latent is faithful enough that reconstruction is not the downstream bottleneck. The paper reports that the latent itself adds almost nothing to end-to-end error, and that stage 2 supervision makes the latent learnable as an LLM target.

Compared with representative discrete tokenizers and continuous representations, the dots.tts VAE sits in the top band on most metrics, while using a much higher input bandwidth than prior continuous baselines. The paper highlights the 0.969 SIM and 4.14 WER as evidence that the latent preserves fidelity while remaining usable for downstream generation.

Seed-TTS-Eval

Seed-TTS-Eval is the main zero-shot voice-cloning benchmark. The paper evaluates test-en, test-zh, and test-zh-hard under the standard reference-prompt protocol and reports the pretrained model, the self-corrected model, and MeanFlow variants with $\mathrm{NFE} \in \{2,3,4\}$.

The strongest reported summary is that dots.tts achieves the best average WER and SIM across the benchmark, with the self-corrected model peaking on similarity and the pretrained model already leading on average WER.

Interpretation in the paper:

• Pretrain already beats all reported baselines on average WER.
• SOAR improves similarity and slightly improves or preserves WER, with the best average SIM at 79.2 and the best per-column SIM values on all three test sets.
• MeanFlow distillation at $\mathrm{NFE}=4$ keeps WER very close to SOAR while lowering inference cost, at the expense of about one SIM point.

MiniMax-Speech multilingual test set

The MiniMax benchmark covers 24 languages and uses two reference speakers per language. The paper’s main findings are that dots.tts (SOAR) achieves the highest average SIM at 83.9, and that a dots.tts variant takes the per-language SIM lead on 19 of 24 languages and ties on 2 more. The average WER is competitive but not uniformly best, with low-resource languages contributing most of the gap.

The authors attribute the low-resource WER gap to insufficient BPE coverage and note that the similarity scores remain strong even when WER rises.

CV3-Eval

CV3-Eval adds a hard Chinese/English split and, importantly, a cross-lingual voice-cloning subset. The paper treats cross-lingual cloning as a difficult test of timbre disentanglement.

Key observations from the paper:

• hard-en is where SOAR helps most, improving from 5.99 to 4.49, and MF inherits that gain with 4.37 at $\mathrm{NFE}=4$.
• On the cross-lingual subset, SOAR gives the best SIM in both directions: 75.0 for en→zh and 72.8 for zh→en.
• Cross-lingual WER still trails CosyVoice 3, which the paper treats as evidence that timbre transfer is strong but lexical coverage is still imperfect.

EmergentTTS-Eval

EmergentTTS-Eval uses a Gemini-based audio judge to compare systems against a fixed gpt-4o-mini-tts reference over six expressiveness-oriented scenarios: Emotions, Foreign Words, Paralinguistics, Complex Pronunciation, Questions, and Syntactic Complexity. The paper reports both WER and head-to-head win rate.

The paper interprets these results as a tradeoff: SOAR tightens text faithfulness and yields the best Syntactic Complexity score in the table, but it reduces expressiveness on the emotion-related columns compared with the pretrained model. Pretrain is the strongest open-source system on overall win rate, while SOAR gives the lowest open-source WER and best syntax score.

Efficiency and streaming

Efficiency is reported on Seed-TTS-Eval using vllm-omni on a single NVIDIA H800 GPU. The LLM runs on vLLM with continuous batching and paged-KV attention, while the AR-FM head and semantic encoder are JIT-compiled with torch.compile. The MeanFlow-distilled student uses $\mathrm{NFE}=4$ per patch.

The reported results are:

• Plain text-only synthesis: 85.4 ms time to first packet and RTF 0.231.
• 1T1A interleaved mode: 54.4 ms time to first packet and RTF 0.245.

The paper’s interpretation is that interleaving lets a conversational upstream LLM drive synthesis token by token, so audio can start as soon as the first text token is emitted rather than waiting for a full response prefix. This is positioned as the mode to use for duplex voice agents and other real-time dialogue settings.

What the paper claims as its main contributions

• A fully continuous, end-to-end autoregressive TTS stack that removes discrete acoustic tokens while staying streamable and prompt-conditioned.
• A semantically structured AudioVAE trained not just for reconstruction but also for downstream learnability, via WavLM alignment and multitask supervision.
• An AR backbone split into semantic planning and acoustic rendering, with a compact 6.25 Hz audio-semantic history to stabilize long rollouts.
• Reward-free self-corrective alignment for the flow-matching head, designed to reduce exposure-bias-like degradation without reward models or preference optimization.
• CFG-aware MeanFlow distillation, which preserves the corrected teacher behavior while reducing inference to a small number of conditional evaluations.

Limitations and open issues

The limitations section is unusually explicit about what remains unresolved:

• Raw BPE text input instead of phonemes inherits text-model strengths but increases data demand. This is linked to the low-resource WER gap on languages such as Arabic, Hindi, Turkish, and Vietnamese in MiniMax.
• The same coverage limitation affects Foreign Words and Complex Pronunciation in EmergentTTS-Eval, especially for loanwords, technical terms, and proper names.
• The released system is evaluated only in canonical zero-shot settings, without explicit style or instruction control. The authors suggest that caption-paired data could support an instruction-tuned variant.
• Although the AudioVAE is in principle modality-agnostic, the release is trained on a speech-heavy mixture, so singing and unified speech-and-sound generation are not covered.
• High-fidelity voice cloning raises misuse risks. The paper recommends consent-aware reference-audio policies, synthetic-speech detection, and watermarking for downstream deployment.

Bottom line

The technical novelty of dots.tts is not just that it uses a continuous latent space, but that it makes continuous AR practical by combining: a learnable semantic latent, a history-aware backbone interface, block-causal training that matches inference exactly, reward-free self-correction, and CFG-aware MeanFlow distillation. The reported results show that this recipe is strong on zero-shot cloning, multilingual similarity, and expressiveness-related syntax, while remaining honest about lexical coverage weaknesses in low-resource and out-of-distribution text conditions.