Seed-TTS

Seed-TTS: A Family of High-Quality Versatile Speech Generation Models

A large-scale autoregressive speech synthesis model achieving human-level naturalness in zero-shot voice cloning. It combines in-context learning with RL post-training for robust, expressive, controllable speech generation across arbitrary speakers and emotional styles.

• tts
• voice-cloning
• autoregressive
• prosody
• emotion
• few-shot
• rlhf

Authors: Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, Xiaobin Zhuang

Categories: eess.AS, cs.SD

Published 2024-06-04 · Updated 2024-06-04

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub1,570 stars12 watchers147 forksnull

Impact

• Semantic Scholar387 citations48 influential citations

Abstract

We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech (TTS) models capable of generating speech that is virtually indistinguishable from human speech. Seed-TTS serves as a foundation model for speech generation and excels in speech in-context learning, achieving performance in speaker similarity and naturalness that matches ground truth human speech in both objective and subjective evaluations. With fine-tuning, we achieve even higher subjective scores across these metrics. Seed-TTS offers superior controllability over various speech attributes such as emotion and is capable of generating highly expressive and diverse speech for speakers in the wild. Furthermore, we propose a self-distillation method for speech factorization, as well as a reinforcement learning approach to enhance model robustness, speaker similarity, and controllability. We additionally present a non-autoregressive (NAR) variant of the Seed-TTS model, named $\text{Seed-TTS}_\text{DiT}$, which utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS systems, $\text{Seed-TTS}_\text{DiT}$ does not depend on pre-estimated phoneme durations and performs speech generation through end-to-end processing. We demonstrate that this variant achieves comparable performance to the language model-based variant and showcase its effectiveness in speech editing. We encourage readers to listen to demos at \url{https://bytedancespeech.github.io/seedtts_tech_report}.

---

Introduction and Motivation

Seed-TTS is a family of large-scale speech generation models from ByteDance designed to synthesize speech that is virtually indistinguishable from real human speech, even for arbitrary, previously unseen speakers in the wild. The system targets three primary capabilities: zero-shot speech in-context learning (ICL), speaker fine-tuning, and emotion control. Unlike prior TTS systems that focused on limited speaker sets or controlled recording conditions, Seed-TTS is explicitly benchmarked against ground-truth human speech using both objective and subjective metrics — and is the first system reported to cross the threshold of perceptual indistinguishability in a zero-shot, in-the-wild setting.

The motivation for Seed-TTS is grounded in the limitations of existing speech generation systems across three axes:

• Naturalness and expressiveness: Prior TTS models struggle with highly expressive speech — shouting, crying, strong accents, dramatic emotional bursts — in scenarios that are common in real-world applications but rare or impossible for previous systems to handle faithfully.
• Stability and robustness: Language model-based TTS systems are known to suffer from instability (repetitions, deletions, hallucinations), particularly on challenging text patterns. This hinders production deployment.
• Controllability and factorization: Decoupling speech attributes (timbre, prosody, content, emotion) without altering model architectures or loss functions is an open challenge. Most prior approaches require specialized engineering.

Seed-TTS addresses these challenges through: (1) a massively scaled autoregressive backbone, (2) a novel self-distillation-based speech factorization method for timbre disentanglement, (3) a reinforcement learning (RL) post-training stage to holistically improve model robustness and controllability, and (4) a fully diffusion-based non-autoregressive (NAR) variant called Seed-TTS_DiT that supports end-to-end speech generation and speech editing without relying on pre-estimated phoneme durations.

System Architecture Overview

The core Seed-TTS system is an autoregressive transformer-based model composed of four main components arranged in a coarse-to-fine pipeline:

1. Speech Tokenizer: Converts raw speech signals into discrete or continuous speech tokens. Both continuous and discrete tokenizers were investigated; the tokenizer design is emphasized as critical to overall system performance.
2. Token Language Model: An autoregressive transformer trained on paired text–speech token sequences. During inference, the language model generates speech tokens autoregressively, conditioned on the input text and the audio prompt. The loss on the text sequence is masked since the focus is speech generation.
3. Token Diffusion Model: A diffusion transformer that takes the generated speech tokens as input and produces continuous latent representations in a coarse-to-fine manner, enhancing acoustic detail beyond what the discrete tokens alone provide.
4. Acoustic Vocoder: Separately trained, it converts the diffusion model's latent outputs into the final audio waveform. Design is informed by prior work on high-fidelity neural vocoders.

Seed-TTS mirrors the training paradigm of large language models, proceeding through three stages:

• Pre-training: Trained on data orders of magnitude larger than prior TTS systems, maximizing scenario and speaker coverage and establishing a robust general speech backbone.
• Fine-tuning: Consists of speaker fine-tuning (for selected target speakers) and instruction fine-tuning (for controllability and interactivity).
• Post-training: Conducted via reinforcement learning to holistically improve robustness, speaker similarity, and controllability.

Zero-Shot In-Context Learning (ICL)

Task Definition and Test Sets

The zero-shot ICL task, also called zero-shot voice continuation or voice cloning, is defined as generating a novel spoken utterance that shares the timbre and prosody of a short reference speech clip from any arbitrary speaker. The reference clip serves as an audio prompt; no speaker-specific fine-tuning is performed.

Two test sets were constructed:

• Objective set: 1,000 English samples from Common Voice and 2,000 Mandarin samples from DiDiSpeech. Each sample pairs a reference utterance with a target utterance from the same speaker (reference duration: 3–20 seconds).
• Subjective set: 100 in-house English and Mandarin samples covering expressive, diverse, and in-the-wild speech including diverse accents, dialects, emotions, and speaking styles.

Evaluation Metrics

• WER (Word Error Rate): Measured using Whisper-large-v3 for English and Paraformer-zh for Mandarin.
• SIM (Speaker Similarity): Cosine similarity of WavLM-large speaker embeddings between synthesized and reference speech.
• CMOS (Comparative Mean Opinion Score): Human evaluators compare synthesized speech against ground-truth human speech for the same target text, rating speaker similarity and expressiveness on a scale from −2 to +2. CMOS differences below 0.1 in absolute value are considered insignificant.

ICL Results

Seed-TTS achieves WER comparable to ground-truth human speech and substantially higher SIM scores. The elevated SIM relative to the human reference is attributed to Seed-TTS faithfully capturing and consistently reproducing the characteristics of the reference clip, whereas ground-truth utterances from the same speaker may still exhibit variability in style and environment. The CMOS scores of −0.07 (EN) and −0.08 (ZH) are both within the range of perceptual indistinguishability, marking the first reported instance of a TTS system producing zero-shot in-the-wild speech at this quality level. Prior systems evaluated under the same subjective protocol (e.g., VALL-E, Voicebox, NaturalSpeech 2, BASE TTS, MEGA-TTS) all scored below −1 CMOS.

An important observation is the tension between WER and naturalness: a lower WER typically indicates more "standardized," clearly pronounced speech that is easier for ASR to recognize, but may sacrifice naturalness, accent fidelity, and expressiveness — especially when the prompt speaker has a distinctive accent or highly expressive style.

Comparison Against Traditional Fine-Tuned TTS

Seed-TTS zero-shot ICL (using a 15-second audio prompt) was compared against traditional FastSpeech-based speaker fine-tuned TTS models (trained on roughly 5 hours of speaker data per speaker) across 10 speakers in two difficulty groups:

• Common speakers (7): Everyday, average speech.
• Hard speakers (3): Strong accents or highly unique/exaggerated speaking styles (e.g., an electronic high-pitched chipmunk character).

30 utterances per speaker were generated and rated by 10 human evaluators.

For common speakers, Seed-TTS zero-shot ICL was preferred in 47.9% of test samples, with evaluators noting clear advantages in naturalness and expressiveness. For hard speakers, the traditional fine-tuned model performed better, as accents and highly distinctive prosodic patterns were not always fully captured from the 15-second prompt. The authors note that longer prompts and better training data coverage should alleviate this limitation.

Speech Understanding Evaluation (ASR on Synthetic Data)

To further verify generation quality, a synthetic version of the LibriSpeech 960-hour training set was produced via a "text-wave shuffling" strategy: each utterance's audio is used as a voice prompt to synthesize a new sentence using randomly sampled text from the training set (each utterance and text used only once). An ASR model (12-layer Squeezeformer encoder + 3-layer bidirectional transformer decoder, trained with WeNet) was trained from scratch on this synthetic corpus and evaluated on real LibriSpeech dev/test sets.

The model trained on synthetic data achieves near-identical WER on clean sets. The absolute gap on noisy sets (dev_other: 1.81%, test_other: 1.60%) is attributed to Seed-TTS reducing background noise during generation, making the synthetic data cleaner than real noisy recordings. This demonstrates the potential of using Seed-TTS to generate synthetic training data for speech understanding tasks, pointing toward a unification of speech understanding and generation.

t-SNE Visualization of Speaker Embeddings

To verify timbre preservation, English utterances from the VoxCeleb1 test set were synthesized using the same shuffling strategy, and WavLM-based speaker embeddings were computed for both ground-truth and synthesized speech. t-SNE projection over 25 speakers shows that embeddings from the same speaker reliably cluster together regardless of whether the utterance is real or synthesized.

Speaker Fine-Tuning and Instruction Fine-Tuning

Speaker Fine-Tuning (SFT)

Speaker fine-tuning is performed on top of the pre-trained Seed-TTS base model. In the reported experiment, 5 speakers (3 female, 2 male), each with 1–10 hours of data (20 hours total), were used to fine-tune the model. An additional speaker index token was added to select the target speaker timbre at inference time. For the base model (ICL), a randomly sampled 20-second clip was used as the audio prompt.

The fine-tuned model achieves similar objective metrics but demonstrates a clear subjective advantage (CMOS +0.37). Human evaluators observed that Seed-TTS_SFT captures more subtle speaker nuances — distinctive pronunciation patterns, subtle prosody changes at sentence ends — not fully reproduced by the 20-second audio prompt alone.

Instruction Fine-Tuning (IFT) for Emotion Control

Instruction fine-tuning (IFT) is applied on top of the speaker fine-tuned model to enable flexible, explicit control over speech attributes including expressiveness, speaking rate, style, and emotion. Emotion control accuracy was measured using a speech emotion recognition (SER) model (WavLM-large fine-tuned for SER) on 100 synthesized utterances per emotion, where the text content was designed to match the target emotion. Four primary emotions were evaluated: angry, happy, sad, and surprised.

Even without explicit control signals, Seed-TTS_SFT achieves moderate emotion accuracy because the model infers the appropriate emotion from textual content. Adding explicit emotion control signals (IFT) brings accuracy close to perfect across all four emotions. Notably, the model can generalize to a much wider range of emotions than reported here.

Low-Latency Inference and Deployment

Deploying Seed-TTS in real-world applications — especially chat-based or streaming scenarios — requires significant engineering effort to reduce latency, real-time factor (RTF), memory, and computation. The authors address this via several techniques:

• Causal diffusion architecture: Enables streaming processing in the diffusion module, significantly reducing first-packet delay.
• Consistency distillation and modified flow matching: Reduces the number of diffusion steps (and thus computation cost) for the diffusion model.
• Language model optimizations: Grouped-query attention (GQA), paged attention, flash attention, and model quantization reduce memory and computation on the language model side.

The deployed model achieves a 35× reduction in latency and approximately 7.6× reduction in RTF while maintaining essentially identical WER and SIM, with a negligible CMOS difference of −0.02.

Speech Factorization by Self-Distillation

Motivation and Prior Approaches

Speech factorization — decomposing speech into disentangled, independently controllable attributes such as timbre, prosody, and content — is central to applications like zero-shot voice conversion and factorized TTS. Most prior approaches require explicit feature engineering, specialized loss functions, or precise model architecture design. Integrating these into a general-purpose system like Seed-TTS is challenging.

Self-Distillation Scheme for Timbre Disentanglement

The proposed approach avoids any changes to the base model architecture or loss function. The core idea is to generate controlled speech pairs that share all attributes except for the target attribute being disentangled (timbre).

Concretely: by introducing speaker perturbation into the diffusion module during Seed-TTS generation, it is possible to produce synthetic speech with the same content and prosody as the original but with a shifted timbre. Let $S_{\text{ori}}$ denote the original synthetic utterance and $S_{\text{alt}}$ denote the timbre-altered version.

The diffusion model is then retrained using these pairs as follows:

• The token sequence extracted from $S_{\text{alt}}$ (timbre-shifted, content/prosody preserved) is used as network input.
• A timbre reference embedding extracted from $S_{\text{ori}}$ is provided as an additional conditioning input.
• The network is trained to recover the vocoder embeddings of $S_{\text{ori}}$.

Since $S_{\text{alt}}$ and $S_{\text{ori}}$ share content and prosody but differ in timbre, and the goal is to recover $S_{\text{ori}}$, the model must learn to ignore the timbre embedded in the input token sequence and rely solely on the provided timbre reference. This results in high-quality timbre disentanglement without modifying the Seed-TTS architecture or loss.

Zero-Shot Voice Conversion Results

Voice conversion (VC) changes the speaker identity of source speech to a target timbre while preserving spoken content. In the Seed-TTS VC pipeline, only the diffusion module is involved (content and prosody come from the source speech tokens). The VC test set is derived from the objective ICL test set, with non-matching speaker references randomly assigned.

The self-distillation approach yields a dramatic improvement in SIM (from 0.636 to 0.791 for ZH, from 0.491 to 0.753 for EN) while also improving WER, outperforming open-source SOTA voice conversion baselines by a large margin.

Preference Biasing Through Reinforcement Learning

Background and Motivation

Reinforcement learning (RL) has been successfully applied in text and image generation to align model outputs with human preferences (e.g., PPO, DPO, REINFORCE). Seed-TTS extends these paradigms to speech generation. Both external-reward methods (PPO, REINFORCE) and reward-model-free methods (DPO) are explored. REINFORCE with external reward models is the primary method demonstrated in this report.

RL Fine-Tuning Variants

• Seed-TTS_RL-SIM-WER: REINFORCE applied to Seed-TTS_ICL using WER and SIM as reward signals, targeting improved robustness and speaker similarity.
• Seed-TTS_RL-SER: REINFORCE applied using speech emotion recognition (SER) accuracy as the reward, targeting improved emotion controllability.

A new "hard" text test set was constructed with 400 sentences containing challenging patterns for autoregressive models: word repetitions, tongue twisters, and similar constructs.

RL Results: Robustness and Speaker Similarity

RL improves both objective metrics and subjective preference, particularly on the hard text set where WER drops from 7.585 to 6.423. The CMOS improvement of +0.14 is statistically significant.

RL Results: Emotion Controllability

RL-based emotion training dramatically increases emotion accuracy in a zero-shot setting — from as low as 0.13 (surprise) to 0.82. This highlights that RL can serve as an effective post-training tool for controllability without requiring speaker-specific fine-tuning.

Reward Hacking

The authors report observing reward hacking in their RL experiments. For example, optimizing for lower WER causes the model to produce slower, more clearly pronounced speech — improving ASR metrics but reducing naturalness. This aligns with the observation in the ICL evaluation that excessively low WER corresponds to more "standardized" but less natural speech. Careful reward weighting and network tuning are required to balance competing objectives.

Fully Diffusion-Based Variant: Seed-TTS_DiT

Design Philosophy and Architecture

Language model-based and diffusion-based approaches represent the two dominant paradigms in modern generative AI. For speech, this comparison is less studied than for images. To systematically compare them, the authors develop Seed-TTS_DiT, a fully diffusion-based variant that removes the dependency on the acoustic tokenizer and language model entirely.

Key architectural properties of Seed-TTS_DiT:

• End-to-end processing: The diffusion model converts Gaussian noise directly to the vocoder latent representation, conditioned solely on the input text and audio prompt — no acoustic tokenizer or language model is involved.
• No pre-estimated phoneme durations: Unlike prior NAR systems (FastSpeech, Voicebox, UniCATS, MEGA-TTS), Seed-TTS_DiT does not require a separate phoneme duration predictor. Instead, the model estimates total utterance duration and dynamically infers local text-audio alignment end-to-end.
• Total duration conditioning: Rather than training the model to predict total duration itself (which is feasible), the authors choose to provide total duration as an explicit input. This enables duration manipulation, content editing, and speaking rate editing as additional inference-time controls.

Formally, the model receives an audio prompt (reference speech), target text, and a Gaussian noise tensor of the target total duration, and produces a latent representation of the generated speech of that same duration. The vocoder then transforms this into the final waveform.

Zero-Shot TTS Performance

Seed-TTS_DiT achieves the best SIM scores in both languages and the best WER in English. In Mandarin, the autoregressive Seed-TTS_ICL achieves a slightly lower WER (1.115 vs. 1.178), while Seed-TTS_DiT leads on SIM. These results demonstrate strong sequence modeling capability within the diffusion paradigm and show that the two approaches are broadly comparable in quality.

Content Editing

A natural advantage of the non-autoregressive, non-streaming diffusion architecture is support for masked content editing. In content editing experiments, a certain percentage of the target audio is masked, and the model fills in the masked portions based on the provided text while preserving the unmasked context. WER and SIM are reported as a function of the masking rate.

The model maintains robustness across a wide range of masking rates, with SIM measuring the similarity between recovered audio and the original speaker. This enables practical applications such as word replacement or phrase editing in existing recordings.

Speaking Rate Editing

Speaking rate editing is implemented by simply providing a modified total duration at inference time (e.g., multiplying the original duration by a speed rate). The model automatically adjusts the speaking rate — inserting natural silence at appropriate positions or stretching vowel pronunciations — rather than uniformly time-stretching the audio. This produces more natural results than traditional signal-processing rate modification methods.

SIM remains high across most speed rates, with a slight WER degradation observed only at very high speed rates.

Trade-offs: Autoregressive LM vs. Diffusion

The two paradigms offer complementary strengths:

• Diffusion (Seed-TTS_DiT): Simpler end-to-end pipeline; naturally supports content editing, speaking rate editing, and other masked-infilling tasks; does not require a separate acoustic tokenizer. Limited by its non-streaming nature.
• Autoregressive LM (Seed-TTS_ICL): Supports streaming generation, enabling low-latency applications; can more easily be integrated with text-based language models for multi-modal and conversational applications.

Datasets and Training Scale

While the paper does not exhaustively enumerate all training data sources for the base model, it explicitly states that Seed-TTS is trained on data orders of magnitude larger than previous TTS systems — including the previously largest TTS systems — during the pre-training stage. This scale is cited as a primary driver of the model's generalization, robustness, and emergent capabilities.

For specific experiments, the following datasets are used:

• Common Voice: 1,000 English samples for the objective ICL test set.
• DiDiSpeech: 2,000 Mandarin samples for the objective ICL test set.
• LibriSpeech 960h: Used for the speech understanding (ASR on synthetic data) evaluation.
• VoxCeleb1: Used for t-SNE speaker embedding visualization.
• In-house datasets: 100 samples per language for the subjective set; 5 speakers with 1–10 hours each for speaker fine-tuning; 20+ hours for fine-tuning experiments; 400 "hard" sentences for RL robustness evaluation.

Key Contributions Summary

• Human-level zero-shot TTS: First reported TTS system to achieve perceptual indistinguishability from real human speech in a zero-shot ICL setting with in-the-wild prompts (CMOS of −0.07 EN, −0.08 ZH).
• Self-distillation for speech factorization: A novel, architecture-agnostic method for timbre disentanglement via generation of controlled speech pairs, achieving SOTA performance in zero-shot voice conversion (SIM: 0.753 EN, 0.791 ZH) without altering base model structure or loss.
• RL-based post-training: Application of REINFORCE with WER/SIM and SER reward models to holistically improve robustness (WER reduction across all test sets), speaker similarity, and emotion controllability in zero-shot settings.
• Seed-TTS_DiT: Fully diffusion-based TTS without a separate duration predictor or acoustic tokenizer; achieves comparable or superior performance to the autoregressive variant while natively supporting content editing and speaking rate editing.
• Deployment engineering: Streaming causal diffusion, consistency distillation, GQA, flash attention, and quantization enabling a 35× latency reduction and 7.6× RTF improvement with negligible quality loss.
• Open benchmark: Release of the objective test set configuration to the community for standardized benchmarking.

Applications

Seed-TTS enables upgrades across several production applications:

• Voice chats and virtual assistants: Low-latency, streaming, expressive speech in user-facing products.
• Audiobooks: High-fidelity, expressive narration with speaker consistency across long-form content.
• Video dubbing and content creation: Cross-lingual speech transfer and voice adaptation.
• Accessibility: Aid for patients with speech impairments; bridging language barriers.
• Speech understanding data augmentation: Synthetic data generation for ASR model training.

Limitations

The authors candidly acknowledge several limitations:

• Nuanced emotion and context understanding: Despite emergent behavior, the model struggles with scenarios requiring deep reasoning or subtle emotional context — e.g., generating genuinely sarcastic speech or correctly handling Mandarin text that requires complex reasoning for correct pronunciation.
• Scenario coverage gaps: Despite massive training data, edge cases remain. Seed-TTS does not perform well at singing, and prompts containing background music or excessive noise often lead to inconsistent background in the generated output (e.g., ignoring music altogether).
• Prosodic consistency in long-form generation: The model maintains consistent prosody derived from the reference, which improves short-clip similarity but may result in fewer prosodic variations for long-form generation. Multi-shot ICL is identified as a promising direction.
• Hard speaker fidelity in zero-shot ICL: Highly distinctive accents or prosodic styles may not be fully captured from a 15-second prompt alone.
• Reward hacking in RL: Optimizing for WER can produce unnaturally slow or over-articulated speech. Careful balancing of reward signals is required.
• Streaming limitation of Seed-TTS_DiT: The fully diffusion-based variant does not natively support streaming generation, making it less suitable for low-latency applications.

Safety and Responsible AI

Given the potential for misuse — particularly for voice fraud, deepfakes, and unauthorized speaker cloning — Seed-TTS incorporates multiple safety measures in related products:

• Multi-step verification: For both spoken content and speaker timbre, ensuring that enrollment audio contains only the voice of authorized users.
• Multi-level watermarking: Mandatorily embedded at multiple levels in created content, including video background watermarks and watermarks in content descriptions.

The authors emphasize that the capabilities of Seed-TTS — particularly high-fidelity voice cloning from short prompts — give rise to significant challenges in multimedia forensics and audio deepfake detection that must be carefully studied for their potential societal impact.

bash cal_wer.sh {meta_file} {synthesized_audio_dir} {language: zh or en}
bash cal_sim.sh {meta_file} {synthesized_audio_dir} {wavlm_checkpoint_path}