NaturalSpeech 3

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

A zero-shot TTS system that factorizes speech into disentangled subspaces (content, prosody, timbre, acoustic details) via a novel codec and separate diffusion models for each attribute. This divide-and-conquer approach overcomes tight coupling in prior methods, achieving human-parity quality.

• tts
• voice-cloning
• prosody
• speech-driven
• multimodal

Authors: Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng Zhao

Categories: eess.AS, cs.AI, cs.CL, cs.LG, cs.SD

Comment: Achieving human-level quality and naturalness on multi-speaker datasets (e.g., LibriSpeech) in a zero-shot way

Published 2024-03-05 · Updated 2024-04-23

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face177 likes

Impact

• Semantic Scholar391 citations35 influential citations

Abstract

While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech in a zero-shot way. Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt. With this factorization design, NaturalSpeech 3 can effectively and efficiently model intricate speech with disentangled subspaces in a divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, and achieves on-par quality with human recordings. Furthermore, we achieve better performance by scaling to 1B parameters and 200K hours of training data.

---

Introduction and Motivation

Text-to-speech (TTS) synthesis has seen remarkable progress in recent years, with large-scale systems demonstrating convincing zero-shot generation for unseen speakers. Yet these systems continue to fall short in three key dimensions: speech quality, speaker similarity, and prosody expressiveness. The root cause is the inherent complexity of speech itself: a single utterance simultaneously encodes content (the linguistic message), prosody (rhythm, intonation, emphasis), timbre (speaker identity and voice character), and fine acoustic details (high-frequency texture, recording conditions, etc.).

Prior approaches to zero-shot TTS can be grouped into four paradigms based on representation type and generation strategy: (1) discrete tokens + autoregressive models (e.g., VALL-E), (2) discrete tokens + non-autoregressive models (e.g., SoundStorm), (3) continuous vectors + autoregressive models, and (4) continuous vectors + non-autoregressive models (e.g., NaturalSpeech 2, Voicebox). The dominant token-based methods rely on Residual Vector Quantization (RVQ) neural codecs (SoundStream, EnCodec), which decompose speech into hierarchical levels but do not effectively disentangle different speech attributes across those levels. As a result, every generation step must cope with tightly coupled, complex information.

NaturalSpeech 3 (NS3) proposes a principled divide-and-conquer strategy: factorize speech into disentangled subspaces — one per attribute — and generate each subspace independently, conditioned on an attribute-specific prompt. This motivates two core innovations: (1) FACodec, a neural speech codec with Factorized Vector Quantization (FVQ) that disentangles content, prosody, timbre, and acoustic details; and (2) a Factorized Diffusion Model that generates duration, prosody, content, and acoustic detail tokens sequentially, each guided by its own prompt and conditions.

System Overview

NaturalSpeech 3 consists of two major components that operate in sequence:

• FACodec — a neural speech codec that encodes a speech waveform into four disentangled subspaces: content tokens $z_c$, prosody tokens $z_p$, acoustic detail tokens $z_d$, and a global timbre vector $h_t$. The codec decoder reconstructs a high-fidelity waveform from these four representations.
• Factorized Diffusion Model — a non-autoregressive generation system that takes phoneme text as input and produces duration, then prosody tokens, then content tokens, then acoustic detail tokens, each step conditioned on the previous outputs and on attribute-specific speech prompts extracted by FACodec from a reference utterance.

At inference time, a reference speech clip (typically 3 seconds) is passed through FACodec to yield attribute prompts. The factorized diffusion model then generates each attribute code sequence guided by the corresponding prompt, and the codec decoder synthesizes the final waveform. Crucially, timbre is never explicitly generated; the timbre embedding $h_t$ is taken directly from the prompt, offloading speaker identity from the generative model entirely.

FACodec: Factorized Neural Speech Codec

Architecture

FACodec follows the general encoder–quantizer–decoder structure of prior neural codecs but replaces the standard RVQ stack with a set of independently learned Factorized Vector Quantizers (FVQs). The pipeline for a waveform $x$ at 16 kHz is:

1. Speech Encoder. Convolutional blocks with a downsampling factor of 200 (each frame = 12.5 ms) map the raw waveform to a pre-quantization latent $h$.
2. Timbre Extractor. A Transformer encoder processes $h$ and produces a single global vector $h_t$ representing timbre. This is the only attribute that is globally pooled rather than frame-wise.
3. Three FVQs. Separate FVQs quantize $h$ into:
   • Prosody codes $z_p$ via $\mathcal{Q}^p$ ($N_{q_p} = 1$ codebook)
   • Content codes $z_c$ via $\mathcal{Q}^c$ ($N_{q_c} = 2$ codebooks)
   • Acoustic detail codes $z_d$ via $\mathcal{Q}^d$ ($N_{q_d} = 3$ codebooks)
   All codebooks have size 1024. Each FVQ first projects $h$ into an 8-dimensional bottleneck space (information bottleneck), quantizes there, then projects back to the original dimension.
4. Speech Decoder. Mirrors the encoder architecture with more parameters. The frame-level representations $z_p + z_c + z_d$ are combined and then conditioned on $h_t$ via Conditional Layer Normalization (AdaSpeech-style) to obtain the decoder input $z$, from which the waveform is reconstructed.

The encoder and decoder backbone follows the DAC architecture and uses the SnakeBeta activation function. The timbre extractor uses Conformer blocks. Total bandwidth is 4.8 kbps at 16 kHz with hop size 200 and 6 total codebooks.

Attribute Disentanglement Techniques

Simply using separate quantizers does not guarantee disentanglement. FACodec employs four complementary techniques:

1. Information Bottleneck

Before each FVQ, the encoder output is projected to an 8-dimensional space. This low-dimensional bottleneck forces each quantizer to discard irrelevant information and focus only on its designated attribute. After quantization, the code is projected back to the full dimension. Ablation confirms that removing this bottleneck causes a 0.13 drop in speaker similarity for zero-shot voice conversion.

2. Supervised Auxiliary Losses

Each attribute subspace is supervised to encode its intended information:

• Prosody ($z_p$): The post-quantization latent predicts frame-level normalized F0 (z-score: $\frac{\text{F0} - \mu_{\text{F0}}}{\sigma_{\text{F0}}}$). Loss weight $\lambda_{\text{f0}} = 5.0$.
• Content ($z_c$): Predicts frame-level phoneme labels from an internal forced aligner. Loss weight $\lambda_{\text{ph}} = 5.0$.
• Timbre ($h_t$): Speaker classification cross-entropy on speaker ID.

3. Gradient Reversal (Adversarial Disentanglement)

Gradient Reversal Layers (GRL) prevent information leakage across subspaces:

• Prosody subspace: phoneme-GRL (removes content leakage from $z_p$). $\lambda_{\text{gr-ph}} = 5.0$.
• Content subspace: F0-GRL (removes prosody leakage from $z_c$). $\lambda_{\text{gr-f0}} = 5.0$.
• Acoustic detail subspace: both phoneme-GRL and F0-GRL (removes content and prosody from $z_d$).
• Sum $z_p + z_c + z_d$: speaker-GRL (removes timbre from the combined representation). $\lambda_{\text{gr-spk}} = 1.0$.

4. Detail Dropout

During training, $z_d$ is randomly masked out with probability $p$. This forces the decoder to reconstruct plausible (if lower-quality) speech from $z_p$, $z_c$, and $h_t$ alone, ensuring that content, prosody, and timbre information is not offloaded into the detail subspace. When $z_d$ is present, the decoder can recover full quality.

Full Training Objective (FACodec)

The complete generator loss combines reconstruction, adversarial, disentanglement, and codebook terms:

Coefficients: $\lambda_{\text{rec}}=10.0$, $\lambda_{\text{adv}}=2.0$, $\lambda_{\text{feat}}=2.0$, $\lambda_{\text{codebook}}=1.0$, $\lambda_{\text{commit}}=0.25$, $\lambda_{\text{f0}}=5.0$, $\lambda_{\text{ph}}=5.0$, $\lambda_{\text{gr-f0}}=5.0$, $\lambda_{\text{gr-ph}}=5.0$, $\lambda_{\text{gr-spk}}=1.0$. The adversarial discriminator uses a Multi-Period Discriminator (MPD) and a Multi-Band Multi-Scale STFT discriminator. Training: 8× NVIDIA V100 32GB GPUs, batch size 32 clips × 16000 frames, 800K steps, Adam optimizer ($\text{lr} = 2\times10^{-4}$, $\beta_1=0.5$, $\beta_2=0.9$).

Factorized Diffusion Model

Generation Order and Conditioning

The factorized diffusion model generates speech attributes in a fixed causal order, each step conditioned on all previously generated attributes plus its specific prompt:

1. Phoneme Encoding: Text phonemes are encoded by a phoneme encoder (6-layer Transformer, 512-dim, 8 heads, filter size 2048).
2. Phoneme-level Prosody + Duration Diffusion: A discrete diffusion model generates phoneme-level duration tokens conditioned on phoneme-level prosody codes (obtained by pooling pre-quantized prosody vectors per phoneme and re-quantizing) and the phoneme encoder output. A length regulator expands phoneme-level features to frame level: $c_{\text{ph}}$.
3. Prosody Diffusion: Generates frame-level prosody codes $z_p$ conditioned on a prosody prompt (no noise) and $c_{\text{ph}}$.
4. Content Diffusion: Generates content codes $z_c$ conditioned on a content prompt, generated prosody $z_p$, and $c_{\text{ph}}$.
5. Acoustic Detail Diffusion: Generates detail codes $z_d$ conditioned on a detail prompt, generated $z_p$, $z_c$, and $c_{\text{ph}}$.
6. Waveform Synthesis: The codec decoder reconstructs speech from $z_p, z_c, z_d$ and the timbre vector $h_t$ extracted from the reference prompt.

Discrete Diffusion Formulation

All four diffusion modules (phoneme-level prosody, duration, prosody, content, acoustic detail) share the same absorbing (mask-based) discrete diffusion formulation.

Forward Process

Given a target discrete token sequence $\mathbf{X} = [x_i]_{i=1}^{N}$, the forward process at time $t$ applies a binary mask $\mathbf{M}_t = [m_{t,i}]_{i=1}^N$ where each $m_{t,i} \stackrel{\text{iid}}{\sim} \text{Bernoulli}(\sigma(t))$. Masked tokens are replaced by a special [MASK] token:

The masking schedule uses a sinusoidal function: $\sigma(t) = \sin\!\left(\frac{\pi t}{2T}\right)$ for $t \in (0, T]$, so that $\mathbf{X}_0 = \mathbf{X}$ (unmasked) and $\mathbf{X}_T$ is fully masked.

Reverse Process and Training Objective

The model $p_\theta$ is trained to predict the original tokens $\mathbf{X}_0$ from partially masked $\mathbf{X}_t$, conditioned on a prompt sequence $\mathbf{X}^p$ (concatenated without noise) and condition $\mathbf{C}$. The loss is the negative log-likelihood over masked positions:

The reverse transition marginalizes over the model prediction:

Inference Procedure

Starting from a fully masked sequence $\mathbf{X}_T$, at each step:

1. Sample $\hat{\mathbf{X}}_0$ from $p_\theta(\mathbf{X}_0 \mid \mathbf{X}_t, \mathbf{X}^p, \mathbf{C})$.
2. Re-mask $\lfloor N \cdot \sigma(t - \Delta t) \rfloor$ tokens with the lowest confidence score to obtain $\mathbf{X}_{t - \Delta t}$. The confidence of masked position $i$ is $p_\theta(\hat{x}_i \mid \mathbf{X}_t, \mathbf{X}^p, \mathbf{C})$; already-unmasked tokens have confidence 1 and are never re-masked.

Top-k sampling ($k=20$) is used with temperature annealing from 1.5 to 0. Gumbel noise is added to token confidences following MaskGIT.

Classifier-Free Guidance (CFG)

During training, the prompt $\mathbf{X}^p$ is dropped with probability $p_{\text{cfg}} = 0.15$. At inference, the logits are extrapolated:

Then rescaled to match the variance of the conditional output:

A guidance scale of $\alpha = 1.0$ is used for prosody, content, and acoustic detail. Duration is generated without CFG. Each diffusion stage runs for 4 iterations. With CFG requiring a double forward pass, the total inference cost is: $4 \times 2$ (phoneme-level prosody) $+ 4$ (duration) $+ 4 \times 2$ (prosody) $+ 4 \times 2$ (content) $+ 4 \times 2$ (acoustic detail) $= 60$ forward passes.

Model Architecture Details

The main diffusion backbone (prosody, content, acoustic details) is a shared 12-layer Transformer with 8 attention heads, 1024-dimensional embeddings, filter size 2048, 1D convolution kernel size 3, and dropout 0.1. Each Transformer block uses Conditional Layer Normalization to incorporate the diffusion timestep. The phoneme-level prosody and duration diffusion use a 6-layer Transformer with the same head count and dimension. Training: 8× A100 80GB GPUs, batch size 10K frames/GPU, 1M steps, AdamW ($\text{lr} = 10^{-4}$, $\beta_1=0.9$, $\beta_2=0.98$, 5K warmup steps with inverse square root schedule).

Connection to the NaturalSpeech Series

NaturalSpeech 3 is the third entry in Microsoft's NaturalSpeech lineage:

• NaturalSpeech 1: Flow-based model; achieved human-level quality on the single-speaker LJSpeech dataset using a VAE-based continuous representation.
• NaturalSpeech 2: Latent diffusion model; leveraged continuous RVQ codec representations for zero-shot multi-speaker synthesis on large-scale datasets.
• NaturalSpeech 3: Introduces factorized discrete diffusion; achieves human-level naturalness on the multi-speaker LibriSpeech test set — the first system to do so — via attribute disentanglement with FACodec.

Experimental Setup

Training Data

The primary training set is Librilight (60K hours, ~7000 speakers, 16 kHz, from LibriVox audiobooks). Transcriptions are produced by an internal ASR system; phoneme conversion uses grapheme-to-phoneme tools; durations are obtained from an internal forced aligner. For scaling experiments, a 1K-hour subset of Librilight and an internal 200K-hour dataset are also used.

Evaluation Benchmarks

• LibriSpeech test-clean: 40 speakers, 5.4 hours. One sentence per speaker; 3-second clips from the same speaker as prompt. Used for quality, similarity, and robustness evaluation.
• RAVDESS: 24 professional actors, 8 emotions × 2 intensities. Strong-intensity samples used. Designed for prosody evaluation: same speaker, same text, different emotions.

Evaluation Metrics

• Sim-O / Sim-R: WavLM-TDCNN speaker similarity to original/reconstructed prompt.
• UTMOS: Automatic MOS surrogate for speech quality.
• WER: Word error rate from a CTC HuBERT ASR model (and additionally a Conformer transducer for WER*).
• CMOS: Comparative MOS (12 native-speaker judges, 20 utterances per benchmark); reference is a system designated score 0.
• SMOS: Similarity MOS (12 judges, 10 utterances).
• MCD (Mel-Cepstral Distortion) / MCD-Acc: RAVDESS prosody metrics. MCD measures spectral distance to ground truth; MCD-Acc measures top-1 emotion accuracy via KNN classifier over MCD distances.

Baselines

VALL-E (AR + discrete), NaturalSpeech 2 (NAR + continuous), Voicebox (NAR + continuous), Mega-TTS 2 (NAR + continuous), UniAudio (AR + discrete), StyleTTS 2 (NAR + continuous), HierSpeech++ (NAR + continuous).

Results

Zero-Shot TTS — LibriSpeech test-clean

NS3 achieves CMOS = 0.00 (on par with ground truth, which scores +0.08), Sim-O = 0.67 (vs. GT 0.68), SMOS = 4.01 (vs. GT 3.85), and WER = 1.81 (better than GT 1.94). This constitutes the first demonstration of human-level naturalness on the multi-speaker LibriSpeech test set in a zero-shot setting. The UTMOS score is 4.30 vs. ground truth 4.14, further confirming perceptual quality parity.

Zero-Shot TTS — RAVDESS (Prosody)

NS3 achieves the lowest average MCD and highest MCD-Acc, demonstrating substantially better prosody cloning across all 8 emotions. The MCD-Acc of 0.52 vs. the next best (StyleTTS 2 at 0.40) shows that the factorized prosody subspace enables far more faithful emotion transfer.

FACodec Reconstruction Quality

FACodec substantially outperforms SoundStream at identical bandwidth (+0.44 PESQ, +0.05 STOI, −0.14 MSTFT, −0.79 MCD) despite adding the disentanglement overhead. FACodec is competitive with or better than EnCodec and DAC on most metrics while operating with a shorter hop size and explicit attribute factorization.

Ablation Studies

Factorization

Removing factorization (replacing FACodec with SoundStream tokens and eliminating factorized generation) causes −0.12 Sim-O, −0.15 Sim-R, +0.68 WER, −0.25 CMOS, and −0.42 SMOS. This is the largest single ablation impact, validating the central importance of the factorization design. Removing CFG causes smaller but still significant drops of −0.03 Sim-O and −0.21 SMOS.

Prosody Representation

Using the learned FVQ prosody tokens from FACodec outperforms manually designed features (low-frequency mel bins used in Mega-TTS, DiffProsody, etc.), demonstrating that the codec's data-driven prosody representation is more expressive and effective for prosody transfer.

Duration Diffusion Design

Each design choice in the duration module contributes incrementally. Duration prompting (providing a reference duration distribution from the prompt) is the single most important component, particularly for WER (+0.89) and speaker similarity (−0.06 Sim-O).

FACodec Component Ablations

• Information Bottleneck: Removing it drops zero-shot voice conversion Sim-O by 0.13 (from 0.86 to 0.73), indicating incomplete timbre disentanglement.
• Gradient Reversal: Without GRL on acoustic details, content and pitch information leaks into $z_d$ (audible as partial phonetic content when reconstructing from $z_d$ alone).
• Acoustic Detail Quantizers: Without $z_d$ (3 codebooks only), PESQ drops from 3.47 to 3.09. Yet even with only 3 codebooks, FACodec outperforms SoundStream with the same 6 codebooks on PESQ (3.09 vs. 3.03) and STOI (0.92 vs. 0.90), showing that $z_p$ and $z_c$ capture most reconstruction-relevant information.

Extensibility: Factorization with Autoregressive Models

The factorization framework is not limited to non-autoregressive generation. To demonstrate this, the authors integrate FACodec tokens into VALL-E (denoted VALL-E + FACodec), where an autoregressive language model generates prosody codes, and a non-autoregressive model generates content and acoustic detail codes.

VALL-E + FACodec consistently outperforms vanilla VALL-E on all metrics, demonstrating that the factorization paradigm provides value independent of the underlying generative model architecture.

Zero-Shot Voice Conversion with FACodec

FACodec enables zero-shot voice conversion directly, without any task-specific fine-tuning. Given source speech and a target-speaker prompt, voice conversion is performed as:

The source speaker's content, prosody, and detail tokens are decoded together with the target speaker's timbre embedding. On VCTK:

FACodec achieves near-SOTA speaker similarity (0.86 vs. 0.87 for UniAudio) while obtaining the best WER (3.46), close to the ground truth WER of 3.25 — all without any dedicated VC training.

Speech Attribute Manipulation

The factorized design allows independent control over each attribute at inference by swapping attribute-specific prompts from different reference utterances. Practically demonstrated manipulations include:

• Timbre swap: Use $h_t$ from a different speaker while keeping the same prosody and content generation conditions.
• Speed control: Use a duration prompt from a slower/faster speaker to regulate speech rate, even though duration and prosody are correlated.
• Cross-attribute combination: Combine timbre from speaker A, prosody from speaker B, and speed from speaker C in a single utterance — mimicking a speaker's voice while adopting a different emotional register and pace.

Scaling Analysis

Data Scaling (500M Parameters)

Performance improves monotonically with data scale. Even at 1K hours, the factorization provides a strong starting point (Sim-O = 0.64). Scaling from 60K to 200K yields a further +0.01 Sim-O and −0.92 WER improvement.

Model Scaling (200K Hours)

Doubling the Transformer depth (12 → 24 layers) yields +0.05 Sim-O and −0.40 WER. The results confirm consistent scaling behavior both in data and model dimensions.

Latency Analysis

NS3 is 15.27× faster than VALL-E and 1.24× faster than NaturalSpeech 2 (on V100 GPU), while achieving superior quality. Reducing to 1 diffusion step per module (15 total NFE) yields a 4.41× speedup with minimal quality degradation (−0.01 Sim-O, −0.01 Sim-R, −0.29 UTMOS).

Limitations and Future Work

The authors identify several limitations:

• Attribute Coverage: The five factorized attributes (content, prosody, duration, acoustic details, timbre) do not cover all speech aspects. Background sounds and energy are notable omissions. Future work plans to extend factorization to energy and environmental acoustics.
• Data Coverage: NS3 is trained exclusively on English LibriVox audiobook data. It does not generalize to diverse real-world voices or multilingual scenarios. Expanding to more diverse and multilingual data is an explicit future direction.
• FACodec Scalability: Content supervision requires phoneme transcriptions, limiting scalability to languages or domains without good ASR or alignment tools. Developing unsupervised disentanglement methods is a stated future goal.
• Task Scope: The disentanglement has only been validated in the TTS task. Generalizing FACodec to other tasks (ASR, speaker diarization, voice conversion at scale) remains unexplored.
• Broader Impact / Misuse: The high speaker similarity achieved by NS3 raises concerns about spoofing and impersonation. The authors call for development of synthetic speech detectors and abuse reporting mechanisms.

Summary of Key Contributions

• FACodec: A novel neural speech codec with factorized vector quantization that learns disentangled subspaces for content, prosody, acoustic details, and timbre, using information bottleneck, supervised auxiliary losses, gradient reversal, and detail dropout — achieving competitive reconstruction quality while enabling attribute-level disentanglement.
• Factorized Diffusion Model: A non-autoregressive generation system using absorbing discrete diffusion with attribute-specific prompts, sequential conditioning, and classifier-free guidance. Generates duration → phoneme-level prosody → frame-level prosody → content → acoustic details, with timbre taken directly from the reference.
• Human-Level Quality on LibriSpeech: For the first time, a zero-shot TTS system achieves on-par CMOS with ground-truth recordings on the diverse multi-speaker LibriSpeech test set.
• SOTA on All Key Metrics: Sim-O 0.67 (vs. prior best 0.64), SMOS 4.01 (vs. 3.71), WER 1.81 (vs. 1.94 for ground truth), MCD-Avg 4.28 (vs. next best 4.44), MCD-Acc 0.52 (vs. next best 0.40 on RAVDESS).
• Scalable Architecture: Demonstrated consistent gains from 1K to 200K hours of training data, and from 500M to 1B parameters.
• Speech Attribute Manipulation: Independent control over timbre, prosody, duration, and acoustic details via attribute-specific prompt swapping.
• Efficient Inference: 60 NFE with RTF = 0.296; degrades gracefully to 15 NFE with RTF = 0.067, still outperforming baselines.