NaturalSpeech 2

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

NaturalSpeech 2 uses latent diffusion on continuous audio codes for natural zero-shot speech and singing synthesis. It overcomes autoregressive model issues and employs speech prompting to capture style and prosody from references, enabling synthesis of unseen voices with high quality and robustness.

• tts
• voice-cloning
• prosody
• speech-to-speech

Demos

NaturalSpeech 2 demos showcase state-of-the-art text-to-speech synthesis with natural voice quality, strong prosody and timbre similarity, and robust zero-shot voice cloning including unseen speakers and singing. Listen for stable, expressive intonation and clear articulation without skipping or repeating, as well as the model's ability to synthesize singing and speech enhancement. The overview diagram and performance table highlight its novel latent diffusion architecture and superior metrics over prior TTS systems.

Authors: Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian

Categories: eess.AS, cs.AI, cs.CL, cs.LG, cs.SD

Comment: A large-scale text-to-speech and singing voice synthesis system with latent diffusion models. Update: NaturalSpeech 2 extension to voice conversion and speech enhancement

Published 2023-04-18 · Updated 2023-05-30

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar388 citations22 influential citations

Abstract

Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

---

Introduction

NaturalSpeech 2 addresses large-scale, multi-speaker, in-the-wild text-to-speech (TTS) and singing synthesis. The paper’s central claim is that the common large-scale recipe of discretizing speech into long token sequences and generating them autoregressively is a poor fit for high-diversity speech: it can produce unstable prosody, word skipping or repetition, and reduced voice quality. NaturalSpeech 2 replaces that pipeline with a continuous latent representation from a neural audio codec and a latent diffusion model that predicts the latent vectors conditioned on text and a speech prompt.

The design goal is not only naturalness, but also zero-shot controllability: the model should synthesize unseen speakers, follow prompt prosody, and generalize to singing and other speech-related tasks. The paper emphasizes that, compared with earlier large-scale systems based on discrete audio tokens and autoregressive language models, NaturalSpeech 2 aims to be more stable, more robust, and more expressive while still scaling to large data.

Problem Setting and Core Idea

The paper frames the main challenge as a trade-off in audio representation. If an audio codec compresses each frame into a single discrete token, token generation becomes easier, but waveform reconstruction can suffer because the compression is aggressive. If the codec uses multiple residual quantizers per frame, reconstruction becomes better, but the flattened token sequence becomes much longer, making autoregressive modeling harder and less robust. NaturalSpeech 2 sidesteps this dilemma by predicting continuous latent vectors rather than a flattened discrete token stream.

The overall pipeline is:

1. Encode speech waveform $x$ into a latent sequence with an audio encoder: $h = f_{\mathrm{enc}}(x)$.

2. Quantize each frame-level latent with a residual vector quantizer, producing a sum of codebook embeddings $z^i = \sum_{j=1}^{R} e_j^i$ and a latent sequence $z$.

3. Train a diffusion model to generate $z$ from text/phoneme input and a prompt-derived condition vector $c$.

4. Decode generated latents back to waveform with the audio decoder.

This setup keeps the generation sequence length at the frame level while preserving more acoustic detail than a highly compressed token stream.

Method Overview

NaturalSpeech 2 consists of two major components: a neural audio codec and a latent diffusion model. The codec turns waveform into frame-wise latent vectors and reconstructs waveform from them. The diffusion model generates those latent vectors in a non-autoregressive way, conditioned on a phoneme encoder, duration predictor, pitch predictor, and speech prompt encoder.

A key theme throughout the method is in-context learning from speech prompts. Rather than conditioning only on text, the system is trained to infer style, timbre, and prosodic information from a short reference speech segment. During training, the model sees randomly cropped prompt segments from target utterances; during inference, it uses a reference utterance from the desired speaker.

Why diffusion instead of autoregression?

The authors argue that autoregressive generation is brittle for long speech sequences because small errors compound over time. This is particularly problematic for TTS because speech has strict monotonic alignment and strong source-target dependency. Diffusion models, by contrast, generate in a denoising process rather than left-to-right token prediction. This makes them a better match for long-form continuous latent generation and removes the classic word skipping and repeating failures associated with autoregressive speech token models.

Neural Audio Codec with Continuous Vectors

The audio codec is trained to compress and reconstruct speech, but the downstream purpose here is synthesis rather than compression. That matters: the paper explicitly notes that it does not need ultra-low bitrate. Instead, it uses the codec to produce a frame-level continuous latent space with enough capacity for high-fidelity reconstruction.

Concretely, the encoder uses convolutional blocks with an overall downsampling rate of $200$ for $16$ kHz audio, so each latent frame corresponds to about $12.5$ ms of audio. A residual vector quantizer with $R$ codebooks maps each frame latent to a sum of residual embeddings. The paper writes this as:

$$h = f_{\mathrm{enc}}(x), \qquad \{e_j^i\}_{j=1}^{R} = f_{\mathrm{rvq}}(h^i), \qquad z^i = \sum_{j=1}^{R} e_j^i, \qquad x = f_{\mathrm{dec}}(z).$$

Although the representation is described as continuous vectors, the implementation still uses a large RVQ structure for regularization and efficiency. The motivation is twofold:

• It avoids storing raw continuous latent tensors during diffusion training; instead, one can store codebook embeddings and token IDs and reconstruct the continuous vectors on demand.
• It enables an auxiliary cross-entropy regularizer over the RVQ codebook assignments, which helps the diffusion model predict more accurate latent vectors.

The appendix specifies the codec configuration as 16 RVQ blocks, codebook size 1024, codebook dimension 256, and a total codec size of about 27M parameters.

Latent Diffusion Model

The latent diffusion model generates the codec latents $z$ from text-conditioned inputs. The authors formulate the diffusion process in stochastic differential equation form. The forward process gradually corrupts the data latent $z_0$ into Gaussian noise using a noise schedule $\beta_t$, and the reverse process reconstructs $z_0$ from noise.

The forward SDE is:

$$\mathrm{d} z_t = -\frac{1}{2} \beta_t z_t\,\mathrm{d}t + \sqrt{\beta_t}\,\mathrm{d}w_t, \qquad t \in [0,1].$$

The reverse SDE is:

$$\mathrm{d} z_t = -\left(\frac{1}{2} z_t + \nabla \log p_t(z_t)\right) \beta_t\,\mathrm{d}t + \sqrt{\beta_t}\,\mathrm{d}\tilde w_t.$$

The paper also gives the corresponding reverse ODE:

$$\mathrm{d} z_t = -\frac{1}{2}\left(z_t + \nabla \log p_t(z_t)\right)\beta_t\,\mathrm{d}t.$$

In practice, the network $s_\theta(z_t,t,c)$ is based on a WaveNet-style denoiser and is trained to predict $\hat z_0$ directly rather than the score, because the authors found this gives better speech quality. Sampling starts from Gaussian noise and numerically solves the reverse process.

The training objective combines three terms:

$$\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{z_0,t}\left[\|\hat z_0 - z_0\|_2^2 + \left\|\Sigma_t^{-1}\big(\rho(\hat z_0,t)-z_t\big)-\nabla \log p_t(z_t)\right\|_2^2 + \lambda_{\mathrm{ce\text{-}rvq}}\mathcal{L}_{\mathrm{ce\text{-}rvq}}\right].$$

The first term is the data reconstruction loss, the second is a score-matching style term, and the third is an RVQ-based cross-entropy regularizer. The paper sets $\lambda_{\mathrm{ce\text{-}rvq}} = 0.1$.

Prior model: phoneme encoder, duration predictor, and pitch predictor

The diffusion model is conditioned on a prior representation $c$ produced from text/phonemes. This prior consists of:

• a phoneme encoder, implemented as a 6-layer Transformer with a convolutional feed-forward network to better capture local dependencies;
• a duration predictor;
• a pitch predictor.

Both predictors share the same general convolutional backbone but use separate parameters. During training, ground-truth duration is used to expand phoneme-level states to frame-level states, and ground-truth pitch is injected into the frame-level hidden sequence. During inference, the predicted duration and pitch are used instead.

The total loss is:

$$\mathcal{L} = \mathcal{L}_{\mathrm{diff}} + \mathcal{L}_{\mathrm{dur}} + \mathcal{L}_{\mathrm{pitch}}.$$

The duration and pitch losses are simple $L_1$ regression losses. This explicit prosody modeling is one of the reasons the system can generalize to singing and better preserve rhythm and intonation in zero-shot settings.

Speech Prompting for In-Context Learning

The paper’s main zero-shot mechanism is speech prompting. The intent is to teach both the diffusion model and the duration/pitch predictors to follow the identity and prosody of a reference speech segment. The prompt is not raw waveform; it is the speech latent sequence produced by the codec encoder.

During training, the authors randomly select a segment $z^{u:v}$ from a target utterance as the prompt $z^p$, and use the rest of the sequence $z^{\setminus u:v}$ as the prediction target. This creates an in-context learning setting where the model must reconstruct the missing content while conditioned on a prompt from the same utterance.

The prompting mechanism differs slightly between components:

• For duration and pitch prediction: the prompt encoder output is injected through Q-K-V attention layers inside the convolutional predictor.
• For diffusion: the paper avoids directly exposing all prompt details to the denoiser. Instead, it uses two attention stages. First, $m$ learnable query tokens attend to the prompt hidden states and compress them into a shorter prompt summary. Second, this summary is attended to by the WaveNet hidden states and then used in a FiLM layer to modulate the denoiser.

This design is meant to preserve the useful prompt information while avoiding over-conditioning that could confuse generation. The default number of query tokens is $m = 32$.

WaveNet denoiser architecture

The denoiser uses a WaveNet-like backbone with 40 layers. Each block contains a dilated convolution, a Q-K-V attention module, and a FiLM layer. The appendix says the WaveNet uses kernel size $3$, dilation $2$, hidden size $512$, and dropout $0.2$. A FiLM layer is inserted every 3 WaveNet layers in the main configuration. The WaveNet has about 183M parameters.

Model Scale and Configuration

The appendix provides the overall parameter breakdown:

• Audio codec: 27M parameters
• Phoneme encoder: 72M parameters
• Duration predictor: 34M parameters
• Pitch predictor: 50M parameters
• Speech prompt encoder: 69M parameters
• Diffusion model: 183M parameters
• Total: 435M parameters

The phoneme encoder uses 8 attention heads, 512 hidden dimensions, convolutional feed-forward size 2048, kernel size 9, and dropout 0.2. The prompt encoder mirrors this configuration. The duration predictor has 30 convolutional layers, kernel size 3, 10 attention layers, 8 heads, hidden size 512, and dropout 0.5. The pitch predictor is similar but uses kernel size 5.

Training and Inference Details

The model is trained on the English subset of Multilingual LibriSpeech (MLS), totaling 44K hours of transcribed speech from LibriVox audiobooks. The training set contains roughly 5,490 speakers overall, split in the paper as 2,742 male and 2,748 female speakers. All audio is at $16$ kHz. Text is converted to phonemes using grapheme-to-phoneme conversion and aligned to speech with an internal alignment tool to obtain phoneme durations. Frame-level pitch is extracted using PyWorld.

Training is staged:

• The audio codec is trained first on 8 NVIDIA Tesla V100 16GB GPUs for 440K steps with batch size 200 audios per GPU and Adam at learning rate $2\times 10^{-4}$.
• The diffusion model is then trained on 16 NVIDIA Tesla V100 32GB GPUs for 300K steps with batch size 6K latent frames per GPU, AdamW at learning rate $5\times 10^{-4}$, and $32$K warmup steps with inverse-square-root scheduling.

At inference time, the authors sample the terminal noise as $z_T \sim \mathcal{N}(0, \tau^{-1} I)$ with $\tau = 1.2^2$. They use the Euler ODE solver and set the diffusion sampling depth to 150 steps for TTS. For singing synthesis, they increase the diffusion steps to 1000 for better quality and lower the learning rate to $5\times 10^{-5}$ when mixing speech and singing data during training.

Evaluation Setup

The evaluation targets zero-shot synthesis on unseen speakers. The main benchmark datasets are:

• LibriSpeech test-clean: 40 speakers, 5.4 hours total, 600 selected utterances for evaluation.
• VCTK: 108 speakers, 540 selected utterances for evaluation.

For each sample, the model conditions on a prompt from a different utterance by the same speaker, cropped to $\sigma = 3$ seconds in the main experiments.

The paper compares against YourTTS as the zero-shot TTS baseline and VALL-E as a strong discrete-token autoregressive baseline. For VALL-E, the paper uses samples from the public demo page. The main objective and subjective metrics are:

• Prosody similarity with prompt: compares pitch and duration statistics between generated speech and prompt speech.
• Prosody similarity with ground truth: uses Pearson correlation and RMSE between generated and ground-truth pitch/duration.
• Word error rate (WER): transcribes synthesized audio with a HuBERT CTC ASR model.
• Intelligibility score: counts word repetitions, word skips, and error sentences on 50 difficult sentences.
• CMOS: comparative mean opinion score for naturalness.
• SMOS: similarity mean opinion score for speaker similarity.

For CMOS, the paper filters multiple diffusion samples with a speech scoring model before human evaluation to improve sample quality.

Main Results: Naturalness, Similarity, and Robustness

Naturalness / CMOS

The CMOS comparison is reported relative to NaturalSpeech 2 as the reference system. The key result is that NaturalSpeech 2 is either on par with or better than the ground-truth recordings, and clearly better than YourTTS.

Interpretation: on LibriSpeech, NaturalSpeech 2 is effectively tied with the ground truth; on VCTK, it is judged more natural than the ground-truth recordings by a noticeable margin in this comparison. In both datasets, it is substantially ahead of YourTTS.

Prosody similarity with the prompt

The paper measures how closely the synthesized speech follows the prompt’s pitch and duration distribution. Lower is better because the metric is the absolute difference in summary statistics of pitch and duration. NaturalSpeech 2 improves over YourTTS on nearly every metric.

The main takeaway is that NaturalSpeech 2 follows prompt prosody more faithfully, especially on pitch statistics. The paper notes that this remains true even on VCTK, where YourTTS has seen 97 of the 108 speakers during its own training but NaturalSpeech 2 still treats all speakers as unseen.

Prosody similarity with ground truth

To measure whether the synthesized prosody matches the reference utterance, the paper also reports Pearson correlation and RMSE against ground truth.

Again, NaturalSpeech 2 improves both pitch and duration tracking, and the gains are larger on pitch correlation and pitch RMSE.

Speaker similarity and WER

The subjective speaker similarity scores show a large advantage over YourTTS, and the ASR-based WER evaluation shows that the generated speech remains intelligible and robust.

NaturalSpeech 2 closes most of the gap to ground truth on speaker similarity, especially on LibriSpeech. The paper reports gains of 1.25 SMOS over YourTTS on LibriSpeech and 0.77 on VCTK.

WER is much lower for NaturalSpeech 2 than for YourTTS. On LibriSpeech it approaches the ground-truth transcription error rate; on VCTK the paper attributes the high WER of ground truth to noisy recordings and the ASR model not being fine-tuned for that dataset.

Robustness on hard sentences

The paper’s robustness stress test uses 50 especially difficult sentences from FastSpeech. The point is to detect classic failure modes such as repeated words, skipped words, or collapsed utterances. NaturalSpeech 2 is fully robust on this benchmark, matching other non-autoregressive systems and outperforming autoregressive ones.

The qualitative conclusion is that diffusion-based non-autoregressive generation avoids the word-level instability that remains common in autoregressive TTS models.

Comparison with VALL-E

VALL-E is used as a strong baseline because it also targets zero-shot synthesis, but it relies on a discrete audio codec and an autoregressive language model. The paper directly compares a small set of demo utterances from VALL-E and reports that NaturalSpeech 2 is better on both similarity and naturalness.

NaturalSpeech 2 therefore improves over VALL-E in both speaker similarity and naturalness, supporting the paper’s thesis that continuous latent diffusion is a stronger design point than long discrete token autoregression for zero-shot speech synthesis.

Ablations

The ablation study isolates three key design choices: speech prompting in diffusion, speech prompting in the duration/pitch predictors, and the RVQ cross-entropy regularizer. The paper also tests a simplified prompt-attention variant that removes the learnable query-token compression stage.

The most important ablation is removing speech prompting from diffusion: the model fails to converge, showing that the prompt mechanism is not a minor refinement but a core enabler of zero-shot synthesis. Removing the duration/pitch prompt causes a large degradation. Removing the RVQ cross-entropy regularizer also hurts, indicating that the discrete codebook supervision stabilizes latent prediction. The simpler prompt attention variant also degrades results, supporting the paper’s claim that compressing the prompt through learned query tokens avoids leaking overly detailed prompt information into the denoiser.

The prompt-length study further shows that longer prompts help, as expected. With LibriSpeech prompts, moving from 3 seconds to 5 or 10 seconds improves the pitch statistics and generally improves duration similarity as well. The paper reports similar trends on VCTK, though the exact metric behavior is not perfectly monotonic across all summary statistics.

Zero-Shot Singing Synthesis

One of the more distinctive results is that NaturalSpeech 2 can synthesize singing voice in a zero-shot setting. The paper builds a singing dataset by crawling singing voices and lyrics from the web, removing accompaniment and backing vocals, and filtering misaligned examples. This yields about 30 hours of singing data, which is mixed with speech data for training.

For singing synthesis, the system can use either a singing prompt or, notably, only a speech prompt. The pitch and duration for the target singing output are taken from another singing voice, while the prompt controls timbre. The paper highlights that this enables novel zero-shot singing synthesis with only a speech prompt, which is a stronger capability than simply voice cloning within the speech domain.

Extensions to Voice Conversion and Speech Enhancement

The authors also describe extensions of the same diffusion framework to voice conversion and speech enhancement.

For voice conversion, the model first performs a source-aware diffusion process that turns the source audio into an informative starting point $z_1$ rather than pure noise. This is intended to preserve some source prosodic information. Then a target-aware denoising process generates the target audio conditioned on the prompt voice and the source-derived phoneme, duration, and pitch information. The result is intended to keep source prosody while changing timbre to match the prompt.

For speech enhancement, the same idea is applied to noisy speech: the source and prompt used in the source-aware step are noisy versions, while the denoising step uses a clean prompt. The paper claims this can remove background noise while preserving prosody and timbre. These extensions are presented as qualitative demonstrations rather than fully benchmarked tasks in the main paper.

Connection to NaturalSpeech

The paper positions NaturalSpeech 2 as an extension of the earlier NaturalSpeech line but with a different focus. NaturalSpeech emphasized very high-quality speech synthesis on single-speaker recording-studio data. NaturalSpeech 2 shifts to diversity, zero-shot generalization, and large-scale multi-speaker/in-the-wild data. Architecturally, it keeps the codec-and-prior structure from NaturalSpeech but replaces the core acoustic generator with diffusion, uses RVQ-based latent regularization, and introduces prompt-based in-context learning.

Limitations, Compute, and Broader Impacts

The paper is candid that the diffusion model is still computationally heavy. In fact, the authors note that the model is underfitting at 300K steps and that longer training would likely improve results. They also explicitly propose future work on faster diffusion alternatives such as consistency models to reduce latency.

Another practical limitation is inference speed: even with the Euler ODE solver and 150 steps for TTS, diffusion is still more expensive than simpler feed-forward or autoregressive decoder-only approaches. Singing synthesis uses 1000 steps, which further underscores the speed-quality trade-off.

On the safety side, the paper’s broader-impact discussion is straightforward: because the system can reproduce speaker identity, it could be misused for spoofing or impersonation. The authors state that experiments assume the user has consent to use the target speaker’s voice, and they recommend consent protocols and synthesized-speech detection if the model is generalized to real-world unseen speakers.

Conclusion

NaturalSpeech 2’s main contribution is the combination of a continuous latent audio codec, a latent diffusion generator, and a speech-prompting mechanism for zero-shot conditioning. The empirical evidence in the paper supports three key claims: it is more robust than autoregressive token-based systems, it better matches prompt and ground-truth prosody, and it achieves much stronger zero-shot speaker similarity and naturalness than the baselines tested. The additional singing, voice conversion, and speech enhancement demonstrations suggest that the architecture is a flexible foundation for a broader family of speech generation tasks.