StyleTTS 2

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

StyleTTS 2 advances text-to-speech synthesis by using text-conditioned style diffusion and adversarial training with large speech language models to generate diverse, natural-sounding speech without relying on reference audio, achieving human-level quality across single- and multispeaker datasets.

• tts
• prosody
• voice-cloning
• llm

Authors: Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, Nima Mesgarani

Categories: eess.AS, cs.AI, cs.CL, cs.LG, cs.SD

Comment: NeurIPS 2023

Published 2023-06-13 · Updated 2023-11-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub464 stars30 watchers70 forksMIT
• Hugging Face0 downloads3 likes

Impact

• Semantic Scholar281 citations25 influential citations

Abstract

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

---

Overview and Core Contributions

StyleTTS 2 is a non-autoregressive text-to-speech system that pushes beyond the original StyleTTS by combining text-conditioned style diffusion, end-to-end waveform synthesis, adversarial training with large speech language models, and a differentiable duration upsampler. The paper’s main claim is that these ingredients together can produce speech judged at or above human recordings on standard single-speaker and multispeaker benchmarks, while also improving zero-shot speaker adaptation.

The paper positions StyleTTS 2 as addressing four persistent TTS pain points:

• Reference dependence: prior StyleTTS needed a reference speech sample to drive style; StyleTTS 2 instead samples style from text.
• Lack of diversity: deterministic or weakly stochastic systems often collapse toward average prosody; the diffusion-based latent style improves variation.
• Training fragmentation: the model removes the need for a separate vocoder stage by synthesizing waveforms directly.
• Limited naturalness and OOD robustness: adversarial supervision from a pre-trained speech language model helps speech sound more human and generalize better to out-of-distribution text.

At a high level, StyleTTS 2 models speech as

$$p(\mathbf{x} \mid \mathbf{t}) = \int p(\mathbf{x} \mid \mathbf{t}, \mathbf{s}) \, p(\mathbf{s} \mid \mathbf{t}) \, d\mathbf{s},$$

where the latent style variable $\mathbf{s}$ captures speech characteristics beyond phonetic content, including prosody, speaking rate, lexical stress, and other paralinguistic factors. The central design goal is to make $\mathbf{s}$ both text-dependent and cheap to sample, so the model can stay fast enough for practical use while still producing varied speech.

The paper’s stated contributions are:

• Modeling speech style as a latent random variable sampled by a diffusion model conditioned on text, eliminating the need for reference audio at inference.
• Using a large pre-trained speech language model (WavLM) as a discriminator, coupled with a new differentiable duration model, to enable end-to-end adversarial training.
• Introducing a prosodic style encoder and a prosodic text encoder to separate style/prosody modeling from acoustic reconstruction.
• Achieving strong subjective results on LJSpeech, VCTK, and LibriTTS, including the paper’s reported first human-level TTS results on both single-speaker and multispeaker public datasets.

Architecture: From StyleTTS to StyleTTS 2

The paper begins from the StyleTTS baseline, which is organized into three groups of modules:

• Acoustic modules: a text encoder, style encoder, and decoder.
• Prediction modules: duration and prosody predictors.
• Utility/training modules: a text aligner, pitch extractor, and discriminator.

In StyleTTS, the training flow is two-stage. First, the model reconstructs mel-spectrograms from aligned phoneme features, style embeddings from reference speech, pitch, and energy. Second, the duration and prosody predictors are trained against targets extracted from the first stage. StyleTTS 2 preserves this backbone but changes the training objective and generation pathway substantially.

The key architectural changes are:

1. Direct waveform synthesis: the decoder is modified to generate waveforms rather than only mel-spectrograms.
2. Style diffusion: the style vector is sampled from a diffusion model conditioned on text, rather than encoded from reference audio alone.
3. Prosodic separation: the model adds a prosodic style encoder $E_p$ alongside the acoustic style encoder $E_a$, and a prosodic text encoder $B$ based on phoneme-level BERT.
4. SLM adversarial training: a frozen WavLM encoder plus a small convolutional head acts as a discriminator on real and generated waveforms.
5. Differentiable duration modeling: the duration predictor is redesigned so gradients can flow through alignment construction during end-to-end training.

For multispeaker training, the style diffusion process is additionally conditioned on a speaker embedding $\mathbf{c} = E(\mathbf{x}_{\text{ref}})$ from a reference clip. The diffusion model then samples a speaker-consistent style vector for the target synthesis.

In the decoder, the paper uses either a HifiGAN-based waveform decoder or an iSTFTNet-based decoder. The decoder uses Snake activations and inserts AdaIN after each activation to preserve style conditioning. For acoustic pretraining, the paper uses the standard GAN discriminators from HifiGAN: a multi-period discriminator and a multi-resolution discriminator.

Style Diffusion: Text-Conditioned Latent Style Sampling

The most important conceptual change in StyleTTS 2 is that style is no longer a deterministic encoding of a reference utterance. Instead, style is treated as a stochastic latent variable sampled from a text-conditioned diffusion model. The paper frames this with a denoising diffusion formulation, using an EDM-style preconditioned denoiser:

$$K(\mathbf{s}; \mathbf{t}, \sigma) = \left(\frac{\sigma_{\text{data}}}{\sigma^*}\right)^2 \mathbf{s} + \frac{\sigma \sigma_{\text{data}}}{\sigma^*} \, V\!\left(\frac{\mathbf{s}}{\sigma^*}; \mathbf{t}, \frac{1}{4}\ln \sigma\right), \quad \sigma^* = \sqrt{\sigma^2 + \sigma_{\text{data}}^2}.$$

The corresponding ODE used for sampling is

$$\frac{d\mathbf{s}}{d\sigma} = \frac{\mathbf{s} - K(\mathbf{s}; \mathbf{t}, \sigma)}{\sigma}.$$

Here, $V$ is a three-layer transformer conditioned on text and noise level. The training objective is a denoising score matching loss on noisy style codes:

$$\mathcal{L}_{\text{edm}} = \mathbb{E}\left[\lambda(\sigma) \left\|K(E(\mathbf{x}) + \sigma \boldsymbol{\xi}; \mathbf{t}, \sigma) - E(\mathbf{x})\right\|_2^2\right],$$

where $E(\mathbf{x}) = [E_a(\mathbf{x}), E_p(\mathbf{x})]$ concatenates acoustic and prosodic style vectors, and $\lambda(\sigma)$ is the preconditioning weight.

The paper uses the EDM noise schedule with $\sigma_{\min} = 0.0001$, $\sigma_{\max} = 3$, and $\rho = 9$. Sampling is performed with the ancestral DPM-2 solver rather than Heun’s method, because the authors prioritize speed and diversity over exact numerical fidelity. In the main experiments, style diffusion steps are randomly sampled from 3 to 5 during training and set to 5 at inference. The appendix also studies the step-quality tradeoff more extensively.

Figure-wise, the paper shows that style vectors sampled from text form meaningful clusters. On LJSpeech, texts generated across five emotions form distinct emotion clusters; on LibriTTS, styles sampled for five unseen speakers form speaker-specific clusters, with some emotion structure still visible within a single speaker.

The authors also interpret the diffusion latent as a practical style-transfer interface: one can extract a style from a text expressing a desired emotion or speaking manner, then synthesize different content using that style. They explicitly emphasize that this relationship is learned self-supervised, without manual emotion labels.

Differentiable Duration Modeling

A major technical bottleneck in end-to-end TTS training is duration prediction. In the original StyleTTS, the duration predictor outputs durations, but converting these durations into a frame-level alignment is not differentiable. That blocks gradients from the waveform-level adversarial objective from reaching the duration predictor.

StyleTTS 2 replaces this with a new non-parametric differentiable upsampler. The duration predictor outputs $q[k,i]$, interpreted as the probability that phoneme $i$ has duration at least $k$, for $k = 1, \dots, L$. The predicted duration is then

$$\hat d_i = \sum_{k=1}^{L} q[k,i], \quad L = 50,$$

which corresponds to a maximum phoneme duration of 50 frames, about 1.25 seconds in the authors’ setup.

The paper models alignment positions recursively via cumulative end positions $\ell_i = \sum_{k=1}^{i} \hat d_k$. To approximate the non-differentiable shift operation, the discrete delta at $\ell_{i-1}$ is replaced by a Gaussian kernel $\mathcal{N}_{\ell_{i-1}}(n; \sigma)$. The resulting alignment probability is then normalized across phonemes with a softmax.

The authors choose $\sigma = 1.5$ after analyzing both sample quality and gradient stability. The appendix argues that smaller $\sigma$ makes the Gaussian closer to a delta function but causes gradients to blow up, while larger $\sigma$ weakens alignment fidelity. Their chosen value balances the two and roughly spans about three speech frames.

The appendix further explains why Gaussian upsampling and attention-based upsamplers are insufficient here: the paper argues that attention-based methods are unstable under adversarial training, and trainable upsampling variants add additional instability. The proposed upsampler is intentionally non-parametric to keep training stable while still allowing gradient flow from the SLM discriminator through alignment construction.

SLM Discriminators and End-to-End Adversarial Training

The second major novelty is the use of a pre-trained speech language model as a discriminator. The paper uses a frozen 12-layer WavLM model, pre-trained on 94k hours of speech, and appends a lightweight convolutional discriminative head to it. The resulting discriminator is trained to distinguish human speech from generated speech in the WavLM representation space.

Conceptually, the authors argue that the discriminator’s features capture both acoustic and semantic information, so forcing the generator to match human speech in this latent space encourages prosody, pauses, intonation, and expressive naturalness rather than just waveform fidelity. The adversarial objective is formulated with a standard least-squares GAN loss during decoder training and an adversarial objective over the full generator during joint training.

A critical detail is that the WavLM encoder itself is kept fixed; only the small convolutional head is trained. Audio is downsampled to 16 kHz before the discriminator. In the joint training stage, the SLM discriminator supervises generated waveforms that come from the full differentiable pipeline, including the duration predictor, so the loss can update the entire model end to end.

The paper emphasizes that this adversarial loss is independent of the ground-truth waveform at the generator side and depends only on text input. This allows the method to use out-of-distribution texts during adversarial training. In practice, the paper samples in-distribution and OOD texts with equal probability to prevent overfitting to content.

The appendix contains a layer-wise analysis of WavLM feature importance. For LJSpeech and LibriTTS, early acoustic layers and mid-level semantic layers are most important, while the final layer contributes least. On VCTK, no strong layer preference emerges, which the authors interpret as a consequence of the dataset’s limited contextual and emotional richness.

Training Objectives and Optimization

The paper uses a two-phase optimization schedule, similar in spirit to the original StyleTTS but extended to the new waveform, diffusion, and SLM components.

Acoustic module pretraining

In pretraining, the model reconstructs mel-spectrograms with an $L_1$ loss and trains the text aligner with transferable monotonic aligner losses. It also applies HifiGAN-style adversarial losses: least-squares GAN loss and feature matching loss, plus a truncated pointwise relativistic loss. The intent is to get the acoustic side and the aligner/pitch extractor in a good regime before adding the harder end-to-end objectives.

The paper writes the pretraining objective as a weighted sum of mel reconstruction, sequence-to-sequence ASR loss, monotonic alignment loss, adversarial loss, relativistic loss, and feature matching. The reported weights are $\lambda_{\text{s2s}} = 0.2$ and $\lambda_{\text{mono}} = 5$.

Joint training

After pretraining, joint training optimizes the waveform decoder, the acoustic and prosodic style encoders, the prosodic text encoder, the duration predictor, the prosody predictor, and the diffusion module. The losses include:

• $\mathcal{L}_{\text{mel}}$ for mel reconstruction from the predicted pitch and energy.
• $\mathcal{L}_{\text{ce}}$ and $\mathcal{L}_{\text{dur}}$ for duration prediction.
• $\mathcal{L}_{f_0}$ and $\mathcal{L}_n$ for pitch and energy regression.
• $\mathcal{L}_{\text{adv}}$, $\mathcal{L}_{\text{fm}}$, and $\mathcal{L}_{\text{rel}}$ for waveform realism.
• $\mathcal{L}_{\text{slm}}$ for SLM adversarial supervision.
• $\mathcal{L}_{\text{edm}}$ for style diffusion.

The paper’s reported joint-training weights are $\lambda_{\text{dur}} = 1$, $\lambda_{\text{ce}} = 1$, $\lambda_{f_0} = 0.1$, $\lambda_n = 1$, $\lambda_{\text{s2s}} = 0.2$, and $\lambda_{\text{mono}} = 5$.

Because adversarial gradients from WavLM can be very large, the authors add gradient scaling safeguards. They report that the maximum gradient norm from the SLM discriminator to the duration predictor can reach around 200 even with their chosen $\sigma$, so they scale the gradient norm by 0.2 when the predictor gradient exceeds 20 and scale the last projection and LSTM layers in the duration predictor by 0.01. This stabilization detail is important: the paper explicitly notes that end-to-end training can otherwise become unstable, especially because the prosodic text encoder is based on BERT and is sensitive to gradient explosion.

The architecture appendix also states that the denoiser uses three transformer blocks with 8-head self-attention, and the SLM discriminative head is a compact CNN that maps pooled WavLM features to a scalar real/fake score.

Datasets, Preprocessing, and Evaluation Protocol

The paper evaluates on three datasets:

• LJSpeech: a single-speaker dataset with 13,100 short clips totaling about 24 hours. The split is 12,500 train, 100 validation, and 500 test.
• VCTK: a multispeaker dataset with nearly 44,000 clips from 109 native speakers. The split is 43,470 train, 100 validation, and 500 test.
• LibriTTS train-clean-460: about 245 hours from 1,151 speakers, split 98% train, 1% validation, 1% test, with test-clean used for zero-shot adaptation evaluation.

All datasets are resampled to 24 kHz for training, while the SLM discriminator receives 16 kHz audio. Text is converted to phonemes using phonemizer. The model uses the LibriTTS training split as OOD text for SLM adversarial training.

The paper’s evaluation protocol is unusually detailed, and the authors explicitly argue that subjective TTS evaluation is highly sensitive to survey design. Their setup includes:

• Native English-speaking raters from the U.S.
• HIT approval rate greater than 95%, more than 50 approved HITs, and location restricted to the United States.
• IP-based verification to reduce proxy/VPN noise.
• Attention checks that filter out inattentive raters.
• Explicit definitions of naturalness and similarity in the survey instructions.
• A MUSHRA-like layout for MOS, where multiple samples are shown together rather than one at a time.
• CMOS tests for finer-grained statistical comparison.

For MOS, each test used 80 random text samples from the test set. In LJSpeech, the authors also added 40 out-of-distribution Librivox utterances read by the same narrator to compare in-distribution and OOD behavior. Each synthesized speech set was evaluated by 5 to 10 raters on a 1 to 5 scale with 0.5 increments. For CMOS, raters judged pairs on a -6 to 6 scale in integer steps. The paper reports excluding six raters via attention checks in total.

Training details are:

• Optimizer: AdamW with $\beta_1 = 0$, $\beta_2 = 0.99$, weight decay $10^{-4}$, learning rate $10^{-4}$, batch size 16.
• Pretraining epochs: 100 on LJSpeech, 50 on VCTK, 30 on LibriTTS.
• Joint training epochs: 60 on LJSpeech, 40 on VCTK, 25 on LibriTTS.
• Waveform segments: random chunks up to 3 seconds.
• SLM training segment length: 3 to 6 seconds.
• Diffusion steps: 3 to 5 during training; 5 during inference.
• Hardware: four NVIDIA A40 GPUs.

Main Results

The main subjective claim is that StyleTTS 2 reaches or exceeds human-level quality on LJSpeech and matches human recordings on VCTK, while also improving zero-shot speaker adaptation on LibriTTS. The authors stress that the results depend on the paper’s careful survey design, especially MUSHRA-style comparison and explicit definitions of naturalness.

Comparative mean opinion scores

Interpretation-wise, the LJSpeech result is the boldest claim: raters preferred StyleTTS 2 over ground truth by a small but statistically significant margin. The paper attributes this partly to LJSpeech’s fragmented audiobook segments: isolated clips can sound less contextually natural than the model-generated speech, especially when the human reference lacks surrounding discourse continuity. On VCTK, which has less narrative context, StyleTTS 2 is statistically indistinguishable from ground truth in naturalness and slightly better in similarity, though the similarity margin is only marginally significant. On zero-shot LibriTTS, StyleTTS 2 surpasses Vall-E in naturalness while losing on similarity, despite using roughly 245 hours of data versus Vall-E’s much larger scale.

LJSpeech MOS and OOD robustness

StyleTTS 2 is the only model in this comparison that does not degrade on OOD text; in fact, its OOD MOS is slightly higher than its in-distribution MOS. The authors take this as evidence that text-conditioned style diffusion and SLM adversarial training improve generalization beyond the training text distribution.

Zero-shot speaker adaptation on LibriTTS

The authors frame this as a favorable efficiency tradeoff: StyleTTS 2 is competitive with very large-scale systems while requiring far less data, though they acknowledge that speaker similarity in zero-shot adaptation still has room for improvement.

Speech diversity and speed

The diversity metrics are an important part of the paper’s argument. StyleTTS 2 has the highest duration and pitch variation among the compared systems, showing that the stochastic style latent indeed expands the expressive support of the model rather than collapsing to a narrow prosodic mode. At the same time, it remains fast: the reported RTF is 0.0185, and the paper emphasizes that the system is faster than several diffusion-based baselines.

The paper uses a 500-sample emotion stress test, with GPT-4 generating text in five emotions, to show that StyleTTS 2 tracks emotional variation in mean $F_0$ and energy more strongly than VITS or JETS. The qualitative interpretation is that probabilistic style sampling produces more expressive speech than deterministic decoding.

Effect of diffusion steps

The appendix concludes that sample quality changes little over a broad range of steps, but diversity increases up to around 16 steps and then largely saturates. The authors therefore regard roughly 16 steps as the best quality/diversity/speed compromise, while still using fewer steps in the main training pipeline for efficiency.

Ablations and Diagnostic Analyses

The ablation study makes the roles of the core components very clear. The paper removes style diffusion, the SLM adversarial objective, the prosodic style encoder, the differentiable upsampler, and OOD text training one at a time. Every removal hurts, and the largest degradation comes from removing style diffusion.

The paper’s interpretation is that style diffusion is the single most important ingredient, because replacing it with random style vectors sharply worsens all metrics. The SLM discriminator helps naturalness, especially for OOD text. The prosodic style encoder improves all measured quantities by separating prosodic and acoustic responsibilities. The differentiable duration upsampler strongly affects pitch-related metrics and is essential for end-to-end gradient flow. OOD text training has a smaller but still visible effect, especially on subjective quality for OOD prompts.

The appendix also reports an objective comparison across diffusion-step counts. That study shows that 4 to 32 steps yield very similar fidelity metrics, while diversity and runtime trade off smoothly. The paper’s practical recommendation is therefore not to chase very large step counts, but to choose a moderate number of steps that preserves speed.

Another diagnostic study looks at participant feedback. Raters often said differences between systems were subtle, and the authors use this to support their claim that StyleTTS 2 is approaching human-like quality. They also caution that the common TTS evaluation setup may be unfair to ground truth audio from long-form contexts, since isolated audiobook snippets can sound less natural when stripped of surrounding narrative continuity.

Additional Capabilities: Long-Form Generation and Style Transfer

The appendix goes beyond benchmarking and discusses two practical applications.

Long-form generation

Because the style latent is sampled independently for each sentence, naïvely generating a paragraph sentence by sentence can create abrupt style shifts. The paper proposes a simple interpolation strategy: split a paragraph into sentences, sample a style for the current sentence, and blend it with the previous sentence’s style using a convex combination with weight $\alpha \in [0,1]$. This keeps long-form narration more coherent without forcing every sentence to use the exact same style.

The authors describe this as evidence that the learned style space is approximately convex, so interpolating between styles can produce smooth transitions rather than discontinuities.

Style transfer

Because style and content are decoupled, the model can transfer style from one piece of text to another. In practice, a user can sample a style vector from a text that expresses a desired emotion, speaking rate, or recording environment, then synthesize a different input text in that style. This is presented as a natural extension of the model’s latent structure rather than a separate module.

Limitations, Safety, and Interpretation Cautions

The paper is unusually explicit about limitations and safety concerns. It notes that while StyleTTS 2 is strong on LJSpeech and VCTK, there is still room for improvement on larger, more heterogeneous datasets such as LibriTTS, where thousands of speakers and many acoustic conditions make the style space harder to disentangle. Speaker similarity in zero-shot adaptation also remains a weakness relative to the model’s naturalness gains.

The authors also caution that the headline claim of exceeding ground truth on LJSpeech may be partly an artifact of the evaluation protocol. Since LJSpeech clips are isolated audiobook segments, ground-truth audio can sound less natural when removed from context, while synthesized sentences can appear more fluent in isolation. They therefore call for context-aware long-form evaluation in future work.

On the safety side, the paper highlights the risk of misuse in zero-shot speaker adaptation, including impersonation, fraud, harassment, and misinformation. The authors state that public code and models should be governed by a code of conduct requiring disclosure of synthesized speech and consent for voice adaptation. They also note the need for future research on speaker-fraud and impersonation detection.

Overall, the paper’s practical message is that StyleTTS 2 is not merely a better TTS model in the narrow benchmark sense; it is a demonstration that text-conditioned style sampling, large self-supervised speech representations, and fully differentiable alignment can be combined into a fast, expressive, and high-quality synthesis system that is much closer to human-level naturalness than earlier end-to-end approaches.