Mega-TTS

Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias

Mega-TTS is a zero-shot text-to-speech system that models speech by decomposing it into content, timbre, prosody, and phase. This intrinsic bias improves naturalness, speaker similarity, and robustness for unseen and cross-lingual speakers, outperforming methods using latent codec tokens.

• tts
• voice-cloning
• prosody
• speech-to-speech

Authors: Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, Zejun Ma, Zhou Zhao

Categories: eess.AS, cs.AI, cs.SD

Published 2023-06-06 · Updated 2023-06-06

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar101 citations11 influential citations

Abstract

Scaling text-to-speech to a large and wild dataset has been proven to be highly effective in achieving timbre and speech style generalization, particularly in zero-shot TTS. However, previous works usually encode speech into latent using audio codec and use autoregressive language models or diffusion models to generate it, which ignores the intrinsic nature of speech and may lead to inferior or uncontrollable results. We argue that speech can be decomposed into several attributes (e.g., content, timbre, prosody, and phase) and each of them should be modeled using a module with appropriate inductive biases. From this perspective, we carefully design a novel and large zero-shot TTS system called Mega-TTS, which is trained with large-scale wild data and models different attributes in different ways: 1) Instead of using latent encoded by audio codec as the intermediate feature, we still choose spectrogram as it separates the phase and other attributes very well. Phase can be appropriately constructed by the GAN-based vocoder and does not need to be modeled by the language model. 2) We model the timbre using global vectors since timbre is a global attribute that changes slowly over time. 3) We further use a VQGAN-based acoustic model to generate the spectrogram and a latent code language model to fit the distribution of prosody, since prosody changes quickly over time in a sentence, and language models can capture both local and long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K hours of speech and evaluate its performance on unseen speakers. Experimental results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior naturalness, robustness, and speaker similarity due to the proper inductive bias of each module. Audio samples are available at https://mega-tts.github.io/demo-page.

---

1. Problem Setting and Core Idea

Mega-TTS targets zero-shot text-to-speech at scale, with the explicit goal of improving naturalness, speaker similarity, and robustness on unseen speakers and cross-domain prompts. The paper starts from a critique of recent large-scale TTS systems that treat speech primarily as a stream of codec tokens and then model those tokens with autoregressive language models or diffusion models. The authors argue that this design overlooks the intrinsic structure of speech and therefore wastes capacity on attributes that do not need to be modeled in the same way.

Their central claim is that speech can be decomposed into distinct components with different temporal behaviors and inductive biases: content is monotonic with text, timbre is global and stable, prosody changes rapidly and benefits from sequence modeling, and phase is highly dynamic but not semantically important in monaural synthesis. Mega-TTS is designed around that decomposition rather than around a single latent space for all speech attributes. This is what the paper means by intrinsic inductive bias.

2. Overall Architecture

Mega-TTS consists of two major parts: a VQGAN-based TTS model that disentangles and reconstructs mel-spectrograms, and a prosody large language model (P-LLM) that predicts discrete prosody codes. The synthesis pipeline keeps the mel-spectrogram as the intermediate representation instead of converting speech to neural codec latents. The authors argue that this choice helps isolate phase from other factors, because the phase can be reconstructed by a GAN vocoder rather than explicitly modeled by the language model.

The architecture has three encoders and one decoder in the first stage:

• Content encoder: maps phoneme sequences to content representations and supports monotonic alignment through a duration predictor and length regulator.
• Timbre encoder: maps a reference utterance to a global speaker vector via temporal averaging.
• Prosody encoder: maps low-frequency mel-spectrogram frames to phoneme-level discrete prosody codes using a vector-quantization bottleneck.
• Mel decoder: generates mel-spectrograms from content, timbre, and prosody, trained with a GAN objective to improve perceptual quality.

3. Speech Disentanglement and VQGAN-Based TTS

The first-stage model is intended to disentangle speech into content, prosody, and timbre representations. The training signal is a reconstruction objective plus a carefully designed bottleneck. The bottleneck is important: it pressures the prosody pathway to exclude content and global timbre, leaving mostly prosodic information.

The prosody encoder uses two convolution stacks, a phoneme-level pooling layer, and a VQ bottleneck. The input is restricted to the first 20 mel bins of each frame, which the authors say preserves most prosodic information while reducing speaker/content leakage. The encoder outputs phoneme-level codes $\mathbf{u} = \{u_1, u_2, \dots, u_T\}$ and hidden states $H_{\text{prosody}}$.

The content encoder is a Transformer-based module operating on phoneme embeddings. To address the one-to-many nature of duration prediction, the paper feeds prosody information into the duration predictor. This is a notable design choice: instead of relying only on text for alignment, Mega-TTS explicitly conditions duration estimation on the extracted prosody.

The timbre encoder is a convolutional network followed by temporal average pooling, producing a single global vector that captures speaker identity. This matches the paper’s assumption that timbre varies slowly and is better modeled as a sequence-invariant or slowly varying factor.

The mel decoder is GAN-based, and the discriminator uses multiple window sizes. The paper emphasizes that phase does not need to be predicted by the language model; instead, a vocoder-like decoder can reconstruct a plausible phase from the mel representation.

The first-stage loss is written as:

$$ \mathcal{L}_{\mathrm{VQ}} = \|y_t - \hat{y}_t\|^2 + \|\operatorname{sg}[E(y_t)] - z_{\mathbf{q}}\|_2^2 + \|\operatorname{sg}[z_{\mathbf{q}}] - E(y_t)\|_2^2, $$

$$ \mathcal{L} = \mathbb{E}\left[\mathcal{L}_{\mathrm{VQ}} + \mathcal{L}_{\mathrm{Adv}}\right]. $$

Here $y_t$ is the target speech, $\hat{y}_t$ is the reconstructed speech, $E(\cdot)$ is the encoder, $z_{\mathbf{q}}$ is the quantized latent, and $\mathcal{L}_{\mathrm{Adv}}$ is an LSGAN-style adversarial loss. The loss combines reconstruction fidelity, VQ commitment/codebook terms, and adversarial realism.

4. P-LLM and Prosody-Oriented Decoding

The second stage is a decoder-only transformer that predicts discrete prosody codes. The paper’s main idea is that prosody is the speech component most suitable for language modeling: it has local and long-range dependencies, it changes quickly in time, and it is not perfectly determined by text. In contrast, content is better handled by monotonic alignment machinery, and timbre is better handled as a global vector.

During training, the P-LLM is teacher-forced and optimized with cross-entropy over the prosody code sequence. The autoregressive factorization is:

$$ p\left(\tilde{\mathbf{u}} \mid \mathbf{u}, H_{\text{content}}, \tilde{H}_{\text{timbre}}, \tilde{H}_{\text{content}}; \theta\right) = \prod_{t=0}^{T} p\left(\tilde{u}_t \mid \tilde{u}_{

The conditioning inputs combine prompt prosody codes, prompt and target content representations, and the prompt timbre vector. The paper calls the resulting inference pipeline prosody-oriented speech decoding: the target speech is generated by first predicting target prosody codes and then decoding them with the target text content and prompt speaker identity.

4.1 Zero-Shot TTS and Cross-Lingual TTS Inference

For zero-shot TTS, the P-LLM uses prompt prosody plus the speaker and content conditioning signals to generate the target prosody codes. The final mel decoder then combines target content, prompt timbre, and predicted prosody to synthesize speech. The paper uses top-$k$ random sampling at inference to increase diversity.

Cross-lingual TTS uses the same mechanism, except the prompt may come from a different language. The paper uses this to demonstrate that the architecture is not tied to English-only prompts or targets.

4.2 Speech Editing Inference

Mega-TTS also supports speech editing by treating the masked region as a prosody completion problem. Instead of blending waveform or mel segments directly, the model works in the discrete prosody space. The authors generate $N$ candidate left-to-right paths from the left boundary of the edit region, then rescore them by also considering the right boundary. The candidate with the highest combined likelihood is selected.

The paper presents this as a way to obtain smooth transitions at both boundaries of the edited region. Compared with an L2-based fusion point, the discrete-prosody approach is intended to better reflect perceptual continuity.

The speech-editing likelihood objective is described as maximizing candidate paths that are consistent with both the left and right context. In the paper’s notation, the right-side ground-truth prosody codes are used to score candidate paths after the left-side proposal stage.

5. Model Size, Training Setup, and Data

The paper scales Mega-TTS to a multi-domain, multi-lingual training set with 20K hours of speech: GigaSpeech and WenetSpeech. The training data are processed with automatic speaker diarization to recover speaker identities and with an external forced-alignment tool to obtain phoneme-level alignments. The authors note that some clips are discarded when diarization confidence is too low or when multiple speakers overlap.

Training is performed on 8 NVIDIA A100 GPUs with a batch size of 30 sentences per GPU. The optimizer is Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, and $\epsilon = 10^{-9}$, following the learning-rate schedule of the Transformer paper. The first-stage VQGAN-based TTS model is trained for 320k steps, and the P-LLM is trained for 100k steps. Inference uses a pre-trained HiFi-GAN V1 vocoder to convert mel-spectrograms into waveform audio.

The model has 222.5M parameters. The appendix provides the following key configuration details:

6. Experimental Protocol

The main evaluation datasets are VCTK and LibriSpeech test-clean. The paper samples 10 utterances for each of 40 speakers, giving 400 evaluation utterances per dataset. For each sample, a different utterance of the same speaker is used as the prompt. The paper emphasizes that all evaluation speakers are unseen during training.

Objective evaluation uses three metrics:

• Pitch distance: average DTW distance between pitch contours of synthesized and reference speech.
• Speaker similarity: cosine similarity between WavLM speaker embeddings of synthesized and reference speech.
• WER for cross-lingual TTS: computed using a fine-tuned HuBERT-Large ASR model.

Subjective evaluation uses MTurk with at least 20 listeners per audio sample and 50 sampled utterances per dataset. The paper reports MOS-Q for audio quality, MOS-P for prosody, MOS-S for speaker similarity, and CMOS-Q / CMOS-P for pairwise comparisons. Listeners are instructed to focus on the target aspect and ignore the others, which is important because the authors are trying to separate quality, prosody, and identity judgments.

7. Main Results

7.1 Zero-Shot TTS

On zero-shot TTS, Mega-TTS outperforms YourTTS on both VCTK and LibriSpeech test-clean in terms of naturalness, prosody, speaker similarity, and pitch distance. The gains are especially clear in speaker similarity and pitch preservation, which are the metrics most directly connected to the paper’s decomposition of speech factors.

The paper also compares Mega-TTS against VALL-E using pairwise subjective metrics. Mega-TTS is reported with CMOS-Q $= 0.00$, CMOS-P $= 0.00$, and MOS-S $= 4.11 \pm 0.21$, while VALL-E has CMOS-Q $= -0.23$, CMOS-P $= -0.27$, and MOS-S $= 4.06 \pm 0.22$. The authors interpret this as showing that introducing the speech-specific decomposition improves perceptual quality and controllability relative to codec-token generation.

7.2 Speech Editing

On speech editing, Mega-TTS improves all three MOS dimensions over EditSpeech and A$^3$T on VCTK. This task is a good stress test for the discrete prosody representation because the model must preserve the unedited context while smoothly synthesizing the missing segment.

The paper attributes these gains to the new discrete-prosody editing strategy. Rather than fusing left and right mel predictions by an L2 criterion, Mega-TTS explicitly reasons over candidate prosody sequences and scores them by boundary consistency.

7.3 Cross-Lingual TTS

Cross-lingual TTS is evaluated against YourTTS and VALL-E X. The subjective comparison uses six speech pairs from LibriSpeech, EMIME, and AISHELL-3, while the objective evaluation uses the full LibriSpeech test-clean set as the target text and an AISHELL-3 prompt for each sentence.

The reported WER reduction from 7.59% to 3.04% is a substantial improvement in intelligibility, and the speaker similarity score also improves. This is consistent with the paper’s claim that modeling prosody separately from content and timbre helps preserve both identity and linguistic correctness across languages.

7.4 Robustness on Hard Sentences

The paper explicitly checks robustness using 50 particularly hard sentences, following the FastSpeech benchmark style. The metric counts repeated words, skipped words, and sentences with errors. Mega-TTS has zero repeats, zero skips, and zero error sentences on this test, matching the robustness of FastSpeech and outperforming Tacotron and VALL-E on this criterion.

8. Ablations and Representation Analysis

The appendix contains several analyses that directly support the paper’s main design choices. These are useful because they connect the theory of inductive bias to concrete behavior under controlled perturbations.

8.1 Information Bottleneck Hyperparameters

The authors vary the vector-quantization bottleneck size and measure disentanglement quality by reconstructing a mel-spectrogram after shuffling timbre embeddings. They report pitch distance and speaker similarity, and find that a channel size of 256 with embedding size 2048 gives the best combination of low pitch error and high speaker similarity.

8.2 Dataset Size Ablation

The paper varies the amount of training data and measures zero-shot performance on LibriSpeech test-clean using pitch distance, speaker similarity, and average absolute duration error. Performance improves with more data, which supports the scaling argument made throughout the paper.

8.3 P-LLM Capacity Ablation

Increasing the hidden size of the P-LLM improves both pitch and speaker similarity. The strongest setting in the appendix is the 512-dimensional hidden state, which performs markedly better than 128 or 256 dimensions.

8.4 Representation Visualizations

The appendix uses T-SNE visualizations to inspect the learned representations on unseen speakers from VCTK. The timbre embeddings cluster by speaker identity, while the prosody embeddings overlap more across speakers. This is direct evidence that the encoders are disentangling speaker identity from prosodic variation, which is exactly the behavior the later P-LLM relies on.

8.5 Diversity Across Random Seeds

The appendix shows multiple mel-spectrograms generated from different random seeds. The visualizations suggest that the model can produce diverse prosody patterns and frequency details rather than collapsing to a single stereotyped output. This is consistent with the use of sampling in the P-LLM stage.

9. Limitations and Broader Impact

The paper is unusually explicit about its limitations. First, although the model is trained on 20K hours of multi-domain data, it still cannot cover every voice; in particular, speakers with extremely heavy accents are not imitated well. The authors propose scaling to 200K hours in future work. Second, reconstruction quality degrades in the presence of background music or very loud reverberation, which suggests the GAN-based reconstruction pipeline is still sensitive to acoustic noise.

The broader-impact discussion also acknowledges misuse risks such as voice spoofing and deepfake generation. The authors suggest detector development and license restrictions as possible mitigations.

10. Technical Takeaways

• Mega-TTS’s key design move is to not model all speech attributes with the same generative mechanism.
• The mel-spectrogram intermediate representation is chosen to isolate phase handling from the learned sequence model.
• Prosody is the only component treated as a language-modeling problem; content and timbre are handled by separate encoders.
• The discrete prosody bottleneck and candidate rescoring strategy are central to speech editing and robust inference.
• Scaling to 20K hours of multi-domain speech materially improves zero-shot behavior, but data coverage remains a limitation.
• Across zero-shot TTS, speech editing, and cross-lingual TTS, the model consistently reports stronger naturalness and speaker similarity than the cited baselines.