MaskGCT

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

A non-autoregressive TTS system using masked generative transformers to predict semantic and acoustic tokens in parallel, without requiring explicit text-speech alignment or duration labels. It achieves state-of-the-art zero-shot speech synthesis by applying the mask-and-predict paradigm across both generation stages.

• tts
• voice-cloning
• one-shot

Demos

MaskGCT demonstrates zero-shot text-to-speech synthesis using a two-stage masked generative approach that eliminates the need for explicit alignment or phone-level duration prediction. Evaluate these demos for naturalness, speaker similarity to the prompt, and the system's ability to handle cross-lingual synthesis and preserve prosody/emotion—key advantages over autoregressive and conventional non-autoregressive baselines claimed in the paper.

Authors: Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, Zhizheng Wu

Categories: cs.SD, cs.AI, cs.LG, eess.AS

Published 2024-09-01 · Updated 2024-10-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub9,913 stars89 watchers819 forksMIT
• Hugging Face330 downloads306 likes

Impact

• Semantic Scholar238 citations25 influential citations

Abstract

The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

---

Introduction and Motivation

Large-scale zero-shot text-to-speech (TTS) systems have achieved remarkable progress by scaling both data and model size. These systems generally fall into two camps: autoregressive (AR) models and non-autoregressive (NAR) models. AR models (e.g., VALL-E, VoiceCraft, XTTS-v2) autoregressively generate discrete speech tokens and implicitly model duration, but suffer from poor robustness, slow inference that scales linearly with output length, and a tendency to hallucinate — particularly on hard inputs like tongue twisters or repeated words. NAR models (e.g., NaturalSpeech 2, VoiceBox, NaturalSpeech 3) are more robust and faster, but traditionally require explicit text-speech alignment supervision and phoneme-level duration prediction, which introduces pipeline complexity and can lead to less prosodically diverse output.

MaskGCT (Masked Generative Codec Transformer) is proposed as a fully non-autoregressive TTS system that eliminates both the need for explicit alignment supervision and phoneme-level duration prediction. It leverages the mask-and-predict learning paradigm — originally popularized in image generation by MaskGIT and Muse — for both stages of speech generation: text-to-semantic and semantic-to-acoustic. The result is a system that achieves human-level similarity, naturalness, and intelligibility across multiple benchmarks, trained on 100K hours of in-the-wild multilingual speech data.

Background: Non-Autoregressive Masked Generative Transformers

Given a discrete token sequence $\mathbf{X}$ of length $N$, MaskGCT applies a binary mask $\mathbf{M}_t = [m_{t,i}]_{i=1}^{N}$, where each $m_{t,i}$ is independently sampled from a Bernoulli distribution with parameter $\gamma(t)$. Tokens with $m_{t,i}=1$ are replaced by a special [MASK] token; the rest remain visible. A canonical choice for the mask schedule is $\gamma(t) = \sin\!\left(\frac{\pi t}{2T}\right)$ for $t \in (0, T]$.

The model is trained to recover the masked tokens from the partially observed sequence and a conditioning signal $\mathbf{C}$:

$$\mathcal{L}_{\text{mask}} = \mathbb{E}_{\mathbf{X} \in \mathcal{D},\; t \in [0, T]} \left[ -\sum_{i=1}^{N} m_{t,i} \cdot \log p_\theta(x_i \mid \mathbf{X}_t, \mathbf{C}) \right]$$

At inference time, decoding proceeds iteratively over $S$ steps, starting from a fully masked sequence $\mathbf{X}_T$. At each step $i$, the model samples $\hat{\mathbf{X}}_0$ from $p_\theta(\mathbf{X}_0 \mid \mathbf{X}_{T-(i-1)\cdot T/S}, \mathbf{C})$. It then re-masks the $\lfloor N \cdot \gamma(T - i \cdot T/S) \rfloor$ tokens with the lowest confidence scores. Tokens that were already unmasked in the previous step receive a confidence of 1 and are never re-masked. This parallel iterative decoding converges in a fixed, small number of steps (25–50) regardless of output length — a substantial advantage over AR models.

System Overview

MaskGCT is a two-stage system with four main components:

1. Speech Semantic Representation Codec: A VQ-VAE that quantizes SSL speech features into discrete semantic tokens.
2. Text-to-Semantic (T2S) Model: A masked generative transformer that predicts semantic tokens from text and a prompt semantic token prefix.
3. Semantic-to-Acoustic (S2A) Model: A masked generative codec transformer (inspired by SoundStorm) that predicts multi-layer acoustic tokens conditioned on semantic tokens and prompt acoustic tokens.
4. Speech Acoustic Codec: A DAC-style RVQ codec that reconstructs waveforms from acoustic tokens.

A key design principle: unlike prior two-stage systems (e.g., VALL-E, SpearTTS, SoundStorm) where Stage 1 is autoregressive, both stages in MaskGCT use the masked generative paradigm. Neither stage requires text-speech alignment labels or phoneme-level duration targets.

Component 1: Speech Semantic Representation Codec

Previous two-stage TTS systems extract semantic tokens using k-means clustering over SSL features. The authors find this introduces information loss that complicates downstream acoustic token prediction, especially for tonally rich languages like Chinese (where proper prosody is hard to recover from k-means tokens).

Instead, MaskGCT trains a VQ-VAE to learn a learned vector-quantization codebook that directly reconstructs the original SSL feature sequence. Given an SSL feature sequence $\mathbf{S} \in \mathbb{R}^{T \times d}$, an encoder $\mathcal{E}$ maps it to a quantized embedding $\mathbf{E}$, and a decoder reconstructs $\hat{\mathbf{S}}$ from $\mathbf{E}$. The total training loss is:

$$\mathcal{L}_{\text{total}} = \frac{1}{Td}\Bigl(\lambda_{\text{rec}}\|\mathbf{S} - \hat{\mathbf{S}}\|_1 + \lambda_{\text{codebook}}\|\text{sg}(\mathcal{E}(\mathbf{S})) - \mathbf{E}\|_2 + \lambda_{\text{commit}}\|\text{sg}(\mathbf{E}) - \mathcal{E}(\mathbf{S})\|_2\Bigr)$$

where $\text{sg}(\cdot)$ denotes the stop-gradient operation.

The SSL features are taken from the 17th layer of W2v-BERT 2.0 (original dimension 1,024). Following improved VQ-GAN and DAC, the encoder projects into a low-dimensional latent space using factorized codes before quantization. The codebook contains 8,192 entries of dimension 8. The encoder and decoder each consist of 12 mirrored ConvNext blocks with kernel size 7 and hidden size 384. Input features are normalized to zero mean and unit variance. The model runs at 16 kHz with a hop size of 320, yielding approximately 50 frames per second. The total model size is 44M parameters.

Component 2: Text-to-Semantic (T2S) Model

The T2S model is trained to learn $p_{\theta_{\text{s1}}}(\mathbf{S} \mid \mathbf{S}_t, (\mathbf{S}^p, \mathbf{P}))$, where $\mathbf{S}$ is the full semantic token sequence, $\mathbf{S}^p$ is the prompt semantic token prefix extracted from a random prefix of the speech, and $\mathbf{P}$ is the text token sequence.

In-context learning: Rather than using cross-attention or separate conditioning modules, the text tokens and prompt semantic tokens are simply prepended to the masked semantic token sequence as a prefix. The model learns to exploit this prefix as context for filling in the masked positions — no explicit alignment is needed.

Architecture: The backbone is a Llama-style Transformer with bidirectional (non-causal) attention, SwiGLU activations, Rotary Position Embeddings (RoPE, $\theta = 10000$), and Adaptive RMSNorm conditioned on the diffusion timestep $t$. Two model sizes are provided:

• T2S-Base: 16 layers, hidden dim 1,024, FFN dim 4,096, 16 attention heads — 315M parameters.
• T2S-Large: 16 layers, hidden dim 1,536, FFN dim 6,144, 16 attention heads — 695M parameters.

Duration control: Because the model generates tokens of a user-specified length, inference naturally supports speed control. A separate flow-matching-based duration predictor (12-layer Transformer, hidden 768) is trained to predict total speech duration given text and prompt duration. It models duration in log space using an ODE: $x_t = (1-t)x_0 + t x_1$, optimized by $\mathbb{E}_{t,x_1}\|v_\theta(x_t,t) - (x_1 - x_0)\|^2$. Phone-level durations from a pretrained duration aligner (using Monotonic Alignment Search over W2v-BERT 2.0 features) provide ground truth for training, but only the total duration is used at inference.

Text tokenization: Both Grapheme-to-Phoneme (G2P) and Byte Pair Encoding (BPE, from Whisper with vocabulary >30,000) are explored. G2P generally outperforms BPE for English; for Chinese, BPE achieves slightly lower WER (1.921 vs. 2.183), likely because the Chinese G2P system has deficiencies in handling polyphonic characters while BPE learns context-dependent pronunciations.

Classifier-free guidance (CFG): During training, the prompt is dropped with probability 0.15, enabling unconditional generation. At inference, CFG with a scale of 2.5 is applied, along with CFG rescaling (factor 0.75) following the method of Lin et al. to prevent over-saturation. The final embedding is a weighted blend: $w_{\text{rescale}} \cdot g^{\text{rescale}} + (1 - w_{\text{rescale}}) \cdot g^{\text{cfg}}$.

Inference settings: 50 steps by default (25 is near-optimal for balancing SIM and WER). Top-k sampling with $k=20$, temperature annealing from 1.5 to 0, and Gumbel noise added to confidence scores during remasking.

Component 3: Semantic-to-Acoustic (S2A) Model

The S2A model is based on SoundStorm but now receives predicted (rather than AR-generated) semantic tokens from the T2S model. It is trained to generate the multi-layer RVQ acoustic tokens conditioned on semantic tokens and prompt acoustic tokens.

Given $N=12$ RVQ layers of acoustic tokens $\mathbf{A}^{1:N}$, at each training step a layer index $j$ is sampled according to a linear schedule:

$$p(j) = 1 - \frac{2j}{N(N+1)}$$

(earlier layers are sampled more frequently, reflecting that they carry more acoustic information.) Layer $j$ tokens are masked at timestep $t$, and the model predicts them conditioned on prompt acoustic tokens $\mathbf{A}^p$, semantic tokens $\mathbf{S}$, and all previous layers $\mathbf{A}^{1:j-1}$:

$$p_{\theta_{\text{s2a}}}(\mathbf{A}^j \mid \mathbf{A}^j_t, (\mathbf{A}^p, \mathbf{S}, \mathbf{A}^{1:j-1}))$$

Since the number of frames in the semantic token sequence equals the combined length of the prompt and target acoustic sequences, the S2A model simply sums the semantic token embeddings with the acoustic token embeddings from layers 1 through $j$ at the corresponding positions.

Architecture: 16-layer bidirectional Transformer, hidden dim 1,024, FFN dim 4,096, 16 attention heads — 353M parameters, same architectural style as T2S.

Inference: Coarse-to-fine decoding layer by layer. Default steps per layer: $[40, 16, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]$. A faster variant $[10, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]$ works well with minor performance degradation. Greedy sampling (rather than top-k) is used for single-step layers.

Component 4: Speech Acoustic Codec

The acoustic codec quantizes 24 kHz waveforms into 12 layers of RVQ tokens (hop size 480 samples, ~50 tokens/second). Each codebook has 1,024 entries of dimension 8. The encoder follows DAC; the decoder follows Vocos, which models amplitude and phase to reconstruct waveforms via inverse STFT — avoiding upsampling layers for more efficient training and inference. Total model size: 170M parameters.

Training losses include: multi-scale mel reconstruction loss $\mathcal{L}_{\text{rec}}$ ($\lambda=10.0$), adversarial loss from Multi-Period Discriminator (MPD) and Multi-Band Multi-Scale STFT discriminator ($\lambda_{\text{adv}}=2.0$), relative feature matching loss ($\lambda_{\text{feat}}=2.0$), codebook loss ($\lambda_{\text{codebook}}=1.0$), and commitment loss ($\lambda_{\text{commit}}=0.25$).

Training Setup

All models are trained on 8 × NVIDIA A100 80GB GPUs. The training dataset is Emilia, a large-scale in-the-wild multilingual speech dataset. For the primary experiments, 50K hours of English and 50K hours of Chinese data are used (100K hours total). Additional languages (Japanese, Korean, German, French) are added for multilingual experiments, with 2,500–8,200 hours each.

Optimization: AdamW optimizer, learning rate $10^{-4}$, 32K warmup steps, inverse square root learning rate schedule.

Evaluation Setup

Three zero-shot TTS benchmarks are used:

• LibriSpeech test-clean: Standard English benchmark.
• SeedTTS test-en: 1,000 samples from Common Voice (English), introduced by Seed-TTS.
• SeedTTS test-zh: 2,000 samples from DiDiSpeech (Chinese), introduced by Seed-TTS.

Objective metrics:

• SIM-O: Cosine similarity between WavLM TDNN speaker embeddings of generated and reference speech.
• WER: Word Error Rate using HuBERT-based ASR for LibriSpeech, Whisper-large-v3 for SeedTTS test-en, Paraformer-zh for SeedTTS test-zh.
• FSD: Fréchet Speech Distance using wav2vec 2.0 features (following VoiceBox).

Subjective metrics:

• CMOS: Comparative Mean Opinion Score for naturalness (scale −3 to +3, relative to ground truth).
• SMOS: Similarity Mean Opinion Score (scale 1–5).

Baselines: VALL-E, NaturalSpeech 3, VoiceBox, VoiceCraft, XTTS-v2, CosyVoice, and an ablation system AR + SoundStorm (MaskGCT with T2S replaced by an AR model).

Main Zero-Shot TTS Results

Numbers in parentheses represent the best result selected from 5 random samples ("rerank 5"). MaskGCT achieves human-level similarity on all three benchmarks: SIM-O improvements of +0.017, −0.002, and +0.027 relative to ground truth on LibriSpeech, SeedTTS test-en, and test-zh respectively; SMOS improvements of +0.28, +0.32, and +0.25. CMOS scores of +0.10, +0.03, +0.05 over ground truth on the three sets indicate human-level naturalness. FSD scores, measuring distributional quality, are consistently best or second-best for MaskGCT.

Autoregressive vs. Masked Generative: Direct Comparison

To isolate the contribution of the masked generative T2S model, an AR + SoundStorm ablation is evaluated — replacing the T2S MaskGCT with an autoregressive T2S model while keeping S2A identical.

MaskGCT consistently outperforms the AR alternative across all metrics and benchmarks, with particularly large CMOS improvements of +0.12, +0.08, and +0.37 on LibriSpeech, test-en, and test-zh respectively. The advantage is amplified on hard cases (tongue twisters, repetitive phrases): AR + SoundStorm achieves WER of 34.16 on SeedTTS test-hard versus MaskGCT's 10.27 (and 6.258 with reranking from 5 samples vs. 17.05 for AR + SoundStorm).

Duration Length Analysis

MaskGCT's unique ability to generate speech of any specified length is analyzed by multiplying the ground-truth total duration by factors from 0.7 to 1.3. WER is lowest at multiplier 1.0 but remains within a reasonable range across the full spectrum. At 0.9× and 1.1×, performance is very close to 1.0×. Even at 0.7× (fast speech) or 1.3× (slow speech), the model maintains intelligibility. This demonstrates robustness to tempo variation and confirms that duration control does not substantially degrade content accuracy.

Speech Style Imitation

Beyond speaker identity cloning (measured by SIM-O), MaskGCT is evaluated on two additional style dimensions: accent imitation (L2-ARCTIC corpus) and emotion imitation (ESD corpus). New metrics are introduced: Accent SIM (using CommonAccent embeddings) and Emotion SIM (using emotion2vec embeddings), alongside corresponding subjective scores.

MaskGCT leads on virtually all style imitation metrics, demonstrating that its in-context learning capability extends beyond speaker timbre to stylistic dimensions including accent and emotional prosody.

Multilingual Zero-Shot TTS

MaskGCT is extended to six languages by retraining only the T2S model on expanded data (adding Japanese, Korean, German, French). The S2A model and tokenizers are not retrained.

MaskGCT outperforms baselines in SIM-O across all four languages and in WER for Korean, French, and German. Only Japanese WER is slightly worse than XTTS-v2. The authors note that retraining all components (including tokenizers and S2A) on the expanded multilingual data would likely improve performance further.

Ablation Studies

Inference Timesteps (T2S)

The impact of T2S inference steps (5–75) is studied. Key findings:

• SIM-O converges quickly: from 0.761 (5 steps) to 0.771 (75 steps) on test-zh; from 0.696 to 0.715 on test-en. Near-peak SIM is achieved at ~10 steps.
• WER improves dramatically up to 25 steps: from 10.19 → 2.507 on test-zh; 8.096 → 2.346 on test-en. Beyond 25 steps, improvements are negligible.
• Practical recommendation: 25 steps is optimal for balancing both metrics.

Inference Timesteps (S2A)

Comparing step schedules $[40, 16, 1, \ldots, 1]$ vs. $[10, 1, \ldots, 1]$ on SeedTTS test-en:

• $[40,16,1,\ldots]$: SIM-O=0.728, WER=2.466, FSD=0.159
• $[10,1,\ldots]$: SIM-O=0.709, WER=2.796, FSD=0.164 (slightly worse but still strong)

Model Size (T2S)

The Large model outperforms Base consistently, though not by wide margins, suggesting that even the Base model achieves strong performance at 100K hours of training data. The authors plan to explore scaling laws more systematically in future work.

Text Tokenizer (G2P vs. BPE)

G2P outperforms BPE for English (SIM-O 0.728 vs. 0.711; WER 2.466 vs. 4.036). For Chinese, G2P leads on SIM-O (0.777 vs. 0.769) but BPE achieves lower WER (1.921 vs. 2.183). The discrepancy for Chinese is attributed to limitations in the G2P system's handling of polyphonic characters. G2P is used by default.

Additional Capabilities

Duration-Controllable Speech Translation (Cross-Lingual Dubbing)

By specifying the target speech duration equal to the source speech duration and using the source speech as the acoustic prompt, MaskGCT can translate speech across six languages while preserving speaker identity and matching total duration. A 30-set experiment (all language pairs among Zh, En, Ja, Ko, De, Fr) demonstrates good SIM-O (0.53–0.75 range) across most pairs. WER is generally low for English as both source and target, and reasonable for Japanese. Higher WER for some pairs (e.g., Korean→Chinese: 31.30) is attributed to the challenge of matching duration across linguistically different languages with limited data.

Emotion Control via Post-Training

After pretraining on Emilia, the T2S model is fine-tuned on the ESD emotion dataset by prepending an emotion label token to the input. This unlocks emotion-controlled zero-shot TTS. A classification accuracy of 72% on the test set confirms that the fine-tuned model generates speech consistent with the target emotion.

Speech Content Editing

The mask-and-predict mechanism naturally supports zero-shot speech editing. A text-speech aligner identifies the boundary of the segment to be edited in the semantic token sequence; those tokens are masked and re-predicted using the edited text and the surrounding unmasked tokens. The authors note this is not yet fully robust, and suggest that a fill-in-the-mask training paradigm (as in VoiceBox) might improve editing performance.

Voice Conversion

The S2A model is fine-tuned for zero-shot voice conversion using a modified training strategy: semantic tokens $\mathbf{S}_{\text{src}}$ are extracted from the source speech (with timbral perturbation applied to the mel-spectrogram input to W2v-BERT 2.0 to reduce timbre leakage), and prompt acoustic tokens $\mathbf{A}_{\text{ref}}$ from the reference speaker are used to condition generation of target acoustic tokens $\mathbf{A}_{\text{tgt}}$.

System Comparison Summary

Broader Impact and Limitations

MaskGCT achieves human-level speaker similarity and naturalness on multiple English and Chinese benchmarks, representing a significant step toward foundational speech generation models. The system's ability to control total speech duration, support multiple languages, and extend to translation, editing, voice conversion, and emotion control makes it versatile.

The authors acknowledge several limitations and areas for future work:

• The speech editing capability is not yet fully robust; a training paradigm more explicitly designed for editing (e.g., fill-in-the-mask) would be needed.
• The Chinese G2P tokenizer has known deficiencies with polyphonic characters, which partially explains why BPE achieves lower WER in Chinese.
• Multilingual performance for French, German, and other low-resource languages could be improved by retraining all components (including S2A and tokenizers) on the full multilingual data.
• The paper notes that the scaling laws for both model size and data are not yet fully characterized, and further investigation is planned.
• Cross-lingual dubbing shows higher WER for certain language pairs (e.g., Korean→Chinese/French/German), attributed to duration matching difficulty and limited training data for some language pairs.

Ethical concerns: High speaker similarity creates risks of voice spoofing and impersonation. The authors emphasize the need for robust speech synthesis detection models and reporting mechanisms for misuse. All experiments assume explicit consent from the target speaker.

Key Contributions Summary

• MaskGCT is the first fully non-autoregressive, alignment-free, duration-prediction-free two-stage masked generative TTS system achieving human-level quality.
• A VQ-VAE-based semantic codec trained on W2v-BERT 2.0 features provides richer semantic tokens than k-means, particularly benefiting tonal languages.
• The T2S model leverages in-context learning with prefix concatenation, bypassing the need for explicit text-speech alignment at train or inference time.
• Both T2S and S2A use parallel iterative decoding converging in a fixed number of steps (25–50) — a major efficiency advantage over AR models.
• Length controllability is a unique property among NAR systems: MaskGCT can generate speech of any specified total duration, enabling tempo control and cross-lingual dubbing.
• Trained on 100K hours of in-the-wild multilingual speech (Emilia), MaskGCT demonstrates strong scaling and generalization, outperforming all evaluated baselines across three benchmarks.