VALL-E
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
A language model for text-to-speech that generates discrete audio codec tokens instead of continuous signals. Trained on 60K hours of speech, VALL-E synthesizes high-quality speech for unseen speakers using only a 3-second acoustic prompt, enabling zero-shot voice cloning without fine-tuning.
Demos
VALL-E and its variants (VALL-E X, VALL-E R, VALL-E 2) demonstrate neural codec language models for zero-shot text-to-speech synthesis. Evaluate these architectures on their ability to synthesize natural, speaker-similar speech from minimal 3-second prompts, preserve speaker emotion and acoustic environment, and handle multilingual scenarios—with VALL-E 2 achieving human parity performance on standard benchmarks.
Links
Paper & demos
Code & resources
Abstract
We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
Introduction and Motivation
Modern text-to-speech (TTS) systems have achieved impressive quality for seen speakers, but they struggle in zero-shot scenarios — synthesizing speech for speakers never seen during training. Existing approaches fall into two camps: speaker adaptation methods, which fine-tune a pre-trained system on a small amount of target-speaker data, and speaker encoding methods, which extract a speaker embedding from a short enrolled recording and condition synthesis on it. Both approaches have fundamental weaknesses: adaptation requires per-speaker fine-tuning that is impractical at scale, while encoding methods suffer from a generalization gap between seen and unseen speakers.
A complementary challenge is the scale of training data. State-of-the-art multi-speaker TTS systems are trained on at most a few hundred hours of high-quality, studio-recorded speech, orders of magnitude less than what is routinely used in natural language processing or speech recognition. The authors argue that, rather than engineering ever more complex model structures, the right path forward is to treat speech synthesis as a language modeling problem and scale training data dramatically — mirroring the trajectory of GPT-style models in NLP, which progressed from 16 GB to over 1 TB of text with consistent quality gains.
This paper introduces VALL-E, a neural codec language model that reframes TTS as a conditional language modeling task over discrete audio codec codes. Instead of predicting a mel spectrogram through continuous signal regression, VALL-E generates a sequence of discrete acoustic tokens derived from an off-the-shelf neural audio codec model (EnCodec). Trained on 60,000 hours of English speech from LibriLight — hundreds of times more data than any prior TTS system — VALL-E acquires strong in-context learning capabilities: given only a 3-second enrolled recording of an unseen speaker plus a text prompt, it synthesizes high-quality, speaker-faithful speech without any fine-tuning.
Background: Speech Quantization and Neural Audio Codecs
Raw audio synthesis is challenging because audio is stored as 16-bit integer values, requiring a model to output $2^{16} = 65{,}536$ probabilities per timestep. With sample rates exceeding 10,000 Hz, sequence lengths become intractable. Speech quantization is therefore essential to compress both the value range and the sequence length.
Earlier approaches include $\mu$-law quantization (used in WaveNet, reducing values to 256 levels but not reducing sequence length), and discrete codes from self-supervised models such as vq-wav2vec or HuBERT (which compress sequence length but discard speaker identity). AudioLM demonstrated that neural codec tokens can represent both content and acoustic identity.
VALL-E adopts EnCodec as its tokenizer — a convolutional encoder-decoder neural codec operating at 24 kHz. EnCodec uses Residual Vector Quantization (RVQ), in which each frame's embedding is quantized by a cascade of codebooks, each quantizing the residual of the previous. VALL-E uses 8 RVQ codebooks with 1024 entries each, corresponding to a 6K bitrate configuration. The encoder produces embeddings at 75 Hz (a 320× reduction in sampling rate). For a 10-second waveform, the resulting discrete representation is a matrix $\mathbf{C} \in \mathbb{Z}^{750 \times 8}$.
A critical property of RVQ: the first quantizer captures the coarsest but most important acoustic properties (including speaker identity), while successive quantizers encode progressively finer acoustic details. This hierarchical structure directly motivates VALL-E's two-stage model design.
Problem Formulation
Given a dataset $\mathcal{D} = \{(\mathbf{x}_i, \mathbf{y}_i)\}$ where $\mathbf{y}$ is an audio sample and $\mathbf{x} = \{x_0, x_1, \ldots, x_L\}$ is its phoneme transcription, VALL-E encodes each audio sample into a discrete acoustic code matrix:
$$\text{EnCodec}(\mathbf{y}) = \mathbf{C}^{T \times 8}$$where $T$ is the downsampled utterance length, the row $\mathbf{c}_{t,:}$ represents 8 codes for frame $t$, and the column $\mathbf{c}_{:,j}$ is the code sequence from the $j$-th codebook ($j \in \{1, \ldots, 8\}$). Reconstruction is performed by the codec decoder: $\text{DeCodec}(\mathbf{C}) \approx \hat{\mathbf{y}}$.
Zero-shot TTS is cast as conditional codec language modeling: train a neural language model to generate an acoustic code matrix $\mathbf{C}$ conditioned on a phoneme sequence $\mathbf{x}$ and an acoustic prompt matrix $\tilde{\mathbf{C}}^{T' \times 8}$ (derived from a 3-second enrolled recording), maximizing:
$$\max \; p(\mathbf{C} \mid \mathbf{x}, \tilde{\mathbf{C}})$$The phoneme sequence constrains the content of the generated speech, while the acoustic prompt constrains the speaker identity. During inference, the acoustic code matrix is estimated by the language model and then decoded to a waveform by the EnCodec decoder.
Model Architecture: Hierarchical Conditional Codec Language Modeling
VALL-E uses two complementary language models arranged hierarchically, reflecting the hierarchical nature of RVQ codes.
Stage 1: Autoregressive (AR) Codec Language Model
The AR model generates tokens from the first quantizer $\mathbf{c}_{:,1}$, which captures the dominant acoustic and speaker properties. It is an autoregressive decoder-only transformer conditioned on the phoneme sequence $\mathbf{x}$ and the first-codebook acoustic prompt $\tilde{\mathbf{C}}_{:,1}$:
$$p(\mathbf{c}_{:,1} \mid \mathbf{x}, \tilde{\mathbf{C}}_{:,1};\, \theta_{AR}) = \prod_{t=0}^{T} p\!\left(\mathbf{c}_{t,1} \mid \mathbf{c}_{Crucially, the AR model is trained with pure causal language modeling — no explicit acoustic prompt segment is extracted during training. Any prefix of the sequence naturally serves as a prompt for the remainder, enabling seamless inference-time prompting. The model concatenates the phoneme sequence of the enrolled recording with the phoneme sequence for the target content as the full phoneme prompt, and uses the enrolled recording's first-level acoustic tokens as the acoustic prefix.
Architecture details: phoneme embedding $W_x$, acoustic embedding $W_a$, a Transformer decoder (12 layers, 16 attention heads, embedding dimension 1024, feed-forward dimension 4096, dropout 0.1). Sinusoidal position embeddings are computed separately for prompt tokens and input tokens. Two special <EOS> tokens are appended after the phoneme and acoustic sequences respectively. The output projection layer shares parameters with $W_a$.
Inference: Sampling-based decoding (not beam search, which was found to cause infinite loops in this setting) is used. Sampling also improves output diversity.
Stage 2: Non-Autoregressive (NAR) Codec Language Model
Given the first-codebook tokens from the AR model, the NAR model generates codes for quantizers 2 through 8 in a non-autoregressive fashion. Its optimization objective is:
$$p(\mathbf{C}_{:,2:8} \mid \mathbf{x}, \tilde{\mathbf{C}};\, \theta_{NAR}) = \prod_{j=2}^{8} p\!\left(\mathbf{c}_{:,j} \mid \mathbf{C}_{:,Architecture details: same Transformer backbone as the AR model but with eight separate acoustic embedding layers $W_a^1, \ldots, W_a^8$. During each training step, a stage $i \in [2, 8]$ is sampled uniformly. The input acoustic embedding is computed as:
$$e_{c_{t,j}} = W_a^j \odot c_{t,j}, \qquad \mathbf{e}_{c_t} = \sum_{j=1}^{i-1} e_{c_{t,j}}$$The acoustic prompt embedding uses all 8 codebooks summed: $\mathbf{e}_{\tilde{c}_t} = \sum_{j=1}^{8} e_{\tilde{c}_{t,j}}$. The Transformer input is the concatenation $(\mathbf{e}_x, \mathbf{e}_{\tilde{c}}, \mathbf{e}_{c_{:,
where $a_i$ and $b_i$ are linear projections of a stage embedding. Unlike the AR model, the NAR Transformer uses full bidirectional self-attention — each token can attend to all input tokens. Greedy decoding (argmax) is used at inference time. The parameters of the $j$-th acoustic embedding layer are shared with the $(j+1)$-th prediction layer.
Full Joint Model
Combining both stages, the full generative model factorizes as:
$$p(\mathbf{C} \mid \mathbf{x}, \tilde{\mathbf{C}};\, \theta) = p(\mathbf{c}_{:,1} \mid \tilde{\mathbf{C}}_{:,1}, \mathbf{x};\, \theta_{AR}) \cdot \prod_{j=2}^{8} p\!\left(\mathbf{c}_{:,j} \mid \mathbf{c}_{:,Inference: In-Context Learning via Prompting
At inference time, the text is first converted to a phoneme sequence via a grapheme-to-phoneme (G2P) model, and the enrolled recording is encoded to an acoustic matrix by EnCodec, forming the phoneme prompt and acoustic prompt.
Two inference modes are defined:
- VALL-E: The transcription phoneme sequence of the enrolled speech is prepended to the target phoneme sequence. The first-codebook acoustic tokens of the enrolled recording serve as the AR prefix. The AR model generates first-codebook tokens for the target content cloning the enrolled speaker's voice; the NAR model then fills in the remaining 7 codebooks.
- VALL-E-continual: The first 3 seconds of the target utterance itself are used as both phoneme and acoustic prompts, and the model generates a semantically continuous continuation. This mode achieves lower WER because the acoustic tokens are ground-truth rather than cross-utterance.
The in-context learning capability of VALL-E parallels GPT-3's few-shot prompting: no parameter updates are required at inference time; the model generalizes to unseen speakers purely through conditioning on the acoustic prompt.
Training Setup
Dataset
VALL-E is trained on LibriLight, a corpus of 60,000 hours of unlabeled English speech from audiobooks with approximately 7,000 unique speakers. Because the data is audio-only, a hybrid DNN-HMM ASR model (trained on 960 hours of labeled LibriSpeech following the Kaldi recipe) is used to generate phoneme-level transcriptions and force-aligned phoneme sequences (30 ms frameshift). The EnCodec model generates the 8-codebook acoustic code matrices for the full 60K hours.
Compared to prior multi-speaker TTS datasets (e.g., LibriTTS at ~500 hours), LibriLight contains noisier speech and less accurate transcriptions, but provides far greater speaker and prosody diversity. The authors argue the language model objective is inherently more robust to such noise than mel spectrogram regression loss.
Training Procedure
Both the AR and NAR models share the same Transformer architecture (12 layers, 16 attention heads, 1024-dim embeddings, 4096-dim FFN, dropout 0.1). Training waveforms are randomly cropped to 10–20 second segments. For the NAR acoustic prompt, a random 3-second segment from the same utterance is selected. Consecutive repeated phonemes in the force-aligned sequence are removed.
Training uses 16 NVIDIA Tesla V100 32 GB GPUs with a batch size of 6,000 acoustic tokens per GPU for 800,000 steps. Optimization uses AdamW with linear warmup over 32,000 steps to a peak learning rate of $5 \times 10^{-4}$, followed by linear decay.
Experiments
Baseline
The primary baseline is YourTTS, the state-of-the-art zero-shot TTS system at the time of writing, trained on a combined dataset of VCTK, LibriTTS, and TTS-Portuguese (totaling around 600 hours — over 100× less data than VALL-E). The released YourTTS checkpoint is used directly.
Evaluation Metrics
- Speaker Similarity (SPK): Cosine similarity predicted by WavLM-TDNN (a state-of-the-art speaker verification model, top-ranked at VoxSRC 2021 and 2022). Range $[-1, 1]$; higher is better.
- Word Error Rate (WER): ASR performed by HuBERT-Large fine-tuned on LibriSpeech 960h (CTC, no LM fusion), measuring synthesis robustness against deletion, insertion, and replacement errors.
- CMOS (Comparative Mean Opinion Score): Human evaluation of naturalness relative to baseline, scale $[-3, +3]$. 12 native speaker raters.
- SMOS (Similarity Mean Opinion Score): Human evaluation of speaker similarity, scale $[1, 5]$ in 0.5 increments. 6 native speaker raters.
LibriSpeech Evaluation
The LibriSpeech test-clean set is used (samples of 4–10 seconds, forming a 2.2-hour subset). There is no speaker overlap between LibriLight training data and LibriSpeech test-clean. For each test sample, a 3-second segment from another utterance of the same speaker is used as the enrolled recording (averaged over 3 runs).
| Model | WER (%) | SPK |
|---|---|---|
| GroundTruth | 2.2 | 0.754 |
| GSLM (speech-to-speech) | 12.4 | 0.126 |
| AudioLM* (speech-to-speech) | 6.0 | — |
| YourTTS | 7.7 | 0.337 |
| VALL-E | 5.9 | 0.580 |
| VALL-E-continual | 3.8 | 0.508 |
VALL-E outperforms YourTTS in both robustness (WER 5.9% vs 7.7%) and speaker similarity (SPK 0.580 vs 0.337), despite YourTTS being trained on speakers from the same audiobook domain. The VALL-E-continual setting achieves the lowest WER (3.8%) because ground-truth acoustic tokens form the prefix. VALL-E also outperforms speech-to-speech LMs: GSLM uses HuBERT codes that discard speaker identity (SPK 0.126), and VALL-E beats AudioLM's reported WER (6.0%). The authors attribute VALL-E's robustness to using pseudo-phoneme labels rather than HuBERT/w2v-BERT codes, which provide better text-acoustic alignment.
| Model | SMOS | CMOS (vs. VALL-E) |
|---|---|---|
| YourTTS | 3.45 ± 0.09 | −0.12 |
| VALL-E | 4.38 ± 0.10 | 0.00 (reference) |
| GroundTruth | 4.50 ± 0.10 | +0.17 |
VALL-E achieves SMOS 4.38 vs. YourTTS's 3.45 — a gain of +0.93 SMOS — nearly matching ground truth (4.50). For naturalness, VALL-E exceeds YourTTS by +0.12 CMOS. Ground truth audio is rated +0.17 CMOS above VALL-E, indicating a small but perceptible gap versus human recordings on LibriSpeech.
VCTK Evaluation
VCTK contains 108 speakers, all unseen during VALL-E training. YourTTS had seen 97 of these 108 speakers in training, so results are reported for both the full 108-speaker set and a 11-speaker unseen subset. Prompts of 3s, 5s, and 10s are tested.
| Model | 3s prompt | 5s prompt | 10s prompt |
|---|---|---|---|
| 108 full speakers | |||
| YourTTS* | 0.357 | 0.377 | 0.394 |
| VALL-E | 0.382 | 0.423 | 0.484 |
| GroundTruth | 0.546 | 0.591 | 0.620 |
| 11 unseen speakers (fair comparison) | |||
| YourTTS | 0.331 | 0.337 | 0.344 |
| VALL-E | 0.389 | 0.380 | 0.414 |
| GroundTruth | 0.528 | 0.556 | 0.586 |
VALL-E outperforms YourTTS even on the full 108-speaker set — where YourTTS has a clear advantage from prior training exposure. In the fair 11-speaker comparison, the performance gap favoring VALL-E is larger, especially with 3-second prompts. Longer prompts consistently improve VALL-E's speaker similarity, as expected.
| Model | SMOS | CMOS (vs. VALL-E) |
|---|---|---|
| YourTTS* | 3.70 ± 0.09 | −0.23 |
| VALL-E | 3.81 ± 0.09 | 0.00 (reference) |
| GroundTruth | 4.29 ± 0.09 | −0.04 |
VALL-E beats YourTTS by +0.11 SMOS and +0.23 CMOS on VCTK, despite seeing none of the 60 evaluated speakers. Notably, VALL-E achieves +0.04 CMOS over ground truth on VCTK — meaning no statistically significant difference from human recordings in naturalness. The authors attribute this to shorter average sentence length in VCTK and the fact that some VCTK ground-truth recordings contain environmental noise, making them easier to match or exceed.
Ablation Studies
NAR Model Ablation
Three NAR variants are evaluated using ground-truth first-level acoustic tokens as input:
| Setting | WER (%) | SPK |
|---|---|---|
| NAR-no prompt | 19.6 | 0.518 |
| NAR-phn prompt (phoneme only) | 3.0 | 0.541 |
| NAR-2 prompts (phoneme + acoustic) | 2.8 | 0.732 |
Without any prompt, the NAR model achieves WER 19.6 even with ground-truth first-codebook input, showing that without guidance it cannot reliably generate coherent content. Adding the phoneme prompt reduces WER dramatically (19.6 → 3.0), confirming that the phoneme prompt primarily controls content accuracy. Adding the acoustic prompt on top of the phoneme prompt significantly improves speaker similarity (0.541 → 0.732), confirming that the acoustic prompt is the key driver of speaker identity cloning.
AR Model Ablation
| Setting | WER (%) | SPK |
|---|---|---|
| VALL-E (full) | 5.9 | 0.585 |
| w/o acoustic prompt | 5.9 | 0.236 |
Removing the acoustic prompt from the AR model maintains robustness (WER unchanged at 5.9%) but collapses speaker similarity from 0.585 to 0.236 — showing that even though the NAR model sees the acoustic prompt, the acoustic prefix in the AR stage is critical for speaker identity in the generated first-codebook tokens. Content (WER) is controlled by the phoneme prompt alone, but speaker identity requires the acoustic prefix in both stages.
Qualitative Analysis
Output Diversity
Because VALL-E uses sampling-based decoding over discrete tokens, its outputs are stochastic: different runs with the same input text and speaker prompt produce genuinely different speech. This contrasts with mel-spectrogram regression TTS, which deterministically maps inputs to outputs. The diversity manifests as differences in speech rate, phrase duration, and prosodic emphasis across runs.
This diversity is practically valuable: diverse synthetic speech with different speakers, acoustic environments, and prosodies is exactly what downstream ASR data augmentation requires — a need that deterministic TTS cannot easily fulfill.
Acoustic Environment Maintenance
When the enrolled 3-second recording contains reverberation or other environmental characteristics, VALL-E preserves those conditions in the synthesized output, whereas the baseline (YourTTS) always produces clean speech. This behavior emerges from large-scale training on diverse acoustic conditions in LibriLight, which allows VALL-E to learn acoustic consistency rather than always defaulting to a studio-clean environment.
Speaker Emotion Maintenance
VALL-E can preserve emotional tone from the acoustic prompt at zero-shot. When acoustic prompts from EmoV-DB (a dataset with five emotion categories — including anger) are used, VALL-E synthesizes speech in the same emotional register without any emotion-specific fine-tuning. This is a purely emergent property of the large-scale conditional language modeling training regime.
Comparison to Prior Art
VALL-E differs from existing systems along four key axes, summarized as follows:
| Aspect | Current Systems | VALL-E |
|---|---|---|
| Intermediate representation | Mel spectrogram | Audio codec codes |
| Objective function | Continuous signal regression | Language model (cross-entropy) |
| Training data | ≤ 600 hours | 60,000 hours |
| In-context learning | No | Yes |
In relation to speech-to-speech models: GSLM and AudioLM both use audio codes but operate without text, making content control impossible. VALL-E is a TTS model that explicitly controls content via phoneme prompts. AudioLM uses both semantic (k-means from a self-supervised model) and acoustic (neural codec) tokens; VALL-E uses only acoustic codec tokens combined with phoneme-level ASR transcriptions, achieving better content fidelity.
Contributions and Novelty
- First LM-based TTS with in-context learning: VALL-E is the first TTS framework to demonstrate GPT-3-style in-context learning, enabling zero-shot voice cloning purely through prompting without fine-tuning or speaker encoder engineering.
- Discrete codec codes as TTS intermediate representation: Replacing mel spectrograms with RVQ audio codec codes enables language model training, is robust to noisy large-scale data, and allows diverse outputs through sampling.
- Unprecedented training scale for TTS: 60,000 hours of semi-supervised data — over 100× more than any prior TTS system — collected via ASR transcription of LibriLight, demonstrating that scaling semi-supervised TTS data has been systematically underestimated.
- Hierarchical AR + NAR model design: The AR model provides flexible length prediction and speaker-faithful first-codebook tokens; the NAR model efficiently generates fine acoustic detail codes for the remaining 7 codebooks, reducing inference complexity from $\mathcal{O}(T)$ to $\mathcal{O}(1)$ for those stages.
- Emergent capabilities: Acoustic environment preservation, zero-shot emotion transfer, and diverse output generation arise naturally from the training approach rather than requiring dedicated modules.
Limitations and Future Work
Synthesis Robustness
The AR language model sometimes produces word deletions, insertions, or duplications, a known failure mode of attention-based autoregressive TTS arising from disordered attention alignments. Non-autoregressive or constrained attention mechanisms could address this in future work.
Data Coverage
Despite 60K hours of training data, LibriLight is predominantly audiobook-style read speech, leaving gaps in coverage of accented speakers, spontaneous conversational speech, and diverse speaking styles. The relatively weaker results on VCTK (which contains many accent speakers) compared to LibriSpeech reflect this. Further scaling of training data to cover more prosody styles, speaking conditions, and accents is identified as a key path forward.
Model Structure
Two separate models (AR + NAR) are used for different codebook stages. A unified large model predicting all codes, or a fully non-autoregressive model for improved inference speed, are promising future directions. The authors believe that sufficient model and data scaling could nearly solve the zero-shot TTS problem.
Broader Impacts
Because VALL-E can synthesize speech that closely matches a target speaker's identity from just 3 seconds of audio, it carries potential risks of misuse for voice spoofing or speaker impersonation. The authors note that detection models discriminating synthesized from real audio, and adherence to responsible AI principles (specifically Microsoft AI Principles), are important mitigations to develop alongside the technology.
Code & Implementation
Repository status: This repository is a placeholder. The VALL-E directory contains only a README noting "Future updates will be posted here" and a link to the arXiv preprint (v1, January 2023).
Paper method: The paper introduces VALL-E, a neural codec language model for zero-shot text-to-speech synthesis. The approach uses discrete codes from an off-the-shelf neural audio codec and frames TTS as conditional language modeling. The model is pre-trained on 60K hours of English speech and exhibits in-context learning, synthesizing personalized speech from a 3-second acoustic prompt of an unseen speaker.
Code availability: At the time of this repository snapshot, the implementation code has not been released. The README indicates that code and updated resources will be posted in the future. A demo is available at https://aka.ms/valle.