Voicebox

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Voicebox is a large-scale speech generation model that unifies tasks like zero-shot TTS, style transfer, denoising, and editing via text-guided speech infilling. It uses past and future context for flexible, high-quality speech generation from massive unfiltered data without task-specific fine-tuning.

• tts
• voice-conversion
• speech-to-speech
• asr
• multimodal
• few-shot

Demos

The demos of Voicebox showcase its versatile speech generation capabilities including content editing, zero-shot text-to-speech synthesis, cross-lingual style transfer, transient noise removal, and diverse speech sampling. Watch for its ability to flawlessly correct audio content without re-recording, preserve style across languages, and generate high-quality speech efficiently and coherently, all validating its strength as a multilingual, non-autoregressive flow-matching model that excels in in-context learning and fast synthesis.

Authors: Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, Wei-Ning Hsu

Categories: eess.AS, cs.CL, cs.LG, cs.SD

Comment: Accepted to NeurIPS 2023

Published 2023-06-23 · Updated 2023-10-19

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face15 downloads1 likes

Impact

• Semantic Scholar531 citations67 influential citations

Abstract

Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

---

Introduction

Voicebox is presented as a general-purpose speech generation model that brings a large-language-model-style framing to speech: train one model on a broad, text-guided infilling task, and then reuse it for many downstream behaviors through in-context prompting rather than task-specific fine-tuning. The core claim is that speech generation has historically been limited by small-scale, highly curated data and by objectives that assume deterministic mappings from text to audio. Voicebox instead trains on large-scale, in-the-wild audiobook speech and learns to predict masked speech segments conditioned on both surrounding audio and transcript.

The paper’s main message is that text-guided speech infilling is a unifying formulation: by changing which parts of the utterance are masked and what context is provided, the same model can do zero-shot text-to-speech, cross-lingual style transfer, denoising, content editing, alignment-preserved style conversion, and diverse sample generation. Voicebox is a non-autoregressive continuous normalizing flow trained with flow matching, which lets it generate high-quality samples with relatively few function evaluations and condition on both past and future context.

Key contributions reported by the paper

• A single text-guided speech infilling model that supports multiple generation and editing tasks without explicit style labels.
• A flow-matching continuous normalizing flow formulation for speech, trained on more than 50K hours of unfiltered, unenhanced speech.
• State-of-the-art monolingual zero-shot TTS, cross-lingual zero-shot TTS, transient noise removal, and diverse speech sampling, plus an application to synthetic-data ASR training.
• A reproducible metric suite for speech generation, including WER, speaker similarity, Fréchet Speech Distance (FSD), QMOS, SMOS, and duration-specific metrics.

Problem formulation and model overview

Voicebox starts from a transcribed speech pair $(\mathbf{x}, y)$, where $\mathbf{x}$ is the audio and $y$ is the transcript. The model is trained on a masked infilling task: given a binary mask $\mathbf{m}$, the model sees the context audio $\mathbf{x}_{\mathrm{ctx}} = (1-\mathbf{m}) \odot \mathbf{x}$ and predicts the missing portion $\mathbf{x}_{\mathrm{mis}} = \mathbf{m} \odot \mathbf{x}$ conditioned on text and audio context. The paper frames this as learning $p(\mathbf{x}_{\mathrm{mis}} \mid y, \mathbf{x}_{\mathrm{ctx}})$.

To make alignment explicit, the model is decomposed into two pieces:

• An audio model that predicts masked acoustic frames conditioned on frame-level phone labels and audio context.
• A duration model that predicts phone durations, optionally conditioned on surrounding duration context.

The text transcript is forced-aligned to speech, yielding a frame-level phone sequence. Each phone is repeated according to its duration to obtain a frame-level transcript. The paper introduces two practical representation choices that matter for inference and masking:

• Ghost silence: zero-duration silence tokens are inserted at eligible word boundaries to let the duration model predict whether a pause should exist.
• Word-position-dependent phones: phones get suffixes such as beginning, intermediate, end, or singleton so the audio model can better recover word boundaries.

Generative modeling with flow matching

Voicebox uses a continuous normalizing flow (CNF). A flow is defined by an ODE:

$$ \frac{d}{dt}\phi_t(\mathbf{x}) = \mathbf{v}_t(\phi_t(\mathbf{x})); \quad \phi_0(\mathbf{x}) = \mathbf{x}. $$

The model learns a time-dependent vector field $\mathbf{v}_t$ that transports samples from a simple prior distribution to the data distribution. Training uses flow matching, which regresses the model vector field toward the vector field of a chosen probability path.

The paper compares the optimal-transport (OT) path against a diffusion-style path. Voicebox adopts the OT path because it is simpler, trains faster, and is more efficient at inference. The paper writes the flow matching objective in conditional form as a regression problem over the vector field, and then further specializes it to the speech setting with audio and duration losses.

For the audio model, the 80-dimensional log-Mel spectrogram $\mathbf{x} \in \mathbb{R}^{N \times 80}$ is modeled framewise. The input to the Transformer is built from three sequences concatenated per frame: the current noisy sample $\mathbf{x}_t$, the audio context $\mathbf{x}_{\mathrm{ctx}}$, and the embedded frame-level phone labels. A sinusoidal embedding encodes the flow time $t \in [0,1]$ and is appended as an extra token-like input.

The audio-model training loss is:

$$ \mathcal{L}_{\mathrm{audio}}(\theta) = \mathbb{E}_{t,\mathbf{m},q(\mathbf{x},z),p_0(\mathbf{x}_0)} \left\| \mathbf{u}_t(\mathbf{x}_t \mid \mathbf{x}) - \mathbf{v}_t(\mathbf{x}_t, \mathbf{x}_{\mathrm{ctx}}, z; \theta) \right\|^2. $$

The paper also uses a masked version:

$$ \mathcal{L}_{\mathrm{audio,m}}(\theta) = \mathbb{E}_{t,\mathbf{m},q(\mathbf{x},z),p_0(\mathbf{x}_0)} \left\| \mathbf{m} \odot \left( \mathbf{u}_t(\mathbf{x}_t \mid \mathbf{x}) - \mathbf{v}_t(\mathbf{x}_t, \mathbf{x}_{\mathrm{ctx}}, z; \theta) \right) \right\|^2. $$

The duration model is implemented in two variants:

• Duration flow matching, which mirrors the audio model and predicts masked durations from text and duration context.
• Regression, which predicts masked durations directly with an $L_1$ loss and no time embedding.

The regression objective is:

$$ \mathcal{L}_{\mathrm{dur,regr,m}}(\theta) = \mathbb{E}_{\mathbf{m},q(\mathbf{l},y)} \left\| \mathbf{m}' \odot \left( \mathbf{l}_{\mathrm{mis}} - g(\mathbf{l}_{\mathrm{ctx}}, y; \theta) \right) \right\|_1. $$

Classifier-free guidance

Voicebox extends classifier-free guidance to the flow-matching setting. The conditioner is the pair of text and context, and it is randomly dropped during training. At inference, the guided vector field is:

$$ \tilde{\mathbf{v}}_t(\mathbf{w}, \mathbf{x}_{\mathrm{ctx}}, z; \theta) = (1+\alpha)\,\mathbf{v}_t(\mathbf{w}, \mathbf{x}_{\mathrm{ctx}}, z; \theta) - \alpha\,\mathbf{v}_t(\mathbf{w}; \theta), $$

where $\alpha$ controls the trade-off between fidelity and diversity. The paper notes that CFG doubles compute at a fixed number of ODE steps because each guided step requires two forward passes.

How the same model covers multiple tasks

The paper’s central design idea is that all supported tasks are just different ways of forming context and masks:

• Zero-shot TTS: keep a prompt audio segment as context and infill the masked continuation from target text.
• Cross-lingual zero-shot TTS: use prompt speech in one language and target text in another language.
• Alignment-preserved style transfer: replace the voice/style while keeping a target alignment sequence fixed.
• Transient noise removal: mask a noisy segment and regenerate it from surrounding clean context and transcript.
• Content editing: change words in the transcript, keep unchanged phones aligned, and regenerate only the modified spans.
• Diverse speech sampling: mask the entire utterance and sample both durations and audio without a prompt.
• Alignment-preserved style shuffling: keep the phone alignment fixed but resample the audio style.

Metrics and evaluation protocol

The paper argues that subjective MOS alone is not enough for comparing speech generators across papers, so it proposes a reproducible metric suite built from public models:

• WER measures intelligibility and text correctness using ASR on synthesized speech.
• SIM-o measures similarity between synthesized speech and the original audio prompt using a WavLM-TDCNN speaker embedding model.
• SIM-r measures similarity to resynthesized prompt audio, matching the VALL-E evaluation style.
• FSD is a speech analogue of FID computed from wav2vec 2.0 features and is used to measure both diversity and quality at the distribution level.
• QMOS and SMOS are subjective quality and similarity mean-opinion-score studies.

For English-only experiments the paper uses a HuBERT-L ASR model fine-tuned on LibriSpeech. For multilingual experiments it uses Whisper large-v2. FSD is validated in the appendix by controlled sampling experiments: subsets with more speakers produce higher FSD, while subsets with different utterance counts but similar speaker sets stay low. The paper also shows that FSD increases with added noise, supporting its use as a quality metric when diversity is fixed.

Training setup, data, and architecture

Voicebox is trained in two main configurations:

• VB-En: an English-only model trained on 60K hours of ASR-transcribed English audiobooks.
• VB-Multi: a multilingual model trained on 50K hours across six languages: English, French, German, Spanish, Polish, and Portuguese.

For multilingual training, the paper uses an upsampling exponent $\beta = 0.25$ to reduce language imbalance. The English model uses 500K audio-model updates; the multilingual model uses 750K. Duration models are trained for 600K updates.

The architecture is Transformer-based for both audio and duration models, with convolutional positional embeddings and symmetric bi-directional ALiBi bias. The audio model is the larger model: 24 layers, 16 attention heads, model dimension 1024, feed-forward dimension 4096, and roughly 330M parameters. It also includes symmetric skip connections in a U-Net-like pattern, connecting early and late layers. The duration models are smaller: 8 layers for English and 10 for multilingual, with model dimensions 512 and 768 respectively, and around 28M or 34M parameters.

Other important training details reported in the appendix and main text include:

• 80-dimensional log-Mel features at 100 Hz, normalized with global mean $-5.8843$ and standard deviation $2.2615$.
• HiFi-GAN vocoder adapted to generate 16 kHz audio from the Mel features.
• Gradient clipping at 0.2 for the audio models.
• Conditional dropout for classifier-free guidance, with unconditional drop probability $0.2$.
• Chunk length capped at 1600 frames for audio-model training.
• Training uses FP16, Adam with learning rate $10^{-4}$, 5000 warmup steps, and linear decay.

The paper also evaluates a number of design choices for the input representation and vocoder. In particular, it compares Mel spectrograms with HiFi-GAN and Parallel WaveGAN, and also compares against Encodec features and decoder reconstruction.

Main experimental results

The paper evaluates Voicebox on monolingual zero-shot TTS, cross-lingual zero-shot TTS, transient noise removal, diverse speech sampling, and ASR data generation from synthetic speech. The headline result is that Voicebox outperforms prior systems in both fidelity and similarity while also being substantially faster than autoregressive alternatives.

Monolingual zero-shot TTS

On filtered LibriSpeech test-clean, Voicebox significantly beats VALL-E and YourTTS. In the cross-sentence prompt setup, it achieves the best reported WER and the best similarity scores; in the continuation setup it also remains best on both WER and similarity.

The paper also reports that Voicebox is up to 20 times faster than VALL-E. In the efficiency study, it can generate a 10-second sample in about 0.31 seconds at $\mathrm{NFE}=2$ without classifier-free guidance, and remains only about 4% slower than VALL-E at $\mathrm{NFE}=64$.

Cross-lingual zero-shot TTS

Voicebox’s multilingual model, VB-Multi, is tested on filtered MLS sets with prompts from six languages. The paper reports that Voicebox outperforms YourTTS across all evaluated directions. For the English target case, the average over six reference languages improves from WER 7.5 and SIM-o 0.356 for YourTTS to WER 4.4 and SIM-o 0.492 for Voicebox, with QMOS also improving.

The broader table in the paper shows that VB-Multi is consistently strong across all six target languages. The average WERs for VB-Multi are 5.0 (German), 4.4 (English), 3.7 (Spanish), 5.5 (French), 5.5 (Polish), and 5.7 (Portuguese), with SIM-o values in the 0.457 to 0.494 range. The authors highlight that prompt length helps speaker similarity but can hurt intelligibility in some cross-lingual directions, especially when transferring from English to non-English targets, likely because English dominates the multilingual training mix.

Transient noise removal

Voicebox is also evaluated on infilling noisy segments. In the main transient-noise setup, noisy speech overlaps with 50% of the utterance at $-10$ dB SNR. Voicebox outperforms Demucs and A3T on intelligibility, similarity, and quality:

Additional ablations in the appendix show that the model remains strong across overlap ratios of 30%, 50%, and 70%, and both speech and non-speech noise. Even in the hardest conditions, Voicebox maintains very low WER and substantially higher similarity than Demucs.

Diverse speech sampling and ASR data generation

For text-only diverse generation on LibriSpeech test-other text, Voicebox produces realistic and diverse samples with low FSD relative to baseline text-to-speech systems. The regression duration model yields the best WER, while the flow-matching duration model yields more diverse speaking styles.

The most striking downstream demonstration is synthetic-data ASR training. The authors generate 281K utterances per system from LibriSpeech training text and train ASR models only on synthetic data. The ASR system trained on Voicebox-generated speech is dramatically better than those trained on prior TTS systems, and the flow-matching duration model gives the best synthetic ASR data.

The paper emphasizes that these synthetic ASR results correlate strongly with FSD: Voicebox generates speech that is not only intelligible but also sufficiently varied and realistic to train recognizers that nearly close the gap to real-data training.

Ablations and deeper analysis

Flow matching versus diffusion versus score matching

The paper performs a controlled comparison among three generative objectives: flow matching with the OT path, flow matching with a variance-preserving diffusion path, and score matching with the same diffusion path. The conclusion is clear: flow matching with the OT path is both faster to train and faster to sample.

The appendix’s training-step sweep shows that FM with OT already outperforms the alternatives at 50K and 100K updates, and reaches its best results faster. The inference-step sweep shows that it yields useful samples even at 8 NFEs, while the other two methods require many more function evaluations to reach comparable quality.

Masked loss versus loss on all frames

The paper compares masked and unmasked training losses for both flow matching and regression. The main conclusion is that masking the loss focuses learning on the missing regions and improves similarity and diversity without materially harming intelligibility.

Data scaling

Scaling from 60 hours to 60K hours consistently improves zero-shot TTS and WER on diverse generation. The paper reports that zero-shot TTS WER drops from 2.30 to 2.05 and SIM-r rises from 0.151 to 0.645 as training data increases. For diverse sampling, WER drops from 3.48 to 2.95. FSD improves strongly up to 6K hours and is slightly worse at 60K than at 6K, which the authors attribute to the reference distribution used for FSD.

Audio feature choice and vocoder choice

The paper tests Mel spectrograms with HiFi-GAN and Parallel WaveGAN, plus Encodec features with the Encodec decoder. All methods give similar WER on LibriSpeech dev-clean, but similarity differs by encoder and evaluator. Parallel WaveGAN is consistently worst. Encodec looks slightly better under the WavLM-based similarity measure, whereas HiFi-GAN looks better under ECAPA-based similarity. Because no single metric settles the question, the paper keeps Mel + HiFi-GAN as its default speech representation.

Duration-model analysis

The duration appendix introduces standalone metrics for duration correctness, speaking-rate correlation, and Fréchet duration distance. The key finding is that the duration-conditional regression model does best on prediction quality and correlation, while flow matching better preserves the distribution of durations and therefore diversity.

For end-to-end generation, duration-conditional regression gives slightly lower WER than duration flow matching, but the flow-matching duration model is better at producing more varied duration samples. The paper interprets this as a trade-off between regularity and diversity.

Inference efficiency and prompt-length behavior

Voicebox exposes a speed-quality knob through the number of ODE function evaluations and the classifier-free guidance strength. The paper reports that lower guidance and fewer steps increase diversity but can reduce quality; after a small number of steps, higher guidance tends to improve speaker similarity. In the zero-shot TTS setting, WER remains relatively stable around 2.0 across many settings, while similarity and diversity vary more strongly.

Ethical analysis, safety, and limitations

The paper explicitly discusses the risks of high-quality speech generation and trains a binary classifier to separate real from generated audio. In the reported detection study, original audio versus Voicebox-generated audio is trivially separable with 100% accuracy across masking levels, while distinguishing Voicebox-generated audio from resynthesized audio is much harder. For the latter task, the classifier reaches 0.704 accuracy at 30% masking, 0.809 at 50%, and 0.907 at 90%.

The stated limitations are important. First, the model is trained on read audiobook speech in up to six written languages, so it may not transfer cleanly to conversational speech with laughter, back-channeling, and other non-verbal phenomena. Second, Voicebox depends on a phonemizer and a forced aligner; the authors note that current word-based phonemizers may miss context-dependent pronunciation, which matters for languages such as French. Third, while the model transfers style attributes such as voice, speaking style, emotion, and acoustic condition, it does not yet provide independent control over these factors.

Takeaway

Voicebox’s main technical advance is to recast speech generation as large-scale text-guided infilling and to pair that formulation with flow matching and explicit duration modeling. This lets one model act as a speech generalist: it can synthesize speech, edit it, denoise it, convert style, and generate diverse samples, all while remaining fast enough for practical use. Across the reported experiments, the OT flow-matching objective, masked training, and large-scale in-the-wild data are the ingredients that most consistently drive the gains.