E2 TTS

E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS

E2 TTS is a non-autoregressive zero-shot text-to-speech system that models TTS as speech-infilling on mel spectrograms with characters plus filler tokens. It achieves top-level speaker similarity and intelligibility without grapheme-to-phoneme conversion, duration models, or alignment complexity.

• tts
• voice-cloning
• speech-to-speech

Demos

The demos showcase E2 TTS, a fully non-autoregressive zero-shot text-to-speech system that synthesizes highly natural speech from any speaker voice prompt with state-of-the-art speaker similarity and intelligibility. Key evaluation points include voice naturalness, accurate emotional expressiveness, flexible speech rate control, and the ability to specify word pronunciations without model retraining—confirming the paper's claims of simplicity and performance. Listen for natural prosody, consistent speaker identity, and clear articulation across diverse contexts.

Authors: Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, Naoyuki Kanda

Categories: eess.AS, cs.SD

Comment: Accepted to SLT 2024. Added evaluation data, see https://github.com/microsoft/e2tts-test-suite for more details

Published 2024-06-26 · Updated 2024-09-12

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub32 stars3 watchers1 forksMIT
• Hugging Face108,011 downloads57 likes

Abstract

This paper introduces Embarrassingly Easy Text-to-Speech (E2 TTS), a fully non-autoregressive zero-shot text-to-speech system that offers human-level naturalness and state-of-the-art speaker similarity and intelligibility. In the E2 TTS framework, the text input is converted into a character sequence with filler tokens. The flow-matching-based mel spectrogram generator is then trained based on the audio infilling task. Unlike many previous works, it does not require additional components (e.g., duration model, grapheme-to-phoneme) or complex techniques (e.g., monotonic alignment search). Despite its simplicity, E2 TTS achieves state-of-the-art zero-shot TTS capabilities that are comparable to or surpass previous works, including Voicebox and NaturalSpeech 3. The simplicity of E2 TTS also allows for flexibility in the input representation. We propose several variants of E2 TTS to improve usability during inference. See https://aka.ms/e2tts/ for demo samples.

---

Introduction

E2 TTS is a fully non-autoregressive zero-shot text-to-speech system designed to be unusually simple while still reaching strong speaker similarity, intelligibility, and naturalness. The central claim of the paper is that a high-quality zero-shot TTS model does not need an explicit grapheme-to-phoneme pipeline, a duration predictor, or complicated alignment machinery such as monotonic alignment search. Instead, the model treats TTS as a conditional speech-infilling problem over mel spectrograms and uses a flow-matching generator conditioned on a character sequence augmented with filler tokens.

The paper positions E2 TTS against two broad families of prior zero-shot TTS systems:

• Autoregressive codec-language-model approaches such as VALL-E, which generate codec tokens sequentially and therefore incur latency and require careful tokenizer choices.
• Fully non-autoregressive approaches such as Voicebox, NaturalSpeech 2/3, Matcha-TTS, and E3 TTS, which avoid sequential decoding but usually require some form of text-audio alignment, duration modeling, or carefully engineered architecture.

E2 TTS argues that the alignment problem can be sidestepped by directly conditioning on a character sequence padded to the mel-frame length with filler tokens. This makes the model “embarrassingly easy” in the sense that the architecture is compact, the training objective is standard conditional flow matching, and the input representation is flexible enough to support practical inference variants.

Core idea and relation to prior work

The paper’s key insight is that the generator itself can implicitly learn what earlier systems factorized into separate components: text normalization, grapheme-to-phoneme conversion, alignment, duration estimation, and acoustic generation. E2 TTS is especially close to Voicebox architecturally, but it replaces Voicebox’s frame-wise phoneme conditioning with a character sequence plus a filler token $? wait invalid.

From the paper’s perspective, the generator can be seen as a joint model of the grapheme-to-phoneme converter, the duration model, and the audio model. The authors report that this joint modeling improves naturalness while preserving strong similarity and intelligibility.

Method

Training formulation

Given a training utterance $s$ with transcription $y=(c_1,c_2,? no, must avoid control chars.

Given a training utterance $s$ with transcription $y=(c_1,c_2,\ldots,c_M)$, the model extracts mel-filterbank features $\hat{s}\in\mathbb{R}^{D\times T}$, where $D$ is the feature dimension and $T$ is the number of frames. The transcription is converted into an extended character sequence $\hat{y}$ by appending the special filler token $\langle F\rangle$ until the length matches $T$:

$$ \hat{y} = (c_1, c_2, \ldots, c_M, \underbrace{\langle F\rangle, \ldots, \langle F\rangle}_{T-M\ \text{times}}). $$

The generator is trained on a speech-infilling objective. A binary mask $m\in\{0,1\}^{D\times T}$ selects which spectrogram entries are hidden, and the model learns the conditional distribution

$$ P(m\odot \hat{s} \mid (1-m)\odot \hat{s}, \hat{y}). $$

The learning rule uses conditional flow matching. This places E2 TTS in the class of flow-based generators rather than diffusion models or autoregressive token decoders.

The paper gives the conditional flow-matching loss as

$$ \mathcal{L}^{\mathrm{CFM}}(\theta)=\mathbb{E}_{t,q(x_1),p_t(x\mid x_1)}\left\|u_t(x\mid x_1)-v_t(x;\theta)\right\|^2, $$

where $v_t(x;\theta)$ is the learned time-dependent vector field, $x_1$ is a sample from the data distribution, and $p_t$ is an optimal-transport probability path. The path is instantiated as

$$ p_t(x\mid x_1)=\mathcal{N}\bigl(x\mid t x_1,\,(1-(1-\sigma_{\min})t)^2I\bigr), $$

with vector field

$$ u_t(x\mid x_1)=\frac{x_1-(1-\sigma_{\min})x}{1-(1-\sigma_{\min})t}. $$

At inference time, an ordinary differential equation solver generates the mel spectrogram from the initial distribution.

Architecture

E2 TTS uses the same overall spectrogram-generator architecture as the audio model of Voicebox, except that the conditioning sequence is a character sequence with filler tokens instead of a frame-wise phoneme sequence. The backbone is a vanilla Transformer with U-Net-style skip connections. The model input includes:

• the masked observed mel spectrogram $(1-m)\odot\hat{s}$,
• the character embedding sequence $\tilde{y}\in\mathbb{R}^{E\times T}$,
• the flow time step $t$, and
• noisy speech $s_t$.

These are stacked into a tensor of shape $(2D+E)\times T$, passed through a linear layer to produce a $D\times T$ tensor, and then augmented with a time embedding $\hat{t}\in\mathbb{R}^D$ appended as an extra token, yielding an input of shape $\mathbb{R}^{D\times(T+1)}$ to the Transformer. The Transformer outputs the vector field $v_t$ used in conditional flow matching.

The vocoder is a BigVGAN-based waveform generator. The paper models 100-dimensional log mel-filterbank features extracted every 10.7 ms from 24 kHz audio.

Inference procedure

For zero-shot synthesis, the model receives an audio prompt $s^{\mathrm{aud}}$ with transcription $y^{\mathrm{aud}}=(c'_1,\ldots,c'_{M^{\mathrm{aud}}})$ and a text prompt $y^{\mathrm{text}}=(c''_1,\ldots,c''_{M^{\mathrm{text}}})$. The user also specifies a target duration, internally represented as a number of frames $T^{\mathrm{gen}}$.

The prompt mel features are extracted as $\hat{s}^{\mathrm{aud}}\in\mathbb{R}^{D\times T^{\mathrm{aud}}}$. The extended input sequence concatenates the prompt transcript, the target transcript, and filler tokens:

$$ \hat{y}'=(c'_1,c'_2,\ldots,c'_{M^{\mathrm{aud}}},c''_1,c''_2,\ldots,c''_{M^{\mathrm{text}}},\underbrace{\langle F\rangle,\ldots,\langle F\rangle}_{\mathcal{T}\ \text{times}}), $$

where $\mathcal{T}=T^{\mathrm{aud}}+T^{\mathrm{gen}}-M^{\mathrm{aud}}-M^{\mathrm{text}}$ ensures that the final length matches $T^{\mathrm{aud}}+T^{\mathrm{gen}}$. The generator then infills the generated portion by modeling

$$ P(\tilde{s}\mid [\hat{s}^{\mathrm{aud}};z^{\mathrm{gen}}],\hat{y}'), $$

where $z^{\mathrm{gen}}$ is an all-zero matrix of shape $D\times T^{\mathrm{gen}}$. The output mel spectrogram is converted to waveform by the vocoder.

Why the formulation is simpler than many baselines

The paper emphasizes three simplifications relative to prior zero-shot TTS systems:

• No grapheme-to-phoneme converter is required. The model can operate directly on characters, or even Unicode bytes as an alternative representation mentioned in the paper.
• No explicit duration model is required. Unlike Voicebox-style systems that rely on phoneme/frame alignment or a separate duration predictor, E2 TTS learns the necessary timing implicitly through the flow-matching generator.
• No monotonic alignment search is required. The model avoids the alignment machinery used by some flow-based TTS methods.

Extensions for practical inference

Extension 1: E2 TTS X1

The basic E2 TTS inference requires a transcription of the audio prompt $y^{\mathrm{aud}}$. To remove that requirement, the paper proposes E2 TTS X1. In X1, the inference sequence excludes the audio-prompt transcript and uses only the text prompt plus filler tokens:

$$ \hat{y}'=(c''_1,c''_2,\ldots,c''_{M^{\mathrm{text}}},\underbrace{\langle F\rangle,\ldots,\langle F\rangle}_{\mathcal{T}\ \text{times}}). $$

The paper notes that the masked-region transcription used during training can be obtained by automatic speech recognition, or more efficiently by forced alignment; in the experiments, the authors used the Montreal Forced Aligner to locate word boundaries so that masked regions would not cut words in the middle.

Extension 2: E2 TTS X2

E2 TTS X2 addresses pronunciation control for specific words or foreign names. During training, a word in $y$ is sometimes replaced by its phoneme sequence enclosed in parentheses; in the implementation, the replacement is performed with 15% probability using phoneme entries from the CMU pronouncing dictionary. At inference, the user can similarly replace a target word with a phoneme sequence in parentheses.

The important design point is that the sequence is still treated as a simple character stream. Whether a token sequence should be read as a word or as phonemes is determined only by the parentheses and their contents. The paper also notes that punctuation adjacent to the replaced word is retained, which preserves useful context for the model.

Training setup

The authors train on Libriheavy, a 50,000-hour read-English dataset with 6,736 speakers and case/punctuation-preserving transcriptions. They also evaluate scaling with a proprietary 200,000-hour unlabeled dataset used for initialization and with a 200,000-hour proprietary labeled training set for one E2 TTS configuration.

The main model configuration is:

• Transformer backbone with U-Net-style skip connections,
• 24 layers,
• 16 attention heads,
• embedding dimension 1024,
• feed-forward dimension 4096,
• character vocabulary size 399,
• about 335 million parameters.

Training details reported in the paper include:

• masking length sampled uniformly between 70% and 100% of the mel-frame length,
• classifier-free guidance conditioning dropout with 20% probability,
• 800,000 mini-batch updates,
• effective batch size of 307,200 audio frames,
• linear learning-rate decay with peak learning rate $7.5\times 10^{-5}$,
• 20,000-step warmup,
• removal of training samples longer than 4,000 frames.

The paper also reports a pretraining-based initialization, following an unsupervised scheme on 200,000 hours of anonymized data for 800,000 updates. This pretraining is used as an initialization option for some E2 TTS runs and for a fair comparison with a Voicebox baseline.

For the comparison with Voicebox, the authors train a separate regression duration model following the Voicebox setup. That duration model uses a Transformer with 8 layers, 8 attention heads, embedding dimension 512, feed-forward dimension 2048, and is trained for 75,000 updates with 120,000 frames.

Evaluation protocol

Evaluation is carried out on the LibriSpeech-PC test-clean subset, which contains case- and punctuation-preserving transcripts. After filtering to utterances between 4 and 10 seconds, the test set contains 1,132 samples from 39 speakers. For each test sample, the audio prompt is the last 3 seconds of a randomly sampled utterance from the same speaker.

The objective metrics are:

• WER, measured using a HuBERT-large ASR model, as an intelligibility metric;
• SIM-o, the cosine similarity between speaker embeddings extracted by a WavLM-large speaker-verification model, as a speaker-similarity metric.

The subjective evaluation consists of:

• CMOS (comparative mean opinion score) for naturalness, rated on a 7-point scale from $-3$ to $3$ against ground truth;
• SMOS (speaker similarity mean opinion score), rated on a 1-to-5 similarity scale.

Each subjective test uses 39 samples, one per speaker, with 12 native English evaluators per sample. The paper averages objective metrics over three random seeds. During inference, both E2 TTS and Voicebox use classifier-free guidance strength 1.0 and a midpoint ODE solver with 32 function evaluations.

Main results

The paper compares E2 TTS against VALL-E, NaturalSpeech 3, and Voicebox. The main quantitative finding is that E2 TTS matches or exceeds the strongest baselines on both intelligibility and speaker similarity, despite the much simpler conditioning design.

The authors highlight several comparisons:

• Compared with their Voicebox reproduction on Libriheavy, E2 TTS improves WER and SIM-o.
• The pretrained E2 TTS configuration achieves the best reported objective scores, with WER 1.9% and SIM-o 0.708.
• Scaling to 200,000 hours preserves the best WER and nearly the best speaker similarity, showing that the simple architecture scales well.
• All E2 TTS variants outperform strong prior systems such as VALL-E and NaturalSpeech 3, and are competitive with or better than Voicebox.

The subjective findings are important because they sharpen the paper’s main claim: the improvement from character-level conditioning is not only visible in objective WER, but also in perceived naturalness. The pretrained E2 TTS variant reaches CMOS close to zero, which the paper interprets as naturalness indistinguishable from ground truth in the cited evaluation convention. At the same time, SMOS remains very close to the stronger baselines.

Extension results

E2 TTS X1: removing the prompt transcript requirement

X1 is intended for settings where the transcript of the audio prompt is unavailable or inconvenient to produce. The paper shows that X1 has nearly the same performance as the base model, especially with pretrained initialization, while eliminating the prompt-transcript dependency at inference.

The paper’s interpretation is that X1 changes usability substantially with only a minor performance cost, if any. In practice, this is a strong engineering advantage because the prompt transcription is often a bottleneck in zero-shot synthesis workflows.

E2 TTS X2: pronunciation control via inline phonemes

X2 is designed for explicit pronunciation control. The model is trained with 15% of words replaced by phoneme sequences in parentheses, and the test-time results show that this augmentation does not harm the baseline case and remains robust even when half of the words are replaced.

The authors interpret these results as evidence that E2 TTS can accept pronunciation hints for new or difficult terms without retraining, while remaining stable under substantial phoneme substitution at inference.

Behavioral analysis

Training progress

The paper examines training curves for Voicebox and E2 TTS from scratch and with pretrained initialization. The main lesson is that Voicebox converges faster early in training on WER because its frame-wise phoneme alignment provides a strong inductive bias. E2 TTS, by contrast, takes more iterations to settle, but eventually reaches better WER and better SIM-o.

This is one of the paper’s most important empirical claims: eliminating the explicit phoneme alignment does not merely simplify the system; it may also improve end quality once the model has enough data and optimization time.

Effect of audio prompt length

The authors vary the length of the audio prompt and observe two trends. First, WER does not show a clear dependence on prompt length, suggesting that E2 TTS can handle prompts as long as 10 seconds without obvious degradation. Second, SIM-o improves as the prompt becomes longer, which is consistent with longer prompts providing more speaker evidence.

This is relevant for practical deployment because the prompt length is often variable in conversational or voice-cloning settings.

Effect of changing speech rate

The paper also evaluates robustness to changes in the total duration input by scaling it by $1/sr$, where $sr$ is a speech-rate factor in the range 0.7 to 1.3. The reported behavior is that E2 TTS keeps high speaker similarity and only moderately worsens WER under faster or slower speech settings, indicating that the duration control is flexible rather than brittle.

Interpretation and limitations

The paper’s main conclusion is that explicit alignment machinery is not necessary for strong zero-shot TTS quality. In the reported experiments, the character-based, filler-token formulation is enough to reach or beat prior fully non-autoregressive systems, especially after large-scale training or unsupervised initialization.

At the same time, the paper makes clear that the basic E2 TTS inference still requires a transcription of the audio prompt, which can be inconvenient in some applications. This is exactly why X1 is introduced. Likewise, the model’s inference needs a target duration $T^{\mathrm{gen}}$, so practical use still requires some notion of desired output length or speech rate. The analysis shows that the system is relatively robust to duration variation, but duration remains an explicit control input rather than something the model infers on its own.

Another practical consideration is that the generator must infer the boundary between the prompt transcript and the target transcript inside the extended sequence. The paper notes that this boundary detection can be challenging when the prompt is long, although the empirical results do not show a clear WER degradation up to roughly 10-second prompts.

Conclusion

E2 TTS demonstrates that a very simple fully non-autoregressive design can achieve state-of-the-art zero-shot TTS performance. The system replaces complicated text and alignment machinery with a character sequence padded by filler tokens and trains a flow-matching spectrogram infiller. Across the reported LibriSpeech-PC evaluation, it achieves the best objective scores among the compared systems and near-human subjective naturalness, while also supporting practical variants for prompt-transcript removal and inline pronunciation control.

The paper’s technical message is therefore twofold: first, the TTS pipeline can be simplified much more aggressively than prior systems assumed; second, this simplification does not necessarily compromise quality, and may even improve final naturalness when trained at scale.