F5-TTS

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

F5-TTS is a non-autoregressive text-to-speech system using flow matching and a Diffusion Transformer. It removes complex alignment steps and adds Sway Sampling at inference for better speech quality and speed. It delivers robust multilingual speech synthesis with code-switching and speed control.

• tts
• multimodal
• voice-cloning
• prosody

Authors: Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen

Categories: eess.AS, cs.SD

Comment: 17 pages, 9 tables, 3 figures

Published 2024-10-09 · Updated 2025-05-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub14,848 stars129 watchers2,170 forksMIT
• Hugging Face748,625 downloads1,183 likes

Impact

• Semantic Scholar466 citations87 influential citations

Abstract

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.

---

1. Problem Setting and High-Level Contribution

This paper studies fully non-autoregressive text-to-speech (TTS) with a flow-matching backbone and argues that strong zero-shot speech synthesis can be obtained without the usual pipeline of phoneme alignment, duration prediction, and a separate text encoder. The proposed system, F5-TTS, keeps the overall interface simple: text in, speech out. The model is trained on a text-guided speech-infilling objective, and at inference it generates speech from noise by integrating a learned vector field with an ordinary differential equation (ODE) solver.

The paper’s main technical claims are: (1) a ConvNeXt V2 text refinement branch helps the model align padded character sequences with acoustic features more robustly than the earlier E2 TTS design; (2) a test-time Sway Sampling rule over flow steps improves both quality and efficiency without retraining; and (3) these choices lead to substantially better robustness than a reproduced E2 TTS baseline, while preserving the simplicity of non-autoregressive flow-matching TTS.

2. Background: Flow Matching and Guided Generation

F5-TTS is built on flow matching, where the model learns a time-dependent vector field that transforms an easy base distribution, typically Gaussian noise, into data. The paper writes the unconditional flow matching objective as

$$\mathcal{L}_{\mathrm{FM}}(\theta)=\mathbb{E}_{t,\,x\sim p_t(x)}\left\|v_t(x)-u_t(x)\right\|^2,$$

and then uses the conditional form for training. With a conditional path $p_t(x\mid x_1)$ and a flow map $\psi_t(x_0)=\sigma_t(x_1)x_0+\mu_t(x_1)$, the conditional flow matching loss becomes

$$\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}_{t,\,q(x_1),\,p(x_0)}\left\|v_t(\psi_t(x_0)) - \frac{d}{dt}\psi_t(x_0)\right\|^2.$$

With the optimal transport path $\psi_t(x_0)=(1-t)x_0+t x_1$, the loss simplifies to

$$\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}_{t,\,q(x_1),\,p(x_0)}\left\|v_t((1-t)x_0+t x_1)-(x_1-x_0)\right\|^2.$$

The paper notes the connection between this formulation and v-prediction under a cosine schedule. At inference, the learned vector field is integrated from noise $x_0$ to a generated mel spectrogram $x_1$ with an ODE solver, and the number of function evaluations (NFE) controls the speed-quality trade-off.

F5-TTS also uses classifier-free guidance (CFG). In the paper’s formulation,

$$v_{t,\mathrm{CFG}} = v_t(\psi_t(x_0),c) + \alpha\bigl(v_t(\psi_t(x_0),c)-v_t(\psi_t(x_0))\bigr),$$

where $\alpha$ is the guidance strength. The paper explicitly notes that CFG doubles inference cost because the model is run once with the condition and once without it.

3. F5-TTS Method: Input Representation, Architecture, and Sampling

3.1 Text-guided speech infilling training

The training task is framed as speech infilling: given surrounding audio and the full transcript, the model predicts a masked speech segment. The acoustic input is a mel spectrogram $x_1\in\mathbb{R}^{F\times N}$, and a binary mask $m\in\{0,1\}^{F\times N}$ indicates which part of the spectrogram is hidden. The model sees the noisy interpolation $(1-t)x_0+t x_1$ together with the visible speech $(1-m)\odot x_1$, and is trained to reconstruct the masked region $m\odot x_1$.

Instead of an explicit duration model, the text is converted to a character sequence and padded with filler tokens $\langle F\rangle$ to match the mel length. If the transcript has characters $c_1,\dots,c_M$, the extended sequence is

$$z=(c_1,c_2,\ldots,c_M,\underbrace{\langle F\rangle,\ldots,\langle F\rangle}_{(N-M)\text{ times}}).$$

This is the same simplifying idea used by E2 TTS, but F5-TTS argues that the earlier design entangles semantics and acoustics too tightly and is therefore hard to optimize robustly. F5-TTS instead gives text its own refinement path before concatenation with the speech features.

3.2 Backbone architecture

F5-TTS uses a Diffusion Transformer (DiT) backbone with zero-initialized adaptive LayerNorm, or adaLN-zero, as the main denoiser. To improve alignment, the padded character sequence is first processed by ConvNeXt V2 blocks. The paper’s key design choice is to let the text representation evolve in a separate modeling space before it is concatenated with the acoustic stream. The authors explicitly contrast this with E2 TTS, which concatenates text and speech directly and thereby mixes the two modalities too early.

Several positional and conditioning choices are also described. The flow step $t$ is embedded as a sinusoidal conditioning signal and injected through adaLN-zero rather than being appended to the token sequence. The text side receives an absolute sinusoidal embedding before the ConvNeXt blocks, while the concatenated input sequence uses convolutional positional embedding and rotary positional embeddings (RoPE) for self-attention. The paper also states that an extra mean-pooled text token used in some other systems was not essential here.

The authors note that they abandoned the U-Net-style skip-connection design used in E2 TTS and switched to a DiT-based structure. In their ablations, this simpler and more modern transformer form is faster to train and infer, but only becomes reliable for TTS after the text refinement branch is added.

3.3 Sway Sampling at inference time

A central contribution is Sway Sampling, an inference-time rule that changes how flow steps are sampled. The paper keeps training uniform, $t\sim\mathcal{U}[0,1]$, but uses a non-uniform schedule during inference:

$$f_{\mathrm{sway}}(u;s)=u+s\cdot\bigl(\cos(\frac{\pi}{2}u)-1+u\bigr),$$

where $u\sim\mathcal{U}[0,1]$ and $s$ is a monotonic control parameter in the range $[-1,\frac{2}{\pi-2}]$. Negative $s$ biases the solver toward earlier flow steps, while $s=0$ reduces to ordinary uniform sampling. The paper’s intuition is that early steps sketch the coarse structure of speech, including alignment and speaker identity, so spending more solver budget there can improve faithfulness and robustness.

Importantly, Sway Sampling is presented as a drop-in inference-only change that can be applied to existing flow-matching TTS systems without retraining.

4. Training and Inference Setup

The base models are trained on the public multilingual in-the-wild dataset Emilia. After filtering transcription failures and misclassified-language samples, the paper retains approximately 95K hours of English and Chinese data. For architecture ablations, the authors also train small models on WenetSpeech4TTS Premium, a 945-hour Mandarin corpus.

The main evaluation sets are LibriSpeech-PC test-clean, Seed-TTS test-en (1088 samples from Common Voice), and Seed-TTS test-zh (2020 samples from DiDiSpeech). The paper also constructs and releases a 4-to-10-second LibriSpeech-PC subset with 1127 samples for fairer comparisons, because many earlier English-only results were reported on different hidden subsets of LibriSpeech test-clean.

The implementation details reported by the paper are:

• Model size: 335.8M parameters for the F5-TTS base model.
• DiT: 22 layers, 16 attention heads, embedding/FFN dimensions 1024/2048.
• ConvNeXt V2: 4 layers, embedding/FFN dimensions 512/1024.
• Input text vocabulary: 2546 symbols including the filler token and all characters present in Emilia, which supports code-switching.
• Text processing: alphabets and symbols for English; jieba and pypinyin to convert Chinese into full pinyin.
• Acoustic features: 100-dimensional log mel filterbanks at 24 kHz with hop length 256.
• Masking: a random 70% to 100% of mel frames is hidden for infilling training.
• CFG dropout during training: first drop masked speech with probability 0.3, then drop masked speech together with text with probability 0.2.
• Optimizer: AdamW with peak learning rate $7.5\times10^{-5}$, 20K warmup updates, linear decay, and gradient norm clipping at 1.
• Training budget: 1.2M updates, batch size 307,200 audio frames, over one week on 8 NVIDIA A100 80G GPUs.
• Inference: EMA weights, Euler ODE solver for F5-TTS, and Vocos as the default vocoder.

The evaluation protocol uses objective and subjective measures. The paper reports word error rate (WER) and speaker similarity (SIM-o) for objective quality, using Whisper-large-v3 for English ASR, Paraformer-zh for Chinese ASR, and a WavLM-large-based speaker verifier for similarity. Subjectively, it reports CMOS and SMOS; the appendix explains that 20 native judges participated, with CMOS on a $-3$ to $+3$ differential scale and SMOS on a 1 to 5 scale in 0.5 increments.

5. Main Results

In the paper’s main comparisons, F5-TTS is evaluated under the cross-sentence zero-shot setting: given a speech prompt and transcript, it synthesizes a new utterance conditioned on the target text while retaining the prompt speaker. The default reported setting is CFG strength 2 and Sway Sampling coefficient $s=-1$.

5.1 LibriSpeech-PC test-clean

The paper’s main takeaway on this benchmark is that F5-TTS substantially improves inference speed over the reproduced E2 TTS baseline while reducing WER. At 16 NFE, it reaches an RTF of 0.15, and at 32 NFE it improves WER to 2.42. The authors emphasize that this is achieved with a relatively compact single-stage pipeline compared with heavier diffusion or multi-stage systems.

5.2 Seed-TTS test-en and test-zh

On both Seed-TTS test sets, F5-TTS improves over the reproduced E2 TTS in WER and subjective ratings, while keeping comparable speaker similarity. The strongest F5-TTS setting reported in the main paper is the 32-NFE run with CFG strength 2 and Sway Sampling coefficient $s=-1$, which yields WER 1.83 and SMOS 3.89 on test-en, and WER 1.56 and SMOS 3.83 on test-zh.

The authors note that Seed-TTS’s best reported results are from a much larger-scale system trained on orders of magnitude more data, so the comparison is not a direct apples-to-apples scaling claim. The point of F5-TTS is that it attains strong zero-shot quality and robustness with a much simpler and smaller training setup.

6. Ablation Studies and Diagnostic Experiments

6.1 Architecture ablations: why ConvNeXt text refinement matters

The paper’s architectural ablations are especially important because they isolate the effect of text refinement and backbone structure. These experiments use approximately 155M-parameter small models trained on WenetSpeech4TTS Premium for 800K updates. The key result is that F5-TTS with ConvNeXt text refinement converges better and is far more robust than the reproduced E2 TTS baseline.

The paper reports the following Seed-TTS test-zh results for the small-model study:

The important patterns are:

• F5-TTS clearly outperforms the reproduced E2 TTS in robustness and WER on the common-input setting.
• Adding a text branch to E2 TTS as a simple plug-in does not fix its weakness; the model becomes even worse in the reported setting.
• Adding an extra speech branch to F5-TTS slightly increases speaker similarity but hurts alignment robustness, which suggests that text-side refinement is the more useful direction.
• Long skip connections are not a universal improvement; they do not rescue the flat U-Net Transformer baseline and only marginally help some settings.

The paper also reports that a pure adaLN DiT without the Conv2Text branch failed to learn alignment reliably on padded character sequences, while MMDiT learned quickly but collapsed quickly, producing repeated utterances and unstable timbre/prosody. The authors interpret this as evidence that TTS needs a model that is flexible enough for alignment but not so flexible that it can drift into degenerate repetition.

6.2 Sway Sampling improves quality and can rescue the baseline

The paper devotes a separate ablation to Sway Sampling and shows that it improves WER, speaker similarity, and naturalness for both F5-TTS and the reproduced E2 TTS. The effect is consistent across LibriSpeech-PC, Seed-TTS test-en, and Seed-TTS test-zh. A representative set of results is:

The authors provide an intuitive "leak and override" experiment. They partially inject ground-truth information into the initial noise state and then ask the model to continue generation under a mismatched text prompt. With uniform flow-step sampling, the leaked information dominates; with Sway Sampling, the model can override the leak and follow the new prompt. This is used as evidence that the earliest flow steps are especially important for coarse alignment and identity selection.

The paper further observes that more negative $s$ values generally help, and uses $s=-1$ as the default. The appendix also compares ODE solvers and concludes that Euler is the fastest choice and usually performs best when paired with Sway Sampling, whereas midpoint and Heun-style solvers are somewhat slower.

6.3 Vocoders and data-scale stability

Two further appendix studies are worth noting. First, the authors compare Vocos and BigVGAN as vocoders. BigVGAN often slightly improves WER and SIM, but the main paper uses Vocos as the default vocoder. Second, they train 158M small models on LibriTTS 585 hours and LJSpeech 24 hours to study stability across dataset scales. The reported trend is that F5-TTS remains trainable and reasonably stable even with limited data, whereas the paper uses this section to argue that the architecture is less brittle than the reproduced E2 TTS baseline.

6.4 Hard-sentence robustness

On the ELLA-V hard-sentence set, F5-TTS again outperforms the reproduced E2 TTS. The paper reports WER 4.40 for F5-TTS versus 8.58 for E2 TTS, with lower substitution and deletion errors, and a very low insertion rate, suggesting that the model is not prone to endless repetition. The authors do note that the higher WER on this benchmark is partly due to difficult pronunciations and ASR ambiguity.

7. What the Paper Concludes About the Design Space

The strongest design lesson in the paper is that removing explicit phoneme alignment is not enough by itself. E2 TTS showed that a padded-text, speech-infilling formulation can work, but its direct concatenation of text and speech features is fragile. F5-TTS keeps the same broad simplification but inserts an intermediate text refinement module, which improves alignment learning without reverting to a more rigid duration-prediction pipeline.

The second lesson is that sampling matters. The model is trained with uniform flow-step sampling, but inference can be improved substantially by redistributing computation toward early steps with Sway Sampling. The paper presents this as a generally useful trick for flow-matching TTS, not a model-specific heuristic.

The third lesson is that robust zero-shot TTS is still constrained by representation length. The paper deliberately keeps the mel-spectrogram representation, but the authors acknowledge that text-to-mel generation remains sequence-heavy and therefore not fully efficient.

8. Limitations and Stated Risks

The paper lists two main limitations. First, mel spectrograms are much longer than text, so even though F5-TTS is more efficient than prior diffusion-style systems, the sequence length remains a major bottleneck. The authors view more efficient continuous representations as an important future direction. Second, while the model can deeply mimic the reference voice and generate expressive zero-shot speech, it lacks fine-grained control over paralinguistic attributes such as emotion.

The ethics statement also warns about misuse: because the model can synthesize highly natural speech with speaker similarity, it could be abused for spoofing or impersonation. The paper explicitly calls for watermarking and detection methods to identify synthetic audio.

9. Bottom Line

F5-TTS is a technically clean non-autoregressive TTS system that combines flow matching, a DiT backbone, ConvNeXt-based text refinement, and a test-time Sway Sampling strategy. Its practical value is that it delivers competitive zero-shot speech quality with substantially improved robustness over the reproduced E2 TTS baseline, while preserving a simple pipeline that avoids phoneme duration models and explicit alignment machinery. The paper’s most useful engineering takeaway is that alignment robustness can be improved by giving text a dedicated refinement stage and by biasing inference toward early flow steps.