Spark-TTS

Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens

Spark-TTS uses a single-stream speech codec to separate linguistic content and speaker attributes, enabling zero-shot voice cloning and fine-grained control with an LLM. It simplifies prior multi-stage pipelines and supports controllable speech synthesis via detailed attribute control.

• llm
• tts
• voice-cloning
• prosody
• autoregressive

Demos

The demos showcase Spark-TTS's advanced zero-shot voice cloning and controlled speech generation capabilities. Viewers should evaluate the naturalness, accuracy, and flexibility of voice replication in various languages, including code-switching scenarios. The control demos highlight how users can adjust speech parameters such as pitch, gender, and speaking rate, demonstrating the model's efficiency and versatility in producing diverse and high-quality synthetic voices with a single LLM-based system.

Authors: Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, Lei Xie, Yike Guo, Wei Xue

Categories: cs.SD, cs.AI, eess.AS

Comment: Submitted to ACL 2025

Published 2025-03-03 · Updated 2025-03-03

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub10,996 stars67 watchers1,163 forksApache-2.0
• Hugging Face722 downloads739 likes

Impact

• Semantic Scholar170 citations18 influential citations

Abstract

Recent advancements in large language models (LLMs) have driven significant progress in zero-shot text-to-speech (TTS) synthesis. However, existing foundation models rely on multi-stage processing or complex architectures for predicting multiple codebooks, limiting efficiency and integration flexibility. To overcome these challenges, we introduce Spark-TTS, a novel system powered by BiCodec, a single-stream speech codec that decomposes speech into two complementary token types: low-bitrate semantic tokens for linguistic content and fixed-length global tokens for speaker attributes. This disentangled representation, combined with the Qwen2.5 LLM and a chain-of-thought (CoT) generation approach, enables both coarse-grained control (e.g., gender, speaking style) and fine-grained adjustments (e.g., precise pitch values, speaking rate). To facilitate research in controllable TTS, we introduce VoxBox, a meticulously curated 100,000-hour dataset with comprehensive attribute annotations. Extensive experiments demonstrate that Spark-TTS not only achieves state-of-the-art zero-shot voice cloning but also generates highly customizable voices that surpass the limitations of reference-based synthesis. Source code, pre-trained models, and audio samples are available at https://github.com/SparkAudio/Spark-TTS.

---

Overview

Spark-TTS is a text-to-speech system that tries to make zero-shot and controllable TTS fit more naturally into the standard large language model (LLM) paradigm. The central design choice is to avoid the multi-stage pipelines and multi-stream code prediction strategies that dominate many current codec-based TTS systems. Instead, Spark-TTS uses a single-stream speech codec called BiCodec and then trains a decoder-only LLM, based on Qwen2.5-0.5B-Instruct, to predict the resulting speech tokens.

The paper’s core claim is that speech can be decomposed into two token types that are complementary rather than entangled: semantic tokens for linguistic content and global tokens for speaker-level attributes. This decomposition makes it possible to preserve the compactness and text-alignment benefits of semantic tokens while also giving the language model a handle on timbre-related control. In the final system, the BiCodec decoder reconstructs waveform audio directly from predicted tokens, so Spark-TTS does not need a separate flow-matching model or vocoder stage after token prediction.

Problem Setting and Main Motivation

The paper starts from three limitations in prior LLM-based TTS systems. First, many systems still require multi-stage processing, such as predicting semantic tokens with one model and acoustic features with another. Second, models that predict multiple codebooks often rely on parallel multi-stream architectures, which are structurally different from standard text LLMs and therefore harder to integrate and scale cleanly. Third, existing zero-shot TTS methods are often strong at voice cloning but weak at controllable voice creation, especially when the user wants a new voice specified by attributes rather than copied from a prompt recording.

Spark-TTS is designed to address these issues with a unified pipeline:

• a single-stream tokenizer that still separates content from speaker-level information;
• an LLM backbone that predicts speech tokens in an autoregressive way similar to text generation;
• a control mechanism that supports both coarse-grained labels and fine-grained numeric values for pitch and speaking rate;
• an open dataset, VoxBox, with attribute annotations to support reproducible research in controllable TTS.

BiCodec: Single-Stream Speech Tokenization with Semantic and Global Tokens

BiCodec is the paper’s most important technical contribution. It is a VQ-VAE-style codec with two tokenizers:

• Semantic Tokenizer: extracts low-bitrate semantic tokens at 50 tokens per second (TPS), intended to preserve linguistic content.
• Global Tokenizer: produces a fixed-length sequence of global tokens, intended to capture time-invariant information such as speaker identity and other global attributes.

The paper presents the codec as follows:

$$ \mathbf{z} = E_s(F(\mathbf{x})), \quad \mathbf{g} = E_g(\operatorname{Mel}(\mathbf{x})), $$

$$ \mathbf{g}_f = \operatorname{CrossAttention}(\mathbf{g}, \mathbf{h}), \quad \mathbf{z}_q = Q_s(\mathbf{z}), \quad \mathbf{g}_q = Q_g(\mathbf{g}_f), \quad \hat{\mathbf{x}} = G(\mathbf{z}_q, A_g(\mathbf{g}_q)). $$

Here, $F(\mathbf{x})$ is a pretrained wav2vec 2.0 feature extractor, $E_s$ and $E_g$ are the semantic and global encoders, $Q_s$ and $Q_g$ are the respective quantizers, $\mathbf{h}$ denotes learnable queries, $A_g$ aggregates the global tokens, and $G$ is the decoder that reconstructs the waveform.

Semantic Tokenizer

The semantic path uses wav2vec 2.0 features rather than raw waveform features directly. The authors state that they average features from the 11th, 14th, and 16th wav2vec 2.0 layers to form the semantic representation, motivated by prior observations that different layers correlate with different linguistic units. The semantic encoder and decoder are fully convolutional networks built from ConvNeXt blocks. The semantic quantizer uses a single-codebook VQ with a codebook size of 8192.

Compared with codecs that directly model acoustic waveforms, this choice is intended to strengthen semantic alignment and improve intelligibility at low bitrates.

Global Tokenizer

The global tokenizer operates on Mel spectrograms. Its encoder is based on ECAPA-TDNN following Wespeaker, and its output is transformed into a fixed-length sequence by a cross-attention mechanism with learnable queries. The global token path uses Finite Scalar Quantization rather than VQ, because the authors want to reduce the risk of VQ collapse for global attributes. In the reported implementation, the global token sequence length is set to 32, and the FSQ module has dimension 6 with 4 levels per dimension, giving a codebook size of 4096.

The paper explicitly argues that this is more expressive than the group-VQ-style approach used by TiCodec-like alternatives, because the learnable-query + FSQ formulation can produce a longer and more flexible global representation.

Codec Training Objective

BiCodec is trained end-to-end with an adversarial objective. The paper combines waveform reconstruction, multi-scale Mel reconstruction, discriminator-based feature matching, codebook and commitment losses for the VQ path, and a wav2vec 2.0 reconstruction loss to preserve semantic relevance. In words, the optimization includes:

• a GAN-style reconstruction objective;
• $L_1$ losses on multi-scale Mel spectrograms;
• multi-period and multi-band multi-scale STFT discriminators;
• VQ codebook and commitment losses;
• an auxiliary wav2vec 2.0 feature reconstruction loss after quantization.

There is also a stabilization trick for the global pathway: early in training, the decoder is not fed the quantized global embedding directly. Instead, the model first uses a teacher-student style update between the pooled pre-quantized global representation and the pooled quantized one. Once training stabilizes, this special treatment is removed.

BiCodec Architecture Details

The appendix gives the following practical settings: the semantic tokenizer uses 12 ConvNeXt blocks and 2 downsampling blocks when the semantic code rate is below 50 TPS; the semantic codebook size is 8192; the ECAPA-TDNN global encoder uses an embedding dimension of 512; and the decoder upsamples with transposed-convolution blocks. For 16 kHz audio, the transposed-convolution upsampling rates are [8, 5, 4, 2]. The paper also reports a 24 kHz variant with upsampling rates [8, 5, 4, 3].

Spark-TTS: LLM-Based Speech Prediction and Control

Spark-TTS keeps the language-model part intentionally close to a standard decoder-only text LLM. The backbone is Qwen2.5-0.5B-Instruct, fine-tuned to predict speech tokens. The key simplification relative to prior systems such as CosyVoice2 is that Spark-TTS does not use flow matching to generate acoustic features; the BiCodec decoder directly reconstructs audio from the predicted tokens.

Tokenization for the Language Model

The LLM uses three token domains:

• Text tokens: standard BPE tokens from the Qwen2.5 tokenizer, supporting multiple languages.
• Attribute tokens: coarse labels and fine-grained values for controllable voice creation.
• Speech tokens: global tokens and semantic tokens from BiCodec.

For attribute control, the paper defines two levels:

• Coarse-grained: categorical labels for gender, pitch level, and speed level.
• Fine-grained: numerical pitch and speed values, rounded to the nearest integer during tokenization.

The labels are discretized into five classes each. Pitch classes are derived from Mel-scale pitch statistics, separately for male and female speakers. Speed classes are derived from syllables per second, after voice activity detection removes silence at the edges.

Training Objectives for the LLM

The language model is trained with next-token prediction. For zero-shot TTS, the model minimizes

$$ \mathcal{L}_{\text{zst}} = -\sum_{t=1}^{T_o} \log p_\theta(o_t \mid \mathcal{T}, \mathcal{G}, o_{

where $\mathcal{T}$ is the tokenized text prompt, $\mathcal{G}$ is the global speech token prompt, and $o_t$ are the semantic tokens to be generated.

For controllable voice creation, the model minimizes

$$ \mathcal{L}_{\text{control}} = -\sum_{t=1}^{T_c} \log p_\theta(c_t \mid \mathcal{T}, \mathcal{A}, c_{

where $\mathcal{A}$ is the attribute-label prompt and the output sequence includes fine-grained values, global tokens, and semantic tokens. The paper mixes the two objectives during training by turning each audio example into two training samples: one for zero-shot TTS and one for attribute-conditioned control.

Inference Modes

The appendix describes two zero-shot inference strategies:

• Reference-global-token prompting: the model receives the target text plus the global tokens extracted from reference audio.
• Reference-prefix prompting: the prompt additionally includes the reference transcript and reference semantic tokens as a prefix.

The second strategy improves speaker similarity, and the main zero-shot numbers in the paper are reported with this prefix-based setup.

For voice creation, Spark-TTS supports both:

• Coarse-grained control: the prompt contains the text and attribute labels, and the model first predicts fine-grained values before generating global and semantic tokens, following a chain-of-thought style.
• Fine-grained control: the prompt includes both labels and explicit values, then the model generates the speech tokens directly under those constraints.

VoxBox: Dataset Construction and Annotation

To support controllable TTS research, the paper introduces VoxBox, a large dataset assembled entirely from open-source corpora. The stated goal is not only scale, but also attribute-rich annotation and cleaner transcripts for training and evaluation. VoxBox covers both English and Chinese and includes speech from TTS-oriented corpora as well as emotional speech datasets, which broadens stylistic diversity.

The final VoxBox statistics reported in the appendix are:

• 47,706,212 utterances in total;
• 102,500.26 hours overall;
• 47,627.15 hours of Chinese speech;
• 54,873.11 hours of English speech;
• data drawn from 29 open datasets.

The dataset is annotated with gender, pitch, and speed for all samples used in the main controllable TTS experiments. The appendix further states that VoxBox also contains age and emotion annotations for broader applicability.

Annotation Pipeline

Gender is predicted using a WavLM-large classifier fine-tuned on datasets with explicit gender labels. The paper reports 99.4% accuracy on the AISHELL-3 test set. This classifier is then used to annotate corpora that do not already contain gender labels.

Pitch is measured using PyWorld. The average pitch of each clip is rounded to the nearest integer for fine-grained pitch tokens. For coarse pitch labels, the authors convert pitch to the Mel scale and set five bins using percentile boundaries, separately for male and female speakers.

Speed is computed using syllables per second after removing leading and trailing silence with VAD. The value is rounded to the nearest integer for fine-grained tokens. Five speed bins are also defined using percentile-based thresholds, separately for Chinese and English.

Text cleaning is important in VoxBox. For Emilia, the authors re-run ASR with FunASR and discard clips whose transcript WER relative to the original script exceeds 0.05. For MLS-English, LibriSpeech, LibriTTS-R, and the emotion corpora, they use Whisper large-v3 and remove samples with insertions or deletions relative to the original scripts.

Additional Attribute Predictors

The appendix reports an age classifier trained on WavLM-large, with five age groups: Child, Teenager, Young Adult, Middle-aged, and Elderly. On the AISHELL-3 test set, the paper reports 95.6% age accuracy and 99.4% gender accuracy. These extra annotations and predictors are not central to the main TTS experiments, but they show that VoxBox was designed as a richer speech resource than a minimal TTS-only corpus.

VoxBox Composition Summary

Training Setup

The training setup is fairly simple given the size of the system:

• BiCodec is trained on LibriSpeech 960 h plus 1,000 h each from Emilia-CN and Emilia-EN, for roughly 3,000 hours total.
• All audio is resampled to 16 kHz.
• The global token length is fixed to 32.
• BiCodec uses AdamW with $\beta_1 = 0.8$ and $\beta_2 = 0.9$ and converges in about 800k steps.
• The Spark-TTS LLM is trained on the full VoxBox training set using AdamW with $\beta_1 = 0.9$ and $\beta_2 = 0.96$.
• The LLM is trained for 3 epochs with batch size 768.

Experimental Results

The paper evaluates two distinct aspects: reconstruction quality of BiCodec as a codec, and synthesis/control quality of Spark-TTS as a TTS system.

BiCodec Reconstruction Quality

On LibriSpeech test-clean, BiCodec is compared against a broad set of codecs and achieves the paper’s best low-bitrate performance in the under-1 kbps regime. The authors emphasize that BiCodec reaches 50 TPS and about 0.65 kbps while still improving reconstruction quality over other sub-1 kbps codecs.

The authors’ interpretation is that BiCodec’s semantic tokens, extracted from wav2vec 2.0 features, give it stronger semantic alignment than waveform-first codecs. That helps it perform especially well on intelligibility-oriented metrics in the low-bitrate regime.

Global Token Length and Quantization Ablation

The paper studies the global token sequence length on LibriSpeech test-clean. Longer global-token sequences improve quality, and length 32 is the best setting reported in the main table. The authors also compare their FSQ-based global tokenization against a group-VQ variant from Ren et al.; the learnable-query + FSQ design is consistently stronger.

Spark-TTS Control Abilities

Spark-TTS is evaluated on three kinds of controllability: gender, coarse pitch/speed labels, and fine-grained pitch/speed values.

Gender Control

On a restructured prompt set derived from PromptTTS, Spark-TTS is compared with VoxInstruct and Parler-TTS. The system achieves the highest gender accuracy by a clear margin:

Pitch and Speed Control

The paper does not report a single scalar table for pitch and speed control. Instead, it provides confusion matrices for coarse labels and scaling plots for fine-grained values. The qualitative conclusion is that Spark-TTS closely follows the specified pitch and speaking-rate targets in both Chinese and English, and for both male and female speakers.

Zero-Shot TTS Results

The main zero-shot evaluation follows Seed-TTS-eval on Chinese and English test sets. The reported metrics are Chinese CER and English WER, plus speaker similarity. Spark-TTS performs especially well on intelligibility, while similarity is somewhat lower than the strongest multi-stage or non-autoregressive systems.

The paper’s interpretation is nuanced: Spark-TTS is not the absolute best on speaker similarity, but it is much stronger on intelligibility than prior one-stage autoregressive methods, and it does so with a much smaller model and less training data. The authors explicitly note that Spark-TTS uses 0.5B parameters and 100k hours of data, whereas Llasa-8B is much larger and is trained on 250k hours.

Reference Prefix Ablation

The appendix compares zero-shot inference with and without the reference-text/semantic prefix. Removing the prefix improves CER/WER but reduces speaker similarity, which matches the intended trade-off between easier transcript prediction and voice cloning fidelity.

Speech Quality on LibriSpeech Test-Clean

The paper also compares mean opinion score prediction on generated speech. On LibriSpeech test-clean, Spark-TTS reaches a UTMOS of 4.35, compared with 4.09 for CosyVoice and 4.23 for CosyVoice2, while the ground truth reference is 4.08 in the table reported by the authors. This is used to argue that the generated speech is not only intelligible but also very high in perceived quality.

Generalization of the Codec

Beyond the main LibriSpeech setting, the appendix evaluates BiCodec on several datasets with different recording conditions. The headline point is that the codec remains competitive on low-quality phone recordings, studio speech, and multilingual in-the-wild data. The paper reports that BiCodec either matches or exceeds X-codec2 on most metrics even when trained on only 3k hours, and the 24 kHz variant gives additional improvements in some settings.

Interpretation of the Contributions

From a research-engineering perspective, Spark-TTS is interesting because it attacks the TTS problem at three different levels simultaneously:

• Representation level: BiCodec disentangles semantic and global attributes into a single-stream tokenization scheme.
• Modeling level: the speech generator is a plain decoder-only LLM, which makes the architecture closer to text generation than many prior TTS systems.
• Control level: Spark-TTS supports both coarse labels and numeric parameters, so users can specify not only what kind of voice they want but also how fast and how high-pitched it should sound.

The paper argues that this design creates a practical middle ground between voice cloning and voice creation. Reference audio can still be used for zero-shot cloning, but the attribute prompts allow the model to synthesize new voices that are not tied to a prompt clip. In that sense, the system is not just a better codec-LLM; it is an attempt to make controllable voice generation more like structured language generation.

Limitations

The authors are explicit that Spark-TTS is not perfect. Their main limitation statement is that speaker similarity in zero-shot TTS is still lower than in multi-stage or non-autoregressive systems. They attribute this partly to the extra speaker variability introduced by the autoregressive language model during inference.

They also point out that BiCodec does not yet impose extra disentanglement constraints between global tokens and semantic tokens. As future work, they suggest perturbing formants or pitch in the semantic-token input so that the decoder can achieve more absolute control over timbre and reduce randomness coming from the AR model. In other words, they see the current system as a strong step toward controllable TTS, but not yet the final answer for perfectly disentangled voice synthesis.

Bottom Line

Spark-TTS combines a semantically aligned single-stream codec with a decoder-only LLM to produce a TTS system that is both efficient and controllable. The paper’s strongest empirical claims are that BiCodec sets a new low-bitrate reconstruction benchmark, that Spark-TTS achieves strong zero-shot intelligibility on Seed-TTS-eval, and that it enables fine-grained control over pitch and speaking rate without abandoning the simplicity of a standard LLM stack. VoxBox is a substantial supporting contribution because it provides a large, cleaned, attribute-annotated open dataset for controllable speech synthesis research.