CosyVoice 2

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

A streaming TTS system combining zero-shot voice cloning with real-time synthesis using a pre-trained LLM and flow matching. It unifies streaming and non-streaming synthesis in one model via interleaved token sequences, achieving human-parity quality with minimal latency for interactive applications.

• tts
• streaming
• voice-cloning
• llm
• prosody
• low-latency
• one-shot

Authors: Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, Jingren Zhou

Categories: cs.SD, cs.AI, cs.LG, eess.AS

Comment: Tech report, work in progress

Published 2024-12-13 · Updated 2024-12-25

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub21,959 stars132 watchers2,528 forksApache-2.0
• Hugging Face3,829 downloads77 likes

Impact

• Semantic Scholar386 citations62 influential citations

Abstract

In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode. We invite readers to listen to the demos at https://funaudiollm.github.io/cosyvoice2.

---

Introduction and Motivation

Neural text-to-speech (TTS) synthesis has advanced dramatically beyond traditional concatenative and statistical parametric methods, reaching levels of fidelity and naturalness that rival human recordings. A particularly active frontier is zero-shot TTS, where models imitate the timbre, prosody, and style of an arbitrary reference speaker using in-context learning. These systems generally fall into three families: codec language models that discretise speech into tokens and model them autoregressively; feature diffusion / flow matching models that operate on continuous speech representations non-autoregressively; and hybrid systems that cascade a token-level language model with a diffusion-based acoustic decoder.

Despite impressive synthesis quality, virtually all existing zero-shot TTS systems operate in non-streaming (offline) mode: they require the full input text upfront and produce the entire waveform before returning any audio. This introduces unacceptably high latency for real-time interactive applications such as multimodal voice chat (e.g., GPT-4o style systems), where the speech synthesis component sits at the very end of a live generation pipeline. While a handful of language-model-based streaming TTS approaches exist, diffusion-based and hybrid architectures lack well-established streaming solutions.

CosyVoice 2 is a comprehensive re-engineering of the original CosyVoice hybrid TTS system to address these gaps. It introduces four principal technical innovations that together enable streaming synthesis at human-parity quality with minimal first-package latency:

• Finite Scalar Quantization (FSQ) replaces vector quantisation (VQ) in the supervised speech tokeniser, achieving 100% codebook utilisation and capturing richer semantic content.
• A simplified unified text-speech language model that directly uses a pre-trained LLM (Qwen2.5-0.5B) as its backbone, dropping the text encoder and speaker embedding of the predecessor, and supporting both streaming and non-streaming generation within a single model via a mixed text-speech token sequence design.
• A chunk-aware causal flow matching model that trains with four different attention masks simultaneously, enabling a single acoustic decoder to operate in offline, full-causal, and chunk-based streaming modes.
• An upgraded instructed TTS capacity that unifies zero-shot and instruction-conditioned synthesis in one model, supporting fine-grained control over emotion, speaking rate, dialect, role style, vocal bursts, emphasis, and laughter.

System Architecture

CosyVoice 2 follows the same high-level design philosophy as its predecessor: it separates semantic and acoustic information and models them independently in a progressive decoding pipeline. The pipeline has four stages:

1. Text tokenisation — raw text is tokenised with a BPE tokeniser (no G2P frontend).
2. Speech tokenisation — input speech is encoded into discrete semantic tokens via an FSQ-based supervised tokeniser built on the SenseVoice-Large ASR model.
3. Text-speech language model — a pre-trained LLM autoregressively maps text tokens to speech tokens.
4. Chunk-aware causal flow matching — a conditional flow matching model decodes speech tokens into a Mel spectrogram conditioned on speaker identity and reference speech; a pre-trained vocoder then converts the spectrogram to waveform.

Text Tokeniser

CosyVoice 2 ingests raw text directly and tokenises it with a BPE tokeniser, eliminating the grapheme-to-phoneme (G2P) frontend required by the original CosyVoice. This simplifies the preprocessing pipeline and lets the model learn context-dependent pronunciation end-to-end. One important design choice: BPE tokens that encode more than one Chinese character are masked out, and each character is encoded separately. This prevents any single token from spanning an excessively long pronunciation sequence and reduces corner cases from data sparsity. Other languages (English, Japanese, Korean) are not subject to this masking.

Supervised Semantic Speech Tokeniser with FSQ

The speech tokeniser inserts a Finite Scalar Quantisation (FSQ) module into the encoder of the SenseVoice-Large ASR model. The encoder is split into two parts: $\mathrm{Encoder}_1$ (six Transformer blocks with rotary positional embeddings) produces intermediate representations, which are quantised by FSQ; the quantised representations then pass through $\mathrm{Encoder}_2$ and the ASR decoder to predict text token posteriors, providing the supervision signal.

In the FSQ module, intermediate representations $H$ are first projected into a $D$-dimensional low-rank space. Each dimension is independently quantised into the integer range $[-K, K]$ via a bounded rounding operation $\mathrm{ROUND}$:

The straight-through estimator is used for gradient computation through the non-differentiable rounding step. The discrete speech token $\mu_i$ is computed as the index of the quantised low-rank vector $\bar{h}_i$ in the $(2K+1)$-ary number system:

This FSQ construction gives a codebook of size $(2K+1)^D$. With the chosen hyperparameters the codebook contains 6,561 entries, all of which are used (100% utilisation) — a dramatic improvement over the VQ baseline which used only 963 of its 4,096 entries (23%). The tokeniser operates at 25 Hz (25 tokens per second).

A t-SNE analysis confirms that the quantised representations are speaker-agnostic: representations of 100 utterances from each of three VoxCeleb1 speakers are intermixed after FSQ, whereas pre-quantisation encoder outputs are clearly speaker-separated. Further validation is provided by a speaker identification (SID) experiment using the S3prl toolkit: a linear SID classifier trained on the quantised tokens fails to converge, demonstrating that speaker information is not recoverable from the tokens alone.

Unified Text-Speech Language Model

CosyVoice 2 uses Qwen2.5-0.5B as the backbone for its text-speech language model. The model is trained in a next-token-prediction scheme. Two important simplifications relative to the original CosyVoice are:

• The text encoder is removed. The pre-trained LLM is found to be powerful enough to handle text-speech alignment directly without a separate encoder.
• The speaker embedding is removed from the LM. Speaker identity is shown to contain not just timbre but also language and paralinguistic information, which interferes with prosody naturalness and cross-lingual transfer when injected at the LM stage. Speaker conditioning is instead handled entirely in the flow matching model.

The key innovation enabling unified streaming/non-streaming synthesis is the sequence construction strategy. The model is trained simultaneously on two types of sequences:

Non-streaming mode: The sequence is constructed as: $[\texttt{S}, \text{text tokens}, \texttt{T}, \text{speech tokens}, \texttt{E}]$, where $\texttt{S}$ is the start-of-sequence token, $\texttt{T}$ is the turn-of-speech token, and $\texttt{E}$ is end-of-sequence. Text token losses are masked (ignored) in the cross-entropy objective.

Streaming mode: Text and speech tokens are interleaved at a ratio of $N{:}M$ (set to $5{:}15$ in experiments). Every $N$ text tokens are followed by $M$ speech tokens. When the model encounters a position that would be a text token, it instead predicts a special filling token, which at inference time signals that the next $N$ real text tokens should be concatenated. Once all text tokens are exhausted, the sequence continues with $\texttt{T}$ followed by the remaining speech tokens.

At inference time, four modes are supported depending on whether in-context learning (ICL) or speaker fine-tuning (SFT) is used and whether streaming or non-streaming output is required:

• ICL Non-Streaming: Prompt text and to-synthesise text are concatenated; prompt speech tokens are fixed as pre-generated context. Autoregressive generation begins after the prompt speech.
• ICL Streaming: Prompt and to-generate text are treated as a whole and interleaved with prompt speech tokens at ratio $N{:}M$. Generation returns $M$ tokens at a time.
• SFT Non-Streaming: No prompt needed; generation starts from $[\texttt{S}, \text{text}, \texttt{T}]$.
• SFT Streaming: Starts from $[\texttt{S}, \text{first } N \text{ text tokens}]$; the model generates $M$ speech tokens, then the next $N$ text tokens are manually padded in, repeating until all text is exhausted. This mode is also applicable to speech-to-speech multimodal LLMs for ultra-low latency.

Chunk-Aware Causal Flow Matching Model

The flow matching model converts semantic speech tokens into a Mel spectrogram (50 Hz frame rate, 24 kHz sampling rate). Because the speech tokens are at 25 Hz and the Mel spectrogram at 50 Hz, the speech tokens are first upsampled by a factor of 2. Before upsampling, a look-ahead convolution layer provides future context: implemented as a right-padded 1-D convolution with pad size $P$ and kernel size $P+1$. Several chunk-aware causal Transformer blocks then align the token representation space with the acoustic feature space.

CosyVoice 2 employs Conditional Flow Matching (CFM) with an Optimal Transport (OT) path. The probability density path is defined by a time-dependent vector field. The OT flow vector field is:

where $\mathbf{v}$ is the speaker embedding, $\mu$ are the upsampled speech tokens, and $\tilde{X}_1$ is the masked Mel spectrogram (serving as a reference speech condition). The training objective minimises the L1 loss between predicted and ground-truth vector fields:

During training, the masked reference $\tilde{X}_1$ is obtained by randomly masking 70%–100% of the final frames of $X_1$. At inference, it is the Mel extracted from the reference speech. The timestep $t$ follows $U[0,1]$ during training, but at inference a cosine schedule is used to allocate more ODE steps to the initial generation phase:

Classifier-Free Guidance (CFG) is applied at inference to improve fidelity:

where $\Psi = \{\mathbf{v}, \mu, \tilde{X}_1\}$. The CFG strength $\beta = 0.7$ and the number of flow estimation steps (NFE) $= 10$ are determined experimentally.

The critical innovation for streaming is the chunk-aware masking strategy. The multi-step flow estimation (10 UNet evaluations unrolled) is treated as a deeper stacked network, and four attention masks are defined:

• Non-causal Mask: All frames attend to all frames. Best offline quality. Used for latency-insensitive scenarios.
• Full-causal Mask: Each frame attends only to past frames. Minimum latency, highest degradation.
• Chunk-$M$ Mask: Each frame attends to past frames plus up to $M$ future frames. A good trade-off for the first generated chunk.
• Chunk-$2M$ Mask: Each frame attends to past frames plus up to $2M$ future frames. Near-offline quality at the cost of additional latency; suitable for subsequent chunks.

During training, each sample in a mini-batch is randomly assigned one of the four masks uniformly. This has two benefits: (1) a single model can handle all deployment scenarios, reducing engineering complexity; (2) masks with more context act as implicit teachers for masks with less context, providing a form of self-distillation.

Latency Analysis for Streaming Mode

The first-package latency for a pure TTS deployment is:

where $d_{lm}$, $d_{fm}$, $d_{voc}$ are the per-token computation times of the LM, flow matching model, and vocoder respectively, and $M$ is the chunk size (number of speech tokens per package). In an LLM-based voice chat scenario where text is also generated on-the-fly, the first-package latency is bounded by:

where $d_{llm}$ is the per-token computation time of the upstream text LLM and $N$ is the number of text tokens per chunk. Importantly, because multi-character Chinese tokens are masked out in CosyVoice 2's tokeniser, each text token in the LLM always represents more raw text characters than each speech-side text token, so the actual latency is strictly below the upper bound.

Instructed Generation

CosyVoice 2 integrates instructed TTS directly into the base model (rather than training a separate instruct variant). 1,500 hours of instructed training data are added to the base training set, covering two types of instruction:

• Natural language instructions: A natural language description is prepended to the input text, followed by a special <|endofprompt|> token. Instructions cover emotion (happy, sad, surprised, angry, fearful, disgusted, calm, serious), speaking rate (fast, very fast, slow, very slow), dialects (Cantonese, Sichuan, Shanghai, Zhengzhou, Changsha, Tianjin), and role-playing (mysterious, fierce, curious, elegant, lonely, robot, Peppa Pig, etc.).
• Fine-grained instructions: Vocal burst markers such as [laughter] and [breath] are inserted between text tokens. Vocal feature tags such as <strong>...</strong> (emphasis) and <laughter>...</laughter> (speaking with laughter) are applied to specific phrases.

Multi-Speaker Fine-Tuning and Reinforcement Learning

Multi-Speaker Fine-Tuning (mSFT)

The pretrained model can be fine-tuned on specific speakers. CosyVoice 2 introduces multi-speaker fine-tuning, where multiple speakers are fine-tuned simultaneously rather than individually. To avoid timbre confusion, speaker-specific prompt tags (e.g., Speaker A<|endofprompt|>) are prepended to input text. Unlabelled samples use a special unknown<|endofprompt|> tag. Unsupervised clustering on speaker embeddings is used to ensure timbre stability within each speaker cluster. The fine-tuning learning rate is fixed at $10^{-5}$. Experiments show that as few as 400 recordings per target speaker suffice for good performance.

Reinforcement Learning for SFT

Speaker fine-tuning can sometimes degrade pronunciation accuracy. CosyVoice 2 addresses this with two complementary reinforcement learning (RL) approaches:

Direct Preference Optimisation (DPO): Preferred ($x^w$) and rejected ($x^l$) synthesis samples are distinguished using ASR WER and speaker similarity (SS) as reward signals. The DPO objective is:

where $\mu^w$ and $\mu^l$ are the speech tokens extracted from preferred and rejected samples. A limitation of DPO is that it requires four forward passes per training step and repeated audio synthesis to form preference pairs.

Differentiable ASR Reward: To avoid the computational overhead of DPO, CosyVoice 2 proposes a more efficient alternative. The LM-predicted token $\mu_i \in \{0, \dots, (2K+1)^D - 1\}$ is inverted back to the quantised low-rank representation $\bar{H}$ by:

Then $\bar{H}$ is projected back to the full-dimensional space $\hat{H} = \mathrm{Proj}_{\text{up}}(\bar{H})$ and fed into the frozen ASR backend of the speech tokeniser to compute the ASR loss:

Since the discrete sampling step $\mu_i \sim P(\mu_i \mid \mu_{1:i-1}, Y; \theta_{LM})$ is non-differentiable, Gumbel-softmax sampling is used to make it differentiable, enabling direct optimisation of $\theta_{LM}$ via $L_{ASR}$. The differentiable ASR reward has better generalisation to out-of-domain text than DPO, because hard samples with repeated or unusual patterns are not accidentally labelled as "rejected." Combining both $L_{ASR}$ and $L_{DPO}$ yields the best overall results.

Experimental Setup

Training Data

The speech tokeniser is trained on 200,000 hours of Chinese and English ASR data (approximately 110,884 hours Chinese, 99,918 hours English). Zero-shot capability for Japanese and Korean is observed despite no exposure to these languages during tokeniser training.

CosyVoice 2's LM and flow matching models are trained on ~166,800 hours of multilingual data: approximately 130,000 hours Chinese, 30,000 hours English, 4,600 hours Japanese, and 2,200 hours Korean. Pseudo-labels are generated by Paraformer (Chinese) and SenseVoice (other languages). An internal force-alignment model filters low-quality data and refines punctuation. An additional 1,500 hours of instructed data is added for instruction fine-tuning.

Evaluation Protocols

Three test sets are used:

• test-clean: From LibriSpeech test-clean; evaluates limited-domain English performance. ASR: Whisper-Large V3. Speaker similarity (SS): ERes2Net. Quality: NMOS.
• SEED test sets: test-zh (~2,000 Chinese samples from CommonVoice), test-en (~1,000 English samples), test-hard (~400 challenging cases with text repetition and tongue twisters). ASR: Paraformer (Chinese/hard), Whisper-Large V3 (English). SS: both WavLM-finetuned SV and ERes2Net.
• test-ja / test-ko: 1,000 Japanese and 1,000 Korean samples from CommonVoice. ASR: Whisper-Large V3. Also evaluates NMOS.

Instructed generation is evaluated on an in-house Chinese test set of 290 samples across 29 instruction types, rated by 10 native Chinese speakers on a MOS-I scale (1–5, in 0.5 increments).

Results

Speech Tokeniser: FSQ vs. VQ

FSQ achieves full codebook utilisation and substantially lower ASR error rates across all benchmarks, confirming that it retains more semantic information than VQ.

Comparison with Baseline Systems on LibriSpeech test-clean

CosyVoice 2 achieves state-of-the-art results across all three metrics on LibriSpeech test-clean, surpassing human-level WER, NMOS, and SS simultaneously, indicating human-parity synthesis quality. Notably, the streaming variant (CosyVoice 2-S) is virtually lossless, even slightly improving on WER and SS.

Comparison on SEED Test Sets

On test-zh, CosyVoice 2 outperforms all open-source models in CER and speaker similarity, narrowly trailing only the closed-source commercial model Seed-TTS. On test-en, CosyVoice 2 ranks fourth in WER and third in SS among open-source systems, likely due to the Chinese/English data imbalance. On the challenging test-hard set, CosyVoice 2 achieves state-of-the-art results among all compared open-source systems. The streaming variant CosyVoice 2-S is essentially lossless on test-zh and test-en, with a mild degradation on test-hard (8.08% vs. 6.83% WER), highlighting robustness of the unified streaming framework.

The paper also notes that speaker similarity rankings differ across SV models (WavLM vs. ERes2Net), pointing to an open research problem in TTS evaluation methodology. ERes2Net is adopted as the primary SS metric in subsequent experiments.

Modular Ablation: LM Improvements

Each successive modification provides clear gains in content consistency. LLM initialisation alone reduces CER on test-zh by a relative 18.5% and on test-hard by a relative 15.4%. Dropping the speaker embedding further reduces errors (particularly on English) while maintaining SS, confirming that speaker information is best handled in the flow matching stage. The largest jump comes from replacing VQ with FSQ, which dramatically lowers error rates by enabling the tokeniser to capture finer phonetic detail. Adding pitch loss as an auxiliary constraint on the tokeniser provides further modest gains and is flagged as a direction for future investigation.

Modular Ablation: Streaming Components

Streaming LM has minimal impact on typical Chinese and English test cases (CER/WER negligibly changes), with the primary degradation appearing on the challenging hard set. Interestingly, streaming flow matching slightly improves speaker similarity, likely because in streaming mode the prompt-to-generation frame ratio in initial chunks is higher than in full-utterance offline mode where many padding frames dilute the prompt signal. The negative effect of streaming FM on content consistency is much smaller than streaming LM, attributable to the semantic-acoustic decoupling design.

Japanese and Korean Results

Korean significantly outperforms Japanese across all metrics. The paper attributes this to character set overlap between Japanese and Chinese, which causes the model to apply Chinese pronunciations in Japanese contexts. Korean uses a completely distinct character set and thus benefits from better linguistic disambiguation. Both languages could further benefit from additional training data. The streaming variant shows slightly lower quality than offline for both languages.

Instructed Generation Results

CosyVoice 2 substantially outperforms CosyVoice-Instruct on instruction following (MOS-I: 4.06 vs. 3.09) while also improving CER and SS. Removing the instruction input from CosyVoice 2 improves CER and NMOS (as expected — instructions introduce stylistic variation that can confuse ASR), but instruction-following accuracy collapses (MOS-I: 2.28), confirming that instruction controllability does not spontaneously emerge from content text alone.

Reinforcement Learning Results

SFT markedly improves speaker similarity but can degrade WER (7.15% vs. 5.34% for the base model) for the challenging Speaker E, which has fast speech rate and only Chinese recordings. Both RL methods reduce WER. The differentiable ASR reward ($L_{ASR}$) has better generalisation: it improves the hard SEED subset (7.30% vs. 7.90%) whereas DPO-only ($L_{DPO}$) worsens it (8.31%), because hard samples with repeated words resemble rejected samples during DPO training. Combining both methods achieves the best overall results across all metrics.

Speaker Fine-Tuning Results

Multi-speaker fine-tuning achieves strong and consistent performance across multiple target speakers, with only slight variation in objective metrics between individuals. Even with as few as 400 audio recordings per speaker, the fine-tuned model achieves good synthesis quality. Most speakers successfully inherit the base model's contextual understanding, enabling natural expression of varied moods and emotions without explicit instruction.

Key Design Insights and Discussion

• FSQ superiority over VQ: VQ leaves 77% of codebook entries unused, limiting its expressivity. FSQ by construction uses every codeword, capturing subtler phonetic variations that directly translate to lower downstream TTS error rates.
• Speaker embedding removal from LM: Utterance-level speaker vectors entangle language, paralinguistic, and speaker identity information, confusing the LM's text-speech alignment. Reserving all speaker conditioning for the flow matching stage produces cleaner separation and better cross-lingual performance.
• Unified streaming/non-streaming framework: Training a single LM on both interleaved and sequential token sequences — and training a single flow matching model with four attention mask types — eliminates the need to maintain separate streaming and non-streaming systems, substantially reducing deployment complexity.
• Implicit self-distillation in chunk-aware FM: Attention masks with more context (e.g., non-causal) provide soft supervision for masks with less context (e.g., full-causal), improving streaming quality without an explicit distillation loss.
• Speaker similarity metric inconsistency: The paper identifies a methodological concern: rankings across WavLM-based and ERes2Net-based SS metrics do not always agree, suggesting the field needs standardised evaluation protocols for TTS speaker similarity.
• Differentiable ASR reward generalises better than DPO: DPO treats hard linguistic patterns (repetition, tongue twisters) as rejection candidates, inadvertently suppressing correct but unusual outputs. The differentiable ASR reward directly optimises phonetic accuracy at the token level without this artefact.

Limitations

The authors explicitly identify three limitations of CosyVoice 2:

1. Limited language support: The model currently supports Chinese, English, Japanese, and Korean. For languages with overlapping character sets (e.g., Japanese and Chinese share many characters), synthesis quality degrades because the model may apply Chinese pronunciations in a Japanese context. Extending to new languages or improving cross-lingual disambiguation remains an open challenge.
2. No textual acoustic control: CosyVoice 2 cannot modify acoustic characteristics such as timbre through natural language instructions. For example, a user cannot instruct the model to "speak in a deep voice" without providing a reference audio. This capability could be valuable for role-playing and character voice applications.
3. Poor singing performance: The model is not designed for singing synthesis and does not perform well on singing tasks, which require precise pitch control and rhythmic alignment beyond what the current architecture provides.

Conclusion

CosyVoice 2 represents a systematic and comprehensive upgrade of the CosyVoice hybrid TTS framework, targeting the specific demands of real-time, interactive multimodal AI applications. Its four core contributions — FSQ-based speech tokenisation, a simplified LLM-backbone text-speech language model, a unified streaming/non-streaming sequence design, and chunk-aware causal flow matching — together deliver human-parity synthesis quality with minimal first-package latency and virtually no streaming quality loss relative to offline mode. The integrated instructed TTS capability further broadens the model's utility for expressive and controllable speech synthesis. The reinforcement learning approaches, particularly the novel differentiable ASR reward, provide a practical and generalisable method for improving pronunciation accuracy in fine-tuned speaker models without sacrificing domain robustness.