CosyVoice

CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

A scalable multilingual zero-shot TTS system that uses supervised semantic tokens from ASR for high-fidelity voice cloning. It combines an LLM for text-to-token generation with flow-matching vocoding, achieving superior content consistency compared to prior unsupervised token approaches.

• tts
• voice-cloning
• llm
• multimodal
• speech-to-speech
• emotion

Demos

CosyVoice demonstrates industry-grade multilingual speech synthesis with zero-shot voice generation and emotional control. Evaluate naturalness, emotional expressiveness, and multilingual quality—from speech-to-speech translation preserving speaker identity across five languages, to dynamic emotional voice chat with controllable tone/speed/pitch, to expressive audiobook narration with character-appropriate delivery. These demos showcase the model's ability to handle both seen speakers (fine-tuned) and unseen speakers (zero-shot in-context learning).

Authors: Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, Zhijie Yan

Categories: cs.SD, cs.AI, eess.AS

Comment: work in progress. arXiv admin note: substantial text overlap with arXiv:2407.04051

Published 2024-07-07 · Updated 2024-07-09

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub21,958 stars132 watchers2,528 forksApache-2.0
• Hugging Face694 downloads9 likes

Impact

• Semantic Scholar469 citations81 influential citations

Abstract

Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.

---

Introduction and Motivation

Large Language Model (LLM)-based Text-to-Speech (TTS) synthesis has emerged as a dominant paradigm owing to its high naturalness and zero-shot voice cloning capability. In this framework, speech signals are first discretized into token sequences; an LLM then models these sequences conditioned on text, and a token-based vocoder reconstructs the waveform. A central design decision — and the primary focus of this paper — is how speech tokens are learned.

Existing approaches derive speech tokens in an unsupervised manner. Examples include EnCodec tokens (learned from speech reconstruction via a VQ-VAE), HuBERT tokens (learned from clustering-based pseudo labels), and W2V-BERT tokens (learned from contrastive loss and masked language modelling). A common weakness of all these methods is the absence of explicit semantic supervision: the tokens lack a direct correspondence to linguistic content, and their alignment to text is imprecise.

The authors argue that this semantic gap hurts content consistency and speaker similarity in downstream zero-shot TTS. Their central hypothesis is that supervised semantic tokens — derived from a discriminatively trained ASR model — will encode richer linguistic content and improve both intelligibility and voice cloning fidelity. CosyVoice is the system built on top of this idea.

The $\mathcal{S}^3$ Tokenizer: Supervised Semantic Speech Tokens

The supervised semantic speech ($\mathcal{S}^3$) tokenizer is obtained by modifying a pre-trained multilingual ASR encoder. Concretely, the encoder is split into two parts and a vector quantization (VQ) layer is inserted between them.

Given a Mel spectrogram $X$ as input, the first encoder block applies positional encoding and produces context-aware representations: $$H = \mathrm{Encoder}_1\!\left(\mathrm{PosEnc}(X)\right)$$ For each frame $l$, the VQ layer finds the nearest codebook entry $\mathbf{c}_n \in C$ by Euclidean distance: $$\mu_l = \mathrm{VQ}(\mathbf{h}_l, C) = \arg\min_{\mathbf{c}_n \in C} \|\mathbf{h}_l - \mathbf{c}_n\|_2$$ The integer index $\mu_l$ is the speech token for frame $l$. Codebook entries are updated during training via exponentially moving average (EMA): $$\mathbf{c}_{\mu_l} \leftarrow \alpha\,\mathbf{c}_{\mu_l} + (1-\alpha)\,\mathbf{h}_l$$ where $\alpha$ is a pre-defined decay coefficient. The quantized representations $\bar{H} = \{\mathbf{c}_{\mu_1}, \dots, \mathbf{c}_{\mu_L}\}$ pass through an additional positional encoding step and then through the second encoder block: $$\tilde{H} = \mathrm{Encoder}_2\!\left(\mathrm{PosEnc}(\bar{H})\right)$$ Finally, a Transformer-based ASR decoder predicts posterior probabilities over text labels in a teacher-forcing scheme: $$P(Y|X) = \mathrm{ASRDecoder}\!\left(\tilde{H},\, Y^{Z-1}\right)$$ where $Y^{Z-1}$ is the left-shifted label sequence.

The key insight is that the ASR training objective provides explicit gradient signal to the codebook: tokens must capture information sufficient for phonemic/linguistic transcription. This contrasts with purely reconstructive or contrastive objectives used in unsupervised tokenisers.

For the small-scale, single-lingual setting the backbone is an ESPNet Conformer ASR model trained on LibriSpeech from scratch. The VQ layer is inserted after the first six encoder layers with a single codebook of 4,096 codes. For the large-scale, multilingual setting the backbone is SenseVoice-Large, a rich-recognition model that supports Chinese (Mandarin), English, Cantonese, Japanese, and Korean. Its pre-trained checkpoint is used for initialisation; the full model (including the inserted VQ) is then fine-tuned for 210,000 steps on eight A800 GPUs.

CosyVoice Architecture Overview

CosyVoice is composed of four components:

• Text Encoder — aligns the semantic space of text with that of speech tokens.
• $\mathcal{S}^3$ Speech Tokenizer — extracts supervised semantic tokens from speech (described above).
• Large Language Model (LLM) — performs auto-regressive text-to-token generation.
• Conditional Flow Matching Model — maps speech tokens to a Mel spectrogram via a learned ODE; HiFi-GAN then converts the spectrogram to a waveform.

Large Language Model for Text-to-Token Generation

The TTS task is framed as an auto-regressive token sequence generation problem. The LLM operates over the following unified sequence: $$\left[\langle S\rangle,\; \mathbf{v},\; \{\bar{\mathbf{y}}_u\}_{u=1}^{U},\; \langle T\rangle,\; \{\mu_l\}_{l=1}^{L},\; \langle E\rangle \right]$$ where $\langle S\rangle$ and $\langle E\rangle$ are learned start- and end-of-sequence tokens, and $\langle T\rangle$ is a "turn-of-speech" boundary token. $\mathbf{v}$ is a speaker embedding (x-vector) extracted from the prompt speech by a pre-trained CAM++ voice-print model and injected directly into the sequence. The text encodings $\bar{Y}$ are computed as: $$\bar{Y} = \mathrm{TextEncoder}\!\left(\mathrm{BPE}(Y)\right)$$ A Byte Pair Encoding (BPE) tokenizer maps raw text to subword indices; a Transformer-based text encoder then projects them into the shared embedding space of the LLM. Including the text encoder bridges the semantic gap between subword tokens and acoustic tokens.

The training uses a teacher-forcing scheme with the standard cross-entropy loss, but only over the speech token and $\langle E\rangle$ positions: $$\mathcal{L}_\mathrm{LM} = -\frac{1}{L+1}\sum_{l=1}^{L+1} \log q(\mu_l)$$ where $\mu_{L+1}$ is the end-of-sequence token and $q(\mu_l)$ is the softmax posterior predicted by the LLM.

Two model sizes are provided: Tiny (6-layer text encoder + 12-layer LM, 512-dim attention, 8 heads, 2,048 FFN units) and Normal (6-layer text encoder + 14-layer LM, 1,024-dim attention, 16 heads, 4,096 FFN units), as detailed below.

Optimal-Transport Conditional Flow Matching (OT-CFM)

The second stage converts a sequence of speech tokens $\{\mu_l\}$ into a Mel spectrogram. The authors use an Optimal-Transport Conditional Flow Matching (OT-CFM) model, which outperforms denoising diffusion probabilistic models (DPMs) in terms of training simplicity and inference speed.

In Continuous Normalizing Flows (CNFs), a time-dependent vector field $\nu_t(X): [0,1]\times\mathbb{R}^{L\times D}\to\mathbb{R}^{L\times D}$ defines a probability density path from a Gaussian prior $p_0 = \mathcal{N}(0, I)$ to the data distribution $q(X)$ (Mel spectrograms) via the ODE: $$\frac{d}{dt}\phi_t(X) = \nu_t\!\left(\phi_t(X),\, t\right), \quad \phi_0(X) \sim \mathcal{N}(0,I)$$ The OT-CFM objective trains a neural network to match the optimal-transport flow: $$\mathcal{L}_\mathrm{OT\text{-}CFM} = \mathbb{E}_{t,\, p_0(X_0),\, q(X_1)}\left|\, \omega_t\!\left(\phi^{OT}_t(X_0,X_1)\mid X_1\right) - \nu_t\!\left(\phi^{OT}_t(X_0,X_1)\mid\theta\right)\right|$$ with the OT interpolant and target vector field: $$\phi^{OT}_t(X_0,X_1) = \bigl(1-(1-\sigma)t\bigr)X_0 + t\,X_1$$ $$\omega_t\!\left(\phi^{OT}_t(X_0,X_1)\mid X_1\right) = X_1 - (1-\sigma)X_0$$

The neural network is conditioned on:

• Speaker embedding $\mathbf{v}$ (x-vector) to capture timbre.
• Semantic tokens $\{\mu_l\}_{1:L}$ to guide linguistic content.
• Masked Mel spectrogram $\tilde{X}_1$: a version of the ground-truth spectrogram where a contiguous segment from a random start point to the end is zeroed out, acting as an acoustic prompt for timbre and environment.
• Intermediate state $X_t$ at the current timestep.

Cosine Timestep Scheduler

Because the generation task is harder at early timesteps (near the noise prior), the authors introduce a cosine schedule that warps the uniform timestep $t \in [0,1]$: $$t \leftarrow 1 - \cos\!\left(\tfrac{1}{2}\pi t\right)$$ This concentrates more denoising steps near $t=0$, where the trajectory is most uncertain.

Classifier-Free Guidance (CFG) for Flow Matching

The authors adapt CFG — previously used in diffusion models — to the flow matching setting. During training, all conditions $\Psi = \{\mathbf{v},\, \{\mu_l\},\, \tilde{X}_1\}$ are jointly dropped with probability 0.2, training both conditional and unconditional flows. At inference, the guided vector field is: $$\tilde{\nu}_t = (1+\beta)\,\nu_t(\cdot\mid\theta;\Psi) - \beta\,\nu_t(\cdot\mid\theta)$$ with guidance strength $\beta = 0.7$.

Zero-Shot In-Context Learning and Cross-Lingual Voice Cloning

CosyVoice achieves zero-shot voice cloning through in-context learning: given a reference speech sample and its transcript, the system is asked to synthesise new text in the same voice.

For same-language scenarios, the prompt text and prompt speech tokens are prepended to the input sequence; the LM treats them as already-generated tokens and auto-regressively completes the sequence for the target text. The prompt tokens extracted from the prompt speech by the $\mathcal{S}^3$ tokenizer are appended before the generated tokens and used as a composite condition for the flow matching stage alongside the speaker embedding and the prompt Mel spectrogram.

For cross-lingual voice cloning, when prompt language and target language differ, the prompt text and its corresponding tokens are omitted from the conditioning sequence to prevent prosodic patterns of the source language from bleeding into the target language output. A language identifier (LID) token is incorporated to disambiguate the target language.

The prompt transcript can be obtained either via human annotation or by running an ASR model (e.g., SenseVoice) on the prompt audio.

Instruction-Following Extension: CosyVoice-Instruct

A fine-tuned variant, CosyVoice-instruct, extends the base model with instruction-following capability. Fine-tuning removes the speaker embedding from the LLM input and instead relies on a natural-language instruction prepended to the text content (separated by a special <endofprompt> token). Three categories of instruction are supported:

• Speaker Identity: descriptive character attributes (e.g., "mysterious, elegant dancer").
• Speaking Style: emotion, gender, speaking rate, and pitch (e.g., "a happy girl with high tone and quick speech").
• Fine-grained Paralinguistics: inline tags for laughter ([laughter]), breath ([breath]), laughing while speaking (<laughter>...</laughter>), and word emphasis (<strong>...</strong>).

The instruction fine-tuning data amounts to approximately 556 hours total: 101 hours for speaker identity, 407 hours for speaking style, and 48 hours for fine-grained paralinguistics.

Datasets

Small-scale Single-lingual (LibriTTS)

Experiments probing the effect of token type use the LibriTTS corpus (585 hours, 2,456 English speakers). The official splits are followed: train-clean-100, train-clean-360, and train-other-500 for training; dev-clean for model selection; test-clean for evaluation.

Large-scale Multilingual (Internal Dataset)

CosyVoice's scalability experiments use a large proprietary corpus collected with in-house tools for speech activity detection, SNR estimation, speaker diarisation, and source separation. Pseudo transcripts are generated by SenseVoice-Large and Paraformer, then refined using forced alignment to remove low-quality utterances and fix punctuation.

Experiments and Results

Effect of Vector Quantisation on ASR Performance

Inserting VQ into the Conformer ASR encoder marginally degrades recognition accuracy. The VQ-augmented model achieves WERs of 3.13% / 3.18% / 7.56% on LibriSpeech dev-clean / test-clean / test-other versus 2.62% / 2.89% / 6.57% for the baseline. The small gap confirms that the supervised tokens retain the bulk of the semantic information encoded by the ASR model.

Multilingual Tokeniser Quality

For the multilingual tokeniser, Table 3 of the paper compares recognition rates from the raw $\mathcal{S}^3$ tokens against SenseVoice-Large and Whisper-Large V3 on Common Voice Chinese (zh-CN) and English (en) benchmarks. The $\mathcal{S}^3$ tokens outperform Whisper-Large V3 on Chinese (12.24% vs 12.82% CER without LID), demonstrating that the single 4,096-entry codebook retains strong multilingual semantic content.

Ablation: Text Tokeniser and Speech Token Type

Table 4 (Comparison with Baselines) ablates the contribution of each tokeniser on LibriTTS test-clean, measuring WER, insertion+deletion counts (#INS+DEL), substitution count (#SUB), and speaker similarity (SS) computed as raw cosine similarity via ERes2Net. The ablation experiments (Exp-1 through Exp-4) are:

• Exp-1: Phone text + HuBERT speech tokens (unsupervised). WER 7.41%, SS 67.85.
• Exp-2: Phone text + $\mathcal{S}^3_{en}$ tokens. WER 5.05%, SS 67.85. Switching to supervised tokens cuts WER substantially.
• Exp-3: BPE$_{en}$ text + $\mathcal{S}^3_{en}$ tokens. WER 3.93%, SS 67.85. BPE text encoder further helps content consistency.
• Exp-4-LibriTTS: BPE (multilingual) + $\mathcal{S}^3$ (multilingual) tokens trained on LibriTTS only. WER 4.76%, SS 65.94. Multi-lingual tokenisers on single-lingual data slightly degrade.
• Exp-4-Large-scale: Same tokenisers + large-scale multilingual training data. WER 3.17%, SS 69.49. Best result; approaches human parity.

The original (human) speech achieves WER 3.01%, SS 69.67. VALL-E (phone + EnCodec) achieves 18.70% WER, confirming the large advantage of supervised tokens over codec tokens for content consistency.

English Zero-Shot Voice Cloning (LibriTTS test-clean)

Five random seeds are used for sampling-based decoding; results are averaged. CosyVoice achieves WER 2.89 ± 0.18%, 88.60 ± 3.88 insertion/deletion errors, and SS 74.30 ± 0.15 — matching human-level WER and exceeding human SS. With 5× ASR re-ranking (selecting the best of five samples by ASR score), WER drops to 1.51% with only 47 ins./del. errors. ChatTTS produces WER 8.32% and 441 insertion/deletion errors, substantially worse on content consistency.

Chinese Zero-Shot Voice Cloning (AISHELL-3 test)

CosyVoice achieves CER 3.82 ± 0.24% and 24.4 ± 2.24 insertion/deletion errors on the Chinese test set, matching the original recordings (2.52% CER, 25 ins./del.). ChatTTS reaches a similar CER of 3.87% but produces 111 insertion/deletion errors, indicative of speaker leakage (unexpected modal particles from a different speaker). CosyVoice is substantially more robust here. With ASR re-ranking, CER drops to 1.84%. Speaker similarity of 81.58 exceeds the original 74.15, demonstrating strong voice cloning.

Emotion Controllability (CosyVoice-Instruct)

100 English utterances per emotion class (happy, sad, angry, surprised, fearful, disgusted) are synthesised. An emo2vec emotion recognition model evaluates predicted emotion accuracy. CosyVoice-instruct with instructions dramatically outperforms CosyVoice-base across all emotions, especially for sad (0.98 vs. 0.45), disgusted (0.93 vs. 0.46), and surprised (0.64 vs. 0.26). Removing the instruction at inference from CosyVoice-instruct reverts most gains, confirming that the instruction text drives the emotional control rather than latent model priors.

CosyVoice as a Data Generator for ASR

The authors evaluate CosyVoice-generated speech as a data augmentation source for ASR training on LibriSpeech. Results show that training on CosyVoice-synthesised data alone matches training on the original 960-hour LibriSpeech set. Combining real and synthesised data further improves recognition. Notably, synthesising speech for the larger MLS text set (beyond LibriSpeech text) yields substantial WER reductions across all four evaluation sets, suggesting that text diversity matters more than audio duration for ASR. The best configuration (Librispeech + Syn on LS + MLS text) achieves dev-clean / dev-other / test-clean / test-other WERs of 1.93 / 4.43 / 2.04 / 4.53%, compared to the baseline of 2.77 / 5.84 / 2.79 / 5.97%.

Key Contributions and Novelty

• First supervised speech tokens for TTS: The $\mathcal{S}^3$ tokenizer is the first to derive discrete speech tokens from a supervised ASR objective (rather than unsupervised reconstruction, clustering, or contrastive training), improving content consistency and alignment to text.
• CosyVoice synthesis pipeline: A complete zero-shot TTS system combining an LLM for text-to-token generation with an OT-CFM model for token-to-spectrogram synthesis, eliminating the need for separate phonemizers and forced aligners that most flow-matching TTS systems require.
• CFG for flow matching: Extension of classifier-free guidance from the diffusion domain to conditional flow matching models, with empirical validation of the guidance strength ($\beta=0.7$) and cosine timestep scheduling.
• Speaker decomposition in the LLM: The x-vector is injected as a sequence element rather than a side-channel feature, allowing the LLM to explicitly model speaker prosody alongside semantic content, while the flow matching model separately captures fine-grained timbre and environment.
• Scalability: Large-scale training on ~172,000 hours across five languages produces human-parity output, and the improvement is consistent with data volume, supporting the scalable nature of the architecture.
• Instruction following: CosyVoice-instruct enables fine-grained natural-language control over identity, emotion, speaking rate, pitch, and paralinguistic features without any additional architectural changes beyond the fine-tuning protocol.

Training Configuration Summary

The tiny model is trained on LibriTTS for 50 epochs on four V100-32G GPUs with a learning rate of $10^{-3}$ and 10,000 warmup steps. The normal (multilingual) model is trained for 800,000 steps on 64 V100-32G GPUs with a learning rate of $10^{-4}$ and the same warmup schedule. The $\mathcal{S}^3$ tokenizer for the multilingual setting is fine-tuned for 210,000 steps on eight A800 GPUs starting from a SenseVoice-Large checkpoint.

Limitations and Future Work

The paper notes several aspects left for future investigation:

• The choice of VQ insertion layer (currently after layer 6) and codebook size (4,096) are fixed engineering choices; a systematic search over these hyperparameters is identified as future work.
• The paper acknowledges text overlap with a related preprint (arXiv:2407.04051), noted in the arXiv admin comment.
• Speaker similarity degradation is observed when switching to multilingual tokenisers without sufficient scale; adequate multilingual data coverage appears necessary for maintaining single-language quality.
• The CosyVoice-instruct evaluation does not include speaker similarity measurements, leaving open questions about the trade-off between instruction adherence and voice preservation.
• ASR re-ranking (5× sampling) substantially improves quality in offline settings but is computationally expensive for real-time deployment.

Conclusion

CosyVoice introduces a principled answer to the question of what makes a good speech token for LLM-based TTS: supervised semantic tokens derived from a discriminatively trained ASR model. The resulting $\mathcal{S}^3$ tokenizer provides tokens that provably retain sufficient linguistic information (near-baseline ASR performance) while supporting large-codebook discrete quantisation. Built on these tokens, the CosyVoice system — combining a BPE-text-conditioned LLM with a speaker-embedding-conditioned OT-CFM model and HiFi-GAN vocoder — achieves human-parity speech generation on both English and Chinese zero-shot voice-cloning benchmarks, exceeds human speaker similarity scores, supports cross-lingual voice cloning, and enables fine-grained emotion and paralinguistic control via natural-language instructions. The scalable performance gains with increased data volume confirm the practical viability of deploying CosyVoice at industrial scale.

from cosyvoice.cli.cosyvoice import AutoModel
cosyvoice = AutoModel(model_dir='pretrained_models/CosyVoice2-0.5B')
for i, j in enumerate(cosyvoice.inference_zero_shot(text, prompt_text, audio_path)):
    torchaudio.save(f'output_{i}.wav', j['tts_speech'], cosyvoice.sample_rate)