CosyVoice 3

CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training

A multilingual text-to-speech model using supervised speech tokenization and differentiable reward optimization. It expands training to 1M hours across 9 languages and scales model size 3×, achieving improvements in content consistency, speaker similarity, and prosody for robust in-the-wild generation.

• tts
• llm
• voice-cloning
• prosody
• rlhf
• streaming
• autoregressive
• multimodal

Authors: Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang, Jieping Ye

Categories: cs.SD, cs.AI, eess.AS

Comment: Preprint, work in progress

Published 2025-05-23 · Updated 2025-05-27

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub21,960 stars132 watchers2,528 forksApache-2.0
• Hugging Face22,563 downloads580 likes

Impact

• Semantic Scholar164 citations17 influential citations

Abstract

In our prior works, we introduced a scalable streaming speech synthesis model, CosyVoice 2, which integrates a large language model (LLM) and a chunk-aware flow matching (FM) model, and achieves low-latency bi-streaming speech synthesis and human-parity quality. Despite these advancements, CosyVoice 2 exhibits limitations in language coverage, domain diversity, data volume, text formats, and post-training techniques. In this paper, we present CosyVoice 3, an improved model designed for zero-shot multilingual speech synthesis in the wild, surpassing its predecessor in content consistency, speaker similarity, and prosody naturalness. Key features of CosyVoice 3 include: 1) A novel speech tokenizer to improve prosody naturalness, developed via supervised multi-task training, including automatic speech recognition, speech emotion recognition, language identification, audio event detection, and speaker analysis. 2) A new differentiable reward model for post-training applicable not only to CosyVoice 3 but also to other LLM-based speech synthesis models. 3) Dataset Size Scaling: Training data is expanded from ten thousand hours to one million hours, encompassing 9 languages and 18 Chinese dialects across various domains and text formats. 4) Model Size Scaling: Model parameters are increased from 0.5 billion to 1.5 billion, resulting in enhanced performance on our multilingual benchmark due to the larger model capacity. These advancements contribute significantly to the progress of speech synthesis in the wild. We encourage readers to listen to the demo at https://funaudiollm.github.io/cosyvoice3.

---

Introduction and Motivation

Text-to-speech (TTS) synthesis has advanced dramatically with generative neural networks, moving well beyond traditional concatenative and parametric methods. In the zero-shot regime — where a model clones a new speaker's voice from just a short audio prompt — three broad paradigms have emerged: (1) large language models (LLMs) over discrete acoustic tokens, (2) diffusion models that learn alignments automatically, and (3) coarse-to-fine hybrid systems combining autoregressive LLMs with non-autoregressive flow-matching or diffusion decoders. This last family has become the dominant choice in industrial deployments due to its favorable trade-offs between synthesis quality, streaming latency, and controllability.

The predecessor of this work, CosyVoice 2, belongs to this hybrid family. It integrates an LLM backbone with a chunk-aware causal flow-matching (FM) model to enable bi-directional streaming synthesis with near-human naturalness and ultra-low latency. Despite these achievements, CosyVoice 2 suffers from identifiable limitations: restricted language coverage, limited domain and style diversity, a modest training data volume, narrow text-format support, and an under-explored post-training regime.

CosyVoice 3 is designed to address all of these gaps simultaneously, targeting what the authors call in-the-wild speech generation — robust, high-quality synthesis across arbitrary speakers, languages, styles, acoustic conditions, and text formats encountered in everyday real-world deployment. The key advances are grouped into four pillars:

• A novel speech tokenizer built on supervised multi-task training atop a large audio-understanding LLM (MinMo), improving paralinguistic fidelity.
• A Differentiable Reward Optimization (DiffRO) method for RL-based post-training that sidesteps the cost of full audio-level reward computation.
• Dataset size scaling: from ~10,000 hours (CosyVoice 2) to one million hours, spanning 9 languages and 18 Chinese dialects.
• Model size scaling: LM grows from 0.5 B to 1.5 B parameters; the conditional flow-matching (CFM) backbone grows from 100 M to 300 M parameters using a diffusion transformer (DiT).

System Architecture Overview

CosyVoice 3 retains the two-stage hybrid architecture of its predecessor: an autoregressive speech-token language model that converts input text into discrete speech tokens, followed by a conditional flow-matching (CFM) model that converts those tokens into mel-spectrograms, which are then rendered to waveforms by a vocoder. What changes fundamentally are the tokenizer, the post-training strategy, and the scale of both data and model.

Speech Tokenizer via Supervised Multi-task Training

Design Rationale

CosyVoice 2 derived its discrete speech tokens by inserting a Finite Scalar Quantization (FSQ) module into the encoder of the SenseVoice-Large ASR model. CosyVoice 3 upgrades the base encoder to MinMo, a multimodal LLM trained on over 1.4 million hours of speech that achieves state-of-the-art results across spoken dialogue, multilingual ASR, and emotion recognition tasks. The intuition is that a richer pre-trained encoder will produce speech tokens that capture more fine-grained paralinguistic information — emotion, style, pronunciation nuance — leading to higher prosody naturalness in the synthesized output.

Multi-task Training

The FSQ module is inserted into the voice encoder of MinMo, specifically after the first 12 Transformer blocks (with rotary positional embeddings, RoPE). A 530,000-hour supervised dataset is used to fine-tune the tokenizer on five tasks simultaneously:

• Automatic Speech Recognition (ASR) — 365,000 h, covering Chinese, English, Japanese, Korean, Russian, French, and German
• Language Identification (LID) — 85,000 h
• Speech Emotion Recognition (SER) — 48,000 h
• Audio Event Detection (AED) — 21,000 h
• Speaker Analysis (SA) — 11,000 h

Quantization Mechanics

Let $H$ denote the intermediate representations produced by the first 12 Transformer blocks of the MinMo voice encoder. The FSQ module projects $H$ into a $D$-dimensional low-rank space and applies bounded rounding:

where each dimension is clamped and rounded to the integer set $[-K, K]$. During training, straight-through estimation approximates the gradients through the rounding operation. The discrete speech token $\mu_i$ at frame $i$ is computed as the index in a $(2K+1)$-ary positional system:

The effective codebook size is $(2K+1)^D$. The tokenizer operates at a fixed rate of 25 tokens per second. The complete tokenizer consists of: (1) the first 12 MinMo voice-encoder blocks, (2) the downward projector of FSQ, (3) the bounded rounding, and (4) the index calculation.

Upstream Recognition Performance

The FSQ-MinMo tokenizer (FSQ-MinMo) is benchmarked against several alternatives on ASR tasks:

FSQ-MinMo surpasses MinMo on Fluers CN because the tokenizer training focuses exclusively on speech tasks, sharpening its representations. On the AIR-Bench benchmark (LID, Gender, Age, Emotion, Vocal Sound, Sound Question), FSQ-MinMo performs comparably to MinMo — confirming that quantization does not destroy the model's broad audio-understanding capability, even though some acoustic tasks (Gender, Age) show modest drops.

Downstream TTS Performance of the Tokenizer

To isolate the tokenizer's contribution, the authors replace only the speech tokens while keeping the LM and CFM architectures identical and evaluate on SEED-TTS-Eval at two data scales (3,000 h and 170,000 h). Baselines include: SoundStream first-VQ (acoustic tokens), HuBERT (self-supervised), W2v-BERT 2.0 (self-supervised), CosyVoice 2.0, and CosyVoice 3.0 tokens.

Scaling from 3,000 to 170,000 hours delivers 63–75% relative WER/CER improvements. Scaling further to one million hours (CosyVoice 3 full training) yields additional gains that begin to plateau, confirming that the tokenizer is scalable but subject to diminishing returns. Supervised tokens consistently outperform acoustic tokens (SoundStream) and match or beat self-supervised tokens (HuBERT, W2v-BERT 2.0) in content consistency while maintaining competitive speaker similarity.

Reinforcement Learning via Differentiable Reward Optimization (DiffRO)

Problem Statement

Recent TTS systems have demonstrated the potential of RL to improve generation quality, but applying RL to LLM-based TTS is non-trivial. The standard pipeline requires downstream CFM and vocoder models to convert discrete tokens to audio before any reward can be computed — an expensive process. More critically, the resulting audio waveforms tend to be very similar to each other, making it hard to differentiate positive from negative feedback and thus to train a reliable reward model.

DiffRO: Bypassing Audio-level Rewards

DiffRO sidesteps the downstream rendering bottleneck by operating directly on the discrete token logits. The key insight is to train an ASR-like Token2Text model on ASR data and use its posterior probability as a differentiable reward signal. To make the reward differentiable with respect to the LLM parameters, DiffRO replaces the argmax token selection with a Gumbel-Softmax relaxation:

where $Y$ is the target text and $\mu_t$ is the ground-truth speech token at timestep $t$. The ASR reward is then:

This reward directly encourages the LLM to produce token sequences from which the target text can be accurately recovered, improving content consistency. The optimization objective combines reward maximization with a token-level KL divergence penalty:

where the KL divergence is computed on the output token-level logits (not sequence-level posteriors), giving finer-grained regularization:

where $Q = (2K+1)^{D-1}$ is the FSQ codebook size.

Multi-task Reward (MTR) Extension

DiffRO is further extended with a Multi-task Reward (MTR) mechanism. A single Token2Text model is augmented with additional downstream heads for SER, MOS prediction, AED, and other audio-understanding tasks. For a desired set of voice attributes $\{A_i\}_{i=1}^{K}$:

This allows the reward model to guide the LLM toward producing tokens that encode not just lexical content but also emotional expressiveness and other paralinguistic attributes. The paper reports a DiffRO-EMO variant that specifically uses SER as an additional reward task to boost emotion cloning capability.

Experimental Impact of DiffRO

DiffRO is applicable both to CosyVoice 2 and to CosyVoice 3, making it a general-purpose post-training technique for discrete-token TTS systems. On the SEED-TTS-Eval benchmark, RL post-training contributes 12–35% relative improvements in content consistency (WER/CER). The gains are especially pronounced in low-resource languages and cross-lingual scenarios, where over 50% relative WER improvement is observed in more than half of the evaluated conditions. CosyVoice 3-0.5B achieves a 68.7% relative Korean WER improvement with DiffRO. The trade-off is a slight reduction in speaker similarity, attributed to the model "hacking" the ASR reward at the expense of timbre fidelity — a known challenge for reward-oriented RL that the authors plan to address by adding a speaker similarity reward task.

Pronunciation Inpainting

LLM-based TTS systems using BPE tokenizers lack explicit controllability over pronunciation of polyphonic characters (words with multiple valid readings depending on context). CosyVoice 3 introduces Pronunciation Inpainting to address this. The vocabulary of the text tokenizer is expanded to include phoneme symbols. An auxiliary training set is constructed by replacing Chinese monophonic characters with their pinyin representations and English monophonic words with their CMU pronunciation-dictionary phonemes. This "mixed" input format — containing both standard graphemes and explicit phoneme annotations — is combined with the base training set.

Three variants are evaluated on a polyphonic test set:

• RepAll + MixPhn: Replace all characters/words (both polyphonic and monophonic) with phonemes. Achieves 69.2% (zh) and 72.7% (en) correction rates but introduces G2P prediction mismatches.
• RepMono + MixPhn: Replace only monophonic characters/words with phonemes. Achieves 100% correction rate for both Chinese and English.
• RepMono + CatPhn: Retain the character and concatenate its phoneme. Achieves 86.7% (zh) / 100% (en). The semantic redundancy complicates the model's prioritization of phoneme over grapheme.

The best strategy, RepMono + MixPhn, achieves perfect pronunciation correction on the evaluation set, demonstrating that mixing grapheme and phoneme inputs — restricted to unambiguous monophonic contexts during training — is the most effective approach for industrial-grade controllability.

Self-training for Text Normalization

Traditional TTS pipelines rely on hand-crafted rule-based text normalization (TN) modules to convert numerals, symbols, and special tokens into their spoken equivalents. CosyVoice 3 instead integrates TN capability directly into the end-to-end model through self-training. Three complementary strategies build an auxiliary dataset of (raw-text, audio) pairs:

1. Pass raw text through an internal rule-based TN module and synthesize audio with CosyVoice 2.
2. Prompt Qwen-Max to normalize the text and synthesize via CosyVoice 2.
3. Prompt Qwen-Max to perform inverse TN on text from existing text-audio pairs, recovering unnormalized raw text to pair with the original audio.

The resulting model handles raw input text directly — including non-standard symbols, digits, abbreviations — without a separate TN pre-processing stage, improving robustness and coverage.

Instructed Speech Generation

CosyVoice 3 expands the instruction-following training data from 1,500 hours (CosyVoice 2) to 5,000 hours, with over 100 distinct speaking-style categories including 64 emotions/personality traits, 14 role archetypes, 4 prosodic modifiers (fast, slow, loud, soft), and 12 Chinese dialects/accents plus 3 non-native English accents (Chinese, Indian, Russian).

The instruction interface supports two modes:

• Natural language instructions: A free-text description is prepended to the synthesis input, followed by a special end-of-prompt token <|endofprompt|>. Example: "You are Speaker A. Please talk to me happily."
• Fine-grained instructions: Inline markers embedded within the text, such as [laughter], [breath], and <strong>word</strong> for emphasis.

On the Expresso and internal expressive datasets, CosyVoice 3 achieves approximately 11% relative improvement in style similarity over CosyVoice 2. WER is higher on Expresso (where emotional speech deviates from the standard pronunciation distributions ASR models favor), while lower on the internal test set. The authors note that singing voice is not yet supported and remains future work. Timbre control via natural language is also identified as an underexplored direction.

Capability Transfer in Speaker Fine-tuning

Turning Monolingual Speakers into Polyglots

A key practical requirement is enabling a speaker fine-tuned (SFT) model — trained to reproduce a specific individual's voice — to speak languages that speaker never recorded. CosyVoice 3 achieves this through continual pretraining with an auxiliary dataset containing studio-quality monolingual speech from randomly selected speakers across all supported languages. Each utterance is annotated with speaker ID and language ID as a natural language instruction (e.g., "You are Speaker B. Please speak German."). The continual pretraining transfers multilingual and instruction-following capabilities from the zero-shot pretrained model to the fine-tuned model without catastrophic forgetting.

Results show CER/WER below 4% for Chinese, English, German, Spanish, French, Italian, and Russian. Japanese is more challenging (~9% CER) due to kanji-to-kana conversion errors and the many-to-one pronunciation mappings of Japanese characters. Korean achieves ~6% CER, limited by available data volume.

Transferring Instructed Generation Capability

The SFT data combines high-quality target-speaker recordings with pretraining instruction-following data. Instruction prompts specify both speaker identity and style. Missing fields are left blank, and during fine-tuning, speaker or style prompts are randomly masked, encouraging the model to generalize to arbitrary combinations and preventing catastrophic forgetting of pretraining capabilities.

The Multilingual Data Pipeline

Collecting large-scale, high-quality multilingual TTS data is substantially harder than for Chinese and English. CosyVoice 3 collects in-the-wild audio from Internet audiobooks, videos, and podcasts, then processes it through a six-stage pipeline:

1. Speech detection and segmentation: Sequential speaker diarization, voice activity detection, and audio event detection to obtain speaker-level segments shorter than 30 seconds.
2. Noise reduction: MossFormer2 model for denoising; utterances with abnormal leading/trailing frames (suggesting incomplete words due to bad cuts) are discarded; silence at edges is trimmed.
3. ASR transcription: Language ID via Faster-Whisper Large-V3; then transcription by three systems (Faster-Whisper Large-V3, NVIDIA NeMo Canary-1B, Meta FAIR SeamlessM4T-V2-large). Cross-validation selects pairs with average pairwise WER below 15%.
4. Punctuation adjustment: Montreal Forced Aligner derives inter-word/clause durations; punctuation is added if gaps $\geq 300$ ms and removed if gaps $\leq 50$ ms.
5. Volume standardization: $\text{normalized\_wav} = \frac{\text{raw\_wav}}{\max(\text{raw\_wav})} \times 0.6$
6. Length-ratio filtering: Speech-token to text-token ratios are computed; the bottom 1% and top 5% by ratio are discarded to remove degenerate samples (e.g., near-silent audio paired with long transcripts).

Dataset and Model Scaling

Dataset Scaling to One Million Hours

CosyVoice 3's training corpus reaches one million hours, compared to roughly 10,000 hours for CosyVoice 2. The expansion is multi-dimensional:

• Domain diversity: e-commerce, navigation, finance, education, conversations, speeches, singing.
• Style diversity: expressive speech including emotion, speed, dialects, accents, role-playing.
• Text format diversity: paired with both normalized and inverse-normalized text for the same audio (TN and ITN augmentation).
• Rare/edge cases: self-training with an early CosyVoice 3 checkpoint to synthesize hard cases not well covered by natural data.
• Language coverage: Chinese and English augmented with Japanese, Russian, French, German, Spanish, Korean, and Italian — 9 languages total (Spanish and Italian are unseen in the tokenizer training data but transfer from other Romance/European languages).
• Chinese dialect coverage: 19 Chinese regional accents/dialects.

Model Size Scaling

The text-to-speech language model (LM) is scaled from 0.5 B to 1.5 B parameters by using a larger LLM backbone. The conditional flow-matching (CFM) model adopts a diffusion transformer (DiT) backbone, growing from 100 M to 300 M parameters. Switching to DiT eliminates the need for a separate text encoder and a length-regularization module that were required in CosyVoice 2; the frame-rate mismatch between 25-Hz speech tokens and mel-spectrogram frames is resolved with a simple interpolation operation.

CV3-Eval: A New Multilingual Benchmark

Existing TTS evaluation benchmarks (e.g., LibriSpeech subsets) are drawn from clean audiobooks, are predominantly English or Chinese, and only measure pronunciation accuracy and audio quality — failing to capture expressiveness, dialect fidelity, cross-lingual transfer, and robustness to noisy prompts. To address this, the authors introduce CV3-Eval, a comprehensive multilingual evaluation benchmark.

Objective Evaluation Subsets

• Multilingual Voice Cloning: 9 languages (zh, en, ja, ko, de, fr, ru, it, es), 500 samples each, sourced from CommonVoice and FLEURS. Noisy backgrounds and long silences are intentionally retained to test robustness. Hard-case sub-sets include rare words, tongue twisters, and domain-specific terminology.
• Cross-lingual Voice Cloning: Source audio and target text from different languages, covering zh, en, ja, and ko. Tests language transfer without timbre loss.
• Emotion Cloning: Sourced from EmoBox and SeCap (zh and en, happy/sad/angry, 100 samples each). Divided into text-related (semantically consistent target text) and text-unrelated (semantically inconsistent target text) to distinguish text-driven from audio-prompt-driven emotion expression.

Subjective Evaluation Subsets

• Expressive Voice Cloning: High-expressivity prompts from news, podcasts, TV drama, academic talks, poetry, and public figures.
• Expressive Voice Continuation: 120 audio samples cut to 3-second prompts; models synthesize the continuation and are evaluated against ground truth for prosody/expressiveness alignment.
• Chinese Accent Voice Cloning: 18 Chinese dialects (Cantonese, Dongbei, Minnan, Shanghai, etc.) from in-house industrial data, evaluated subjectively due to the absence of reliable automated dialect-authenticity metrics.

Experimental Results

SEED-TTS-Eval: Zero-shot Chinese and English TTS

CosyVoice 3 is benchmarked against 10 competitive systems, covering both non-autoregressive (MaskGCT, E2 TTS, F5-TTS, F5R-TTS) and autoregressive (Seed-TTS, FireRedTTS, Qwen2.5-Omni, CosyVoice, CosyVoice 2, Spark TTS) baselines. Content consistency is measured by CER/WER (lower is better); speaker similarity (SS) is measured by both WavLM-based and ERes2Net-based cosine similarity (higher is better).

Key findings:

• Compared to CosyVoice 2, CosyVoice 3 achieves 44% relative CER reduction on test-zh and 51% relative WER reduction on test-en (base 0.5B model). With DiffRO, test-hard CER drops by 26% relative (from 6.83% to 5.09%).
• CosyVoice 3-1.5B + DiffRO achieves the lowest WER on test-en (1.45%) and the lowest CER on test-zh (0.71%) of all evaluated systems.
• Speaker similarity exceeds CosyVoice 2 and all non-Seed-TTS baselines. The remaining gap to Seed-TTS is attributed to differences in speaker diversity and pretraining data volume.
• The 1.5B model occasionally underperforms the 0.5B model on the hard set, attributed to insufficient hard-case data coverage for larger-model pretraining at the current data scale.

CV3-Eval: Multilingual Voice Cloning

CosyVoice 3 is the only system capable of covering all 9 languages in this subset. Competing open-source systems (F5-TTS, Spark-TTS, GPT-SoVits) only support Chinese and/or English. DiffRO brings large additional gains across all languages.

CV3-Eval: Cross-lingual Voice Cloning

CosyVoice 3 substantially outperforms CosyVoice 2 in cross-lingual scenarios. A notable CosyVoice 2 failure — transferring a Japanese voice prompt to Chinese output — is resolved in CosyVoice 3 by converting all Japanese text to kana before synthesis, eliminating the character overlap ambiguity between Japanese kanji and Chinese hanzi. Model size scaling also helps: the 1.5B variant consistently achieves lower WERs than the 0.5B variant across cross-lingual conditions, demonstrating that larger capacity is especially beneficial for challenging tasks.

CV3-Eval: Emotional Voice Cloning

All models perform significantly better on text-related than text-unrelated emotion cloning, revealing that current TTS systems predominantly infer emotional tone from text semantics rather than audio prompt acoustics. "Happy" is the easiest emotion to clone across all systems. DiffRO-EMO dramatically improves emotion accuracy on text-unrelated tasks (e.g., happy: 0.64 → 0.98) but at the cost of some pronunciation accuracy — illustrating the multi-objective tension inherent in reward-based training.

Subjective Evaluation: MOS

A 200-sentence MOS evaluation (10 native speakers, 5 male/5 female, scores 1–5 in 0.5-point steps) shows:

• Chinese: All three CosyVoice models score above 4.45, close to but slightly below human speech. The gap is traced to occasional unstable synthesis outputs.
• English: CosyVoice 2 falls below human MOS; CosyVoice 3-0.5B matches human MOS; CosyVoice 3-1.5B exceeds the human MOS score.
• Overall: CosyVoice 3-1.5B > CosyVoice 3-0.5B > CosyVoice 2, illustrating clear benefits of both data and model scaling.

Instructed Generation

CosyVoice 3 shows approximately 11% relative improvement in style similarity over CosyVoice 2. The elevated WER on Expresso (despite lower ground-truth ASR accuracy on that set) reflects ASR model bias against emotional pronunciation patterns. On the internal expressive dataset, CosyVoice 3 achieves lower WER than CosyVoice 2.

Ablation Studies

Tokenizer Ablation (Upstream and Downstream)

The upstream ablation (Table comparing SenseVoice, MinMo, VQ-SenseVoice, FSQ-SenseVoice, FSQ-MinMo on CommonVoice and FLEURS) confirms that FSQ-MinMo retains strong multilingual ASR capability and outperforms vanilla MinMo on the FLEURS CN set by focusing training on speech-only tasks. The downstream TTS ablation (comparing acoustic, self-supervised, and supervised tokenizers at 3K and 170K hours) confirms that supervised tokenizers are the best foundation for LM-based TTS, with CosyVoice 3's FSQ-MinMo tokens outperforming CosyVoice 2's FSQ-SenseVoice tokens at equivalent data scales.

DiffRO Ablation

DiffRO consistently yields 20–50% relative WER improvement across all tested models and languages. The gains are largest for low-resource languages and cross-lingual scenarios (e.g., 68.7% relative improvement in Korean for CosyVoice 3-0.5B). Speaker similarity sees a small decline under DiffRO, suggesting a reward-hacking effect where the model over-optimizes for ASR accuracy at the cost of timbre fidelity. DiffRO improves less on hard-sample test sets, as rare words and tongue twisters remain challenging for the Token2Text reward model. DiffRO-EMO (adding SER as a reward task) boosts emotion accuracy — especially for text-unrelated emotions — but can adversely affect WER, confirming the multi-objective balancing challenge.

Pronunciation Inpainting Ablation

The ablation (RepAll+MixPhn vs. RepMono+MixPhn vs. RepMono+CatPhn) demonstrates that replacing only unambiguous monophonic characters/words with phonemes (RepMono+MixPhn) is strictly superior, achieving 100% correction rate in both Chinese and English. Replacing all characters regardless of ambiguity (RepAll) introduces G2P prediction noise and reduces accuracy. Concatenating the phoneme with the retained character (CatPhn) makes it harder for the model to prioritize pronunciation over spelling.

Limitations

The authors explicitly acknowledge the following unresolved limitations of CosyVoice 3:

• Timbre control via text: CosyVoice 3 cannot adjust acoustic timbre (as opposed to style or emotion) through natural language instructions. This is a promising but underexplored direction, relevant for role-playing applications where users want to specify voice characteristics textually.
• Singing voice generation: The model does not produce high-quality singing. Improving this would require adding singing data to both the tokenizer and LM training stages.
• Hard-sample synthesis stability: Rare words, tongue twisters, and repeated-word sequences remain challenging, and DiffRO provides less benefit on these cases due to the difficulty of training reward models for them.
• Large-model data hunger: The 1.5B LM occasionally underperforms the 0.5B LM on hard subsets because the available dataset — especially for edge cases — is insufficient for the larger model. The authors plan to scale data to tens of millions of hours to address this.
• Japanese and Korean: Higher CER/WER for these languages in the polyglot SFT setting, due to kanji-to-kana conversion errors and limited training data, respectively.
• Speaker similarity vs. Seed-TTS: CosyVoice 3 does not yet match Seed-TTS in speaker similarity, attributable to differences in speaker diversity and the total pretraining data volume.
• Emotion from prompt vs. text: Current models primarily derive emotion from text sentiment rather than from the audio prompt, causing large performance drops in text-unrelated emotion cloning tasks.

Conclusion

CosyVoice 3 represents a comprehensive evolution of the CosyVoice family toward true in-the-wild speech synthesis. Its four principal innovations — a supervised multi-task speech tokenizer built on the MinMo audio LLM, the DiffRO differentiable RL post-training method, one-million-hour multilingual dataset scaling, and 0.5B-to-1.5B model size scaling with a DiT-based CFM backbone — combine to deliver state-of-the-art results across content consistency, speaker similarity, and prosody naturalness on both established benchmarks (SEED-TTS-Eval) and the newly introduced CV3-Eval benchmark. The work validates that both data and model scaling laws hold in the speech synthesis domain, that differentiable reward optimization is an effective and broadly applicable alternative to full RL loops for discrete-token TTS, and that supervised tokenizer design is a critical bottleneck for downstream synthesis quality. Future directions include further data scaling to tens of millions of hours, timbre control via natural language, singing voice synthesis, and improved handling of rare linguistic edge cases.