PilotTTS

PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

A lightweight autoregressive TTS system that achieves competitive performance through meticulous data engineering rather than scale or architectural novelty. Using only 200K hours of openly processed data, PilotTTS matches systems trained on orders of magnitude more proprietary data while remaining fully reproducible.

• tts
• voice-cloning
• prosody
• emotion
• autoregressive
• asr

Demos

PilotTTS demonstrates competitive speech synthesis using a lightweight modular architecture trained on only 200K hours of data. Evaluate the demos for synthesis quality, speaker similarity fidelity, and the system's versatility across zero-shot voice cloning, 11 emotional categories, paralinguistic features (laugh, cough, cry, breath), and 13 Chinese dialects. Key metrics: WER/CER accuracy, speaker similarity preservation, and naturalness across diverse speaking styles and linguistic variations.

Authors: Bowen Li, Shaotong Guo, Zhen Wang, Yang Xiang, Mingli Jin, Yihang Lin, Jiahui Zhao, Weibo Xiong, Dongrui Zhang, Keming Chen, Yunze Gao, Zeyang Lin, Yuze Zhou, Yue Liu

Categories: cs.SD, cs.AI

Published 2026-05-26 · Updated 2026-05-27

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub199 stars2 watchers24 forksApache-2.0
• Hugging Face0 downloads7 likes

Abstract

Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.

---

Introduction and Motivation

Text-to-speech (TTS) synthesis has advanced dramatically with the rise of large-scale generative models. Zero-shot voice cloning — generating speech in an arbitrary speaker's voice from only a few seconds of reference audio — is now achievable by large language model (LLM)-based systems and non-autoregressive flow-based systems alike. Yet this progress has come at a steep cost: state-of-the-art systems increasingly rely on millions of hours of proprietary speech data, multi-codebook tokenizers, hierarchical prediction modules, and collections of specialized sub-models for features like emotion and dialect synthesis.

This trajectory creates serious barriers for resource-constrained research and engineering teams. First, acquiring and processing millions of hours of audio requires proprietary infrastructure and computational resources that few teams can access. Second, the architectural complexity of modern systems — residual vector quantization, streaming-specific designs, multi-stage training — inflates engineering effort and deployment difficulty. Third, advanced controllability features (emotion, paralinguistics, dialect) are typically developed as entirely separate systems, further fragmenting expertise and resources. The cumulative effect is that most teams cannot build, extend, or refine competitive TTS systems on their own.

PilotTTS is presented as a pragmatic response to these challenges. Rather than pursuing architectural novelty or data scale, it achieves competitive performance by combining well-established open-source components with meticulous data engineering. The system is trained on approximately 200,000 hours of speech data — processed entirely with publicly available tools — and demonstrates that disciplined integration and rigorous data curation can match or outperform systems trained on an order of magnitude more data.

Contributions at a Glance

• Reproducible data processing pipeline. A multi-stage pipeline built entirely from publicly available tools, covering quality assessment, label annotation, and quality filtering. Any team can obtain the constituent modules and assemble a comparable pipeline.
• Compact autoregressive architecture with decoupled conditioning. A Q-Former-based conditioning mechanism paired with cross-sample paired training that disentangles static speaker identity from dynamic speaking style. This design achieves the highest speaker similarity on the Seed-TTS Eval benchmark (SIM 0.862 on test-zh, 0.815 on test-en) using only 200K hours of data.
• Multi-dimensional controllability under low-resource constraints. Within the same framework, PilotTTS supports zero-shot voice cloning, emotion synthesis across 11 categories, paralinguistic synthesis across 4 categories (plus a combined mode), and Chinese dialect synthesis across 14 dialects — all via targeted post-training.

Data Processing Pipeline

The pipeline operates in three sequential stages, transforming raw, heterogeneous internet audio into a clean, richly annotated training corpus. A distinguishing property is that every constituent module is publicly available, making the pipeline fully reproducible.

Stage 1: Quality Assessment and Enhancement

All recordings are first standardized to a unified format and sampling rate to reduce source-level heterogeneity. Speech activity detection (SAD) and speaker change detection (SCD) modules then segment the audio into valid speech regions and identify speaker transitions, using pyannote-based tools.

Three quality signals are estimated in parallel for each segment:

• DNSMOS — a neural MOS predictor that estimates perceptual speech quality. Segments with predicted MOS ≤ 3.5 are flagged as acoustically deficient.
• Speech/non-speech classification — a classifier (SenseVoiceSmall) distinguishes speech from music, environmental noise, or other non-speech events.
• Signal-to-noise ratio (SNR) estimation — segments with insufficient SNR are flagged.

Segments identified as low quality are then passed through a denoising and enhancement module (Resemble Enhance) to suppress background noise and improve speech intelligibility. This dual assessment-then-enhancement design ensures cleaner inputs for downstream annotation.

Stage 2: Label Annotation

Each processed segment is annotated with textual, temporal, speaker, and prosodic information. Multiple ASR systems — Paraformer, FireRedASR, Whisper, and internal models — transcribe each segment, and cross-system consistency checking is used to improve label reliability.

An overlapping speech detection (OSD) module identifies segments with simultaneous multi-speaker speech. A forced alignment module establishes temporal correspondence between text and audio, and a prosody annotation system built on Qwen3-Force-Alignment annotates hierarchical prosodic structures. Speaker tagging is performed using the 3D-Speaker-Toolkit to assign speaker identity metadata, enabling the construction of speaker-consistent training samples. Spectral rolloff analysis is also applied to identify low-bandwidth recordings with insufficient high-frequency content.

Stage 3: Quality Filtering

A truncation detector removes utterances with incomplete beginnings or endings, which are common artefacts of imprecise segmentation. A synthetic speech detector flags segments likely to be artificially generated rather than naturally spoken — an important check when crawling large-scale internet data.

A final filtering module aggregates all available quality signals and annotations — acoustic quality, speech validity, transcription reliability, overlap status, speaker consistency, truncation risk, synthesis likelihood, and spectral quality — and retains only samples that satisfy all criteria. Importantly, excluded samples are preserved with their quality tags, enabling flexible dataset construction for future tasks with different quality requirements.

After filtering, approximately 200,000 hours of Chinese and English speech data are retained for pre-training.

Model Architecture and Training Strategy

PilotTTS consists of four components: a speech tokenizer, an autoregressive text-to-semantic module, a Conditional Flow Matching (CFM) decoder, and a HiFi-GAN vocoder. Each component builds on established open-source modules; the system's competitiveness derives from careful integration and data curation rather than novel architectural design.

Speech Tokenizer

PilotTTS directly adopts the single-codebook speech tokenizer of CosyVoice 3, based on Finite Scalar Quantization (FSQ). Unlike multi-codebook residual vector quantization (RVQ) approaches that require hierarchical prediction modules, FSQ offers a favorable trade-off between simplicity and information density.

In FSQ, an intermediate representation $H$ is projected into a $D$-dimensional low-rank space and each dimension is independently quantized into $[-K, K]$ via bounded rounding:

$$\tilde{H} = \operatorname{ROUND}(\operatorname{Proj}_{\text{down}}(H))$$ $$\hat{H} = \operatorname{Proj}_{\text{up}}(\tilde{H})$$

This yields a discrete codebook of size $(2K+1)^D = 6{,}561$. The token index $\mu_i$ for each frame is computed as:

$$\mu_i = \sum_{j=0}^{D-1} \tilde{h}_{i,j} \cdot (2K+1)^j$$

where $\tilde{h}_{i,j}$ is the $j$-th quantized dimension of frame $i$. The tokenizer operates at 25 Hz, producing one discrete token per 40 ms of speech.

The CosyVoice 3 tokenizer was trained with multi-task supervision across five objectives: automatic speech recognition (ASR), language identification (LID), speech emotion recognition (SER), audio event detection (AED), and speaker analysis (SA). This diverse supervision encourages tokens to encode not only linguistic content but also paralinguistic attributes such as emotion, speaker characteristics, and acoustic events. PilotTTS reuses this pretrained tokenizer without modification.

Autoregressive Text-to-Semantic Module

Decoupled Speaker and Style Encoding

A central design question is how to extract speaker-related information from reference audio. Two established paradigms exist:

• Audio token continuation — concatenates prompt tokens with target text tokens and autoregressively predicts target audio tokens. It leverages fine-grained acoustic cues for high-fidelity cloning, but degrades with noisy or short prompts, and long prompts substantially increase inference cost.
• Speaker embedding — compresses the reference into a global fixed-dimensional vector, improving robustness but inevitably discarding timbral details and failing to capture dynamic style features such as speaking rate and prosodic contours.

PilotTTS addresses both limitations through a dual-pathway conditioning design:

• Style pathway (Q-Former conditioner): A frozen w2v-BERT 2.0 encoder extracts continuous acoustic features from the reference speech. A set of 32 learnable query vectors attends to these features through cross-attention in a Q-Former architecture, compressing arbitrary-length reference speech into a fixed set of 32 condition tokens at constant computational cost. Continuous features are preferred over discrete tokens for stronger noise robustness.
• Identity pathway (CAMPPlus speaker encoder): A frozen CAMPPlus encoder extracts a global speaker embedding optimized for speaker discrimination. Since it already carries speaker identity, this pathway allows the Q-Former to focus on dynamic speaking style rather than static timbre.

The two pathways thus assume complementary roles: Q-Former tokens encode dynamic speaking habits and expressive style, while the CAMPPlus embedding captures static timbre identity.

Autoregressive Generation and Input Sequence

Text-to-semantic prediction is formulated as a conditional autoregressive task. The input sequence to the Qwen3-0.6B language model is:

$$\mathbf{x} = [\mathbf{s},\; \mathbf{c},\; e_{\text{BT}},\; |\text{lang}|,\; |\text{emo}|,\; \mathbf{e}_{\text{Text}},\; e_{\text{ET}},\; e_{\text{BA}},\; \mathbf{e}_{\text{Audio}},\; e_{\text{EA}}]$$

where:

• $\mathbf{s}$ — speaker embedding from CAMPPlus
• $\mathbf{c} = \{\mathbf{c}_i\}_{i=1}^{32}$ — style condition tokens from the Q-Former conditioner
• $e_{\text{BT}} / e_{\text{ET}}$ and $e_{\text{BA}} / e_{\text{EA}}$ — boundary markers for the text and audio regions, respectively
• $|\text{lang}|$ and $|\text{emo}|$ — control tags specifying target language and emotion
• $\mathbf{e}_{\text{Text}} / \mathbf{e}_{\text{Audio}}$ — text and audio token embeddings

The model autoregressively predicts the audio token sequence:

$$p(\mathbf{e}_{\text{Audio}} \mid \mathbf{x}_{<\text{Audio}}) = \prod_{i=1}^{N_s} p(\mathbf{e}_{\text{Audio},i} \mid \mathbf{x}_{<\text{Audio}},\; \mathbf{e}_{\text{Audio},

Cross-Sample Paired Training Strategy

To disentangle speaker-related attributes from linguistic content, PilotTTS employs a cross-sample paired training strategy: for each training sample, a different utterance from the same speaker is used as the reference for extracting both the speaker embedding $\mathbf{s}$ and the style condition $\mathbf{c}$. Since the reference and target utterances differ in content and acoustic realization, the model is forced to encode only speaker attributes that generalize across utterances rather than memorizing local acoustic patterns.

This strategy confers two benefits: (1) it suppresses overfitting to reference content, forcing text-independent speaker encoding; and (2) it promotes functional specialization — CAMPPlus focuses on stable timbre while Q-Former tokens capture dynamic style. This disentanglement also serves as the foundation for downstream emotion control and dialect synthesis post-training.

Emotion Control

The Q-Former conditioner captures rich speaker-related attributes from reference audio, including emotional characteristics, and the Qwen3 backbone contributes implicit emotion inference from textual context. However, implicit control lacks precision and stability. PilotTTS therefore introduces explicit emotion control through post-training on approximately 2,200 hours of emotion-labeled data (1,000 hours of high-quality data and 1,200 hours of augmented data from open-source datasets, internal annotations, and model-augmented sources).

The unified label system supports:

• Primary categories (7): happy, sad, angry, fear, contempt, serious, surprise
• Extended categories (4): concern, blue (melancholy), disgust, psychology (inner monologue)

Paralinguistic Generation

PilotTTS supports four paralinguistic phenomena — laughter (LAUGH), breathing (BREATH), crying (CRY), and coughing (COUGH) — plus a wrapped laughter mode (LAUGH_SPAN) in which laughter is temporally coupled with speech throughout an utterance. The system operates in two modes:

• Implicit mode: Infers appropriate paralinguistic behavior from textual context, producing varied laughter types (restrained chuckle, soft giggle, hearty laughter) depending on the situation.
• Explicit mode: Users specify behaviors through textual onomatopoeia.

Both modes support zero-shot synthesis with arbitrary speaker prompts. These capabilities are realized through supervised fine-tuning on approximately 200 hours of paralinguistic data from the data processing pipeline outputs and internal collections.

Dialect Synthesis

Dialect synthesis generates target-dialect speech using Mandarin audio as the speaker prompt and a dialect tag ($|\text{lang}|$) to specify the variety. The central challenge is data scarcity: dialect recordings are far less abundant than Mandarin.

PilotTTS exploits an observation that after bilingual pre-training, the model generates Mandarin robustly even when the prompt contains dialectal features. This enables construction of parallel data: three Mandarin utterances per dialect speaker are synthesized using the pretrained model, yielding large-scale "dialect–Mandarin" pairs that alleviate data sparsity.

During fine-tuning on approximately 16,000 hours of data covering 14 Chinese dialects (all derived from dialectal ASR corpora), mixed-prompt sampling is employed: the target is always a dialect utterance, while the conditioning prompt is drawn with equal probability from either a Mandarin or dialect utterance of the same speaker. This forces the model to extract speaker identity from stylistically diverse prompts and map it to the target dialect rather than copying prompt style directly.

Speech Decoder

The speech decoder converts discrete semantic codes into mel spectrograms using Conditional Flow Matching (CFM) with a Diffusion Transformer (DiT) backbone of 300M parameters. The conditioning input is:

$$[\mathbf{M}_{\text{ref}},\; \mathbf{s},\; \mathbf{e}_{\text{Audio}}^{\text{tgt}}]$$

comprising the reference mel spectrogram, the CAMPPlus speaker embedding, and the predicted semantic features. Through 10-step iterative denoising, the decoder progressively transforms a noise sample into the target mel spectrogram conditioned on these inputs. The generated mel spectrogram is then converted to a waveform by HiFi-GAN.

Experiments

Experimental Setup

Training data: Pre-training uses approximately 200,000 hours of Chinese and English speech from publicly available sources, processed as described above. Post-training data is organized by capability: emotion (~2,200 hrs), paralinguistic (~200 hrs), and dialect (~16,000 hrs across 14 Chinese dialects).

Model: The autoregressive module is built upon Qwen3-0.6B. The CFM decoder contains approximately 300M parameters.

Evaluation metrics:

• Zero-shot synthesis: Character Error Rate (CER) for Chinese (using Paraformer-zh) and Word Error Rate (WER) for English (using Whisper), plus Speaker Similarity (SIM) via cosine similarity of speaker embeddings.
• Emotion, paralinguistic, and dialect: Human evaluation with task-specific success criteria.

Baselines (zero-shot): Seed-TTS, F5-TTS, FireRedTTS-2, CosyVoice-3-0.5B, VoxCPM-0.5B, Qwen3-TTS-25Hz-0.6B, MiniMax-Speech, VibeVoice-1.5B — spanning diverse architectures and training scales. Comparisons focus on systems with comparable model scale (~0.6B) evaluated in their base zero-shot setting, alongside larger systems for broader context.

Zero-Shot Speech Generation Results

PilotTTS achieves the highest speaker similarity on both test sets (0.862 on test-zh, 0.815 on test-en), substantially outperforming all baselines with reported SIM scores. The improvement over the second-best system (Seed-TTS) is +0.066 on test-zh and +0.053 on test-en. On content accuracy, PilotTTS attains a CER of 0.87% on test-zh — just 0.04% behind MiniMax-Speech — and achieves the lowest WER of 1.50% on test-en.

These results are particularly notable because PilotTTS uses only ~200K hours, substantially less than several competing systems that leverage larger-scale proprietary corpora (e.g., MiniMax-Speech). The paper attributes this data efficiency to the rigorous data pipeline and the decoupled dual-pathway conditioning design.

Emotion Control Results

Emotion control is evaluated using 51 speaker prompts (15 expressive voices from anime/film characters, 36 ordinary speakers). Human evaluators listen to three utterances per test case: the original prompt, a neutral synthesis, and an emotion-controlled synthesis. A sample is counted as successful only when both conditions hold: (1) speaker timbre is consistent with the original prompt, and (2) the target emotion is clearly recognizable.

PilotTTS achieves the highest average success rate on primary emotions (88.1%), surpassing CosyVoice 3 (83.8%). On the overall average across all eleven categories, PilotTTS leads at 85.7%. IndexTTS supports only seven of the eleven categories and exhibits excessive timbre variation under emotion conditioning, lowering its success rates under the joint evaluation criterion.

PilotTTS achieves the highest speaker similarity both without emotion control (0.8101) and with emotion control (0.7329), and exhibits the smallest drop between the two conditions among all systems. This indicates that the decoupled conditioning design effectively modulates emotional expressiveness while preserving speaker timbre — a key advantage of the cross-sample paired training strategy.

Paralinguistic Synthesis Results

The test set covers five categories: LAUGH, BREATH, COUGH, LAUGH_SPAN, and CRY. For each category, 21 distinct speaker prompts are used for zero-shot synthesis. Human evaluators judge whether the target paralinguistic behavior is successfully generated. PilotTTS is compared against CosyVoice 3 and Fish-Speech S2 on the three commonly supported categories (LAUGH, COUGH, BREATH).

PilotTTS achieves the best overall success rate (85.1%) across the three common categories, outperforming CosyVoice 3 (80.4%) and Fish-Speech S2 (64.3%). It substantially leads on LAUGH (97.6% vs. 83.3% for CosyVoice 3). For BREATH, CosyVoice 3 achieves the highest rate (95.2%) while PilotTTS reaches 81.0%. COUGH is the most difficult for all systems (~60%), attributable to high acoustic variability and limited training data for cough events.

PilotTTS uniquely supports LAUGH_SPAN (94.6%) and CRY (61.9%), neither of which is available in the baseline systems. LAUGH_SPAN demonstrates the model's ability to maintain coherent speech while simultaneously generating natural laughter throughout an utterance.

Dialect Synthesis Results

Three evaluation scenarios of increasing difficulty are defined:

1. Same-Dialect: Reference audio and target speech share the same dialect.
2. Mandarin-to-Dialect: Standard Mandarin audio serves as speaker prompt to generate dialect speech.
3. Cross-Dialect: Reference audio is in dialect A while the target output is in dialect B.

Dialect control accuracy is measured by human evaluation; a synthesized sample is counted as a failure if the proportion of non-target-dialect pronunciation exceeds 10%.

PilotTTS achieves 91.8% in the Same-Dialect scenario, 86.46% in the Mandarin-to-Dialect setting, and 85.38% in Cross-Dialect synthesis. These results validate the parallel data construction and mixed-prompt sampling strategy as effective means to alleviate dialect data scarcity and enable robust cross-lingual style transfer.

Ablation Study on Conditioning Components

To assess the individual contribution of each conditioning component, an ablation study is conducted on a cleaned 60K-hour subset of the full 200K-hour dataset. Three settings are compared, each trained for 200K optimization steps:

• Full: Both the CAMPPlus speaker embedding ($\mathbf{s}$) and the Q-Former condition tokens ($\mathbf{c}$).
• w/o spk: Removes $\mathbf{s}$, keeps $\mathbf{c}$.
• w/o both: Removes both, leaving only text input.

Finding 1: Q-Former condition tokens are indispensable for content accuracy

Removing the Q-Former condition tokens (the "w/o both" setting) leads to substantial degradation in pronunciation accuracy across all three test sets. The effect is most pronounced on the hard-case subset (test-hc), where CER rises from 7.83% to 10.62% — a relative increase of approximately 35%. On test-zh and test-en, content errors also increase meaningfully (1.13% → 1.41% and 1.94% → 2.71%). These results confirm that fine-grained content- and prosody-related cues in the Q-Former condition tokens are essential for stable autoregressive generation, particularly under challenging acoustic conditions.

Finding 2: CAMPPlus speaker embedding plays a complementary role

The effect of the speaker embedding is more nuanced. Under a fixed training budget, removing $\mathbf{s}$ slightly reduces content errors on test-zh and test-en (as the model allocates more capacity to condition tokens), though this gap narrows with more training. In contrast, the speaker embedding consistently improves speaker similarity across all test sets, with the largest gain on the hard-case subset (0.8355 → 0.8470). Beyond enriching timbre information, $\mathbf{s}$ encourages the Q-Former to specialize in timbre-independent prosodic and stylistic cues, consistent with stronger robustness on the hard-case subset. The full dual-pathway design strikes the best balance across content accuracy, speaker fidelity, and robustness.

Conclusion and Limitations

PilotTTS demonstrates that a lightweight, modular system built entirely from publicly available components can achieve competitive — and in some metrics, state-of-the-art — performance on challenging benchmarks, using only 200K hours of training data. The key enablers are a rigorous, reproducible data processing pipeline and a Q-Former-based decoupled conditioning mechanism that disentangles speaker identity from speaking style. The same unified framework supports zero-shot voice cloning, emotion control (11 categories), paralinguistic synthesis (5 modes), and dialect synthesis (14 Chinese dialects) through targeted post-training.

The authors identify three concrete limitations and corresponding future directions:

1. Insufficient explicit style modeling. The current architecture relies on the Q-Former conditioner to implicitly capture style-related factors without a dedicated, high-capacity style modeling module. This limits the granularity of expressive detail in synthesis. Future work will develop a representation module for joint modeling of global and local (fine-grained) style.
2. Single-codebook quantization ceiling. The FSQ tokenizer is architecturally simpler than multi-codebook RVQ or continuous latent representations, but has a lower information-capacity ceiling. This makes it more challenging to extend the model to complex scenarios such as singing synthesis or background music generation.
3. Lossy mel-spectrogram reconstruction. The decoding pipeline relies on mel spectrograms and a separate HiFi-GAN vocoder — an indirect reconstruction process that may introduce distortion compared to end-to-end waveform generation. Future work will explore end-to-end waveform generation architectures.

# Zero-shot voice cloning
python inference.py \
    --checkpoint pretrained_models/pilot_tts.pt \
    --prompt-wav assert/prompt.wav \
    --text "synthesis text" \
    --output output/clone.wav

# Emotion synthesis (instruct model)
python inference.py \
    --config configs/infer_pilot_tts_instruct.yaml \
    --checkpoint pretrained_models/pilot_tts_instruct.pt \
    --prompt-wav assert/prompt.wav \
    --text "some text" \
    --emotion happy \
    --output output/emotion.wav