Llasa

Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis

A unified speech synthesis system that replaces multi-stage TTS with a single Llama-aligned Transformer, systematically studying training-time and inference-time compute scaling. By applying scaling laws from text LLMs to speech generation, Llasa achieves improved prosody, voice cloning, and emotional expressiveness.

• tts
• llm
• voice-cloning
• prosody
• autoregressive
• streaming

Authors: Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, Hongzhan Lin, Jianyi Chen, Xingjian Du, Liumeng Xue, Yunlin Chen, Zhifei Li, Lei Xie, Qiuqiang Kong, Yike Guo, Wei Xue

Categories: eess.AS, cs.AI, cs.CL, cs.MM, cs.SD

Published 2025-02-06 · Updated 2025-02-22

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub661 stars24 watchers49 forksNOASSERTION
• Hugging Face595 downloads31 likes

Abstract

Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute. However, current state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring separate models (e.g., diffusion models after LLM), complicating the decision of whether to scale a particular model during training or testing. This work makes the following contributions: First, we explore the scaling of train-time and inference-time compute for speech synthesis. Second, we propose a simple framework Llasa for speech synthesis that employs a single-layer vector quantizer (VQ) codec and a single Transformer architecture to fully align with standard LLMs such as Llama. Our experiments reveal that scaling train-time compute for Llasa consistently improves the naturalness of synthesized speech and enables the generation of more complex and accurate prosody patterns. Furthermore, from the perspective of scaling inference-time compute, we employ speech understanding models as verifiers during the search, finding that scaling inference-time compute shifts the sampling modes toward the preferences of specific verifiers, thereby improving emotional expressiveness, timbre consistency, and content accuracy. In addition, we released the checkpoint and training code for our TTS model (1B, 3B, 8B) and codec model publicly available.

---

Introduction & Motivation

The text-domain large language model (LLM) revolution—exemplified by the GPT series and more recently by reasoning-focused models such as o1—has established two complementary scaling dimensions: training-time compute (larger models, more data) and inference-time compute (search, self-correction, tree search). Both dimensions have been shown to yield consistent performance gains in natural language tasks.

Text-to-speech (TTS) research has historically taken a different path: rather than converging on a common architecture and studying scaling laws, the field has pursued ever-more-elaborate model designs—multi-stage pipelines that combine an autoregressive (AR) language model with a non-autoregressive (NAR) residual stage or a diffusion model. While these systems achieve high quality, they make it difficult to study scaling cleanly: which component should be scaled? By contrast, the text LLM community settled on a single architecture (Transformer + tokenizer) and has since been free to explore training-time scaling laws, inference-time behaviors, fine-tuning, pruning, and quantization.

Llasa is motivated by closing this gap. The paper makes the following key contributions:

• A simple, unified TTS framework consisting of exactly one speech tokenizer (X-codec2) and one Transformer (initialized from Llama), fully aligned with standard LLM architecture.
• A systematic study of train-time compute scaling for TTS: both model-size scaling (1B → 3B → 8B parameters) and data scaling (80k → 160k → 250k hours).
• A systematic study of inference-time compute scaling for TTS using speech understanding models as verifiers in beam-search (PRM) and best-of-N (ORM) strategies.
• Full open-source release of codec models, TTS model checkpoints (1B, 3B, 8B), training code, and inference-time scaling code.

System Overview

The Llasa framework keeps exactly two components: a tokenizer and a single Transformer-based LLM. The Transformer parameters $\phi$ are initialized from an existing Llama checkpoint, and its text tokenizer is inherited directly. The central design challenge is converting raw speech waveforms into discrete 1-D sequences of tokens that can be modeled autoregressively—analogous to how word/subword tokens are modeled in text LLMs.

Formally, let:

• $\operatorname{Tokenize}_{\text{text}}(X) = \{x_1, \dots, x_T\}$ be the text tokenizer mapping input text $X$ to $T$ tokens.
• $\operatorname{Tokenize}_{\text{speech}}(Y) = \{y_1, \dots, y_S\}$ be the speech tokenizer mapping a waveform $Y$ to $S$ discrete tokens.
• $\operatorname{Detokenize}_{\text{speech}}(\{y_1, \dots, y_S\}) = \hat{Y}$ be the codec decoder reconstructing the waveform from tokens.

Each training pair $(X_i, Y_i)$ is represented as a single concatenated token sequence $(x_1, \dots, x_T, y_1, \dots, y_S)$. The model learns the conditional distribution of speech tokens given text tokens:

Training minimizes the negative log-likelihood over speech tokens only:

This is identical to the standard next-token prediction objective used in text LLMs, making Llasa a clean, architecture-aligned TTS system. No separate NAR stage, diffusion model, or vocoder is required at the generative modeling level—the codec decoder handles the final waveform reconstruction from the predicted discrete token sequence.

X-codec2: The Speech Tokenizer

The speech tokenizer, X-codec2, is the foundational piece that enables the single-Transformer architecture. Its central requirement is that all speech information—content, prosody, and timbre—must be captured in a single flat sequence of discrete tokens, with no auxiliary features needed at decoding time. This is in contrast to RVQ-based codecs that produce multiple parallel sequences (one per codebook layer) and require multi-step prediction strategies in the LM.

X-codec2 builds on X-codec and consists of three stages: Encoder → Vector Quantizer → Decoder.

Encoder

Two separate encoders are applied to the raw waveform $\mathbf{Y}$:

• Semantic Encoder $\operatorname{Enc}_s$: A pre-trained Wav2Vec2-BERT model (from SeamlessM4T) extracts multilingual semantic features capturing content and emotional cues.
• Acoustic Encoder $\operatorname{Enc}_a$: Multiple residual convolutional blocks with Snake activation functions (following BigCodec and DAC designs) encode low-level acoustic details including timbre.

Their outputs are concatenated into a fused embedding:

Vector Quantization

Rather than residual vector quantization (RVQ), X-codec2 uses Finite Scalar Quantization (FSQ):

FSQ is chosen for its training stability and high codebook usage efficiency. It does not require a codebook commitment loss term, simplifying training. The codebook size is 65,536 (substantially larger than standard 1024-entry codebooks used in prior work), with a projection dimension of 8 in the VQ module. The total downsampling ratio is $R = 320$ at 16 kHz, yielding a frame rate of 50 tokens/second—so a 2048-token context window corresponds to ~40 seconds of audio, which is directly interpretable from the LLM's perspective.

Decoder

Two reconstruction branches operate on $\mathbf{H}_q$:

• Semantic Reconstruction: A semantic decoder predicts the original Wav2Vec2-BERT features via $\ell_2$ loss. This component is only used during training to encourage the codebook to retain semantic information; it is discarded at inference time.
• Acoustic Reconstruction: Following Vocos, a Transformer-based decoder (replacing ConvNeXt) predicts STFT magnitude and phase, which are then converted back to waveforms via an inverse STFT (iSTFT) head.

Codec Training

Codec training simultaneously optimizes semantic and acoustic reconstruction with adversarial losses from:

• A multi-period discriminator (MPD)
• A multi-scale STFT (MS-STFT) discriminator
• A spectral discriminator with FFT sizes $\{78, 126, 206, 334, 542, 876, 1418, 2296\}$

A perceptual loss is additionally applied during the final 0.2 million of the total 1.4 million training steps to enhance intelligibility. The codec is trained on approximately 150k hours of multilingual speech from the Emilia dataset (En/Zh/De/Fr/Ja/Ko) and MLS (En/Fr/De/Nl/Es/It/Pt/Pl), all at 16 kHz. Training uses a learning rate of $1 \times 10^{-4}$ with a 3,000-step warmup; 6-second random crops are used during training.

Codec Evaluation Results

X-codec2 is evaluated on LibriSpeech test-clean (2,620 utterances at 16 kHz) using WER (HuBERT-based ASR), STOI, PESQ-WB, PESQ-NB, speaker similarity (WavLM-based SPK SIM), and UTMOS.

The table below summarizes key results. X-codec2 at token rate 50 outperforms all competing single-codebook codecs on most metrics:

Key observations from the codec comparison:

• At token rate 50 (single codebook), X-codec2 achieves best WER (2.47), STOI (0.92), PESQ-WB (2.43), PESQ-NB (3.04), and SPK SIM (0.82)—the best single-VQ codec at this token rate by a wide margin.
• Its UTMOS (4.13) closely matches the ground truth (4.09), indicating perceptually faithful reconstruction.
• A notable remaining gap exists relative to high-rate multi-layer codecs (e.g., DAC at 600 tokens achieves SPK SIM 0.95, PESQ-WB 4.01). Single-VQ codecs at low token rates inherently sacrifice some acoustic detail—this is acknowledged as a limitation that affects the SIM-O metric in TTS evaluation.
• Some models (e.g., Mimi, X-codec) achieve low WER at low token rates by incorporating semantic information into the codec, which helps intelligibility even if acoustic fidelity is lower.

Llasa TTS System

Training Details

All TTS models are trained for 3 epochs with a batch size of 2 million tokens and a peak learning rate of $5 \times 10^{-5}$ using a cosine schedule. Warmup covers 3% of an epoch; the final learning rate is 10% of the peak. Sequences are formed by concatenating text tokens (left) with speech tokens (right), then cropped to a maximum of 2,048 tokens.

The training corpus totals 250,000 hours of mixed Mandarin Chinese and English speech, integrating:

• Libriheavy
• Emilia corpus (Chinese-English subset)
• WenetSpeech4TTS
• Internal data

All textual content preserves original punctuation. Three model sizes are investigated: Llasa-1B (Llama 3.2 1B), Llasa-3B (Llama 3.2 3B), and Llasa-8B (Llama 3.1 8B).

Scaling Strategies

Two orthogonal scaling axes are studied:

• Fix data, vary model size: Llasa-1B-250k, Llasa-3B-250k, Llasa-8B-250k.
• Fix model size, vary data: Llasa-1B-80k (random 1/3 of full data), Llasa-1B-160k (random 2/3), Llasa-1B-250k (full dataset).

Scaling Train-Time Compute: Text Understanding Ability

A critical TTS capability is text understanding: the ability to go beyond mechanical phoneme rendering and produce speech that reflects the actual meaning of the text—correct emotion, intonation for questions, appropriate prosody for poetry, correct pronunciation of polyphonic or rare characters, etc.

Two evaluation protocols are used:

English: Seven categories (following BASE TTS): Questions, Emotions, Compound Nouns, Complex Syntax, Foreign Words, Paralinguistics, and Punctuation. Each sentence is synthesized five times without a speech prompt (random speaker/style). A linguistic expert rates each output on a 1–3 scale.

Chinese (novel contribution): Seven categories tailored to Chinese-specific challenges: Questions, Emotions, Paralinguistics, Chinese Poetry, Rare Characters, Polyphonic Words, and Tongue Twisters.

Key findings on text understanding:

• Both data and model scaling improve performance consistently across nearly all categories in both languages.
• Simple tasks (e.g., Questions) already reach high scores with smaller models, with only marginal gains from further scaling.
• Hard tasks requiring deep semantic understanding (Emotions in Chinese, Chinese Poetry, Tongue Twisters) benefit most from larger models (8B is substantially better than 1B).
• Data-driven tasks (Rare Characters in Chinese; Compound Nouns and Foreign Words in English) benefit primarily from more training data rather than model size, suggesting that lexical coverage and pronunciation correctness are learned from exposure to diverse examples.

Scaling Train-Time Compute: In-Context Learning Ability

In-context learning ability is evaluated as the model's zero-shot TTS capability: given a short speech prompt from an unseen speaker, can the model clone the speaker's voice, emotion, and style without any fine-tuning?

Three test sets are used:

• Seed-TTS-Eval: Three subsets (test-zh, test-en, test-hard). Metrics: CER/WER (Paraformer-zh for Chinese, Whisper-large-v3 for English) and speaker similarity SIM-O (WavLM-large fine-tuned on speaker verification). SIM-R is also reported (similarity after codec resynthesis).
• LibriSpeech test-clean: Continuation task (zero-shot speech synthesis). Metrics: WER-H (HuBERT-Large ASR), SIM-O, SIM-R.
• ESD (Emotional Speech Dataset): 10 English + 10 Chinese speakers, 5 emotion categories. The longest utterance per speaker/emotion is used as the prompt; the second longest as ground truth. 100 samples total (50 English, 50 Chinese). Metric: Emotion2Vec-Plus-Large emotion similarity.

Results on Seed-TTS-Eval (train-time scaling only, no inference search):

Consistent trends: both CER/WER and SIM-O improve monotonically as model size or data amount increases. However, SIM-O values for Llasa under direct inference remain below those of competing SOTA systems that use mel-based or RVQ-based decoders—a known limitation of single-VQ codec reconstruction at 50 tokens/second.

Results on LibriSpeech test-clean (continuation task):

A particularly important finding is that Llasa-8B-250k's SIM-R (0.626) is very close to the codec resynthesis ceiling (0.638). Since SIM-R is computed after passing predicted tokens through the codec decoder—eliminating codec distortion from the equation—this demonstrates that from a pure generative modeling perspective, a single Transformer is not inherently inferior to AR+NAR hybrid architectures. The gap in SIM-O is attributable to codec reconstruction quality, not to the generative model itself.

Emotion similarity results (ESD dataset):

Emotion similarity also improves monotonically with both model size and training data, showing that in-context emotion cloning benefits from scaling.

Scaling Inference-Time Compute

The inference-time scaling framework generates multiple speech candidates and selects the best one using speech understanding models as verifiers. The framework distinguishes between two types of reward models:

• Output Reward Models (ORMs): Evaluate the fully generated speech segment holistically. Simple and commonly used via a Best-of-N strategy: generate $N$ independent outputs, score each with a verifier, select the highest scorer.
• Process Reward Models (PRMs): Evaluate the generation step by step (e.g., every 0.5 seconds). Enable beam search: maintain $B$ beams, expand each to $N = 16$ new candidates, score at each step, prune back to $B$ beams, repeat until EOS or maximum length 2048. This provides fine-grained control but risks collapsing diversity and falling into local optima.

Verifiers that can be plugged in include:

• Speaker verification models (WavLM-finetuned) — for timbre/speaker similarity
• Emotion recognition models (Emotion2Vec) — for emotional expressiveness
• ASR models (Whisper Large v3) — for content accuracy / WER
• Speech quality metrics (UTMOS/SpeechMOS) — for naturalness
• Prosodic analyzers — for rhythm and intonation

Beam Search and Best-of-N Mechanics

For beam search: $B$ candidate sequences are maintained. At each step of $M = 25$ tokens (0.5 seconds), each beam is expanded into $N = 16$ new candidates, yielding $B \times N$ candidates. The top $B$ are retained by verifier score. This repeats until EOS or length 2048. For Best-of-N, $B \times N$ candidates are generated independently and the best is selected.

Findings from Inference-Time Scaling Experiments

All experiments use Llasa-1B-250k on the Seed-TTS-Eval test-hard subset. Key results:

• Best-of-N (ORM, speaker sim verifier) — orange line: SIM increases markedly and monotonically as more candidates are generated. Straightforward and effective.
• PRM beam search (speaker sim verifier) — blue line: Under the same compute budget ($B \times N$ total evaluations), PRM beam search achieves higher SIM than Best-of-N. More focused exploration pays off.
• PRM beam search with WER selection at end — red line: When trying to simultaneously optimize WER by selecting the lowest-WER candidate from the final $B \times N$ pool, WER is worse than the baseline (especially at large beam widths). The PRM's aggressive speaker similarity optimization reduces diversity, leaving no high-quality candidates in terms of WER.
• Partial PRM strategy (PRM first $n=2$ seconds, then ORM) — green line: Applying PRM only in the first portion of generation and switching to ORM for the remainder avoids the diversity collapse. SIM is higher than Best-of-N, and WER is near ground truth.
• Partial PRM (speaker sim) + ORM (WER verifier) — purple line: Replacing the ORM step with a WER-based verifier simultaneously improves both SIM and WER as compute scales up. This is the proposed best strategy.

Inference-Time Scaling Results

With inference-time scaling (partial PRM for speaker similarity + ORM for WER, beam width = 16), Llasa surpasses all baselines on Seed-TTS-Eval:

The test-hard WER of 3.12 (achieved via chunking long utterances and applying search) is particularly notable. This subset contains challenging long-form speech where previous models have struggled substantially. Allocating more inference compute proves especially beneficial for hard cases.

On LibriSpeech test-clean (continuation task, with partial PRM + ORM WER, beam width = 16):

Inference-time scaling dramatically improves SIM-O from ~0.48 (direct inference) to ~0.71 and WER-H from ~2.3 to 1.49, outperforming all reported baselines on both metrics simultaneously.

Emotion Similarity via Inference-Time Scaling

Using Emotion2Vec as the PRM verifier on ESD:

Emotion similarity reaches or exceeds 0.93 in English and 0.97 in Chinese across all model sizes when using emotion-based PRM—compared to 0.75–0.86 with direct inference alone. This shows that inference-time scaling can dramatically shift generation toward specific verifier preferences (here, emotion), essentially controlling expressiveness without any explicit conditioning mechanism or fine-tuning.

Cross-Model Inference-Time Scaling Observations

• Larger models generally benefit more from inference-time scaling—the gain in SIM-O from search is larger for 8B than for 1B models.
• For simpler tasks and metrics, the gap between small and large models after inference-time scaling narrows considerably, suggesting that for some use cases it may be more compute-efficient to train a smaller model and invest compute at inference time rather than always scaling training.
• This parallels findings in the text LLM domain and is presented as an open direction for TTS research.

Extending to Speech Understanding: ASR

To demonstrate the generality of the single Transformer + tokenizer framework, Llasa is adapted to ASR by simply swapping the token order: speech tokens come first (left), text tokens follow (right), and the cross-entropy loss is applied to text tokens only. The same X-codec2 tokenizer is used.

ASR models are trained on Libriheavy, MLS English, and GigaSpeech for 2 epochs with learning rate $2 \times 10^{-5}$, batch size 1M tokens, and 0.03 warmup ratio.

On test-clean, Llasa-ASR-3B (WER 1.9) is competitive with Whisper Large v3 (WER 1.8) despite using quantized discrete tokens rather than continuous Mel features. Performance on test-other is weaker (7.2 for 1B, 5.9 for 3B vs. 3.6 for Whisper Large v3), likely due to the smaller and cleaner training set and absence of data augmentation. These results confirm that fully discrete token-based ASR is viable and that the same Transformer architecture can be used bidirectionally for both TTS and ASR within the same framework.

Comparison with Prior TTS Baselines

Under direct inference (no search), Llasa achieves WER performance competitive with SOTA systems on both Seed-TTS-Eval and LibriSpeech, but its SIM-O is lower due to the single-VQ codec's reconstruction limitation. The key architectural trade-off is:

• Advantage: Simpler, scalable, unified framework. SIM-R (post-codec similarity) is very high, showing the generative model itself is competitive. Amenable to inference-time scaling which other multi-stage systems are not.
• Limitation: Single-VQ codec at 50 tokens/second cannot fully reconstruct acoustic detail that multi-layer RVQ or mel+vocoder pipelines achieve. SIM-O gap is codec-level, not generative-model-level.

Selected baselines on Seed-TTS-Eval include: Seed-TTS, MaskGCT, E2-TTS (32 NFE), F5-TTS (32 NFE), CosyVoice, CosyVoice 2, FireRedTTS. Selected baselines on LibriSpeech test-clean include: ELLA-V, VALL-E R, CLaM-TTS, VALL-E, VALL-E 2, Voicebox, MELLE.

Architecture Alignment Analysis

A central argument of the paper is that the SIM-R metric (which controls for codec quality by measuring similarity in the speaker embedding space after both ground truth and generated speech have been passed through the codec) isolates the quality of the generative model from the quality of the codec. The finding that Llasa-8B-250k achieves SIM-R = 0.626 on LibriSpeech test-clean—very close to the codec resynthesis ceiling of 0.638—provides strong evidence that:

From a pure generative modeling perspective, a single Transformer architecture is not inferior to carefully designed AR+NAR hybrid architectures. The remaining performance gap in SIM-O is a codec-level limitation, not a modeling limitation.

Discussion and Limitations

Acknowledged Limitations:

• Single-VQ codec reconstruction quality: At 50 tokens/second with a single codebook layer, acoustic reconstruction (especially timbre precision) is inherently limited compared to multi-layer RVQ codecs or mel+vocoder pipelines. This is reflected in lower SIM-O scores under direct inference compared to some baselines.
• Inference-time scaling fairness: Comparisons using search are not compute-matched with baselines that use single-pass inference. The paper acknowledges this but frames search-augmented inference as a legitimate strategy when quality is the primary objective regardless of cost.
• Diversity vs. quality trade-off in PRM: Pure PRM beam search with aggressive speaker-similarity reward collapses diversity, leading to poor WER. The partial PRM strategy mitigates this but introduces a hyperparameter (the $n=2$ second switching point) that may need tuning.
• ASR performance on noisy/accented speech: The ASR model underperforms on LibriSpeech test-other due to the training data being relatively clean, highlighting sensitivity to data distribution mismatch.

Broader Implications:

• The paper argues that TTS is at a similar inflection point to where text LLMs were before the field converged on a standard architecture—and that converging on a simple Transformer + tokenizer framework will unlock the study of TTS scaling laws, inference strategies, and downstream adaptations at the community level.
• Inference-time scaling via verifier-guided search offers a new control dimension: rather than conditioning on explicit emotion or speaker labels at training time, the model can be steered at inference time toward any attribute for which a differentiable or scorable verifier exists.
• The partial PRM + ORM strategy provides a practical recipe for simultaneous multi-objective optimization at inference time (e.g., jointly improving speaker similarity and content accuracy).

Summary of Key Results