Synthetic ASR

How to Leverage Synthetic Speech for LLM-Based ASR Systems?

This work probes where LLM-based ASR models distinguish real from synthetic speech, then leverages those insights to train effective mixed-data systems. It shows that room impulse response augmentation and learned layer-wise pooling can match real-data baselines using just 25% real speech.

• asr
• tts
• llm
• prosody

Demos

This paper investigates using synthetic speech to train LLM-based ASR systems. ELOQUENCE demonstrates this across four pilots: smart home AI with Telefónica, bias detection with CNR, virtual agents with Omilia, and emergency call center AI with UNS. Evaluate demos for synthesis quality, ASR robustness with synthetic training data, multilingual support, bias mitigation, and deployment readiness in safety-critical contexts prioritizing privacy and European values.

Authors: Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso, Séverin Baroudi, Shashi Kumar, Esaú Villatoro-Tello, Srikanth Madikeri, Manjunath K E, Oldřich Plchot, Kadri Hacioğlu, Petr Motlicek, Andreas Stolcke

Categories: cs.CL, cs.AI

Comment: Submitted to SLT 2026

Published 2026-06-27 · Updated 2026-06-27

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar0 citations

Abstract

In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.

---

Introduction and Motivation

Speech data in regulated domains such as banking and healthcare is extraordinarily difficult to collect and retain at scale. Customer–agent telephone calls contain personally identifiable and financial information; data-protection regimes such as the GDPR constrain how such recordings may be stored, shared, and reused for model training, since voice recordings potentially qualify as biometric data. Synthetic speech from modern text-to-speech (TTS) systems offers an appealing alternative — utterances can be generated on demand without exposing real customers, sidestepping the collection and retention of sensitive audio.

However, the promise of TTS-as-a-substitute only holds if synthetic speech can actually replace real recordings at training time. A persistent distributional gap between synthetic and real speech limits how far this substitution can go: modern LLM-based ASR models can still separate real from synthetic signals with high accuracy, reducing the effectiveness of naive mixing strategies. Prior work largely treats this gap as a black box to be engineered around — through data filtering, weight merging, or closed-loop ASR/TTS refinement — without asking where inside the model the distinction is actually encoded.

This paper takes a fundamentally different approach: rather than bypassing the gap, the authors open it up. They probe a SLAM-ASR architecture layer by layer to localise where its LLM backbone separates real from synthetic speech, identify which signal-level perturbations most effectively disrupt that separation, and then translate those findings into practical training recipes. The study is organized around four concrete research questions:

• RQ1 – Where is real/synthetic discrimination encoded? Which layers of the LLM backbone most strongly separate the two classes, and which signal-level perturbations disrupt that separation most?
• RQ2 – Do interpretability-guided filters improve ASR? Does disrupting the real/synthetic boundary at the representation level translate to downstream word error rate (WER) gains?
• RQ3 – How much real data can synthetic replace? What real/synthetic mix preserves or improves over an all-real baseline?
• RQ4 – How much synthetic data helps on top of full real data? Does raw or RIR-augmented synthetic audio provide gains when added to a complete real corpus?

Background and Related Work

Speech LLMs and LLM-Based ASR

Coupling a speech encoder to a pre-trained large language model (LLM) through a trainable projector has become the dominant recipe for general-purpose audio understanding. Systems such as SALMONN, Qwen-Audio, and AudioPaLM connect self-supervised or supervised encoders to a frozen or lightly adapted text LLM. SLAM-ASR, a key reference for this paper, demonstrates that a single trainable projector between a frozen WavLM-Large encoder and a frozen LLM is sufficient for competitive ASR performance. While substantial work has been done on the linguistic performance of these architectures, how the LLM backbone encodes non-linguistic properties — such as whether speech is real or synthetic — remains largely unexamined.

Training ASR with Synthetic Speech and Its Limits

TTS-generated speech is widely used to augment low-resource ASR. Modern multi-speaker TTS has made this a practical foundation, yet two obstacles remain under-examined. First, performance gains are often dependent on the amount of data generated rather than the diversity of voices or acoustic conditions. Second, a systematic acoustic mismatch separates synthetic from real speech: open-source TTS outputs tend to be cleaner and more uniform than real recordings, so a model trained on them learns cues specific to the TTS system and transfers poorly to real audio.

Recent methods reduce this mismatch through weight merging of models trained separately on real and synthetic data (task arithmetic), by filtering and optimising the generation pipeline, or by letting ASR and TTS models refine one another in a closed loop. All of these act on the data, the TTS model, or the trained weights — treating the model's internal training dynamics as a black box, and none ask where inside the model the synthetic/real distinction is actually represented.

Interpretability of Speech Models

The anti-spoofing community (ASVspoof and related work) has shown that synthetic speech carries systematic, learnable signatures in pitch, prosody, and spectral envelope, making it reliably detectable. Layer-wise probing of speech encoders has revealed a progression from acoustic to semantic features across encoder layers. Learnable weighted sums over encoder layers (popularized by SUPERB) are now standard practice. However, attention has only recently turned to the LLM decoder backbone in SpeechLLM systems, and existing work has focused on the modality gap (accuracy drop between text and speech inputs) rather than acoustic distinctions within speech itself. To the authors' knowledge, no prior work examines how a speech LLM internally represents the synthetic/real gap, nor uses such an analysis to inform how synthetic data is exploited for training.

Model Architecture: SLAM-ASR with Layer-Wise Weighted Pooling

Base SLAM-ASR Architecture

The foundation is the SLAM-ASR framework, which couples a frozen WavLM-Large speech encoder to a Llama-3.2-3B-Instruct backbone through a single-hidden-layer projector that downsamples audio representations by a factor of 5. Domain adaptation uses LoRA adapters ($r=16$, $\alpha=32$, dropout $= 0.05$) applied to the q_proj and v_proj modules of the LLM, while all other parameters remain frozen.

For the layer-wise interpretability analysis, representations are read directly from all 28 Llama transformer layers at the base checkpoint (projector trained, LLM frozen, before any domain fine-tuning), so the analysis reflects generic acoustic encoding rather than domain-specific adaptation.

Layer-Wise Weighted Pooling (LWP) Module

The standard SLAM-ASR decoder feeds only the final LLM transformer layer's hidden states to the language modeling head. The authors replace this with a Layer-Wise Weighted Pooling (LWP) module that learns, per token and per utterance, a softmax-weighted combination of all LLM layers:

where $\mathbf{s} \in \mathbb{R}^D$ is a zero-initialized score vector — the only new trainable parameter introduced. Zero initialization provides a neutral prior: at the start of training, the softmax produces a uniform $1/L$ mixture over all layers (equivalent to a simple average), without biasing toward any particular depth. An additional speech residual optionally re-injects the projected encoder output at speech-token positions after pooling, preserving a direct acoustic information stream to the LM head.

Experimental Setup

Dataset: DefinedAI Banking Corpus

All experiments use DefinedAI, a corpus of manually transcribed English customer–agent telephone calls. Key statistics:

• The base model checkpoint is pre-trained on a mixed-domain set of approximately 38 hours (spanning banking, insurance, and healthcare), achieving a starting WER of 10.90% on the held-out banking test set.
• The banking test set consists of 3,164 utterances (6.55 hours).
• The banking training partition used for domain fine-tuning contains 26,457 utterances totalling 54.43 hours of real speech.
• The all-real baseline (100% real speech fine-tuning) achieves 8.68% WER.

Synthetic Speech Generation

Synthetic utterances are produced with Qwen3-TTS VoiceDesign, conditioned on a natural language voice prompt rather than reference speaker audio. Qwen3-TTS was selected after preliminary comparative listening against several open-source alternatives (CosyVoice, XTTS, Parler-TTS, IndexTTS2, OmniVoice, Chatterbox), based on highest perceived naturalness, fewest artefacts, natural prosody, and the most controllable voice characteristics. This controllability allows matching synthetic voices to real data persona metadata (gender, race) without any reference recordings.

A per-role prompt is built from persona attributes and fixed style tokens ("clear articulation, naturally"), yielding a distinct synthetic voice per speaker role in every dialog. The synthetic corpus totals approximately 51 hours (including RIR-augmented versions), generated from the same transcripts as the real banking training data.

Speaker Diversity Analysis

Despite using no real speaker recordings, VoiceDesign yields a synthetic corpus with higher within-corpus acoustic diversity than the real data. Pairwise cosine distances between pyannote speaker embedding vectors show the synthetic corpus (dominant mode ≈ 0.88) is shifted substantially right of real (dominant mode ≈ 0.68). The synthetic distribution is also bimodal — a secondary mode (≈ 0.6) overlaps with real, so synthetic keeps a real-like core while adding more diverse vocal characteristics, a direct consequence of generating a distinct voice per speaker role in each dialogue.

Room Impulse Response (RIR) Augmentation

Convolving clean speech with measured room impulse responses (RIRs) is an established robustness augmentation technique. For synthetic speech specifically, RIR convolution injects the room reverberation and channel variability absent from pristine TTS recordings, masking their clean-condition signature and narrowing the synthetic/real gap. The BUT Speech@FIT Reverb Database is used throughout.

Audio quality metrics (Table 1 in the paper) reveal a key insight:

Raw synthetic speech scores far higher than real on UTMOS (4.36 vs. 2.08), confirming Qwen3-TTS produces perceptually cleaner output. RIR convolution sharply degrades naturalness (UTMOS drops to 1.34) while only marginally changing PESQ (1.26 vs. 1.12). This is a critical finding: the ASR gains from RIR augmentation come not from improving audio quality, but from making synthetic speech acoustically messier — reproducing the acoustic irregularities of real telephone recordings and bridging the domain gap rather than the perceptual one.

Audio Filters Studied

Six signal-level perturbations are applied to synthetic audio:

For downstream ASR experiments, four combined conditions are additionally evaluated: High Pass or Low Pass, each paired with either Pitch Shift or Time Stretch.

Training Protocol

Training proceeds in two stages:

• Stage 1 (projector pre-training): Trains only the speech encoder projector on the full mixed-domain partition (38.17 hours). The LLM backbone and WavLM encoder are frozen.
• Stage 2 (domain adaptation): Loads the Stage 1 projector and fine-tunes LoRA adapters on banking-specific data for each experimental condition. Only LoRA weights (and the LWP score vector, where applicable) are trained.

Shared hyperparameters: AdamW optimizer, learning rate $1 \times 10^{-4}$, linear warmup over 1,000 steps followed by linear decay to 0, batch size 10, gradient accumulation 1, BF16 precision, random seed 42. All experiments run on a single NVIDIA H100 GPU for 5 epochs.

Where is Real/Synthetic Discrimination Encoded? (Layer-Wise Analysis)

Overlap Metrics

The authors quantify the synthetic/real gap at each of the 28 Llama layers using four complementary metrics:

• Silhouette score (full representation space): measures class separation; lower values mean more overlap between real and synthetic distributions.
• Silhouette score (PCA-2D projection): same metric after projecting to 2D via PCA.
• Wasserstein-1 distance on PC1: normalized by the pooled PC1 standard deviation for scale-invariance; lower values indicate more distributional overlap along the first principal component.
• PCA-2D KDE overlap coefficient: fits a kernel density estimate per class in 2D PCA space and integrates $\min(p_\text{real}, p_\text{synth})$ over the space; higher values indicate greater overlap and less separation.

Key Findings from Layer-Wise Analysis

Table 2 in the paper reports PCA-2D KDE overlap coefficients per filter at three representative Llama layers (3, 14, 28):

Four key observations emerge:

1. Discrimination decreases monotonically with depth. Silhouette scores converge near zero by layer 28 across all filters. Overlap coefficients reach 0.75–0.82 at layer 28, confirming that the final LLM layers collapse the synthetic/real gap regardless of perturbation. The discriminative signal is concentrated in early-to-middle layers (0–14).
2. High Pass is the least effective filter. Across all metrics and layers, High Pass maintains the highest silhouette scores — spectral attenuation of low frequencies alone is insufficient to confuse the decoder.
3. Time Stretch achieves the broadest early-layer disruption. It produces the lowest silhouette at layer ≈ 3, most effectively increasing overlap in early representations. Pitch Shift and Band Pass achieve similar disruption at middle layers (layer ≈ 14).
4. Frequency-band filters match prosodic ones. Band Pass and Low Pass reduce discrimination nearly as well as Pitch Shift and Time Stretch in the mid-layer range, suggesting that restricting frequency content targets the same underlying cues as prosodic modification.

These observations guide filter selection for downstream experiments: Time Stretch and Pitch Shift are prioritised individually, and their combination with High Pass tests whether spectral and prosodic perturbations are complementary at the ASR level even when High Pass alone fails.

Downstream ASR Experiments

How Much Real Data Can Be Substituted by Synthetic?

Two experimental regimes are studied at a constant total training budget:

• Fixed-budget substitution: varies the real/synthetic split in 10% increments (real + synth ≈ 100%).
• Fixed-synth augmentation: keeps the full synthetic set (100%) and progressively adds real data fractions.

The relationship between synthetic proportion and WER is non-monotonic. Only the 90/10 (8.46%) and 70/30 (8.45%) mixtures with RIRs improve over the all-real baseline (8.68%). Mixtures exceeding 40% synthetic data degrade performance, reaching 9.31% at equal proportions. Even in the extreme 10%-real / 90%-synthetic regime, however, the resulting system (10.26% with RIRs) still outperforms the unadapted base model (10.90%), indicating that RIR-augmented synthetic data remains beneficial under severe data scarcity. The authors conclude that a modest synthetic proportion of 10–30% with RIR augmentation is optimal for fixed-budget substitution.

How Much Synthetic Data Can Augment Real Speech?

In the augmentation setting, the full real corpus (100%) is kept fixed and synthetic data is progressively added:

All RIR-augmented conditions outperform the real-only baseline. Notably, even 10% synthetic with RIRs (8.43%) outperforms any quantity of raw synthetic data. Raw synthetic data shows diminishing returns beyond 50%, while RIR-augmented data consistently improves with scale up to 100% synthetic. This confirms that acoustic environment diversity — not sheer data volume — is the driving factor behind the gains.

Do Interpretability-Guided Filters Improve ASR?

The layer-wise analysis flagged Time Stretch, Pitch Shift, and Band Pass as effective at reducing synthetic/real separation, while High Pass alone was not. This section tests whether this translates to ASR gains on a 50% real / 50% synthetic training split:

Several nuanced findings emerge:

• Pitch Shift gives the largest individual gain ($-0.23$% WER without RIRs, $-0.40$% with), matching the interpretability finding that pitch is a key early-layer cue.
• Time Stretch without RIRs slightly hurts ($+0.15$% WER), suggesting temporal distortion introduces mismatches that outweigh its layer-level masking benefit.
• Layer 28 overlap does not predict ASR benefit — all conditions converge there regardless of their effect at earlier layers, confirming the interpretability signal lives in early-to-middle layers, not the final one.
• Low Pass + Time Stretch gives the best no-RIR result (8.68%), matching the all-real baseline without any acoustic augmentation. However, it benefits little from RIRs, suggesting low-pass filtering and reverberation target overlapping cues.
• High Pass + Pitch Shift is the best condition with RIRs (8.58%): retaining high-frequency content is more compatible with reverberation than suppressing it, indicating some filters act on cues not captured by the silhouette analysis alone.

The key takeaway is that representation-level separability helps but does not directly predict downstream ASR gains — the relationship between internal discrimination and WER is not straightforward.

Layer-Wise Weighted Pooling: Low-Resource Results

Substitution Setting

The LWP module is evaluated in the substitution scenario (full synthetic corpus + varying fractions of real data):

Without RIRs, LWP helps only at low real-data fractions. The score vector $\mathbf{s}$ learns useful layer preferences from as little as 13.6h (25% real, $-0.48$% WER) but lacks enough signal at 10% (5.4h). With RIRs, LWP is far stronger: at 25% real data, LWP + RIRs reaches 8.70%, nearly matching the 100% real baseline (8.68%), and every setting with $\geq 25$% real data surpasses the all-real baseline. The best result overall is 8.01% at 100% real + 100% synthetic with RIRs (a 7.72% relative improvement over the all-real 8.68% baseline).

The authors attribute the strong LWP + RIRs interaction to RIRs narrowing the acoustic gap, which allows the score vector $\mathbf{s}$ to learn stable layer preferences that transfer to the real test domain.

Residual Ablation

Ablating the speech-token residual (the optional re-injection of projected encoder output at speech token positions) has no meaningful impact at any real data fraction ($\leq 0.01$% absolute WER change). The pooling mechanism itself captures the relevant acoustic information, and the residual stream is redundant: WavLM features already enter via the projector before the LLM layers and are reachable into the initial layers of the LLM.

Augmentation Setting with LWP

In the augmentation setting (100% real + varying synthetic), LWP without RIRs mostly hurts. With 54h of real data, the decoder already has well-calibrated last-layer representations, so the pooling weights only add noise. Adding RIRs recovers small gains at 10–50% synthetic (e.g., 8.73% → 8.37% at 10%) but degrades slightly at $\geq 75$% synthetic, where the processed synthetic utterances dominate and disrupt score learning.

LWP therefore helps most in low-resource, substitution-heavy settings, not in augmentation where abundant real data already anchors the decoder.

Layer Weight Analysis

The mean softmax weights learned by LWP for the 25%-real / 100%-synthetic condition, evaluated on both real and synthetic test sets, are strikingly concentrated on layer 28:

Speech tokens receive mean weight 0.977 (synthetic test) and 0.930 (real test) on layer 28, far above the uniform $1/28 \approx 0.036$, with entropy as low as 0.161 nats (vs. 3.332 uniform), confirming near-deterministic selection of the final layer. Text tokens are less peaked (0.652/0.643; ~1.303 nats), spreading residual weight near-uniformly over layers 1–27.

This result is complementary to, but distinct from, the interpretability findings. Section 4 showed that early-to-middle layers (0–14) are most discriminative between real and synthetic speech. The LWP weights show the final layer is most useful for decoding: by layer 28, the model has refined acoustic input into a linguistically rich form optimal for transcription, regardless of origin. Early layers encode the domain gap; the final layer encodes what matters for the output task. The near-identical weight profiles on both test sets confirm LWP learns domain-agnostic preferences, explaining why it generalises once RIRs reduce the acoustic gap.

Synthesis: Why RIRs Work and What the Interpretability Tells Us

The paper draws a coherent picture from the combination of interpretability analysis and downstream experiments:

• The synthetic/real gap is an early-layer phenomenon. Discrimination concentrates in layers 0–14 of the Llama backbone. The final layer (28) collapses the gap regardless of perturbation — this is where the model performs linguistically useful computation, not domain discrimination.
• RIR convolution bridges the domain gap, not the perceptual gap. UTMOS drops from 4.36 to 1.34 after RIR convolution — the audio sounds worse perceptually. Yet PESQ changes only marginally (1.12 → 1.26). The mechanism is not quality improvement but acoustic irregularity reproduction: RIRs inject the room reverberation and channel artifacts characteristic of real telephone recordings, masking the pristine TTS signature that early layers detect.
• Representation-level separability does not directly predict ASR gains. Filters that most disrupt early-layer discrimination (Time Stretch, Pitch Shift) do not uniformly yield the best WER. The relationship is mediated by how well the filter interacts with RIR augmentation and the full training distribution.
• LWP acts as a soft layer selector that learns to rely on the final layer. This is the right layer for decoding, but getting there requires that the acoustic gap (present in early layers) is already reduced — which is precisely what RIR augmentation achieves. Together, LWP + RIRs provides a compounding benefit in low-resource settings.

Limitations

The authors are explicit about several important limitations:

• Single domain and corpus: All experiments use a single banking-domain corpus (DefinedAI). Generalization to other domains (insurance, healthcare), languages, and conversational styles remains unvalidated.
• Single TTS system: All synthetic speech is generated with Qwen3-TTS VoiceDesign. The findings — particularly which filters are most effective — are likely tied to the specific artefacts of this system. Different TTS systems with different spectral characteristics may show different patterns.
• Single ASR architecture: The probing is specific to the WavLM-Large + Llama-3.2-3B architecture. Architectures with different spectral front-ends (e.g., Whisper's log-mel encoder, filterbank-based systems) may exhibit different layer-wise discrimination patterns and respond differently to the same perturbations.
• Single-seed experiments: Given the large number of experimental conditions, multi-seed significance testing was computationally prohibitive. All results are reported for a single run (seed 42) and should be interpreted with appropriate caution.

Conclusions and Future Directions

This paper presents one of the first interpretability-driven analyses of how a Speech LLM internally encodes the real-vs-synthetic speech distinction, and demonstrates that the resulting insights yield practical improvements in ASR with synthetic data augmentation.

Concretely, the best system (combining interpretability-guided filters, specifically Pitch Shift or High Pass + Pitch Shift, with RIR augmentation) reaches 8.01% WER on the banking test set — a 7.72% relative improvement (0.67% absolute) over the all-real baseline of 8.68%. Adding layer-wise weighted pooling allows a substitution-oriented system to match the all-real baseline with only 25% real data (13.6h) and surpass it at every higher real-data fraction.

Future directions identified by the authors include:

• Scaling findings to larger synthetic mixtures, additional TTS systems, alternative encoders (including Whisper-based architectures), and multilingual settings.
• Moving beyond supervised fine-tuning to leverage synthetic data through reinforcement learning — optimising the ASR policy directly against WER using methods such as Group Relative Policy Optimization (GRPO), which could exploit large pools of synthetic and RIR-augmented utterances to extract further signal from synthetic data.