Bridging Synthetic and Real Speech for LLM-Based ASR
Today's digest examines how LLM-based ASR systems perceive synthetic speech and how to close the gap with real data — achieving real-data parity with just 25% genuine recordings through smart augmentation and pooling strategies.
Layer-wise Weighted Pooling inside of Llama architecture. All LLM hidden states ( ) are weighted by a trainable parameter in order to select how each layer is kept before, optionally, the addition to the residual stream from the speech. Once done, it's passing through RMS~Norm and to output the textual tokens of the transcripts. From Synthetic ASR.
Modern ASR pipelines increasingly rely on large language models as their backbone, but training data remains a bottleneck. Today's digest focuses on a timely question: can synthetic speech meaningfully substitute for real recordings, and if so, how?
ASR & Speech Synthesis
Making synthetic speech work for LLM-based ASR — closing the gap between generated and real audio without a full real-data budget.
A persistent challenge in scaling ASR is that LLMs are surprisingly sensitive to the acoustic provenance of their input. Understanding where in the model that sensitivity lives is the first step to engineering around it. This paper digs into exactly that question, probing layer-wise representations in LLM-based ASR to locate how and where models distinguish real from synthetic speech — then uses those findings to design a practical training recipe. The key ingredients are room impulse response (RIR) augmentation, which narrows the acoustic domain gap, and learned layer-wise pooling, which lets the model weight representations from different depths rather than committing to a fixed layer. The result: mixed-data systems that match real-data baselines using only 25% real speech, a substantial reduction in the costly data collection burden.
Synthetic ASR · 2026-06
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
This work probes where LLM-based ASR models distinguish real from synthetic speech, then leverages those insights to train effective mixed-data systems. It shows that room impulse response augmentation and learned layer-wise pooling can match real-data baselines using just 25% real speech.
Abstract
In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.
Takeaway: Knowing where a model notices synthetic artifacts is just as important as knowing that it does — and that diagnostic insight directly informs augmentation strategy.