Synthetic L2 Speech for Automated English Proficiency Assessment
Today's digest focuses on a Cambridge study using TTS and voice cloning to synthesize second-language learner speech, showing that proficiency-matched synthetic data significantly boosts automated speaking assessment models.
Real$_ Spoken$ vs Matched$_ CEFR$+Speechify predictions. From University of Cambridge.
Today's digest is a focused look at a clever application of TTS technology to a real-world scarcity problem: the chronic shortage of labeled second-language (L2) learner speech data for automated assessment systems.
TTS & Voice Synthesis
Using synthetic speech to close the L2 data gap
Automated speaking assessment for language learners depends on large pools of annotated learner audio — data that is expensive, slow, and ethically complex to collect at scale. A new paper from the University of Cambridge tackles this head-on by generating synthetic L2 speech with modern TTS and voice cloning pipelines, purpose-built to mimic the non-native characteristics that assessment models need to handle.
The key methodological contributions are two-fold. First, the authors introduce "speechification" — a text transformation step that converts written exam prompts into spoken-style transcripts, capturing the disfluencies, contractions, and informal structures typical of real spoken L2 output, before feeding text into the TTS engine. Second, they demonstrate that proficiency-level-matched speaker pairing — selecting voice cloning targets whose L2 proficiency aligns with the target synthetic sample's intended level — is critical: mismatched pairing produces synthetic speech that fails to carry the right prosodic and fluency signals, while matched pairing yields data that measurably moves the needle on downstream model performance.
Data Augmentation for L2 English Speaking Assessment using TTS
This paper synthesizes L2 learner speech for automated assessment via TTS and voice cloning, using "speechification" to transform written text into spoken-style transcripts. Proficiency-level-matched speaker pairing yields synthetic speech that significantly improves assessment model performance.
Abstract
Automated assessment of second language (L2) speaking proficiency relies on large-scale annotated speech data, which remains scarce compared to widely available written learner corpora. A promising direction for addressing this imbalance is to use text-to-speech (TTS) and voice cloning to convert written L2 production into synthetic speech. However, written and spoken L2 differ fundamentally: spontaneous speech includes disfluencies and discourse markers, while writing is more planned and complex. This raises the question of what is required to generate synthetic L2 speech suitable for assessment. We address this through a systematic analysis of speaker-text relationships using COREFL, a publicly available corpus containing paired spoken and written responses from the same L2 learners to the same questions across modalities. In our proposed framework, we first address the structural differences between written and spoken language by transforming written responses into spoken-style transcripts ("speechification") using a large language model. These transcripts are then converted into speech using a TTS/voice-cloning model. To assign a voice to each synthetic response, we investigate different speaker-text pairing strategies based on shared learner attributes (proficiency level, first language, both, or neither). We evaluate our data augmentation techniques on the language assessment task, with improvements shown in both wav2vec2 (audio-based) and ModernBERT (text-based) scoring systems. Results show that matching speakers and texts by proficiency level yields the most robust synthetic speech. Moreover, raw written text leads to a strong mismatch with spoken language, while speechification substantially reduces this gap and improves grading performance.
The result is a data augmentation recipe that significantly improves automated L2 English speaking assessment model performance — a meaningful step toward more accessible, scalable language testing infrastructure built on synthetic speech.
Trending on Hugging Face
Vidu S1: A Real-Time Interactive Video Generation Model
Vidu S1 is a real-time interactive video generation model that supports voice-controlled digital character animation with infinite-length output and high frame rate on consumer hardware.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.