Beyond Transcripts: Speech Grounding, Voice Privacy, and Cross-Lingual Prosody
Today's digest probes what gets lost — or hidden — when speech is reduced to text: benchmarks exposing transcript bias in speech LLMs, a voice cloning model repurposed for anonymization, and the first fine-grained cross-lingual prosody analysis for expressive translation.
Structured acoustic evidence used by the agentic-style reasoner. Audio Twin converts speech into time-aligned, LLM-compatible evidence entries that expose paralinguistic and interactional cues relevant to the question. From Johns Hopkins University.
A recurring theme in today's papers is the gap between what is said and what is heard. Speech carries information that transcripts strip away — paralinguistic cues, prosodic contours, and speaker identity — and this digest covers four papers that each grapple with that gap from a different angle: benchmarking LLMs' speech grounding, stress-testing input validation, repurposing voice cloning for privacy, and quantifying how prosody survives cross-lingual translation.
SpeechLLMs & Spoken Dialogue
When models read speech instead of listening to it — and when that goes wrong.
A persistent blind spot in dialogue-capable speech LLMs is transcript bias: models that can simply read the ASR output will exploit that shortcut, ignoring paralinguistic signals that contradict the words. Johns Hopkins University surfaces this failure mode head-on, constructing the ContraTalk benchmark around deliberate text–audio disagreements. Their "Audio Twin" framework treats paralinguistic signals as explicit evidence that reasoning models must consult, enabling rigorous measurement of whether a model is actually grounded in the audio or just reading along.
When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
This paper identifies transcript shortcuts as a failure mode in dialogue understanding and proposes ContraTalk, a benchmark for text–audio disagreements. An "Audio Twin" framework exposes paralinguistic signals as evidence for reasoning models, enabling rigorous evaluation of speech grounding versus transcript bias.
Abstract
Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.
On the input-validation side, speech LLMs are typically evaluated on what they say — not on whether they should say anything at all. Speech AI Research's SURE-Challenge fills that gap by benchmarking a model's ability to abstain when the audio evidence is insufficient (noisy, ambiguous, or simply absent). Crucially, they show that a straightforward rule combining an energy signal with an ASR confidence score substantially boosts rejection of bad inputs without hurting accuracy on valid queries — a practical, deployable baseline that exposes just how much low-hanging fruit remains in robustness evaluation.
SURE-Challenge: Evaluating Speech Evidence Before Speech-LLM Generation
This paper introduces a benchmark for evaluating speech LLMs' ability to reject unsupported inputs before generation. It demonstrates that a simple energy-plus-ASR-score rule substantially improves rejection of noise and ambiguous audio while preserving accuracy on valid queries.
Abstract
Speech LLMs are usually graded after they answer, although an operating system first has to decide whether to send a waveform to the model. We define the Speech-Unsupported Rejection Evaluation Challenge (SURE-Challenge) for this admission step. The benchmark pairs LibriSpeech-derived transcription and first-word question answering with unsupported silence, colored noise, synthetic tones, and source-ambiguous babble under disjoint source splits. Front-end ablations use Qwen2-Audio; the selected energy-plus-Whisper-score rule is then replayed before six speech/audio LLMs. On the leakage-screened 474-example SURE-Extended test set, raw Qwen2-Audio rejects 15/204 unsupported inputs, whereas the fixed rule rejects 196/204 and leaves supported accuracy unchanged. External evaluations qualify this result: Common Voice retention drops as the Whisper-score threshold is tightened, and no-speed babble gives 18 to 24 rejected clips out of 54 across regenerated seeds. The result identifies a pre-generation error mode missed by answer-only scoring.
TTS & Voice Synthesis
Finding privacy in a voice cloning model.
Voice cloning and speaker anonymization are usually treated as opposing goals, but the University of Zurich flips that intuition: XTTSv2, a multilingual voice cloning model, turns out to be a surprisingly strong anonymizer when you decouple prosody from speaker identity. Their iterative refinement strategy progressively suppresses identity-linked features while preserving speech quality, outperforming dedicated anonymization baselines across seven languages on both privacy and fidelity metrics. It's a compelling demonstration that large TTS systems already contain the machinery needed for privacy — they just haven't been pointed that direction.
Your Voice Cloning System is Secretly a Voice Anonymizer
This work repurposes XTTSv2, a multilingual voice cloning model, for speaker anonymization by decoupling prosody from speaker identity. An iterative refinement strategy balances privacy and speech quality across seven languages, achieving stronger anonymization and better fidelity than dedicated baselines.
Abstract
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER $\approx$ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Prosody & Expressive Speech-to-Speech
What pitch, energy, and rhythm look like when they cross a language boundary.
Expressive speech-to-speech translation demands that the feel of an utterance survive the language switch, but there's been little systematic evidence of how much prosody actually transfers. The University of Edinburgh delivers the first fine-grained cross-lingual prosody analysis, examining pitch, energy, and temporal patterns across language pairs using multilingual dubbing data. The findings pinpoint which linguistic factors erode prosodic similarity and offer concrete guidance for engineers building expressive S2ST systems — grounding design decisions in empirical measurement rather than intuition.
Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages
The first fine-grained cross-lingual analysis of prosody, examining how pitch, energy, and temporal patterns transfer across languages using multilingual dubbing data. The findings reveal linguistic factors affecting prosody transfer and provide guidance for expressive speech-to-speech translation systems.
Abstract
Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.
Trending on Hugging Face
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.