Multilingual ASR Distillation and Melody-Preserving Singing Voice Editing
Today's digest covers two focused advances: a multi-teacher RL distillation strategy that resolves cross-lingual conflicts in LLM-based ASR, and a continuous-latent autoregressive model for editing lyrics in singing voice synthesis without disrupting melody.
generates 64-D continuous AudioVAE latent patches every 100~ms. Target lyrics and reference-melody tokens persist in the causal planner's cache. Each patch updates semantic feedback and becomes the previous latent patch for the next step; the concatenated sequence is decoded once. Repeated Flow-DiT blocks share parameters. From Tsinghua University.
Today brings two tightly scoped but technically rich contributions — one pushing the frontier of multilingual speech recognition with smarter knowledge distillation, and another tackling the surprisingly tricky problem of editing lyrics in a synthesized singing voice without mangling the tune.
ASR & Spoken Language Models
Resolving cross-lingual interference in LLM-based speech recognition through divide-and-conquer distillation.
Multilingual LLM-based ASR systems routinely suffer from cross-lingual interference: optimizing a single model across many languages forces uncomfortable parameter-sharing trade-offs. The Speech Research Lab attacks this with a language-specialized multi-teacher on-policy distillation framework. The recipe is two-stage: first, a set of language-expert teachers are fine-tuned independently via reinforcement learning so each one can focus entirely on its target language; then a generalist student is trained by routing each utterance to its corresponding teacher and distilling from that specialist signal. Notably, the student achieves emergent performance gains beyond any individual teacher, suggesting the multi-teacher setup creates positive cross-lingual transfer rather than merely inheriting the ceiling of each specialist.
Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR
This paper addresses cross-lingual conflicts in multilingual LLM-based ASR by training language-specialized teachers via RL and distilling them into a generalist student through language routing and multi-teacher distillation. The approach achieves emergent performance gains beyond individual teachers.
Abstract
Modern LLM-based ASR systems have established multilingual capability as a standard feature, leveraging large-scale multilingual corpora and LLMs' cross-lingual knowledge to achieve competitive performance across multilingual benchmarks. However, jointly modeling languages with heterogeneous acoustic, phonological, and lexical characteristics inevitably introduces optimization conflicts, undermining language-wise specialization. To address this challenge, we propose Language-Specialized Multi-Teacher On-Policy Distillation (LS-MOPD), which decouples language-specific knowledge acquisition from multilingual capability integration: language-specialized teachers are independently optimized via reinforcement learning (RL), with their expertise then integrated into a generalist multilingual student through language routing and token-level multi-teacher distillation, thereby reducing direct cross-lingual optimization conflicts. We further explore static and dynamic acoustic-prefix configurations to examine how teacher-student prefix consistency influences the efficacy of on-policy distillation. Experiments on benchmarks covering Mandarin, Mandarin subdialects, Cantonese, and English demonstrate that LS-MOPD substantially outperforms RL baselines and surpasses the empirical performance envelope defined by the best-performing RL teachers on nearly all benchmarks, revealing its potential to generalize beyond all teachers in multilingual ASR.
TTS & Voice Synthesis
Continuous-latent autoregression with State-Control-Transition routing for artifact-free lyric editing in singing.
Lyric editing in singing voice synthesis is harder than it first appears: because autoregressive models condition each step on generation history, replacing a lyric segment risks propagating source-lyric artifacts forward through the entire sequence. Tsinghua University's CLASVS (Continuous-Latent Autoregression for Singing Voice Synthesis) addresses this head-on. The model operates in a continuous latent space and uses a State-Control-Transition (SCT) routing mechanism to ensure that revised lyrics cleanly override source-lyric cues at the appropriate generation states, preventing contamination of the subsequent output while faithfully preserving the original melody contour. This is a meaningful step toward practical, fine-grained post-hoc editing of generated or synthesized song recordings.
CLASVS: Continuous-Latent Autoregression for Melody-Preserving Lyric Editing in Singing Voice Synthesis
This work enables melody-preserving lyric editing in singing voice synthesis via continuous-latent autoregression with State-Control-Transition routing. It addresses the unique editing challenge where revised lyrics must override source-lyric cues without propagating artifacts through the generation history.
Abstract
Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/Liyric-SVS/.
Trending on Hugging Face
LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
LeapTalk enables real-time, long-form talking-head generation via single-step bridge distillation with Brownian-bridge transport, heterogeneous SNR-aligned distillation, and audio-driven guidance.
PersonaLive! Expressive Portrait Image Animation for Live Streaming
PersonaLive is a diffusion-based portrait animation framework that improves real-time performance through hybrid implicit signals, appearance distillation, and autoregressive streaming generation.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.