Duet Animation, Self-Listening Dialogue, and Acoustic Bottlenecks in LLMs
Today's digest covers 3D duet singing animation with explicit interaction dynamics, a self-listening mechanism for full-duplex speech models, and a diagnostic study on why audio-conditioned LLMs fail to act on the acoustic information they encode.
InterSing generates duet-singing animations from audio, with interaction scores either given or predicted from the same audio. The framework supports tasks including (a) dyadic generation, (b) partner generation, and (c) score-guided generation. In the partner generation and score-guided illustrations, the gray head denotes the given reference motion sequence. From Shenzhen University.
Today's papers probe the boundaries of how AI systems perceive, generate, and act on audio — from animating coordinated musical performers in 3D, to making spoken dialogue models aware of what they've actually said, to diagnosing why language models routinely ignore the acoustic information sitting right inside them.
Talking Avatars & Audio-Driven Animation
Bringing multi-performer realism to audio-driven 3D animation.
Generating believable animations for a single performer driven by audio is hard enough — but duet performance introduces a new layer of complexity: the subtle, rhythmically-locked coordination between two people. Most prior audio-driven animation methods treat each body independently and miss these inter-performer dynamics entirely. InterSing from Shenzhen University tackles this directly by learning explicit interaction logits — a lightweight signal that captures when two singers are coordinating rather than acting independently. The approach explicitly models the sparse, rhythm-dependent engagement patterns unique to musical performance, producing 3D duet animations that are notably more cohesive and realistic than those from single-body baselines.
InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond
InterSing generates realistic 3D duet singing animations by modeling inter-performer engagement through learned "interaction logits" — a signal capturing when singers coordinate. Unlike prior audio-driven methods, it captures sparse, rhythm-dependent coordination patterns unique to musical performance.
Abstract
We present InterSing, a framework for generating realistic 3D head animations for duet singing performances. Unlike solo singing, duet performance requires each singer to balance individual expressiveness with intermittent interaction at musically salient moments, such as phrase boundaries, synchronized rhythms, and call-and-response passages. Because these interactions are sparse and rhythm-dependent, existing audio-driven animation methods and conversational interaction models do not adequately capture their structure. Our key insight is that duet coordination can be represented as a time-varying signal that reflects how strongly performers engage with one another throughout a song. Based on this observation, we introduce interaction logits, an interpretable latent representation that models the degree of cross-performer engagement at each time step. We learn these logits using weak supervision and use them to condition an interaction-aware diffusion model jointly driven by audio features and interaction dynamics. This formulation enables unified multi-mode generation, spanning independent motion, coordinated behavior, and smooth transitions between them. Experiments show that InterSing generates realistic and expressive singing head animations with stronger coordination and musical alignment than existing methods, while preserving each performer's characteristic motion style. We further demonstrate that the same formulation generalizes to multi-singer performances and provides intuitive control over when and how performers engage.
SpeechLLMs & Spoken Dialogue
Closing feedback loops and diagnosing information bottlenecks in speech-aware models.
Full-duplex speech models — systems that simultaneously speak and listen, handling interruptions in real time — face a subtle but critical problem: they often don't know exactly what audio the user has actually heard. Network jitter, buffering, and playback delays mean the model's intended output and the realized output can diverge significantly. Self-Listening, from CUHK Shenzhen, closes this loop by feeding the model its own actually played audio as an additional input stream. This grounds the model's understanding of the conversational state — particularly important for interruption recovery, where knowing precisely where the user cut in is essential. The method outperforms prior full-duplex approaches on anchoring benchmarks by virtue of this simple but underexplored feedback signal.
What Did I Just Say? Self-Listening for Full-Duplex Speech Models
Self-Listening feeds a full-duplex speech model its actual played audio to ground awareness of what the user has heard, enabling accurate interruption recovery. By closing the gap between intended and realized speech, the approach outperforms prior full-duplex models in anchoring performance.
Abstract
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.
A complementary diagnostic question comes from KAIST: audio-conditioned LLMs demonstrably contain rich acoustic information inside their intermediate representations — yet they frequently fail to act on it in their outputs. Why? Layer-wise probing and targeted interventions reveal that the bottleneck is not in the audio encoder or the early fusion stages; instead, it lies in LM head readout alignment — the final mapping from hidden states to token predictions simply isn't calibrated to surface acoustic features, even when those features are faithfully preserved deeper in the network. This reframes the design challenge for audio-LM systems: the problem isn't getting acoustic information in, it's making the output head use it.
Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
This paper investigates why audio-conditioned LLMs underutilize acoustic cues despite containing this information. Through layer-wise probing and interventions, it shows the bottleneck lies not in encoding loss but in LM head readout alignment—reshaping how audio-LM systems should be designed.
Abstract
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Trending on Hugging Face
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to overcome scaling bottlenecks.
Omni Interaction Agent Technical Report
Gander is an end-to-end framework that integrates continuous multi-modal streaming, real-time full-duplex interaction, and agentic reasoning through a Cerebellum-Brain architecture and a chunk-level token stream design.
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
AuK is an open-source foundational model that unifies speech generation and editing via natural-language instructions and audio context, using a multimodal language model, joint VAE, hybrid rectified-flow Transformer, and efficient distillation for fast inference.
