Emotion Control Across Avatars, TTS, and Speech-LLMs
Today's digest covers three papers tackling emotional control and representation: real-time emotional talking heads via 3D Gaussian Splatting, geometric analysis of emotion steering in TTS models, and a speech-text interleaving strategy that keeps LLM priors intact during ASR training.
Given the audio, emotion category, and intensity, our method can real-time render high-fidelity, emotion-driven avatars with accurate lip-syncing for different emotions and intensities. It outperforms the current state-of-the-art methods in video quality (PSNR) and achieves real-time generation efficiency (FPS). From GaussianEmoTalker.
A common thread runs through today's papers: how to represent, steer, and preserve meaningful structure — whether that's emotion in a talking avatar, speaker-invariant geometry in a TTS latent space, or textual priors in a speech-conditioned LLM. Three distinct problems, one shared ambition.
Talking Avatars & Lip Sync
Real-time, controllable emotional faces push 3D Gaussian Splatting into production-grade territory.
Generating expressive talking heads that separate lip motion from emotional expression has long been a pain point — most approaches entangle the two, making precise intensity control difficult. GaussianEmoTalker tackles this with a two-stage pipeline built on 3D Gaussian Splatting: the first stage drives phoneme-aligned lip motion from audio, and the second applies a neutral-to-emotional residual deformation via blendshape-based control. The decomposition means emotion intensity can be dialed independently of speech content, while the Gaussian backbone keeps everything running in real time at high fidelity.
GaussianEmoTalker · 2026-07
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
Nanjing University of Science and Technology
A real-time emotional talking head framework using 3D Gaussian Splatting that decomposes emotion generation as neutral-to-emotional residual deformation. This two-stage approach separates phoneme-driven lip motion from controllable emotional expression, enabling high-fidelity synthesis with precise intensity control.
Abstract
Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/
TTS & Voice Synthesis
Understanding where emotion lives in a hybrid TTS model unlocks principled, composable control.
Steering emotion in a text-to-speech system is easy to do naively and hard to do well — especially across speakers. Emotion Geometry takes a rigorous geometric approach, comparing emotion representations at two intervention sites in a hybrid TTS architecture: the language model and the acoustic decoder. The key finding is that the language model hosts clean, speaker-invariant emotion subspaces that compose naturally across speakers, while the acoustic decoder's representations suffer from speaker entanglement that limits cross-speaker generalization. This has direct implications for where practitioners should intervene when building controllable, multi-speaker TTS systems.
Emotion Geometry · 2026-07
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
The University of Melbourne
This work compares emotion steering sites in hybrid TTS through representation geometry, revealing that the language model has clean, speaker-invariant emotion subspaces ideal for composable control, while the acoustic decoder suffers speaker entanglement limiting cross-speaker generalization.
Abstract
While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.
SpeechLLMs & ASR Architectures
Naive joint training of speech and text breaks the LLM — interleaving fixes it.
Scaling ASR data in a speech-LLM tends to wash out the textual priors the language model originally learned, hurting performance on entity-rich, domain-specific inputs. JSTIP (Joint Speech-Text training by Interleaving) addresses this by interleaving speech tokens and text tokens within aligned training pairs, rather than treating them as separate streams. This preserves the LLM's generative capacity while maintaining effective speech conditioning, yielding measurable gains on domain-specific entity recognition without sacrificing general ASR quality.
JSTIP · 2026-07
Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
Microsoft, USA
JSTIP interleaves speech and text within aligned pairs to preserve both LLM generative capacity and speech-conditioning during joint training. This addresses the failure of naive joint training to maintain textual priors as ASR data scales, improving entity recognition on domain-specific tasks.
Abstract
Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We observe that as supervised ASR training data increases, the contribution of LLM priors becomes less evident, and simple speech-text joint training under-utilizes textual knowledge. We therefore propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs. Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines. JSTIP achieves on-par entity recognition performance using domain transcription text compared to synthetic speech-text pairs, simplifying domain adaptation. Benefiting from textual pretraining and domain text data, JSTIP is competitive with open-source ASR and Speech-LLM systems in medical entity recognition. The zero-shot speech question answering behaviors further suggest that interleaving reduces the speech-text modality gap and preserves the LLM generative prior, which is likely the reason for the entity improvements on the ASR task.