Smarter 3D Talking Heads: Continuous Emotion Control and Human-Preference Alignment
Today's digest focuses on audio-driven 3D facial animation, covering continuous valence-arousal emotion control for expressive talking heads and a new human preference reward model that moves evaluation beyond ground-truth metrics.
Continuous emotion controllability via VA-space interpolation. With the same audio clip, we linearly interpolate the input Valence--Arousal condition from $(-1,-1)$ to $(1,1)$. From Tsinghua University.
Today's papers both tackle the same frontier in audio-driven 3D face animation — but from complementary angles. One asks how to control emotion more expressively; the other asks how to measure whether the result actually looks good to humans. Together, they push talking-head synthesis closer to the nuanced, preference-aware systems real applications demand.
Talking Avatars & Audio-Driven Face Animation
Richer emotion, better evaluation — two new directions for 3D facial animation.
Discrete emotion labels (happy, sad, angry…) have long been a crutch in expressive talking-head generation, forcing continuous human affect into a handful of buckets. CETalk from Tsinghua University breaks out of this mold by grounding emotion control in the Valence-Arousal (VA) space — a continuous 2D representation of affect — enabling fine-grained, smoothly interpolated emotional expressions. A key architectural insight is the decoupling of high-frequency lip-sync dynamics from low-frequency emotional dynamics: the model keeps mouth movements tight to the audio signal while letting the broader emotional envelope evolve naturally and controllably across time. The result is a system that can transition fluidly through emotional states rather than snapping between discrete categories.
CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation
CETalk generates expressive 3D talking heads using continuous Valence-Arousal emotion control rather than discrete categories. It decouples high-frequency lip-sync from low-frequency emotional dynamics, enabling both accurate synchronization and smooth, controllable emotion transitions.
Abstract
Emotional 3D talking head generation aims to synthesize expressive facial animations with accurate lip synchronization. However, existing methods often rely on discrete emotion categories, which fail to capture the continuous evolution of affect. They also overlook the temporal frequency mismatch between audio articulation and emotional expression. In this paper, we propose CETalk, an audio-driven 3D facial animation framework conditioned on continuous Valence--Arousal (VA) representations for fine-grained emotion control. CETalk predicts a sequence of FLAME parameters through three key components: a Dynamic Emotion Modulation Module that adaptively scales emotional intensity using audio-derived cues; a Multi-Scale Temporal Modeling mechanism that employs parallel branches to decouple high-frequency articulatory movements from low-frequency emotional dynamics; and a Dynamic Fusion Mechanism that integrates these multi-scale features via an adaptive gating network. To support training and evaluation, we construct 3D-VA-MEAD, a large-scale dataset with automatically estimated VA annotations and reconstructed 3D facial motions. Extensive experiments demonstrate that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness, while enabling smooth and controllable emotion transitions.
Even a perfectly expressive animation pipeline is only as reliable as its evaluation metric. Most audio-driven 3D facial animation systems are benchmarked against ground-truth reconstructions — but human perception of "naturalness" doesn't always track reconstruction error. FMReward from Shanghai Jiao Tong University addresses this head-on by introducing the first human preference dataset for audio-driven 3D facial animation and training a perceptual reward model on it. The reward model is then used to fine-tune diffusion-based animation models via a preference-based fine-tuning algorithm, directly optimizing for what people actually find natural rather than what minimizes vertex error. This RLHF-style approach to facial animation is a meaningful methodological shift for the field.
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
This work introduces the first human preference dataset and perceptual reward model (FMReward) for audio-driven 3D facial animation, moving beyond ground-truth errors to align with human preferences. A preference-based fine-tuning algorithm optimizes diffusion models for more natural facial animations.
Abstract
Audio-driven 3D facial animation is essential for advancing immersion and interactivity in virtual experiences. Although recent advances have shown promising capabilities, the training and evaluation of existing methods typically rely on ground-truth-based errors, which fall short of aligning with human preferences. To address this, we present a comprehensive framework that learns an automatic perceptual model from human preference data and leverages it to improve and evaluate the perceptual quality of audio-driven 3D facial animation. To begin with, we construct FMPair (Facial Motion Pairwise preference), the first human preference dataset for audio-driven 3D facial animation, which is built through a systematic annotation pipeline and comprises 65,574 annotated 3D facial motion pairs from 8,834 distinct in-the-wild audio clips. Based on the pairwise comparison dataset, we propose a Facial Motion Reward model, termed FMReward, which takes audio and 3D facial motion as inputs and predicts a perceptual quality score aligned with human preferences. Building upon FMReward, we further introduce Facial Motion reward Feedback Learning (FMFL), a direct fine-tuning algorithm that leverages a pretrained reward model to optimize diffusion-based audio-driven 3D facial animation models for better alignment with human preferences. Extensive experiments demonstrate the superiority of FMReward over other metrics in aligning with human preferences and the effectiveness of FMFL in improving the perceptual quality of audio-driven 3D facial animation.
Trending on Hugging Face
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
4DAnyone reconstructs 4D humans from monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, using reference and target context designs to overcome scaling bottlenecks.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
dots.tts Technical Report
dots.tts is a 2B-parameter continuous autoregressive text-to-speech model that generates speech in a semantically structured continuous latent space. Innovations include full-history conditioning and self-corrective post-training for robust, expressive, and low-latency multilingual speech.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.