GaussianEmoTalker
GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
A real-time emotional talking head framework using 3D Gaussian Splatting that decomposes emotion generation as neutral-to-emotional residual deformation. This two-stage approach separates phoneme-driven lip motion from controllable emotional expression, enabling high-fidelity synthesis with precise intensity control.
Links
Paper & demos
Impact
Abstract
Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/
Introduction and Motivation
Audio-driven talking head synthesis has made remarkable strides in recent years, enabling high-fidelity video generation synchronized with speech. Yet most existing methods concentrate primarily on lip synchronization and general visual quality, leaving two critical gaps: expressively controllable emotions and real-time rendering performance. Generating talking head videos that faithfully reflect specific emotion categories and their discrete intensity levels—while doing so at interactive frame rates—remains a fundamentally open challenge.
Earlier GAN-based approaches learn direct mappings from audio to video frames or use intermediate landmark representations. While these achieve decent lip sync, they largely ignore emotional expressiveness. Datasets with multi-intensity emotional annotations (notably MEAD) have spurred a new generation of methods that attempt to control emotion, but they either rely on external detector accuracy, lack intensity control, or—as in the case of diffusion-model-based approaches—suffer from generation speed bottlenecks incompatible with real-time use.
3D Gaussian Splatting (3DGS) has emerged as a powerful representation for real-time novel-view synthesis, with recent works demonstrating its applicability to talking head generation. The key insight of GaussianEmoTalker is to reframe the emotional talking head problem as a neutral-to-emotional residual deformation task rather than a direct end-to-end prediction of an emotional avatar from audio. This decomposition is principled: phoneme-related lip articulation is largely emotion-independent and can be anchored in a neutral space first, while emotion acts as a structured residual modifying brows, cheeks, eyelids, and mouth corners. The result is a significantly reduced search space for the second stage, more interpretable Gaussian offsets, and a system that achieves real-time rendering.
Background and Preliminaries
3D Gaussian Splatting
3D Gaussian Splatting (3DGS) represents a scene as a set of anisotropic 3D Gaussian primitives. Each Gaussian is characterized by a center position $\mathbf{x}$, a covariance matrix $\Sigma$, an opacity $\alpha$, and a view-dependent color $c$ (typically encoded in spherical harmonics). The influence of a Gaussian at a 3D point is: $$G(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x}^T \Sigma^{-1} \mathbf{x}}$$ For rendering, 3D Gaussians are projected onto the 2D image plane. The projected covariance in camera coordinates is: $$\Sigma' = J W \Sigma W^T J^T$$ where $W$ is the view transform and $J$ is the Jacobian of the affine approximation of the projective transformation. Training optimizes the Gaussian parameters—positions, covariance (factored into rotation $r$ and scale $s$), opacity, and color—via differentiable rasterization.
3D Gaussian Blendshapes (GaussianBlendshapes)
GaussianBlendshapes extend static 3DGS to dynamic facial animation by defining a neutral base model $B_0$ and a set of $K$ expression blendshapes $\{B_1, B_2, \dots, B_K\}$, derived via PCA from the FLAME parametric face model. Each model is a collection of 3D Gaussians with attributes: position $x$, opacity $\alpha$, rotation $r$, scale $s$. The deviation of a blendshape from neutral is: $$\Delta B_k = B_k - B_0$$ The head avatar for expression coefficients $\psi = \{\psi_1, \ldots, \psi_K\}$ is then: $$B_\psi = B_0 + \sum_{k=1}^{K} \psi_k \Delta B_k$$ While GaussianBlendshapes can handle common expressions well, the linearity of the blendshape model limits its ability to capture exaggerated, high-intensity emotional expressions—a key motivation for adding the emotional deformation stage.
Method Overview
GaussianEmoTalker is a two-stage framework:
- Stage 1 – Neutral Gaussian Initialization (NGI): Given a neutral-expression video of the target identity, GaussianBlendshapes are fitted to construct an identity-specific neutral state space providing high-fidelity Gaussian attributes anchored to phoneme-driven motion.
- Stage 2 – Emotional Gaussian Deformation (EGD): An emotion-conditioned residual deformation is predicted by combining mesh displacement cues (the difference between neutral and emotional FLAME meshes), audio features, CLIP-encoded emotion category and intensity text, and a null token. A spatial-audio-emotion cross-attention module fuses these heterogeneous signals to output per-Gaussian attribute offsets.
Stage 1: Neutral Gaussian Initialization (NGI)
Data Preparation
Following the GaussianBlendshapes methodology, a face tracker is used to compute the FLAME mesh for the neutral expression of the target identity, along with FLAME meshes for 50 basis expressions extracted from the input neutral video clips.
Gaussian Initialization
The neutral base model $B_0$ is initialized by placing Gaussian centers on the neutral FLAME mesh $M_0$ via Poisson disk sampling. Each Gaussian's nearest triangle on $M_0$ is identified, and Linear Blend Skinning (LBS) weights are assigned by linearly interpolating the LBS weights of the triangle's vertices. Mouth interior Gaussians $B_m$ model the upper and lower teeth via two predefined billboard planes—upper teeth rigidly attached to the skull, lower teeth bound to the jaw joint vertex. Expression blendshapes $B_k$ are initialized by applying deformation gradients from $M_0$ to each expression FLAME mesh $M_k$.
Audio-to-Expression Transformer (A2ET)
Building on the EAT framework, the A2ET is adapted to predict a FLAME expression vector $\psi$ and a local pose vector $\theta = (\theta^{\text{jaw}}, \theta^{\text{neck}})$ covering jaw and neck rotations. A fixed canonical neutral frame $I_s$ of the target identity is used as a source image during both training and inference to preserve identity-specific mouth geometry when mapping audio to FLAME coefficients. The model takes as input audio $A$, source image $I_s$, and an emotion label (neutral or one of seven emotional categories):
$$\mathcal{F}_{A2ET}(A, I_s, \text{neu}) \to \psi_{\text{neu}},\, \theta_{\text{neu}}$$ $$\mathcal{F}_{A2ET}(A, I_s, \text{emo}) \to \psi_{\text{emo}},\, \theta_{\text{emo}}$$Since MEAD is captured from a nearly fixed viewpoint, global head pose and eye pose are fixed to the canonical values estimated by the tracker for each identity; only local jaw and neck motion are modeled dynamically. The neutral expression coefficients $\psi_{\text{neu}}$ are then fed into the pre-fitted GaussianBlendshapes to produce the initialized neutral mesh $M_{\text{neu}}$ and Gaussian attributes $G_{\text{neu}}$: $$\mathcal{F}_{GB}(\psi_{\text{neu}}) \to M_{\text{neu}},\, G_{\text{neu}}$$
Stage 2: Emotional Gaussian Deformation (EGD)
Neutral Mesh to Emotion Mesh via LBS
The neutral mesh is transformed into an emotional mesh through FLAME's standard LBS pipeline. Drawing inspiration from IMavatar, the deformation is: $$M_{\text{emo}} = \text{LBS}\!\left(M_{\text{neu}} + B_P(\theta_{\text{emo}};\, \rho) + B_E(\psi_{\text{emo}};\, \varepsilon),\; J(\psi_{\text{emo}}),\; \theta_{\text{emo}},\; \omega\right)$$ where $\text{LBS}$ and $J$ are the standard FLAME skinning function and joint regressor; $B_P$ and $B_E$ are the pose-dependent and expression-dependent blendshapes parameterized by $\rho$ and $\varepsilon$; $\omega$ are the skinning weights. The residual motion cue for Stage 2 is the mesh displacement: $$\Delta M = M_{\text{emo}} - M_{\text{neu}}$$ This displacement encodes where and by how much each facial region needs to move to express the target emotion, providing spatial grounding for the cross-attention module.
Disentangling Speech-Related Motion
Audio alone does not capture all facial dynamics—eye blinks, wrinkles, and subtle scene changes are speech-independent. To disentangle these, the framework incorporates:
- AU45 (FACS blink unit): Sinusoidal positional encoding of the eye-blink action unit to model blink dynamics.
- Camera parameters $c_n$: Tracked viewpoint parameters of the aligned crop to absorb residual frame-wise viewpoint variation.
- Null token $z_\emptyset$: An all-zero vector of the same dimensionality as other conditioning channels, acting as a global stability anchor that reserves network capacity for motion-independent appearance (skin tone, hair color) and long-term dynamics (breathing rhythms, head inertia).
The initial query for the attention module at frame $n$ is formed by concatenating and projecting these signals: $$q_n = W_q\!\left[f(\Delta M_n) \;\|\; \text{PE}(\text{AU45}_n) \;\|\; c_n \;\|\; z_\emptyset\right]$$ where $\|$ denotes concatenation and $W_q$ is a learned linear projection. The null token prevents the network from forcing high-frequency audio fluctuations onto regions of the face that are genuinely static.
Emotion and Intensity Text Encoding
Emotion categories and intensity levels are encoded using the text encoder of the CLIP model. The emotion text $t_e$ covers seven categories: happy, sad, fear, disgusted, contempt, angry, surprised. The intensity text $t_s$ covers three levels: level1, level2, level3. Both are encoded independently: $$f_e = \text{TextEncoder}(t_e), \quad f_s = \text{TextEncoder}(t_s)$$ Using CLIP's text encoder rather than simple one-hot vectors allows the model to leverage rich semantic representations of emotion and intensity.
Spatial-Audio-Emotion Cross-Attention Module
A key innovation of GaussianEmoTalker is the Spatial-Audio-Emotion Cross-Attention (SAECA) module, which fuses the heterogeneous signals—spatial mesh displacement, audio features, emotion embeddings, and intensity embeddings—to produce a unified feature for Gaussian deformation prediction. Previous region-aware audio/emotion approaches apply element-wise multiplication of weights at fixed 3D coordinates, which breaks down when a fixed coordinate migrates to a different facial region across frames. The cross-attention formulation avoids this problem by operating in feature space.
The module consists of $L$ stacked cross-attention layers $F_{ca}$ and feed-forward layers $F_{fd}$, each connected via skip connections:
$$z_n^0 = q_n$$ $$z_n^{\prime\,l} = F_{ca}(z_n^{l-1},\; a_n,\; f_e,\; f_s) + z_n^{l-1}, \quad l = 1, \ldots, L$$ $$z_n^{l} = F_{fd}(z_n^{\prime\,l}) + z_n^{\prime\,l}, \quad l = 1, \ldots, L$$The query is derived from the spatial displacement features, while the audio features $a_n$, emotion features $f_e$, and intensity features $f_s$ serve as keys and values. The resulting features $z_n^L$ integrate spatial, acoustic, and affective signals with the rich facial detail captured by the Gaussian representation.
Gaussian Attribute Deformation
The fused cross-attention output $z_n^L$ is passed through a set of MLP regressors $\mathcal{F}_d$ that predict per-Gaussian residual offsets: $$(\Delta r_{\text{emo}},\; \Delta s_{\text{emo}},\; \Delta c_{\text{emo}},\; \Delta \alpha_{\text{emo}}) = \mathcal{F}_d(z_n^L)$$ The final emotional Gaussian attributes are:
- Rotation: $r_{\text{emo}} = r_{\text{neu}} + \Delta r_{\text{emo}}$
- Scale: $s_{\text{emo}} = s_{\text{neu}} + \Delta s_{\text{emo}}$
- Color: $c_{\text{emo}} = c_{\text{neu}} + \Delta c_{\text{emo}}$
- Opacity: $o_{\text{emo}} = o_{\text{neu}} + \Delta o_{\text{emo}}$
All neutral attributes $r_{\text{neu}}, s_{\text{neu}}, c_{\text{neu}}, o_{\text{neu}}$ are derived from $G_{\text{neu}}$ produced by Stage 1. Crucially, position is controlled via the LBS-driven mesh, while all other Gaussian attributes are refined by the residual network—enabling both geometric expressiveness and appearance-level detail.
Gaussian Splatting Rendering
With all emotional Gaussian parameters computed, rendering follows the standard 3DGS alpha-compositing formula. For a pixel, the blended color from $\mathcal{S}$ sorted Gaussians is: $$C_{\text{pix}} = \sum_{i \in \mathcal{S}} c_{\text{emo}}^i \,\Pi(f_{\text{emo}}^i) \prod_{j=1}^{i-1}\!\left(1 - \Pi(f_{\text{emo}}^j)\right)$$ where $\Pi(f_{\text{emo}}^i)$ is the projected 2D influence of the $i$-th Gaussian on the pixel and the product term is the accumulated transmittance. This formulation enables fully differentiable, real-time rendering.
Training Objectives
Both stages are trained with the same composite loss function. The total loss is: $$\mathcal{L} = \lambda_{\text{rgb}}\,\mathcal{L}_{\text{RGB}} + \lambda_{\text{flame}}\,\mathcal{L}_{\text{flame}} + \lambda_{\text{vgg}}\,\mathcal{L}_{\text{vgg}} + \lambda_{\text{D-SSIM}}\,\mathcal{L}_{\text{D-SSIM}}$$
- RGB Loss $\mathcal{L}_{\text{RGB}}$: Per-pixel $L_1$ difference between the rendered image $I_{GS}$ and the ground truth $I_{GT}$, ensuring pixel-level color alignment: $\mathcal{L}_{\text{RGB}}(I_{GS}) = \|I_{GS} - I_{GT}\|$.
- VGG Perceptual Loss $\mathcal{L}_{\text{vgg}}$: Feature-space difference using the first four layers of a pre-trained VGG network, promoting perceptual fidelity and high-frequency detail: $\mathcal{L}_{\text{vgg}}(I_{GS}) = \|F_{\text{vgg}}(I_{GS}) - F_{\text{vgg}}(I_{GT})\|$.
- FLAME Loss $\mathcal{L}_{\text{flame}}$: Supervises the deformation network with pseudo ground-truth values derived from nearest FLAME vertices: $$\mathcal{L}_{\text{flame}} = \frac{1}{N}\sum_{i=1}^{N}\!\left(\lambda_e \|\mathcal{E}_i - \hat{\mathcal{E}}_i\|_2 + \lambda_p \|\mathcal{P}_i - \hat{\mathcal{P}}_i\|_2 + \lambda_w \|\mathcal{W}_i - \hat{\mathcal{W}}_i\|_2\right)$$ where $\mathcal{E}_i, \mathcal{P}_i, \mathcal{W}_i$ are the predicted values and $\hat{\mathcal{E}}_i, \hat{\mathcal{P}}_i, \hat{\mathcal{W}}_i$ are pseudo ground truths from the nearest FLAME vertex. This loss injects prior geometric knowledge from FLAME about plausible expression and pose deformations.
- D-SSIM Loss $\mathcal{L}_{\text{D-SSIM}}$: Differentiable structural similarity loss for perceptual consistency.
Loss weights: $\lambda_{\text{rgb}} = 1$, $\lambda_{\text{D-SSIM}} = 0.25$, $\lambda_{\text{flame}} = 1$, $\lambda_{\text{vgg}} = 0.1$. Within the FLAME loss: $\lambda_e = 1000$, $\lambda_p = 1000$, $\lambda_w = 1$.
Experimental Setup
Dataset
All training and evaluation is performed on the MEAD dataset, a high-quality emotional talking-head video dataset. MEAD includes multiple actors covering 8 emotion categories (neutral, happy, sad, fear, disgusted, contempt, angry, surprised), each with three intensity levels (level 1–3). Videos are cropped and resized to 512×512. For each identity, video clips are split into training and testing sets at a 9:1 ratio at the clip level, ensuring test sequences are entirely unseen. The person-specific monocular training setting means each model is trained on a single subject's data from a fixed viewpoint.
Evaluation Metrics
Two evaluation settings are used:
- Self-driven setting: Audio and emotion labels from the same sequence drive generation. Metrics include:
- Video quality: FID, FVD, PSNR, SSIM, LPIPS, CPBD
- Facial motion accuracy: Landmark Distance (LD) and Landmark Velocity Difference (LVD) for mouth (M-) and face (F-) regions
- Lip synchronization: SyncNet confidence score ($\text{Sync}_{\text{conf}}$)
- Rendering speed: FPS
- Cross-driven setting: Entirely different audio tracks (from other identities) drive generation. Since ground truth is unavailable, lip sync is measured via Landmark Distance (LMD), SyncNet confidence (Sync), and Action Unit Error (AUE).
Baselines
GaussianEmoTalker is compared against:
- SadTalker: GAN-based audio-driven talking head synthesis.
- EAT: Emotion-aware talking head using 3D landmark prediction and an audio-to-expression transformer.
- EDTalk: Decouples lip motion, head pose, and emotion for flexible synthesis.
- GSBS (GaussianBlendshapes): The Stage 1 baseline—3DGS with linear expression blendshapes, no emotional deformation stage.
- EmoTaG: An emotion-aware Gaussian baseline using FLAME-guided animation with gated residual motion in a pretrain-and-adapt few-shot paradigm.
Implementation Details
The framework is implemented in PyTorch on an RTX 3090 GPU. Stage 1 follows the GaussianBlendshapes training procedure exactly. Stage 2 trains for 5,000 iterations with an initial learning rate of $10^{-4}$ decaying to $10^{-5}$. The Adam optimizer is used throughout.
Results
Qualitative Results
In the self-driven setting, GaussianEmoTalker produces more realistic emotional portraits that align more closely with ground truth, particularly around lip closure, cheek lifting, and high-frequency details near the eyes. Compared with SadTalker, EAT, EDTalk, GSBS, and EmoTaG in the self-driven comparison, the method preserves sharper identity details and more faithful emotional deformation.
In the cross-driven setting, GaussianEmoTalker achieves better lip-sync accuracy across diverse phonemes. Compared to baselines, it preserves tighter lip closure for plosives (e.g., "keep", "had") and more temporally stable mouth corners. This is attributed to the Spatial-Audio-Emotion cross-attention mechanism anchoring emotional deformations to the neutral talking state while linking audio features to the Gaussian structure.
All qualitative results are composited into the aligned MEAD crop using the original static background plate of the corresponding target clip—the background is not synthesized by the network and is kept identical across compared methods.
Quantitative Results — Self-Driven Setting
| Method | FID ↓ | FVD ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | CPBD ↑ | M-LD ↓ | M-LVD ↓ | F-LD ↓ | F-LVD ↓ | Sync ↑ | FPS ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SadTalker | 49.53 | 344.19 | 26.56 | 0.65 | 0.37 | 0.33 | 2.66 | 2.45 | 3.23 | 3.01 | 2.56 | 2.2 |
| EAT | 54.22 | 389.74 | 28.39 | 0.72 | 0.36 | 0.41 | 2.90 | 2.66 | 3.67 | 3.33 | 3.27 | 2.0 |
| EDTalk | 64.76 | 432.90 | 23.43 | 0.61 | 0.43 | 0.29 | 3.11 | 2.88 | 3.99 | 3.56 | 2.93 | 4.8 |
| GSBS | 32.33 | 169.32 | 30.78 | 0.75 | 0.27 | 0.41 | 2.57 | 2.10 | 3.18 | 2.56 | 4.34 | 51.0 |
| EmoTaG | 27.95 | 136.82 | 32.22 | 0.82 | 0.19 | 0.44 | 2.41 | 1.78 | 3.02 | 1.88 | 5.08 | 33.0 |
| w/o A2ET | 28.54 | 147.75 | 31.32 | 0.77 | 0.25 | 0.41 | 2.47 | 1.89 | 3.13 | 1.94 | 4.80 | — |
| w/o EGD | 32.40 | 165.54 | 30.90 | 0.75 | 0.25 | 0.42 | 2.45 | 1.80 | 3.08 | 1.90 | 4.46 | — |
| w/o Null Vector | 28.51 | 140.32 | 32.11 | 0.81 | 0.23 | 0.44 | 2.38 | 1.79 | 3.01 | 1.88 | 4.96 | — |
| Ours-Combine | 27.77 | 134.46 | 32.35 | 0.82 | 0.17 | 0.43 | 2.37 | 1.77 | 3.04 | 1.86 | 5.19 | — |
| Ours | 26.36 | 132.34 | 32.40 | 0.84 | 0.18 | 0.46 | 2.35 | 1.75 | 2.98 | 1.83 | 5.23 | 40.0 |
GaussianEmoTalker achieves the best scores on nearly all video quality metrics, lip synchronization, and facial motion accuracy. Compared to GSBS (Stage 1 alone), adding Stage 2 emotional deformation brings substantial improvements across all metrics at a modest FPS cost (51 → 40 FPS). EmoTaG is the strongest competitor but runs at only 33 FPS and uses a few-shot pretrain-and-adapt paradigm; GaussianEmoTalker outperforms it on all metrics while running faster.
Quantitative Results — Cross-Driven Setting
| Method | A-I Sync ↑ | A-I LMD ↓ | A-I AUE ↓ | A-II Sync ↑ | A-II LMD ↓ | A-II AUE ↓ | A-III Sync ↑ | A-III LMD ↓ | A-III AUE ↓ | A-IV Sync ↑ | A-IV LMD ↓ | A-IV AUE ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SadTalker | 3.813 | 15.234 | 5.125 | 3.727 | 16.044 | 5.315 | 3.933 | 14.852 | 5.075 | 4.013 | 14.574 | 4.935 |
| EAT | 4.121 | 14.565 | 4.847 | 4.034 | 15.274 | 4.961 | 4.246 | 14.427 | 4.786 | 4.365 | 13.883 | 4.672 |
| EDTalk | 3.451 | 16.821 | 5.534 | 3.426 | 17.567 | 5.676 | 3.773 | 16.345 | 5.456 | 3.734 | 15.834 | 5.342 |
| GSBS | 4.762 | 12.817 | 4.345 | 4.459 | 13.521 | 4.456 | 4.648 | 12.347 | 4.234 | 4.149 | 11.931 | 4.123 |
| EmoTaG | 5.176 | 11.598 | 4.033 | 5.104 | 13.471 | 4.102 | 5.388 | 10.732 | 3.891 | 5.317 | 9.711 | 3.776 |
| w/o A2ET | 4.678 | 12.123 | 4.234 | 4.543 | 13.845 | 4.345 | 4.789 | 11.168 | 4.123 | 4.991 | 11.234 | 3.987 |
| w/o EGD | 4.589 | 12.349 | 4.278 | 4.478 | 13.923 | 4.389 | 4.712 | 11.845 | 4.167 | 4.823 | 11.564 | 4.056 |
| w/o Null Vec. | 5.121 | 11.728 | 4.076 | 5.021 | 13.689 | 4.152 | 5.302 | 10.985 | 3.976 | 5.187 | 9.932 | 3.865 |
| Ours-Combine | 5.212 | 11.405 | 4.008 | 5.066 | 13.535 | 4.089 | 5.429 | 10.849 | 3.909 | 5.255 | 9.774 | 3.801 |
| Ours | 5.235 | 11.436 | 3.940 | 5.164 | 13.354 | 4.054 | 5.431 | 10.575 | 3.848 | 5.453 | 9.655 | 3.719 |
In the cross-driven setting, GaussianEmoTalker achieves state-of-the-art performance on Sync, LMD, and AUE across all four unseen audio tracks, confirming that the SAECA module generalizes well to novel speech content.
User Study
| Method | Emotional Accuracy ↑ | Lip Synchronization ↑ | Identity Consistency ↑ | Video Quality ↑ |
|---|---|---|---|---|
| SadTalker | 0% | 5% | 10% | 0% |
| EAT | 15% | 0% | 10% | 0% |
| EDTalk | 0% | 5% | 0% | 0% |
| GSBS | 10% | 20% | 25% | 5% |
| EmoTaG | 35% | 10% | 20% | 20% |
| Ours | 40% | 60% | 35% | 75% |
The user study involved 20 participants (master's and PhD students in computer vision) evaluating videos on four criteria. GaussianEmoTalker achieves the highest preference across all four categories by substantial margins, most notably in video quality (75%) and lip synchronization (60%).
Ablation Study
Neutral Gaussian Initialization (NGI)
Removing Stage 1 (NGI) and instead training Stage 2 to directly predict absolute Gaussian attributes (not residuals) leads to severe training instability. Without neutral initialization, the model cannot converge meaningfully. Training time comparison:
| Method | Stage 1 (NGI) | Stage 2 (Emotional Gaussian Deformation) |
|---|---|---|
| w/o NGI | 0 h | ~10 h+ |
| With NGI | ~0.5 h | ~3 h |
The NGI stage costs only ~0.5 hours and reduces Stage 2 training time from 10+ hours (with convergence failure) to ~3 hours, delivering a 3× speedup and reliable convergence.
Emotional Gaussian Deformation (EGD)
Without Stage 2 (EGD)—relying solely on GaussianBlendshapes with emotional expression coefficients from A2ET—the model is unable to capture exaggerated, high-intensity emotional expressions. This confirms the fundamental limitation of linear blendshape models for extreme emotions. The facial landmark displacement analysis shows that with EGD, displacement magnitudes match ground truth much more closely at all intensity levels, with the gap being largest at intensity level 3.
Audio-to-Expression Transformer (A2ET)
Without A2ET, expression coefficients must be derived purely from tracking techniques without considering audio content or emotion labels. This leads to less accurate expression coefficients, imprecise LBS mesh deformations, and ultimately weaker emotional expressions. Quantitative results across both self-driven and cross-driven settings confirm degraded performance in every metric when A2ET is removed.
Null Vector ($z_\emptyset$)
Removing the globally constant null token from the spatial-audio-emotion attention module leads to significant inter-frame instability and convergence difficulty. Without this global anchor, the attention mechanism cannot decouple motion-independent static attributes (skin tone, hair color) and long-term dynamics (breathing rhythms) from high-frequency audio signals. The result is visual jittering as the network erroneously maps stationary facial regions to fluctuating speech features. The quantitative gap between "w/o Null Vector" and the full model confirms the component's importance for temporal stability.
Combined Identity Training (Ours-Combine)
An additional variant jointly trains using all identity subjects to assess generalization. Although performance slightly decreases compared to identity-specific training, it still outperforms all non-Gaussian baselines. This suggests the framework can be adapted for multi-identity settings with modest quality trade-offs.
Discussion
GaussianEmoTalker operates under an identity-specific monocular training regime: each model is trained on video clips from a single identity captured at a consistent viewpoint (MEAD's near-fixed camera). Global head pose is not predicted during inference; instead, local jaw and neck rotations from the FLAME kinematic chain maintain anatomical realism for mouth openings and head-neck transitions. The dynamic expressiveness—subtle mouth corner twitching, characteristic eyebrow motion—emerges from the SAECA module mapping audio and emotion labels to Gaussian attribute offsets. Because the model is person-specific, it successfully memorizes the unique micro-expression habits of a particular individual, reproducing these habits deterministically whenever specific audio frequencies and emotional cues are encountered. The null token further enables the attention mechanism to capture motion-independent static appearance and long-term dynamic patterns such as breathing rhythms or subtle head inertia, collectively contributing to lifelike quality without requiring a stochastic generator.
Limitations and Future Work
The authors identify two primary limitations:
- Person-specific training: Each model is trained for a single identity, making cross-identity generalization impossible without retraining. While this is a deliberate design choice for high-fidelity personalized scenarios (digital avatars, personal digital twins), extending to identity-generalizable Gaussian models is identified as a promising future direction.
- Discrete intensity levels only: The current system supports only three discrete intensity levels. Smooth interpolation between levels (e.g., intensity 1.5 or 2.5) is not possible. Future work could explore linear facial prior interpolation or Gaussian mixture model-based interpolation for continuous intensity control.
Conclusions
GaussianEmoTalker presents a novel two-stage framework for real-time, audio-driven emotional talking head synthesis based on 3D Gaussian Splatting. The core insight—formulating emotional animation as neutral-to-emotional residual deformation rather than direct regression from audio—yields a system that is both more stable to train and more interpretable at inference time. Stage 1 (Neutral Gaussian Initialization) constructs an identity-specific, high-fidelity neutral talking space via GaussianBlendshapes in roughly 0.5 hours. Stage 2 (Emotional Gaussian Deformation) introduces the Spatial-Audio-Emotion Cross-Attention module that fuses FLAME mesh displacement, audio features, and CLIP-encoded emotion and intensity to predict per-Gaussian residual offsets for rotation, scale, color, and opacity. A null token provides a global stability anchor that decouples speech-independent facial dynamics from audio-driven signals.
On the MEAD benchmark, GaussianEmoTalker achieves state-of-the-art performance in video quality (PSNR 32.40, SSIM 0.84, FID 26.36, FVD 132.34), lip synchronization (SyncNet confidence 5.23), facial motion fidelity, and real-time rendering at 40 FPS on an RTX 3090—outperforming prior Gaussian-based, NeRF-based, GAN-based, and diffusion-based emotional talking head methods, with strong user study preference margins across all four perceptual dimensions.