TalkingGaussian

TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting

TalkingGaussian synthesizes 3D talking heads by modeling facial motion as smooth deformations of Gaussian primitives, preserving facial features and reducing distortions. It separates face and inside-mouth motion to enhance lip sync and mouth detail, improving quality and speed over prior methods.

• talking-head
• 3d-avatar
• lip-sync
• audio-driven
• speech-driven
• gaussian-splatting

Demos

These demos highlight TalkingGaussian's strength in synthesizing high-fidelity, lip-synchronized 3D talking heads with persistent face and mouth structures. Focus on the clarity and precision of facial motions, especially around the mouth area, and assess the detailed rendering quality compared to state-of-the-art baselines. The videos and images illustrate the framework's separate handling of face and mouth motions, ensuring structural consistency and improved motion accuracy.

Authors: Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, Lin Gu

Categories: cs.CV

Comment: Accepted at ECCV 2024. Project page: https://fictionarry.github.io/TalkingGaussian/

Published 2024-04-23 · Updated 2024-07-05

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads0 likes

Impact

• Semantic Scholar77 citations26 influential citations

Abstract

Radiance fields have demonstrated impressive performance in synthesizing lifelike 3D talking heads. However, due to the difficulty in fitting steep appearance changes, the prevailing paradigm that presents facial motions by directly modifying point appearance may lead to distortions in dynamic regions. To tackle this challenge, we introduce TalkingGaussian, a deformation-based radiance fields framework for high-fidelity talking head synthesis. Leveraging the point-based Gaussian Splatting, facial motions can be represented in our method by applying smooth and continuous deformations to persistent Gaussian primitives, without requiring to learn the difficult appearance change like previous methods. Due to this simplification, precise facial motions can be synthesized while keeping a highly intact facial feature. Under such a deformation paradigm, we further identify a face-mouth motion inconsistency that would affect the learning of detailed speaking motions. To address this conflict, we decompose the model into two branches separately for the face and inside mouth areas, therefore simplifying the learning tasks to help reconstruct more accurate motion and structure of the mouth region. Extensive experiments demonstrate that our method renders high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency compared with previous methods.

---

Introduction

TalkingGaussian addresses a specific failure mode in radiance-field-based talking head synthesis: when a model tries to represent facial motion by directly predicting rapidly changing appearance for each point in space, the learned colors and densities can become unstable in dynamic regions, producing distorted mouths, blurry lips, transparent eyelids, and other local artifacts. The paper argues that this is not just an optimization issue, but a representation issue: appearance changes on a human face can be steep and discontinuous, which is hard for continuous neural fields to fit faithfully.

The proposed solution is to represent talking head motion as smooth deformation of persistent 3D Gaussian primitives rather than as point-wise appearance modification. The core idea is to keep a stable, explicit head structure using 3D Gaussian Splatting (3DGS) and to model facial dynamics by deforming those primitives in 3D. This decouples structure from motion: the canonical Gaussians preserve geometry, color, and opacity, while a motion field predicts only offsets and shape changes. The paper further observes a face-mouth motion inconsistency that hurts fine-grained speech motion learning, and proposes a two-branch decomposition to separate the face region from the inside-mouth region.

The paper’s main claims are:

• a deformation-based talking head representation built on persistent Gaussian primitives improves facial fidelity;
• a Face-Mouth Decomposition module reduces conflicting motion learning for lips and inside-mouth structures;
• an incremental sampling strategy stabilizes deformation learning by scheduling easier-to-harder motion examples;
• the full system is significantly faster than prior NeRF-based baselines while improving quality and lip synchronization.

Method

Problem setting and 3DGS preliminaries

The training data is a few-minute monocular speech video of a single person. A 3DMM-based face model estimates head pose, which is then used to infer camera pose. Audio is encoded using a pre-trained DeepSpeech model for the main experiments, giving an audio feature vector that conditions motion synthesis.

3D Gaussian Splatting represents the scene using a set of Gaussian primitives. Each primitive $\mathcal{G}_i$ has center $\mu_i \in \mathbb{R}^3$, scaling $s_i \in \mathbb{R}^3$, rotation quaternion $q_i \in \mathbb{R}^4$, opacity $\alpha_i$, and a color feature $f_i$. Its basis function is:

$$ \mathcal{G}_i(\mathbf{x}) = \exp\left(-\frac{1}{2}(\mathbf{x}-\mu_i)^T \Sigma_i^{-1} (\mathbf{x}-\mu_i)\right), $$

with covariance $\Sigma_i$ derived from $s_i$ and $q_i$. For a pixel $\mathbf{x}_p$, color is rendered by alpha-compositing the projected Gaussians:

$$ \mathcal{C}(\mathbf{x}_p)=\sum_{i\in N} c_i\,\tilde{\alpha}_i\prod_{j=1}^{i-1}(1-\tilde{\alpha}_j), \qquad \tilde{\alpha}_i = \alpha_i \mathcal{G}^{\text{proj}}_i(\mathbf{x}_p). $$

The opacity is computed similarly:

$$ \mathcal{A}(\mathbf{x}_p)=\sum_{i\in N} \tilde{\alpha}_i\prod_{j=1}^{i-1}(1-\tilde{\alpha}_j). $$

The method inherits the standard 3DGS optimization pipeline, including densification and pruning during training.

Deformable Gaussian Fields: persistent structure plus motion

The core representation is a Deformable Gaussian Field consisting of two parts:

• Persistent Gaussian Fields, which store the canonical parameters $\theta_C = \{\mu, s, q, \alpha, f\}$ and preserve a stable head structure;
• Grid-based Motion Fields, which predict point-wise deformation conditioned on audio and facial control signals.

The persistent field is initialized with vanilla 3DGS from the training video to obtain a coarse static head representation. The motion field uses a tri-plane / hash-style position encoding $\mathcal{H}$ with an MLP decoder. For each Gaussian primitive, the motion module predicts deformation

$$ \delta_i = \operatorname{MLP}(\mathcal{H}(\mu_i) \oplus \mathbf{C}), $$

where $\mathbf{C}$ denotes the conditioning features and $\oplus$ is concatenation. Importantly, deformation is applied only to geometry-related variables, not to color or opacity. The deformed parameters are

$$ \theta_D = \{\mu + \Delta\mu,\; s + \Delta s,\; q + \Delta q,\; \alpha,\; f\}. $$

This design is the paper’s central departure from prior NeRF talking-head methods: instead of forcing the network to learn abrupt appearance changes at each sample point, TalkingGaussian keeps the appearance persistent and uses deformation to represent motion. The authors show this conceptually in a comparison between deformation traces and appearance traces: offsets evolve smoothly, whereas color changes can be sudden and difficult to fit.

Incremental sampling for stable deformation learning

The paper identifies a practical optimization issue: if the target primitive positions are too far from the current prediction, gradients can vanish and the motion field can fail to update effectively. To reduce this problem, TalkingGaussian introduces incremental sampling. The idea is to schedule training frames from easier to harder motion states using a motion metric $m$ such as action units or landmarks.

At the $k$-th training iteration, a frame $j$ is sampled from a sliding window that satisfies

$$ m_j \in [B_{\text{lower}} + kT,\; B_{\text{upper}} + kT], $$

where $B_{\text{lower}}$ and $B_{\text{upper}}$ are the initial bounds and $T$ is the step length. This makes the deformation task progressively more difficult and helps avoid catastrophic forgetting when the strategy is applied every $K$ iterations.

Face-Mouth Decomposition

The second main contribution is a Face-Mouth Decomposition module. The paper argues that the face and inside-mouth regions are spatially close but not motion-consistent: lips, teeth, and inner mouth can move differently, and a single interpolation-based motion field has difficulty representing these conflicting dynamics. This can degrade both motion quality and static reconstruction quality.

To address this, the authors split the head into two branches using a semantic mouth mask computed from off-the-shelf face parsing. Each branch is trained on a masked image: one for the surface face/head region, and one for the inside mouth region. The segmentation pipeline is further detailed in the supplement, where a BiSeNet parser is combined with a tooth parser to improve mouth coverage.

Face branch

The face branch handles the main head region, including facial motion except the inside mouth. Its motion field uses region attention and is conditioned on both audio feature $\mathbf{a}$ and upper-face expression feature $\mathbf{e}$. The expression feature is explicitly chosen to be mouth-independent and is composed of seven action units: 1, 2, 4, 5, 6, 7, and 45. For primitive $i$, the deformation is

$$ \delta_i^{\text{F}} = \operatorname{MLP}(\mathcal{H}^{\text{F}}(\mu_i) \oplus \mathbf{a}_{r,i} \oplus \mathbf{e}_{r,i}), $$

with region-aware features $\mathbf{a}_{r,i} = V_{\mathbf{a},i} \odot \mathbf{a}$ and $\mathbf{e}_{r,i} = V_{\mathbf{e},i} \odot \mathbf{e}$. The Hadamard product $\odot$ and attention vectors $V_{\mathbf{a},i}$, $V_{\mathbf{e},i}$ allow the model to focus on local regions.

During incremental sampling for this branch, the paper schedules lip opening from closed to open using mouth-height measurements from facial landmarks and uses AU45 to describe eye closure. This helps the face branch learn both lip motion and audio-independent blinking.

Inside-mouth branch

The inside-mouth branch is intentionally lightweight because the inner mouth motion is simpler and driven only by audio. It predicts only translation for each primitive:

$$ \delta_i^{\text{M}} = \{\Delta\mu_i^{\text{M}}\} = \operatorname{MLP}(\mathcal{H}^{\text{M}}(\mu_i) \oplus \mathbf{a}). $$

To improve teeth reconstruction, incremental sampling is also applied using AU25, which describes mouth opening/teeth exposure.

Rendering and branch fusion

The final head image is produced by rendering the two branches separately and then compositing them under a physical depth assumption: the mouth branch lies behind the face branch. If $\mathcal{C}_{\text{face}}$ and $\mathcal{A}_{\text{face}}$ are the face branch color and opacity, and $\mathcal{C}_{\text{mouth}}$ is the mouth branch color, then the fused color is

$$ \mathcal{C}_{\text{head}}(\mathbf{x}_p)=\mathcal{C}_{\text{face}}(\mathbf{x}_p)\,\mathcal{A}_{\text{face}}(\mathbf{x}_p) + \mathcal{C}_{\text{mouth}}(\mathbf{x}_p)\bigl(1-\mathcal{A}_{\text{face}}(\mathbf{x}_p)\bigr). $$

This compositing makes the inside mouth visible only where the face branch is not opaque, which is consistent with the intended facial layering.

Training objectives and optimization stages

The paper trains the model in three stages: static initialization, motion learning, and final fine-tuning.

Static initialization. Vanilla 3DGS is first used to initialize the persistent field with a coarse head structure. The reconstruction loss for each branch uses pixel-wise $L_1$ and D-SSIM supervision against the masked ground-truth image $\mathcal{I}_{\text{mask}}$:

$$ \mathcal{L}_C = \mathcal{L}_1(\hat{\mathcal{I}}_C, \mathcal{I}_{\text{mask}}) + \lambda\,\mathcal{L}_{\mathrm{D\text{-}SSIM}}(\hat{\mathcal{I}}_C, \mathcal{I}_{\text{mask}}). $$

Motion learning. The motion field is then activated, and the deformed parameters $\theta_D$ are rasterized to obtain $\hat{\mathcal{I}}_D$ with the same $L_1$ and D-SSIM objective:

$$ \mathcal{L}_D = \mathcal{L}_1(\hat{\mathcal{I}}_D, \mathcal{I}_{\text{mask}}) + \lambda\,\mathcal{L}_{\mathrm{D\text{-}SSIM}}(\hat{\mathcal{I}}_D, \mathcal{I}_{\text{mask}}). $$

Fine-tuning. The final stage optimizes the fused head image $\hat{\mathcal{I}}_{\text{head}}$ against the full-frame ground truth $\mathcal{I}$ with $L_1$, D-SSIM, and LPIPS:

$$ \mathcal{L}_F = \mathcal{L}_1(\hat{\mathcal{I}}_{\text{head}}, \mathcal{I}) + \lambda\,\mathcal{L}_{\mathrm{D\text{-}SSIM}}(\hat{\mathcal{I}}_{\text{head}}, \mathcal{I}) + \gamma\,\mathcal{L}_{\mathrm{LPIPS}}(\hat{\mathcal{I}}_{\text{head}}, \mathcal{I}). $$

At this final stage, only the color feature $f$ is updated, and the 3DGS densification strategy is stopped for stability. The paper reports $\lambda = 0.2$ and $\gamma = 0.5$.

Implementation Details

The implementation uses PyTorch. For a given portrait, the two branches are trained in parallel for 50,000 iterations and then jointly fine-tuned for another 10,000 iterations. Adam and AdamW are used as optimizers. The paper reports that training on an RTX 3080 Ti takes about 0.5 hours per subject, which is far faster than the compared NeRF-based methods.

In the supplement, the authors provide more detail on the motion-field parameterization: the Grid-based Motion Fields are implemented with three 2D hash encoders and a 3-layer MLP decoder. The face branch uses encoder resolutions from 16 to 256 and hidden dimension 64; the inside-mouth branch uses encoder resolutions from 64 to 384 and hidden dimension 32. Separate optimizers are maintained for the persistent Gaussian fields and the motion fields, with Adam for the Gaussians and AdamW for the motion module.

The preprocessing pipeline also includes a CNN-based attention module for audio feature smoothing and OpenFace action-unit extraction. The upper-face expression feature is built from action units 1, 2, 4, 5, 6, 7, and 45. Head pose is estimated using a BFM-based face model and converted into camera pose. For semantic masking, the supplement combines a BiSeNet parser trained on CelebAMask-HQ with a ResNet-FPN tooth parser trained on EasyPortrait to improve the inside-mouth mask.

Experimental Setup

Dataset

The experiments use four high-definition portrait speech clips gathered from previous publicly released video sets: three male portraits, Macron, Lieu, and Obama, and one female portrait, May. The clips average about 6,500 frames at 25 FPS. Three videos are cropped and resized to $512\times512$, while Obama is resized to $450\times450$.

Baselines

The paper compares against both person-specific and general-purpose talking-head methods:

• NeRF-based person-specific baselines: AD-NeRF, DFRF, RAD-NeRF, GeneFace, ER-NeRF;
• 2D generative baselines: Wav2Lip, IP-LAP, DINet;
• other person-specific baselines: SynObama, NVP, and LSP.

Evaluation settings and metrics

The paper evaluates in two settings:

• Self-reconstruction: each video is split into train/test sets, and the test audio, expression, and pose sequences are used to reconstruct the held-out frames of the same identity;
• Lip-synchronization: models trained in the first setting are driven by cross-domain audio tracks from other videos, including the challenging audio samples A and B used by prior work.

Metrics include PSNR, LPIPS, and SSIM for rendering quality; LMD for landmark motion; Sync-C and Sync-E / Sync-D style SyncNet-based measures for lip synchronization; and AUE-U / AUE-L for upper-face and lower-face action-unit errors. The paper also measures training time and inference FPS.

Quantitative Results

Self-reconstruction results

In the self-reconstruction setting, TalkingGaussian achieves the best overall balance of image quality, motion quality, and efficiency. It is particularly strong on LPIPS, SSIM, LMD, and action-unit errors, while also reaching the fastest reported training and inference speed.

The paper highlights several takeaways. First, 2D generative baselines can reach decent lip-sync scores, but they do not preserve person-specific realism and are not evaluated with the same static-image metrics in the Wav2Lip setup. Second, NeRF-based methods can achieve strong reconstruction but often suffer from facial artifacts in dynamic regions. Third, TalkingGaussian improves both fidelity and temporal quality while being much faster to train and render.

Lip-synchronization generalization

In the cross-domain lip-synchronization setting, TalkingGaussian is reported to generalize better, especially on the harder cross-gender case involving the female portrait May. The paper argues that previous NeRF-based methods tend to overfit audio-to-appearance mappings, whereas deformation-based motion is smoother and therefore more robust to unseen input audio.

Under Audio A, the model reaches 8.635 / 5.962 on Obama and 9.368 / 4.774 on May. Under Audio B, it reaches 8.627 / 6.737 on Obama and 9.273 / 5.441 on May. The authors emphasize that the hardest case is the cross-gender May setting, where many prior methods degrade strongly.

Human evaluation

The user study uses 32 generated videos from 8 methods and asks 16 participants to rate lip-sync accuracy, video realness, and image quality on a 1-5 scale. TalkingGaussian receives the best ratings on all three dimensions.

Qualitative Findings

The qualitative figures reinforce the numeric results. In the synchronization comparison, TalkingGaussian reconstructs more accurate lip motions and can also capture some audio-independent facial motions such as blinking, which many baselines fail to model. In the detail comparison, the paper shows that NeRF-based methods such as RAD-NeRF and ER-NeRF can produce distorted or blurry facial regions in dynamic areas, whereas the deformation-based representation preserves a more intact face.

Ablation and Supplementary Analysis

The ablation study isolates three ideas: the choice of motion representation, the Face-Mouth Decomposition, and Incremental Sampling. The paper uses both a Tri-Hash backbone from ER-NeRF and a 3DGS backbone to show that deformation only becomes clearly advantageous when it is paired with the explicit 3DGS structure. On Tri-Hash, deformation underperforms appearance modification because the model lacks stable point-wise control. On 3DGS, deformation better preserves geometry and becomes competitive or superior in image quality and motion quality.

The full ablation shows that Face-Mouth Decomposition especially helps lip-sync and mouth reconstruction, while Incremental Sampling stabilizes optimization and improves structural smoothness.

The supplementary material also studies a hybrid motion representation. Predicting opacity in addition to deformation does not help, and directly predicting RGB on top of deformation yields slightly higher PSNR but worse LPIPS/SSIM, confirming the design choice to keep color and opacity persistent. The best result remains the pure deformation model with $\delta$ only.

The audio encoder study shows that stronger encoders can further improve the framework without changing the architecture. Using Wav2Vec 2.0 or HuBERT gives slightly better motion metrics and lip-sync scores than DeepSpeech in the reported self-reconstruction setting. The strongest Sync-C among these variants is obtained by HuBERT at 6.667.

The supplementary cross-lingual and cross-gender experiments further support the generalization claim: on female German, male German, and male Chinese test audios, TalkingGaussian outperforms ER-NeRF or GeneFace using the same audio extractor in each comparison group. The paper also states that the method can synthesize singing faces when driven by a song, despite never seeing such training audio.

Limitations and Ethical Considerations

The paper reports two main limitations. First, because 3DGS uses densification, noisy primitives can sometimes appear; incremental sampling reduces but does not completely remove this problem. Second, the face and inside-mouth branches are aligned only through audio conditioning, which is not always strong enough for difficult cross-domain inputs. In such cases, the inside mouth can become misaligned with the face region.

From an ethical perspective, the authors explicitly acknowledge misuse risk. They recommend informed consent for training data, disclosure of synthesized content, and support for deepfake detection research. The dataset declaration says the multimedia data were taken from prior public works and were manually checked to reduce offensive content.

Takeaway

TalkingGaussian’s key technical contribution is a shift from appearance prediction to deformation prediction for talking head synthesis, implemented on top of an explicit 3D Gaussian representation. That design makes the head structure persistent, reduces distortion in dynamic regions, and simplifies motion learning. The two-branch face-mouth decomposition and incremental sampling are practical additions that improve mouth fidelity and training stability. Across reconstruction, lip synchronization, user study, and efficiency, the paper reports strong gains over NeRF-based and 2D baselines, while also showing that the framework is extensible to stronger audio features and harder cross-domain conditions.