HHAvatar

HHAvatar: Gaussian Head Avatar with Dynamic Hairs

HHAvatar creates high-fidelity 3D head avatars with dynamic hair motion using controllable 3D Gaussians. It separates hair dynamics from facial expressions and uses a geometry-guided initialization for stable training, enabling realistic 2K rendering from sparse views with superior detail and hair animation.

• talking-head
• avatar
• head-avatar
• 3d-avatar
• gaussian-splatting

Demos

The demos highlight HHAvatar's ability to create ultra high-fidelity 3D head avatars with dynamic hair motion and precise expression control, even under sparse multi-view setups. Watch for the sharp rendering details like beards, teeth, and hair movements that adapt naturally with head poses and expressions. The videos demonstrate cross-identity reenactment showcasing realistic expression transfer and hair dynamics, while the images compare HHAvatar against other state-of-the-art methods, emphasizing superior quality in both static and dynamic hair rendering.

Authors: Zhanfeng Liao, Yuelang Xu, Zhe Li, Qijing Li, Boyao Zhou, Ruifeng Bai, Di Xu, Hongwen Zhang, Yebin Liu

Categories: cs.CV, cs.GR

Comment: Project Page: https://liaozhanfeng.github.io/HHAvatar

Published 2023-12-05 · Updated 2024-11-20

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar43 citations2 influential citations

Abstract

Creating high-fidelity 3D head avatars has always been a research hotspot, but it remains a great challenge under lightweight sparse view setups. In this paper, we propose HHAvatar represented by controllable 3D Gaussians for high-fidelity head avatar with dynamic hair modeling. We first use 3D Gaussians to represent the appearance of the head, and then jointly optimize neutral 3D Gaussians and a fully learned MLP-based deformation field to capture complex expressions. The two parts benefit each other, thereby our method can model fine-grained dynamic details while ensuring expression accuracy. Furthermore, we devise a well-designed geometry-guided initialization strategy based on implicit SDF and Deep Marching Tetrahedra for the stability and convergence of the training procedure. To address the problem of dynamic hair modeling, we introduce a hybrid head model into our avatar representation based Gaussian Head Avatar and a training method that considers timing information and an occlusion perception module to model the non-rigid motion of hair. Experiments show that our approach outperforms other state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering quality at 2K resolution even under exaggerated expressions and driving hairs reasonably with the motion of the head

---

Introduction and Problem Setting

HHAvatar addresses the problem of reconstructing a high-fidelity, animatable 3D head avatar from lightweight sparse-view video while also modeling dynamic hair motion. The paper is motivated by a gap in existing head avatar systems: methods can often reconstruct faces or full heads, but they still struggle to simultaneously deliver (1) ultra high-frequency detail at 2K resolution, (2) accurate control under exaggerated expressions, and (3) physically plausible hair dynamics driven by head motion. The authors position 3D Gaussian Splatting as an attractive explicit representation for this setting because it supports high-quality rendering and efficient optimization, but they argue that prior Gaussian head avatar methods still treat hair as if it were rigidly attached to the head and therefore cannot model the non-rigid, inertia-dependent motion of hair.

The core idea of HHAvatar is to represent the head with controllable 3D Gaussians and to split the avatar into components that are modeled differently according to their dynamics: the face/head region is driven by facial expression and pose, while the hair additionally depends on temporal history, including previous hair positions and head motion. The method also introduces a geometry-guided initialization stage built around implicit signed distance fields and Deep Marching Tetrahedra to stabilize training and provide a good starting geometry for both the head and hair.

High-Level Contributions

• A Gaussian head avatar representation that uses 3D Gaussians plus a learned deformation field to model expressive human heads at high fidelity.
• A hair-specific dynamic model that separates hair from the rest of the head and conditions its motion on temporal history and head movement, enabling physically plausible hair animation under sparse-view supervision.
• A geometry-guided initialization strategy that first learns an SDF-based mesh guidance model and then transfers it to Gaussian initialization, improving convergence and enabling better coverage of both head and hair regions.
• Extensive experiments on self-reenactment, cross-identity reenactment, and novel-view synthesis, including a self-captured dataset with dynamic hair motion, showing stronger perceptual quality and better 3D consistency than the reported baselines.

Method Overview

The pipeline has two stages. First, a mesh guidance model is optimized to obtain a plausible neutral geometry, per-vertex features, and deformation/color predictors. Second, these learned quantities are used to initialize a dynamic Gaussian avatar, which is then trained end-to-end with differentiable rendering, a super-resolution network, and mask supervision.

The model takes as driving signals facial expression coefficients $\theta_t$, head pose $\beta_t$, and for hair also temporal history through the previous expressive Gaussian states at times $t-1$ and $t-2$. The paper emphasizes that this is not just pose-driven articulation: hair motion is treated as a temporally conditioned deformation problem because hair depends on inertia and prior states.

Avatar Representation

HHAvatar uses the standard 3D Gaussian representation with point positions, colors, rotations, scales, and opacities. Rendering is written as

where $\mu$ denotes camera parameters and $I$ is the rendered image. The paper then extends this static formulation to a dynamic, expression-controlled avatar. The canonical neutral model stores Gaussian positions $\mathbf{X}_0 \in \mathbb{R}^{N \times 3}$, point features $\mathbf{F}_0 \in \mathbb{R}^{N \times 128}$, rotations $\mathbf{Q}_0$, scales $\mathbf{S}_0$, and opacities $\mathbf{A}_0$. Notably, there is no fixed neutral color field; instead, color is predicted dynamically from the learned point features.

The dynamic generator $\Phi$ maps the neutral Gaussian state plus driving signals to time-varying Gaussian attributes:

The authors explicitly separate facial-expression-driven deformation from pose-driven deformation, and they use spatial weights to localize which Gaussians should be affected more by expression versus pose.

Head Deformation and Appearance

For the head region, the paper predicts expression-conditioned and pose-conditioned displacements with separate MLPs. In the canonical space, the deformed positions are

with $\lambda_{pose}(x) = 1 - \lambda_{exp}(x)$. The weighting function is distance-based: points close to the fitted facial landmarks are more affected by expression, while points farther away are more influenced by pose. The paper uses thresholds $t_1 = 0.15$ and $t_2 = 0.25$ (with head length normalized to about 1) and defines

where $\mathbf{P}_0$ are canonical 3D facial landmarks. For the head appearance, color and other attributes are also predicted by expression- and pose-conditioned MLPs:

This design is intended to preserve high-frequency detail while avoiding the expressiveness limits of linear blend skinning, which prior methods often rely on via morphable models such as FLAME.

Dynamic Hair Modeling

The key novelty of HHAvatar is that hair is not modeled as a rigid extension of the face. Instead, the hair deformation MLP takes into account temporal history and head motion. The paper assumes that hair at time $t$ depends on the previous hair positions $X'_{t-1}$ and $X'_{t-2}$, the current and past head poses $\beta_t$, $\beta_{t-1}$, $\beta_{t-2}$, and the implicit hair speed derived from those states.

where $X'_{t-1,t-2}$ and $\beta_{t-1,t-2,t}$ denote the previous expressive Gaussian positions and head pose history used by the hair deformation module. The hair weight $\lambda_{hair}$ is again distance-based, but this time measured from the scalp: points close to the scalp are more strongly affected by head motion, while points farther away are less constrained and can swing more freely. The paper sets $t_3 = 0.05$ and $t_4 = 0.15$.

Here $\mathbf{P}_1$ denotes the scalp estimate from FLAME preprocessing. Hair attributes are then updated by separate hair color and hair attribute MLPs:

Finally, the canonical-space Gaussians are transformed into world space using the head pose, but only directional quantities are rigidly transformed; color, scale, and opacity remain unchanged. The authors stress that this temporal conditioning is essential because hair motion is not determined by pose alone, but by motion history and inertia.

Training Objective

HHAvatar is trained with multi-view RGB videos and hair/head masks. In each iteration, the model renders a 32-channel $512 \times 512$ feature image and then uses a super-resolution network to produce a $2048 \times 2048$ RGB output. This super-resolution stage is intended both to recover fine detail and to suppress nuisance appearance artifacts such as uneven illumination and camera chromatic aberration.

The overall loss combines a reconstruction loss and a mask loss:

The reconstruction term uses foreground RGB supervision with an $L_1$ loss plus a VGG perceptual loss, and also applies the same loss to the low-resolution RGB channels extracted from the first three channels of the rendered 32-channel image:

The reported weights are $\lambda_{vgg} = 0.1$ and $\lambda_{lr} = 1$.

The mask loss uses face parsing to supervise head and hair separation. The head and hair Gaussians are temporarily assigned distinct colors, rendered, and compared against parsed masks:

The paper sets $\lambda_{mask} = 0.1$.

Geometry-Guided Initialization

A central claim of the paper is that naive Gaussian initialization is unstable because Gaussian points are discrete and their gradients do not diffuse across the whole space as they do in continuous implicit representations. Similarly, directly initializing from a FLAME template is insufficient because FLAME does not adequately cover long hair and shoulders. To solve this, the authors first train a mesh guidance model based on implicit SDFs and then transfer it to Gaussian initialization.

The guidance model learns two SDF fields, one for the head and one for the hair, and also predicts per-point feature vectors. The SDF is converted into meshes using Deep Marching Tetrahedra, and the mesh is optimized under RGB, silhouette, landmark, offset, and Laplacian constraints. Importantly, the initialization stage temporarily ignores the extra physical inertia of hair; this is reserved for the Gaussian avatar stage.

The guidance stage uses the following losses:

The reported weights are $\lambda_{sil} = 0.1$, $\lambda_{def} = 1$, $\lambda_{offset} = 0.01$, $\lambda_{lmk} = 0.1$, $\lambda_{lap} = 100$, and $\lambda_{mask} = 0.1$.

After the guidance model converges, the neutral mesh vertices and their per-vertex features are transferred directly into the neutral Gaussian positions and Gaussian features. The learned expression color and expression/pose deformation MLPs are retained, while the other Gaussian attributes use the standard Gaussian Splatting initialization. The hair-specific MLPs and the super-resolution network are randomly initialized at this stage.

The authors report that this initialization procedure takes only around 10 minutes.

Datasets and Experimental Setup

The paper evaluates on 15 datasets: 10 identities from NeRSemble, 2 identities from HAvatar, and 3 self-captured datasets with dynamic hair motion. The self-captured sets are included because NeRSemble does not contain complete non-rigid dynamic hair trajectories such as nodding and swinging at different speeds.

• NeRSemble: 10 identities, each with 2500 to 3000 frames, 16 cameras distributed about 120 degrees in front, and synchronized 2K-resolution video. The sequences marked FREE are used for evaluation, with the rest for training.
• HAvatar: 2 identities, each with 3000 frames, 8 cameras distributed about 120 degrees in front, and 4K-resolution capture that is later cropped to the face area and resized to 2K.
• Self-captured dynamic-hair data: 3 identities, about 1000 frames per identity, 4 cameras distributed about 90 degrees in front, and 4K-resolution capture cropped and resized to 2K.

Preprocessing includes background removal, face parsing for hair/head masks, extraction of 68 2D facial landmarks, and BFM fitting to obtain 3D landmarks, expression coefficients, and head pose per frame. The paper defines the canonical 3D landmarks as the usual 68 landmarks indexed as multiples of 100 in the BFM vertices.

Training uses Adam. In the geometry-guided initialization stage, the learning rate is $10^{-3}$ for all networks and $10^{-4}$ for the neutral 3D landmarks, trained for 10,000 iterations with batch size 4. In the Gaussian model stage, the learning rates are $10^{-4}$ for the color, deformation, and attribute MLPs, $10^{-5}$ for neutral positions and point-wise features, $10^{-4}$ for the neutral rotation, $3 \times 10^{-4}$ for the neutral scale, $10^{-3}$ for the neutral opacity, and $10^{-4}$ for the super-resolution network. The Gaussian model is trained for 600,000 iterations with batch size 1.

Quantitative Results

The paper reports results on three evaluation settings: self reenactment on NeRSemble/HAvatar-style data, self reenactment with dynamic hairs, and 3D consistency / novel-view synthesis. Across all settings, HHAvatar is reported to improve perceptual quality and to better preserve fine detail.

Self Reenactment

The authors note that the full model does not necessarily maximize PSNR or SSIM relative to the no-super-resolution variant, but it substantially improves perceptual metrics, especially LPIPS and FID at 2K, which they interpret as evidence that the super-resolution network recovers more convincing high-frequency detail.

Self Reenactment with Dynamic Hairs

This setting is particularly important because the paper argues that existing methods treat hair as rigidly attached to the head and therefore fail to capture motion blur-free, physically plausible swing and inertia. HHAvatar is reported to improve all metrics in this setting and to reconstruct sharper hair while preserving the face behind occluding strands.

3D Consistency / Novel View Synthesis

The paper uses an 8-camera hold-out evaluation for 3D consistency and also shows novel-view synthesis for both the multi-view facial data and the dynamic-hair self-captured data. HHAvatar is reported to outperform the baselines in 3D consistency while preserving details under viewpoint change.

Ablation Studies

The ablations isolate three design choices: geometry-guided initialization, fully learned deformation, and the hair-specific modules.

Initialization Strategy

The FLAME-initialization baseline struggles to cover long hair and shoulders, leading to sparse or blurred hair strands. The proposed SDF + DMTet initialization yields denser, better-shaped geometry for these regions and also improves convergence stability.

Hair Dynamics and Occlusion Perception

The paper reports that removing the dynamic-hair branch eliminates realistic swing and inertia, while removing occlusion perception can create holes in the face when hair moves in front of it. The full model is therefore required to obtain both plausible hair motion and robust head/hair separation.

Qualitative Behavior and Failure Modes

Qualitatively, the method is reported to reconstruct fine structures such as beards, teeth, eyes, and detailed hair strands more faithfully than prior NeRF-based and Gaussian-based baselines. On dynamic hair datasets, the paper emphasizes that other methods tend to produce blurry hair and sometimes blurry facial regions behind the hair because they ignore occlusion and kinematic state.

The paper also includes a failure case showing that the method can produce less exaggerated results when the target expressions are beyond the training distribution. This limitation is consistent with the authors' broader observation that the avatar can only render expressions that are sufficiently represented in the training set.

Limitations and Ethical Considerations

The authors explicitly discuss several limitations. First, tongue and teeth inside the mouth can become blurry because the method does not include dedicated tracking for these regions. Second, the reconstructed avatar cannot generate expressions that are not present in the training set, so very exaggerated motions may be damped. Third, if the training data contains head motions that are too fast and cause hair to fully cover facial features, fitting the 3DMM expression coefficients can fail, so the method benefits from avoiding overly rapid head movement.

The paper also notes an ethical concern: the method can create artificial portrait videos and could therefore be misused to spread misinformation or reduce trust in media authenticity.

Conclusion

HHAvatar extends Gaussian-based head avatar reconstruction to the harder setting of dynamic hair modeling. Its main technical ingredients are a fully learned expression- and pose-conditioned deformation field, a temporal hair module conditioned on past states and head motion, an occlusion-aware hair/head separation mechanism, and a geometry-guided initialization scheme based on SDFs and DMTet. Within the scope of the reported experiments, the method improves perceptual quality at 2K resolution, preserves fine appearance details, and produces more plausible hair motion than the compared sparse-view head avatar baselines.