GAGAvatar

Generalizable and Animatable Gaussian Head Avatar

GAGAvatar reconstructs animatable 3D head avatars from a single image using a novel dual-lifting Gaussian approach, enabling real-time reenactment without test-time optimization. It uniquely combines high-fidelity reconstruction with efficient expression control for generalized one-shot avatars.

• head-avatar
• avatar
• face-reconstruction
• face-animation
• face-reenactment
• 3d-avatar
• gaussian-splatting
• realtime

Demos

The demo GIF showcases GAGAvatar's ability to reconstruct and animate 3D head avatars from a single image with high quality and real-time reenactment. Watch for the smoothness of facial animation, natural head movements, and how well identity and expression are preserved across different poses and expressions. The teaser highlights the model's generalizability and effectiveness of 3D Gaussian splatting for detailed, controllable head avatar synthesis.

Authors: Xuangeng Chu, Tatsuya Harada

Categories: cs.CV, cs.GR

Comment: NeurIPS 2024, code is available at https://github.com/xg-chu/GAGAvatar, more demos are available at https://xg-chu.site/project_gagavatar

Published 2024-10-10 · Updated 2024-10-10

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub583 stars13 watchers56 forksMIT
• Hugging Face9 downloads1 likes

Impact

• Semantic Scholar95 citations35 influential citations

Abstract

In this paper, we propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. Existing methods rely on neural radiance fields, leading to heavy rendering consumption and low reenactment speeds. To address these limitations, we generate the parameters of 3D Gaussians from a single image in a single forward pass. The key innovation of our work is the proposed dual-lifting method, which produces high-fidelity 3D Gaussians that capture identity and facial details. Additionally, we leverage global image features and the 3D morphable model to construct 3D Gaussians for controlling expressions. After training, our model can reconstruct unseen identities without specific optimizations and perform reenactment rendering at real-time speeds. Experiments show that our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy. We believe our method can establish new benchmarks for future research and advance applications of digital avatars. Code and demos are available https://github.com/xg-chu/GAGAvatar.

---

1. Problem setting and high-level idea

GAGAvatar targets one-shot animatable head avatar reconstruction: given a single source image, the model must reconstruct a head avatar that preserves identity, supports expression and pose reenactment, and renders fast enough for interactive use. The paper is motivated by a gap in prior work: many high-quality head avatar systems rely on Neural Radiance Fields (NeRFs) or similar implicit representations, which often require either test-time optimization, identity-specific training, or expensive rendering. In contrast, this paper uses 3D Gaussian splatting as the representation and aims to infer the avatar in a single forward pass from one image.

The core claim is that the method is both generalizable and animatable: it is trained on many monocular portrait clips and then applied to unseen identities without per-subject optimization. Reenactment is performed at real-time speed, while the reconstruction branch captures static identity details and the expression branch transfers facial motion using a 3D morphable model prior.

2. Main contributions

• A generalizable 3D Gaussian head avatar framework that reconstructs an animatable avatar from a single image in one forward pass and supports real-time reenactment.
• A novel dual-lifting strategy that lifts Gaussians from the image plane in two directions, reducing ambiguity and helping recover a more complete 3D head structure from a single view.
• A 3DMM-guided expression branch that combines global source-image identity information with 3DMM vertex priors to generate expression-controlled Gaussians efficiently.
• A training objective that combines reconstruction losses with a lifting distance loss to constrain the lifted 3D structure using 3DMM priors.

3. Method overview

The full system has two main branches. The reconstruction branch takes a source image and reconstructs a static 3D Gaussian head avatar. The expression branch uses the source identity and a driving 3DMM to produce expression Gaussians that encode motion. The output from both branches is first rendered by Gaussian splatting into a coarse image, and then refined by a neural renderer to obtain the final reenacted frame.

The overall design is intentionally asymmetric: most identity-specific content is produced once per source image, while only a smaller driving-related component must be updated for reenactment. This is what enables the method to run fast at test time.

4. Reconstruction branch: dual-lifting from a single image

The reconstruction branch is the key technical component. The paper argues that directly regressing millions of 3D Gaussian parameters from a single image is difficult, especially without test-time optimization. Instead, the authors predict lifting distances for image-plane pixels and then map the plane back into 3D based on the camera pose.

Let $F_{\text{local}}$ denote the local feature map from a frozen DINOv2 backbone. For each pixel, the model predicts two lifting directions: one forward and one backward relative to the image plane. In simplified form, the 3D Gaussian positions are formed as

$$ \mathbf{G}_{\text{pos}} = \big[\mathbf{p}_s + f_0(F_{\text{local}})\,\mathbf{n}_s,\; \mathbf{p}_s - f_1(F_{\text{local}})\,\mathbf{n}_s\big], $$

where $\mathbf{p}_s$ is the image plane mapped into 3D space using the estimated source camera pose, $\mathbf{n}_s$ is the plane normal, and $f_0$, $f_1$ are convolutional predictors for the two lifting directions. The same branches also predict Gaussian attributes such as color, opacity, scale, and rotation.

This is called dual-lifting. The paper contrasts it with a one-plane lifting baseline that tries to lift all points from a single directional offset. The issue with a single lift is ambiguity: for a side-view face, a point might reasonably be lifted toward the visible cheek surface or toward the hidden opposite surface. Dual-lifting separates these directions, makes optimization more stable, and tends to recover a more closed 3D point distribution.

The authors emphasize that the dual-lifting prediction is performed only once per identity. After the source avatar has been reconstructed, reenactment does not require recomputing this branch for every frame.

4.1 Why dual-lifting helps

• Reduces ambiguity compared with single-direction lifting.
• Improves completeness by producing an almost closed set of lifted Gaussians.
• Better preserves facial edges and accessories, especially at large view changes.
• Leverages local image detail rather than asking the network to directly hallucinate a dense 3D volume.

4.2 Representation size and Gaussian features

In the supplementary material, the authors specify that the DINOv2 Base backbone takes $3\times 518\times 518$ input and produces a $296\times 296$ local feature map plus a 768-dimensional global feature. The reconstruction branch outputs a 41-dimensional Gaussian parameter vector per point: 32 dimensions for color features, 1 for opacity, 3 for scale, 4 for rotation, and 1 for lifting distance.

The paper also notes that the reconstruction branch produces 175,232 points, which is far fewer than the millions of points typically used by dense 3D Gaussian models. To compensate for this sparsity, the model relies on a neural renderer to refine the output.

5. Expression branch: 3DMM-guided motion control

Expression transfer is handled separately from static reconstruction. The method uses the FLAME 3D morphable model as a geometry and expression prior. Each FLAME vertex is assigned learnable features, and these features are concatenated with the global source-image identity feature $F_{\text{id}}$ extracted by DINOv2. A multilayer perceptron then predicts Gaussian parameters for each 3DMM vertex, while the vertex positions themselves are taken directly from the tracked 3DMM mesh.

In other words, the expression branch does not need to infer positions from scratch. It uses the tracked 3DMM geometry as a stable anchor and predicts the remaining Gaussian attributes from the source identity and vertex-level learnable embeddings. This design is what lets the model transfer facial motion efficiently while keeping the generated avatar tied to the source identity.

The paper explicitly says that injecting the global source identity feature into the expression branch improves identity consistency during reenactment. An ablation confirms this claim: removing $F_{\text{id}}$ degrades identity similarity and overall reenactment quality.

5.1 Driving and reuse during inference

During inference, only the expression-related computations need to react to a driving frame. The static avatar reconstructed from the source image is reused, which is one of the reasons the method achieves real-time performance. The paper also notes that the driving-related part can be precomputed or inferred separately, and the reported FPS excludes the time for obtaining driving parameters.

6. Neural renderer and coarse-to-fine synthesis

Because the reconstructed Gaussian sets are sparse compared with fully dense 3D point clouds, the paper does not rely on raw RGB splatting alone. Instead, each Gaussian carries a 32-dimensional feature vector, and the model first renders a coarse image from the combined Gaussians. A neural renderer then refines the coarse output into the final image.

The supplementary material states that the neural renderer is based on StyleUNet and maps $32\times 512\times 512$ features to a $3\times 512\times 512$ output image. The authors explicitly say this renderer is not used as a super-resolution module; unlike some prior work, it is not intended to increase image resolution, but rather to refine appearance and resolve conflicts between the static and dynamic Gaussian sets.

This renderer is trained from scratch and has no pretrained initialization. It also serves as a fusion mechanism: when dual-lifting and expression Gaussians produce conflicting RGB evidence, the renderer can reconcile them using the richer 32D features.

6.1 Per-part behavior

The supplementary material provides a per-part rendering visualization showing that the dual-lifting branch reconstructs the base head structure and facial details, while the expression branch contributes motion-related changes. Importantly, the visualization is done with only the first three dimensions of the Gaussian features, so it is not meant to be interpreted as the full behavior of the model before neural rendering.

7. Training strategy and losses

Training is done from scratch except for the frozen DINOv2 backbone. The model is trained on pairs of frames from the same video: one source image and one driving/target image. This allows the reconstruction and reenactment outputs to be supervised by a ground-truth target frame with the same identity.

The total loss combines pixel reconstruction, perceptual similarity, and the lifting constraint:

$$ \mathcal{L} = \|I_c - I_t\|_1 + \|I_f - I_t\|_1 + \lambda_p\Big(\|\phi(I_c) - \phi(I_t)\|_1 + \|\phi(I_f) - \phi(I_t)\|_1\Big) + \lambda_l\mathcal{L}_{\text{lifting}}, $$

where $I_t$ is the target image, $I_c$ is the coarse splatting result, $I_f$ is the final refined image, and $\phi$ denotes the perceptual feature extractor.

The lifting loss uses 3DMM priors to constrain the lifted geometry. For each tracked 3DMM vertex, the model finds the nearest lifted Gaussian point and penalizes their distance. The intent is to encourage the lifted cloud to align with facial anatomy while still allowing unconstrained regions such as hair and accessories to be modeled by the rest of the network.

The paper explicitly notes that the lifting loss helps improve the 3D structure and large-view-change behavior. In ablations, removing this term reduces performance and makes the structure flatter, especially near facial edges.

8. Datasets and preprocessing

The main training dataset is VFHQ, a large interview-style dataset. The authors use 15,204 video clips and sample between 25 and 75 frames per clip depending on video length, resulting in 586,382 training frames. All frames are resized to $512\times 512$. Backgrounds are removed, and both camera poses and FLAME parameters are tracked.

For testing on VFHQ, the paper uses 100 videos from the official test split, with 5,000 frames total. The first frame of each video is used as the source image, and the remaining frames are used as driving and target frames.

The paper also evaluates on HDTF, following the test split used in prior work. This test set includes 19 videos and 1,900 total frames after uniform sampling. The supplementary material states that the same background removal, resizing, and 3DMM refinement pipeline is used there as well.

The authors also note that the 3DMM annotations are automatically extracted and therefore somewhat noisy, but the scale of the dataset helps mitigate this issue.

9. Implementation details

• Framework: PyTorch.
• Optimizer: Adam with learning rate $10^{-4}$.
• Training length: 200,000 iterations.
• Batch size: 8 total.
• Backbone: DINOv2 is frozen and not fine-tuned.
• Driving 3DMM: FLAME.
• Training hardware: NVIDIA Tesla A100 GPU.
• Training time: about 46 GPU hours.
• Inference speed: 67 FPS on an A100 with 2.5 GB VRAM.

The supplementary material further specifies that the MLP in the expression branch has 6 layers and takes a 1024-dimensional input formed by concatenating global features with per-vertex features. FLAME has 5,023 vertices, each assigned a 256-dimensional learnable feature.

10. Main experimental results

The paper compares against a broad set of prior methods, including StyleHeat, ROME, OTAvatar, HideNeRF, GOHA, CVTHead, GPAvatar, Real3DPortrait, Portrait4D, and Portrait4D-v2. The authors use the official code and official preprocessing for each baseline, and they additionally realign all outputs so that face crops are compared fairly.

The evaluation metrics are:

• PSNR, SSIM, and LPIPS for reconstruction quality on self-reenactment.
• CSIM for identity similarity via face recognition features.
• AED and APD for expression and pose accuracy from a 3DMM estimator.
• AKD for landmark-based keypoint error on self-reenactment.

10.1 VFHQ results

On VFHQ, GAGAvatar obtains the best scores in every self-reenactment metric reported in the table. Compared with the strongest prior method, Portrait4D-v2, it improves PSNR, SSIM, LPIPS, CSIM, AED, APD, and AKD on self-reenactment.

The one caveat the authors highlight is cross-identity identity consistency: on VFHQ, the method’s cross-reenactment CSIM is slightly below the strongest prior methods. The paper attributes this to the fact that FLAME identity and expression parameters are not perfectly decoupled, which can limit identity consistency under cross-identity driving.

10.2 HDTF results

On HDTF, GAGAvatar again performs strongly across the self-reenactment metrics and achieves the best PSNR, SSIM, LPIPS, CSIM, APD, and AKD values in the table, with AED tied at the best value. On cross-reenactment, the method remains competitive on expression/pose metrics while keeping real-time speed.

10.3 Qualitative findings

Qualitatively, the paper reports that GAGAvatar preserves identity better than the baselines, handles subtle mouth and eye motions more accurately, and maintains much better appearance consistency under large pose changes. The authors also show improved multi-view consistency and richer details in both VFHQ and in-the-wild examples.

11. Ablation studies

The ablations isolate four design choices: dual-lifting versus one-plane lifting, the identity feature $F_{\text{id}}$ in the expression branch, the neural renderer, and the lifting distance loss $\mathcal{L}_{\text{lifting}}$.

The conclusions are consistent across metrics and visualizations: each component contributes to final quality, and the full model is best overall.

11.1 Interpretation of the ablations

• Dual-lifting vs. one-plane lifting: the two-direction strategy improves reconstruction quality and reduces ambiguous geometry, especially for side views.
• Global identity feature in the expression branch: removing $F_{\text{id}}$ lowers identity similarity and hurts reenactment quality.
• Neural renderer: without it, coarse splatting is still reasonable, but fine detail, expression fidelity, and overall metrics drop substantially.
• Lifting distance loss: removing the 3DMM-based constraint weakens the recovered 3D structure and degrades large-view performance.

The paper also visualizes lifted point clouds and notes that without the lifting loss, the recovered structure is noticeably flatter. With the full method, the lifted cloud looks more reasonable in 3D and contains better off-plane structure.

12. Speed and efficiency

A major practical result of the paper is runtime. The authors report 67.12 FPS for reenactment on an A100 GPU, which is far faster than the compared baselines. The speed table in the paper shows that many NeRF-based methods are far below real-time, while GAGAvatar is the first real-time method in this one-shot animatable head avatar setting, according to the authors.

The reported memory usage is also modest: 2.5 GB of VRAM during inference.

13. Robustness, failure cases, and limitations

The paper includes qualitative evidence that the method is robust to several challenging conditions, including low-quality images, difficult lighting, heavy occlusion, sunglasses, and extreme expressions. It can also work on in-the-wild images with large view changes.

At the same time, the paper is explicit about its limitations. The most important ones are:

• Unseen regions lack detail. When the source image does not reveal a region, the reconstructed avatar often falls back to a statistically average guess.
• 3DMM coverage is incomplete. The expression branch cannot control parts that the 3DMM does not model well, such as hair, tongue, or certain asymmetric facial motions.
• Cross-identity consistency is not perfect. The authors attribute this to entanglement between identity and expression in the 3DMM/tracker pipeline.

The paper also suggests future directions such as learning expression embeddings directly from images, using better data generation, collecting more expressive data, and extending the method from head avatars to full-body avatars.

14. Ethical discussion and broader impact

The paper acknowledges the risk that a system like GAGAvatar could be misused to generate fake videos or otherwise spread false information. The authors propose watermarking and strict licensing as mitigation strategies, including visible watermarks, invisible watermarks, and restrictions on commercial use or unauthorized synthesis of real individuals.

This is not a method limitation in the algorithmic sense, but it is a practical deployment concern for any high-fidelity talking-head generation system.

15. Takeaway

The central technical insight of GAGAvatar is that a single-image head avatar can be made both generalizable and real-time if reconstruction is split into a dual-lifting 3D Gaussian branch plus a 3DMM-guided expression branch, and if sparse Gaussian rendering is followed by a learned neural refinement stage. The method trades some cross-identity identity consistency for substantial gains in reconstruction quality, motion fidelity, and runtime, and the paper presents this as a practical step toward interactive digital avatars.