GaussianAvatars

GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

GaussianAvatars creates photorealistic, animatable head avatars by rigging 3D Gaussian splats to a parametric morphable face model. This approach combines high-fidelity rendering and precise control over expressions and poses without relying on complex implicit deformation fields.

• head-avatar
• avatar
• face-animation
• face-reenactment
• 3d-avatar
• gaussian-splatting
• talking-head

Demos

These demos showcase GaussianAvatars' capability to generate fully controllable, photorealistic head avatars rigged with 3D Gaussian splats. Watch for the avatar's high fidelity in expression and pose reenactment from real driving videos, as well as its seamless novel view synthesis that highlights the underlying 3D representation and precise animation control. These results emphasize end-to-end optimization and superior animation quality compared to prior methods.

Authors: Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, Matthias Nießner

Categories: cs.CV

Comment: Project page: https://shenhanqian.github.io/gaussian-avatars

Published 2023-12-04 · Updated 2024-03-28

Links

Paper & demos

• arXiv
• Project page

Abstract

We introduce GaussianAvatars, a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint. The core idea is a dynamic 3D representation based on 3D Gaussian splats that are rigged to a parametric morphable face model. This combination facilitates photorealistic rendering while allowing for precise animation control via the underlying parametric model, e.g., through expression transfer from a driving sequence or by manually changing the morphable model parameters. We parameterize each splat by a local coordinate frame of a triangle and optimize for explicit displacement offset to obtain a more accurate geometric representation. During avatar reconstruction, we jointly optimize for the morphable model parameters and Gaussian splat parameters in an end-to-end fashion. We demonstrate the animation capabilities of our photorealistic avatar in several challenging scenarios. For instance, we show reenactments from a driving video, where our method outperforms existing works by a significant margin.

---

Problem setting and core idea

GaussianAvatars addresses the problem of reconstructing photorealistic, animatable head avatars from multiview video. The paper is motivated by a recurring trade-off in head-avatar work: high-fidelity rendering methods such as NeRF-style models often lack direct controllability, while more controllable mesh-based or point-based avatars can struggle to reproduce the visual richness of real faces, especially in difficult regions such as hair, wrinkles, the mouth interior, and the eyes.

The key idea is to combine two complementary ingredients: (1) a parametric morphable face model (FLAME) that provides explicit control over expression, pose, and view-dependent animation, and (2) a 3D Gaussian splatting representation that can model the radiance field with high visual quality and real-time rendering. Instead of treating the Gaussian field as a free-floating static point cloud, the method rigs each Gaussian to a triangle of the FLAME mesh. This gives each splat a stable semantic anchor while still allowing the splat to deviate locally from the mesh surface so that it can capture details the mesh cannot represent well.

The paper’s high-level claim is that this hybrid representation preserves the photorealism and efficiency of 3D Gaussian Splatting while adding precise animation control through a structured parametric face model. It supports reenactment from a driving sequence, novel expressions, and novel viewpoints without requiring a learned deformation field or a canonical-space implicit representation.

Representation: rigged 3D Gaussians on a parametric face mesh

The starting point is a standard 3D Gaussian Splatting scene representation. Each Gaussian is an anisotropic primitive with mean $\boldsymbol{\mu}$ and covariance $\Sigma$, where the covariance is parameterized via rotation and scale as

$$\Sigma = R S S^\top R^\top.$$

For rendering, pixels are formed by depth-sorted alpha blending of projected Gaussians. If $\mathbf{c}_i$ is the color of the $i$-th splat and $\alpha_i'$ its effective opacity after projection, the pixel color is

$$\mathbf{C} = \sum_{i=1}^{N} \mathbf{c}_i\,\alpha_i'\prod_{j=1}^{i-1}(1-\alpha_j').$$

GaussianAvatars keeps this splatting renderer, but changes the coordinate system and attachment mechanism. For each FLAME triangle, the method defines a local frame whose origin is the triangle centroid $T$, whose axes are built from an edge direction, the triangle normal, and their cross product, and whose scale factor $k$ reflects the triangle size. A Gaussian associated with that triangle is parameterized in the triangle’s local frame by a local position $\mu$, a local rotation $r$, and a local anisotropic scale $s$. At render time, those local quantities are mapped to the global frame by

$$r' = Rr, \qquad \mu' = kR\mu + T, \qquad s' = ks.$$

This local parameterization is important for two reasons. First, it gives every splat a persistent correspondence to a particular part of the tracked face mesh, which makes animation controllable. Second, because the parameters live in triangle-local coordinates, optimization behaves more consistently across triangles of different size: the same learning rate in local space corresponds to different effective movements in global space depending on the triangle scale. The authors explicitly use this to make local position and scale updates easier to interpret and to stabilize optimization.

At initialization, the method places one Gaussian at the center of each triangle, with identity local rotation and unit scale. The splats then learn offsets and anisotropy as reconstruction proceeds. Conceptually, the Gaussians act as a radiance field layered on top of the mesh: the mesh supplies the controllable coarse geometry, while the Gaussians absorb appearance details and compensate for mesh inaccuracies or missing geometry.

Adaptive density control and binding inheritance

The paper argues that one Gaussian per triangle is not enough to represent a detailed head. Hair strands, reflective eye highlights, teeth, and sharp local appearance changes often require a much denser set of splats than the raw FLAME tessellation provides. To address this, GaussianAvatars adopts the adaptive density control strategy from 3D Gaussian Splatting: Gaussians with large view-space positional gradients are either split into two smaller splats if they are large, or cloned if they are small. Gaussians with very low opacity are periodically pruned.

The novelty is that densification must not break the rigging to the face mesh. The paper therefore introduces binding inheritance: every Gaussian stores the index of its parent triangle, and newly created Gaussians inherit the binding of the parent splat that triggered the split/clone operation. In other words, densification happens in local space, but the new splats remain attached to the same triangle so they continue to follow the face during animation.

This binding inheritance matters especially for regions that are under-observed or frequently occluded, such as the eyeballs. Standard pruning can remove all splats from such triangles, leaving the avatar unable to reconstruct those regions during animation. GaussianAvatars counters this by tracking the number of splats per triangle and enforcing that every triangle retains at least one attached Gaussian. This keeps the mesh-rigged representation complete even after repeated densification and pruning.

The paper’s qualitative and ablation results indicate that this adaptive density mechanism is important not only for fidelity but also for keeping the model controllable under novel expressions and poses.

Optimization objective and regularization

Training uses image supervision only. The rendered images are compared against the ground-truth views using a standard reconstruction loss

$$L_{\text{rgb}} = (1-\lambda)L_1 + \lambda L_{\text{D-SSIM}}, \qquad \lambda = 0.2.$$

The authors report that this photometric supervision is sufficient for strong reconstruction quality, thanks to the expressive Gaussian renderer, and that no depth or silhouette supervision is required.

However, if the splats are left unconstrained, they can drift away from the underlying mesh during optimization. The paper observes that this produces severe artifacts when the avatar is later animated with novel expressions or poses: spikes, floating blobs, cracks, and other misalignment-driven distortions. To mitigate that, GaussianAvatars adds two thresholded regularizers on the local splat parameters:

• Position regularization. The local offset of each Gaussian is penalized unless it remains within a tolerance of the triangle scale. The paper sets the threshold to $\epsilon_{\text{position}} = 1$.
• Scale regularization. The local anisotropic scale is also penalized unless it remains below a threshold relative to the parent triangle scale. The paper sets the threshold to $\epsilon_{\text{scaling}} = 0.6$.

These thresholded terms are important. If the scale regularizer is used without tolerance, the splats shrink excessively, which hurts rendering speed because more splats must be traversed before transmittance vanishes. If the position regularizer is omitted entirely, the model can overfit the training frames and achieve strong image metrics while becoming brittle under animation. The paper therefore uses the final loss

$$L = L_{\text{rgb}} + \lambda_{\text{position}}L_{\text{position}} + \lambda_{\text{scaling}}L_{\text{scaling}},$$

with $\lambda_{\text{position}} = 0.01$ and $\lambda_{\text{scaling}} = 1$, applied only to visible splats so that the regularization is coupled to actual image supervision.

Implementation-wise, optimization uses Adam. The paper reports learning rates of $5\times 10^{-3}$ for position and $1.7\times 10^{-2}$ for scaling of the Gaussians, while the remaining Gaussian parameters follow the learning-rate choices of standard 3D Gaussian Splatting. In parallel, the per-frame FLAME translation, joint rotation, and expression parameters are also fine-tuned, with learning rates $10^{-6}$, $10^{-5}$, and $10^{-3}$, respectively. The full optimization runs for 600,000 iterations. The splat position learning rate decays exponentially to $0.01\times$ its initial value by the end of training. Adaptive density control with binding inheritance is activated every 2,000 iterations starting at iteration 10,000, and opacity resets are performed every 60,000 iterations.

FLAME tracking, preprocessing, and geometry tweaks

The method relies on an upstream photometric head tracker based on Face2Face-style tracking to fit FLAME to every time step using multiview observations and known camera parameters. The appendix provides additional details: the tracker optimizes per-frame translation $t_i$, joint pose $\theta_i$, and expression $\psi_i$, together with shared shape $\beta$, vertex offset $\Delta v$, and an albedo map. The fitting uses a landmark loss, a color loss, and regularizers, and it is refined over multiple stages.

More specifically, the appendix states that 68 facial landmarks are extracted in 300-W format, with 17 contour landmarks excluded to reduce inconsistencies from occlusion. The color-driven mesh fitting uses NVDiffRast so that gradients of vertex positions can be obtained by differentiable rendering, including texel interpolation in the interior and anti-aliasing on boundaries. The optimization is first run to convergence on the first time step, then for 50 iterations on each subsequent time step using the previous fit as initialization, and then globally fine-tuned for 30 epochs by randomly sampling time steps.

The authors also modify FLAME’s template to better handle mouth and eye details. They use the 2023 FLAME revision for improved eye regions and manually add 168 triangles for teeth. Those teeth triangles are rigidly attached to the neck and jaw joints, which helps the Gaussian splats reconstruct the mouth interior more faithfully.

Before splat optimization, the raw images are preprocessed to focus the reconstruction on the head region. The paper removes the background using Background Matting V2 and also removes pixels below a line projected from the tracked FLAME mesh’s bottom vertices, which effectively crops away the shoulders and lower body.

This preprocessing and mesh augmentation are part of the overall reconstruction pipeline: the avatar is not just a free-floating radiance field, but a tightly controlled head-centric model whose geometry is made more suitable for capturing teeth, eyes, and the face boundary.

Experimental protocol

The paper evaluates the method on 9 subjects from the NeRSemble dataset. Each recording contains 16 views covering the front and sides of the head, and each subject has 11 sequences. The videos are downsampled to a resolution of $802 \times 550$. Ten of the sequences are prescribed expressions or emotions, and the last one is a free-performance sequence.

The authors evaluate three settings:

• Novel-view synthesis: drive the avatar with poses and expressions from the training sequences and render from a held-out view.
• Self-reenactment: drive the avatar with unseen poses and expressions from a held-out sequence of the same subject and render all 16 views.
• Cross-identity reenactment: drive one reconstructed avatar with pose and expression parameters from another subject.

For the quantitative comparison, the paper trains GaussianAvatars and three baselines on 9 of the 10 prescribed sequences and 15 of the 16 available cameras, leaving one prescribed sequence out for self-reenactment evaluation. The free-performance sequence is reserved for cross-identity qualitative evaluation.

The compared baselines are:

• INSTA, which warps points according to the nearest FLAME triangle and uses a mouth-focused strategy for the inner mouth region.
• PointAvatar, a point-based avatar that uses pose and expression conditioning for deformation and applies a coarse-to-fine strategy on the point cloud.
• AvatarMAV, which uses voxel grids and learned motion bases conditioned on expression vectors.

Quantitative results

GaussianAvatars reports the strongest overall results on novel-view synthesis and competitive results on self-reenactment. The paper emphasizes that self-reenactment is partly limited by tracking inaccuracies in FLAME, so pixel-wise measures such as PSNR are not always the best reflection of perceptual quality.

On novel-view synthesis, the paper reports a clear margin over all baselines: PSNR rises to 31.6, SSIM to 0.938, and LPIPS falls to 0.065. The authors interpret this as evidence that the rigged Gaussian representation is both expressive and stable enough to recover fine details while retaining view consistency.

On self-reenactment, the method achieves the best SSIM and LPIPS, while PSNR is slightly below INSTA. The paper attributes this to the fact that the reenactment target is itself generated from tracked FLAME meshes, which can be imperfectly aligned to the ground-truth images; thus perceptual metrics are more informative than PSNR alone.

The qualitative examples show the same pattern: GaussianAvatars reconstructs sharper eyes, better specular highlights, more detailed hair, and clearer teeth than the baselines. The results are especially strong where the face model must be supplemented by local radiance-field detail.

Qualitative behavior and comparison to prior work

The paper’s visual analysis highlights several recurring failure modes in competing methods. PointAvatar can produce dotted or speckled artifacts because point size is fixed or aggressively tuned. INSTA often looks clean on the facial region but becomes noisier around the neck and shoulders because its warping depends on the nearest triangle, which is vulnerable to FLAME tracking errors in those regions. AvatarMAV can look reasonable on training-like views but struggles with expressions far from the training distribution, because its expression-to-motion mapping must be learned rather than explicitly constrained by a mesh rig.

GaussianAvatars is presented as a more robust compromise: the mesh provides direct controllability and better generalization, while the Gaussians recover appearance details that the mesh alone cannot represent. The paper specifically notes strong reconstruction of reflective eye highlights, hair strands, teeth, and wrinkles, all of which are difficult for purely mesh-driven avatar pipelines.

In cross-identity reenactment, GaussianAvatars transfers eye blinks, mouth motion, and dynamic wrinkles convincingly. The paper uses this setting to demonstrate that the avatar can respond to pose and expression from a different person while preserving the identity-specific appearance reconstructed from the target subject.

Ablation study

The ablation study is performed on subject #304 and isolates the effects of adaptive density control, scale regularization, position regularization, thresholding, and FLAME fine-tuning. The table below summarizes the reported values.

The ablations support several concrete conclusions:

• Adaptive density control with binding inheritance is essential for detail. Without it, the model cannot add enough splats in complex regions, so large Gaussians become blurry proxies and image quality drops sharply.
• Scale regularization suppresses spike-like artifacts. Without it, the learned splats can become visually unstable; without the threshold, the scale penalty becomes over-constraining and can collapse the splats to very small sizes, harming both quality and efficiency.
• Position regularization trades a small amount of training-frame fit for much better controllability. Removing it can improve raw novel-view metrics, but the resulting avatar is overfit and produces cracks and floating blobs under unseen expressions and poses.
• FLAME fine-tuning improves alignment between the mesh and the input images, which in turn improves both novel-view synthesis and perceptual reenactment quality.

The paper’s takeaway from the ablation is that the method’s best results come from the combination of mesh rigging, adaptive densification, and explicit geometry-aware regularization. None of these pieces alone is sufficient to guarantee controllable photorealistic animation.

Efficiency and rendering speed

GaussianAvatars is designed to remain efficient despite the dynamic density control. In the appendix, the authors report that the number of Gaussians grows from the initial FLAME triangle count of 10,144 to roughly 100,000 on average, and then stabilizes. Even with this increase, the runtime of a training iteration grows by less than 100%, which the paper attributes to the efficiency of the tiled splatting renderer and to the scale threshold that prevents splats from shrinking excessively.

The method can be trained on an NVIDIA RTX 2080 Ti with 12 GB of VRAM. At inference time, it supports arbitrary output resolution. The reported rendering speed for subject #306 is 187 FPS at both $401 \times 225$ and $802 \times 550$, 156 FPS at $1604 \times 1100$, 95 FPS at $3208 \times 2200$, and 36 FPS at $6416 \times 4400$. The paper presents this as evidence that the representation is suitable for real-time applications, especially at training and display resolutions near the native dataset resolution.

Limitations and potential negative impacts

The paper is explicit about several limitations. First, because the method directly models a radiance field, it does not decouple material from lighting, so relighting is not supported. Second, controllability is limited to the regions that FLAME models well. Hair, accessories, and other non-FLAME structures are not explicitly controlled, and the authors identify those as promising future directions.

The paper also discusses the broader ethical risks of photorealistic avatar technology. Such models can enable privacy violations, unauthorized manipulation of a person’s likeness, deepfake generation, misinformation, defamation, identity theft, and impersonation. The authors explicitly condemn malicious or unauthorized use of the technology.

Conclusion

GaussianAvatars shows that a rigidly organized Gaussian radiance field can be made both photorealistic and animatable by attaching the splats to a parametric FLAME mesh. The local triangle-based parameterization, binding inheritance during densification, and thresholded position/scale regularization are the central technical ingredients that make the system work. On the NeRSemble benchmark, the method produces the best reported novel-view metrics and strong reenactment quality, while also remaining efficient enough for real-time rendering. The paper’s overall message is that explicit geometric control and high-fidelity Gaussian rendering can be combined without resorting to a canonical implicit deformation field.