FlashAvatar
FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding
FlashAvatar is a fast, high-fidelity 3D head avatar reconstruction method from short monocular videos. It uniquely combines Gaussian fields embedded on a face mesh with learned offsets to capture detailed facial and non-surface features for real-time photorealistic rendering at 300 FPS.
Demos
The demos showcase FlashAvatar's ability to reconstruct high-fidelity digital head avatars from monocular videos in minutes, highlighting its efficient 3D Gaussian embedding and fast rendering at over 300 FPS. Viewers should focus on the photo-realistic detail fidelity, subtle facial expressions, and the smooth, high-speed animation and novel view synthesis across different poses. The comparison video reveals superior visual quality and personalized detail retention versus prior works, and the training demo emphasizes the method's remarkable efficiency.
Links
Paper & demos
Abstract
We propose FlashAvatar, a novel and lightweight 3D animatable avatar representation that could reconstruct a digital avatar from a short monocular video sequence in minutes and render high-fidelity photo-realistic images at 300FPS on a consumer-grade GPU. To achieve this, we maintain a uniform 3D Gaussian field embedded in the surface of a parametric face model and learn extra spatial offset to model non-surface regions and subtle facial details. While full use of geometric priors can capture high-frequency facial details and preserve exaggerated expressions, proper initialization can help reduce the number of Gaussians, thus enabling super-fast rendering speed. Extensive experimental results demonstrate that FlashAvatar outperforms existing works regarding visual quality and personalized details and is almost an order of magnitude faster in rendering speed. Project page: https://ustc3dv.github.io/FlashAvatar/
1. Problem Setting and Core Idea
FlashAvatar targets the reconstruction of a personalized, animatable head avatar from a short monocular video, with an emphasis on two constraints that are usually hard to satisfy simultaneously: high-fidelity visual quality and very fast rendering. The paper positions this as a building block for real-time multimodal digital humans, where head rendering should be efficient enough to leave compute budget for other tasks such as speech or language processing.
The central design choice is to represent the avatar as a mesh-embedded 3D Gaussian field rather than a purely implicit radiance field or a purely mesh-based deformation model. The tracked face mesh provides strong geometric priors for expression and pose, while Gaussian splatting provides a non-neural, fast-rendering representation that can capture high-frequency appearance. FlashAvatar then learns extra spatial offsets on top of this surface-attached initialization to account for non-surface regions such as hair, glasses, and subtle facial details such as wrinkles and mouth interiors.
The paper’s main claim is that if the Gaussian field is initialized uniformly on the face surface using UV-space sampling, the number of Gaussians can be kept in the roughly $10^4$ range while still producing photo-realistic outputs at around $300$ FPS on a consumer GPU at $512 \times 512$ resolution.
2. Motivation and Relation to Prior Approaches
The introduction contrasts FlashAvatar with three major lines of prior work. First, 3D morphable model and mesh-based methods are compatible with the graphics pipeline and extrapolate well to unseen deformations, but coarse topology makes them weak on complex hair, accessories, and other non-surface structures. Second, implicit neural representations can model fine detail and 3D consistency, but training and inference are usually much slower. Third, acceleration techniques for NeRF-like models, including voxel grids and hash-based structures, improve speed but still rely on volume sampling and alpha compositing, which limits inference efficiency.
The paper treats 3D Gaussian Splatting as a better primitive for real-time avatar rendering because it is non-neural, differentiable, and fast to rasterize. However, the authors argue that a generic dynamic-scene strategy of the form “canonical Gaussian field + deformation field” is not robust enough for complex head motion and expression changes, even if expression codes are used instead of timestamps. FlashAvatar therefore couples Gaussians directly to a tracked face mesh and reserves a separate learned offset field for the residual dynamics that the mesh cannot express.
3. Background: Gaussian Splatting Formulation Used by the Paper
Each 3D Gaussian is defined by a center $\mathbf{\mu}$ and covariance matrix $\mathbf{\Sigma}$, with density
$$g(\mathbf{x}) = \exp\!\left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^T \mathbf{\Sigma}^{-1}(\mathbf{x}-\mathbf{\mu})\right).$$
Following 3D Gaussian Splatting, the covariance is decomposed into rotation and scaling factors:
$$\mathbf{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^T\mathbf{R}^T,$$
where $\mathbf{R}$ is parameterized by a learnable quaternion and $\mathbf{S}$ by a learnable scaling vector. Under a viewing transform $W$ and the projective Jacobian $J$, the covariance is projected to screen space as
$$\mathbf{\Sigma}' = J W \mathbf{\Sigma} W^T J^T.$$
Each Gaussian also carries opacity $o$ and spherical harmonic coefficients $\mathbf{h}$ for color. Final pixel color is obtained by sorted alpha blending:
$$\mathbf{C} = \sum_{i \in N} \mathbf{c}_i \alpha_i \prod_{j=1}^{i-1}(1-\alpha_j).$$
The paper’s key insight is that head avatars have a constrained spatial support compared with unconstrained scenes, so the Gaussian set should be initialized and kept in a stable, surface-aware configuration rather than grown by aggressive densification.
4. Method Overview
The input is a monocular video $I = \{I_i\}$ together with camera intrinsics $\mathbf{K}$, camera poses $\mathbf{P} = \{P_i\}$, tracked FLAME meshes $\mathbf{M} = \{M_i\}$, and corresponding expression codes $\Psi = \{\psi_i\}$. The output is a personalized avatar that can be animated by new tracked expressions and posed for novel viewpoints.
The method has three main components:
- Surface-embedded Gaussian initialization on the mesh via UV sampling.
- Gaussian offset prediction conditioned on expression code to model non-surface and subtle dynamic details.
- Training with photometric and perceptual losses, with special emphasis on the mouth region.
4.1 Surface-Embedded Gaussian Initialization
Rather than attaching Gaussians directly to vertices or sampling them naively over faces, FlashAvatar uses UV-space sampling. The FLAME mesh is rasterized into UV space so that each UV pixel corresponds to a point on the mesh surface. Sampling in UV space produces a much more uniform spatial distribution of Gaussians and makes it easy to control density by changing UV resolution, sampling interval, or semantic coverage of specific facial regions.
Because the FLAME topology is fixed, the UV parameterization can be reused across frames. The 3D position of a sampled Gaussian on a deformed face is recovered by barycentric interpolation of the tracked mesh vertices. The initial surface-attached position is denoted $\mathbf{\mu}_M$, while the underlying canonical mesh location is denoted $\mathbf{\mu}_T$.
The paper also explicitly modifies the mesh to better represent the interior mouth: it adds faces to close the mouth cavity and broadens the corresponding UV region so the model can place Gaussian support in the interior mouth area instead of forcing it to be approximated by nearby lip Gaussians.
4.2 Gaussian Offset Network
Surface attachment handles the bulk motion caused by expression changes, but it cannot represent non-surface geometry or subtle local deformations. FlashAvatar therefore adds an MLP offset field $F_\theta$ that takes as input the positional encoding $\gamma(\mathbf{\mu}_T)$ and the tracked expression code $\psi$, and predicts residual offsets for position, rotation, and scale:
$$\Delta \mathbf{\mu}_\psi, \Delta \mathbf{r}_\psi, \Delta \mathbf{s}_\psi = F_\theta(\gamma(\mathbf{\mu}_T), \psi).$$
The final spatial parameters become
$$\mathbf{\mu}_\psi, \mathbf{r}_\psi, \mathbf{s}_\psi = (\mathbf{\mu}_M \oplus \Delta\mathbf{\mu}_\psi, \mathbf{r} \oplus \Delta\mathbf{r}_\psi, \mathbf{s} \oplus \Delta\mathbf{s}_\psi).$$
The authors emphasize that scaling should also be dynamic, not only position and rotation. Their motivation is that the effective influence region of a Gaussian can change with facial motion and with the deformation of the underlying mesh, especially for structured regions such as teeth. Allowing learned scale residuals helps the model capture these effects.
The offset MLP uses $D=5$ layers and hidden width $W=256$. The supplementary material indicates that all hidden linear layers use ReLU activations except for the final layer.
4.3 Training Objective
For each expression instance, the Gaussian field is rendered into an image $\hat{I}$. The paper uses a robust Huber photometric loss with threshold $\delta = 0.1$:
$$\mathcal{L}_H(x, \hat{x}) = \begin{cases} \frac{1}{2}(x-\hat{x})^2, & \text{if } |x-\hat{x}| < \delta \\ \delta \left((x-\hat{x}) - \frac{1}{2}\delta\right), & \text{otherwise.} \end{cases}$$
The color loss augments the global photometric term with extra weight on the mouth region using a mouth mask $\mathcal{M}$:
$$\mathcal{L}_C = \mathcal{L}_H(I, \hat{I}) + \lambda_{\text{mouth}}\, \mathcal{L}_H(I \cdot \mathcal{M}, \hat{I} \cdot \mathcal{M}).$$
A perceptual term $\mathcal{L}_{\text{lpips}}$ is then added, with a VGG backbone as the LPIPS feature extractor:
$$\mathcal{L} = \mathcal{L}_C + \lambda_{\text{lpips}}\,\mathcal{L}_{\text{lpips}}.$$
The paper notes that the perceptual loss both improves fine detail and stabilizes training by providing structural supervision.
4.4 Implementation Details
The system is implemented in PyTorch, uses PyTorch3D for mesh rasterization, and keeps the differentiable Gaussian rasterizer from 3D-GS. FLAME tracking is performed with the analysis-by-synthesis tracker from MICA, further modified in INSTA. The expression code is the concatenation of tracked expression coefficients, eye pose, jaw pose, and eyelid coefficients.
For initialization, the UV map resolution is set to $128$, the head region including the neck is sampled, and the total number of Gaussians is $13{,}453$. The paper also mentions that only the head region is sampled and that this is fewer than the FLAME vertex count of 5023 only in the sense of sampled points on the head region representation described in the figure caption? More importantly, the paper’s key point is that the sample distribution is much more uniform than direct vertex attachment.
The optimization uses Adam with $\beta=(0.9,0.999)$. The learning rate for the offset network is $10^{-4}$. The loss weights are $\lambda_{\text{mouth}} = 40$ and $\lambda_{\text{lpips}} = 0$ for the first $15{,}000$ steps and $0.05$ afterward. For each epoch, the training sampler draws $2000$ frames from the training set.
The supplementary material additionally states that the FLAME mesh boundary is excluded from Gaussian sampling via a mask, limiting Gaussians to the head-and-neck region.
5. Datasets and Evaluation Protocol
The paper mainly evaluates on monocular videos released by prior head-avatar works, specifically data associated with previous reconstruction and rendering methods such as Gao et al., INSTA, Neural Head Avatar, and NHA-related setups. Each video is cropped, downsampled to $25$ FPS, and resized to $512^2$. The processed sequences are between $1$ and $3$ minutes long, and the last $500$ frames are used as test data.
Foreground segmentation is obtained with RVM, and mouth-region parsing is done with an off-the-shelf face parsing framework. These preprocessing steps support the training losses and the region-specific evaluation of appearance quality.
The paper reports standard image-quality metrics for head avatar reconstruction: MSE, L1, PSNR, SSIM, and LPIPS. For the reported benchmark table, the authors compute averages across test videos.
6. Experimental Results
6.1 Comparison with Representative Methods
FlashAvatar is compared with three representative baselines: Neural Head Avatar (NHA), PointAvatar, and INSTA. The paper uses NHA as a mesh-based explicit baseline, PointAvatar as a point-based explicit representation, and INSTA as a recent efficient implicit method with tracked neck region support. The authors also note that PointAvatar’s full training is heavy enough to require an $80$ GB A100 GPU according to the original setup, whereas their own experiments are done on a single $24$ GB RTX 3090, with PointAvatar trained under the authors’ recommended reduced settings.
Qualitatively, the paper reports that FlashAvatar reconstructs facial details, thin structures, and subtle expressions more faithfully than the baselines. NHA can produce geometry artifacts due to the limits of explicit mesh deformation. INSTA tends to oversmooth results and struggles with accessories and thin hair structures. PointAvatar can recover some accessories but still fails on subtle expressions and clear teeth, while using substantial memory. FlashAvatar is claimed to preserve personalized appearance in hair, eyes, and mouth regions while remaining fast.
| Metrics | NHA | PointAvatar | INSTA | Ours |
|---|---|---|---|---|
| MSE ($10^{-3}$) ↓ | 1.49 | 2.47 | 0.95 | 0.66 |
| L1 ($10^{-2}$) ↓ | 0.99 | 1.52 | 0.89 | 0.83 |
| PSNR ↑ | 28.80 | 27.03 | 30.54 | 32.33 |
| SSIM ($10^{-1}$) ↑ | 9.31 | 9.00 | 9.40 | 9.42 |
| LPIPS ($10^{-2}$) ↓ | 4.01 | 5.89 | 3.76 | 3.23 |
The quantitative table shows that FlashAvatar is best on all reported metrics: it achieves the lowest MSE, L1, and LPIPS, and the highest PSNR and SSIM. The reported improvement is especially notable for perceptual quality, where LPIPS improves from $3.76$ for INSTA to $3.23$ for FlashAvatar.
6.2 Facial Reenactment and Novel View Synthesis
Because the Gaussian field is explicitly tied to expression code and the tracked face mesh, FlashAvatar can be used for facial reenactment without changing the identity representation. The paper reports that this works naturally and at very high frame rate. It also demonstrates novel-view rendering by changing the global camera pose on the non-neural 3D Gaussian field, showing strong 3D consistency.
6.3 Comparison with the "Canonical + Deformation" Strategy
The paper explicitly compares against the common dynamic-scene paradigm of a canonical Gaussian field plus deformation field. The authors follow the typical setup of initializing Gaussians randomly in a ball and training a canonical field first, then a deformation field. They report that this strategy struggles to produce acceptable avatars with low Gaussian count and often exhibits head-edge artifacts. When they improve the initialization by placing canonical Gaussians on the mesh surface, many artifacts disappear, but subtle expression details are still not captured as well as in FlashAvatar.
The interpretation given is that FlashAvatar benefits from two forms of guidance simultaneously: a strong mesh prior for large-scale deformation and a dynamic offset field for residual local effects. This is presented as a better fit to head avatars than a generic canonical/deformation factorization.
6.4 Training Efficiency and Rendering Speed
A major result of the paper is that the Gaussian count can be kept around the $10$K level while achieving a stable rendering speed of around $300$ FPS at $512 \times 512$ resolution on an RTX 3090. The training process is also reported to be efficient: coarse appearance is recovered within seconds and fine detail such as hair strands and teeth is reconstructed within a few minutes.
The paper attributes the speedup to two factors: a fixed, compact Gaussian set initialized in a good surface distribution, and a rendering primitive that avoids the expensive volumetric sampling of NeRF-style methods. The ablation table on UV resolution shows the trade-off between quality and speed more clearly.
7. Ablation Studies
7.1 Gaussian Sampling Density
The main density knob is the UV resolution. As expected, denser Gaussian sampling improves reconstruction quality but reduces frame rate. The paper uses UV resolution $128$ as its default comparison setting.
| UV Resolution | PSNR ↑ | LPIPS ($10^{-2}$) ↓ | FPS | # GS |
|---|---|---|---|---|
| 64 | 30.35 | 4.47 | 394 | 3348 |
| 128 | 30.80 | 3.47 | 304 | 13453 |
| 256 | 31.07 | 2.99 | 112 | 53678 |
The trend is simple: increasing density slightly improves PSNR and LPIPS, but the cost in rendering speed is steep. This supports the paper’s emphasis on careful initialization instead of brute-force densification.
7.2 Surface Embedding Method
The paper compares UV-based initialization against direct vertex attachment. Vertex attachment fails to converge to satisfactory results because vertex locations are unevenly distributed and do not cover the surface uniformly enough. UV-space initialization provides much more even coverage and yields more photo-realistic results with fewer Gaussians.
7.3 Dynamic Offset versus Static Offset
A static offset field can help reconstruct relatively fixed regions such as hair, but it cannot accurately model facial changes driven by expressions. When the offset depends on expression code, the method achieves noticeably better fidelity and better reproduces fine facial motion.
7.4 Additional Ablations from the Supplement
The supplementary material adds two more findings. First, closing the mouth cavity of the FLAME mesh helps the network model the interior mouth; without this step, the upper and lower teeth tend to blend together and the rendering becomes blurry. Second, adding perceptual loss improves the preservation of personalized facial attributes and increases photo-realism.
8. Main Contributions and Claimed Novelty
- The paper proposes a mesh-guided Gaussian avatar representation that couples 3D Gaussian splats with a parametric face model, using surface attachment plus learned residual offsets.
- It introduces a uniform UV sampling strategy for initializing Gaussians on the face surface, which supports density control and compresses the representation to about the $10$K Gaussian scale.
- It achieves an explicit trade-off of high fidelity, strong personalized detail, and very fast rendering, reporting about $300$ FPS at $512 \times 512$ on a consumer GPU.
- The paper claims that this is the first mesh-guided Gaussian field for head avatar modeling, and argues it combines the strengths of geometric priors and Gaussian splatting more effectively than canonical-plus-deformation dynamic Gaussian methods.
9. Limitations and Future Directions
The paper is explicit about two limitations. First, FlashAvatar still depends on a good surface-embedded Gaussian initialization, so large tracking errors, especially global pose errors, can cause detail loss or misalignment. Second, because the representation conditions on tracked expression code, it cannot model dynamically changing hair with heavy non-rigid deformation.
The authors suggest future work in extending the method toward mobile and mixed-reality settings, where the fast rendering speed could free resources for other real-time digital-human components. They also note the broader risk of misuse for deepfakes and emphasize responsible use.
10. Bottom Line
FlashAvatar is a compact, explicit, and geometry-guided avatar representation that uses a FLAME-driven Gaussian field to reconstruct head avatars from monocular video. Its key technical contribution is not just that it uses Gaussians, but that it places them carefully on a mesh surface in UV space, then learns a residual expression-conditioned offset to capture what the mesh misses. According to the paper’s experiments, this design gives a strong combination of visual fidelity, detailed reenactment, and real-time rendering speed.