SplitAvatar

SplitAvatar: One-shot Head Avatar with Autoregressive Gaussian Splitting

One-shot head avatar reconstruction using autoregressive Gaussian splitting to progressively refine expression details. It addresses scale mismatches in prior 3DGS methods by progressively splitting Gaussians with learned density control, allocating more detail to expressive regions like eyes and mouth.

• one-shot
• head-avatar
• gaussian-splatting
• autoregressive
• avatar
• face-animation
• realtime

Authors: Hongzhe Liao, Chuhua Xian, Hongmin Cai, Haiyang Liu, Fa-Ting Hong

Categories: cs.CV

Published 2026-05-25 · Updated 2026-05-25

Links

Paper & demos

• arXiv

Abstract

3D Gaussian Splatting (3DGS) provides an efficient method for high-quality scene reconstruction using anisotropic Gaussians. Recently, 3DGS-based methods have significantly improved the rendering quality of human avatars while enabling real-time performance. However, existing methods suffer from a magnitude mismatch in the number of Gaussians generated by image-based and 3DMM-based approaches. This discrepancy results in reconstructed expressions that lack fine-grained detail. In this paper, we introduce a novel method for reconstructing an animatable head avatar from a single image. We propose a Graph splitting network to progressively generate Gaussians from coarse to fine using an autoregressive architecture. To address the graph inconsistency caused by split Gaussians, we employ a mesh topology extension method to align the GNN's connectivity with the increased Gaussian count. Furthermore, we introduce a novel density control method that includes a gating mechanism that generates soft masks for Gaussians, preventing over-densification after the splitting operation. This allows for dynamic control over Gaussian density across different facial regions. For smooth and rapid training, we employ a delayed filtering strategy to avoid re-computing the graph topology during training. Experimental results demonstrate that our autoregressive structure effectively improves expression representation ability by progressively splitting Gaussians. This process, enabled by the GNN-guided splitting, synthesizes more precise facial details and achieves higher reconstruction quality.

---

Introduction and Motivation

Generating a high-quality, animatable 3D head avatar from a single source image — a setting commonly called one-shot head avatar reconstruction — is a critical capability for applications such as virtual digital humans, telepresence, and conversational AI. The core challenge is the fundamental ambiguity of lifting a single 2D image into a full 3D representation that can simultaneously preserve the source person's identity and faithfully adopt the pose and expression of an arbitrary driving signal.

Prior art broadly divides into 2D-based methods, which warp source image features using estimated 2D motion fields, and 3D-based methods, which explicitly reconstruct 3D geometry. 2D methods inherently struggle with multi-view consistency and large pose changes. 3D methods, especially those built on Neural Radiance Fields (NeRF), achieve strong quality but are too slow for interactive use due to volumetric rendering costs. 3D Gaussian Splatting (3DGS) has emerged as a powerful alternative: it is differentiable, tile-rasterizable, and supports real-time rendering.

Despite the promise of 3DGS-based avatar methods, two critical problems remain unsolved:

• Scale mismatch: Expression-driven branches typically rely on 3D Morphable Models (3DMMs) such as FLAME, which has a fixed mesh of 5,023 vertices. The resulting sparse set of expression Gaussians is orders of magnitude smaller than the dense Gaussians produced by image-based identity reconstruction branches. Naive fusion strategies fail to reconcile this discrepancy, leading to reconstructed expressions lacking fine-grained detail.
• Uncontrolled densification: Hierarchical or coarse-to-fine Gaussian synthesis methods use fixed replication rules that lack active filtering, causing rampant redundancy and excessive Gaussian density while failing to allocate sufficient resolution to critical expressive regions such as the eyes and teeth.

SplitAvatar addresses both problems with a unified framework built around an autoregressive Graph Neural Network (GNN) that progressively splits expression Gaussians from coarse to fine, paired with a learned gating mechanism that dynamically prunes redundant Gaussians after each split.

Background: 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) represents a scene as a collection of anisotropic 3D Gaussians. Each Gaussian is parameterized by its mean (center) $\mu$ and covariance matrix $\Sigma$:

$$G(\mathbf{x}) = e^{-\frac{1}{2}(\mathbf{x}-\mu)^T \Sigma^{-1} (\mathbf{x}-\mu)}$$

To ensure $\Sigma$ is positive semi-definite and to enable gradient-based optimization, it is decomposed into a rotation matrix $R$ and a scaling matrix $S$:

$$\Sigma = R S S^T R^T$$

Each Gaussian additionally carries a color $c$ and opacity $\alpha$. The color $C$ of a pixel is computed via tile-based rasterization by alpha-compositing Gaussians in depth order:

$$C = \sum_{i=1}^{N} c_i \alpha_i' \prod_{j=1}^{i-1}(1 - \alpha_j')$$

SplitAvatar uses a neural renderer on top of 32-dimensional color features to generate the final image, following the practice of prior generalizable avatar work.

Method Overview

SplitAvatar decomposes the head avatar reconstruction problem into two parallel branches:

• Identity branch: A frozen DINOv2 backbone extracts rich visual features from the source image. A trainable Gaussian feature adapter — consisting of two Vision Transformers (depth=12, heads=8) and transposed convolutions that upsample to $296 \times 296$ — lifts these features into identity Gaussians via a dual-lifting strategy. Rather than directly predicting Gaussian positions, the adapter predicts forward and backward offsets relative to a feature plane produced by the frozen DINOv2, following the approach of GAGAvatar.
• Expression branch: A 3DMM estimator (FLAME) tracks the driving image to obtain mesh vertices, pose, and expression parameters. A Gaussian generator — an MLP conditioned on DINOv2 identity features — predicts one Gaussian per mesh vertex, yielding exactly 5,023 initial expression Gaussians. This count is much smaller than the identity Gaussian count, motivating the splitting network.

The key innovation lies in the autoregressive Graph Splitting Network (GSN), which progressively increases the number of expression Gaussians through a series of splitting layers, each guided by a GNN that exploits the FLAME mesh topology.

Autoregressive Graph Splitting Network (GSN)

Gaussian Splitting as an Autoregressive Sequence

The expression Gaussians are treated as a sequence of $L+1$ discrete layers: $G_\text{exp} = (G_0, G_1, \ldots, G_L)$. The initial 5,023 Gaussians from the generator form $G_0$. At each step $i$, a single Gaussian in layer $i$ produces $k$ new Gaussians in layer $i+1$ — a Gaussian splitting operation. The total number of Gaussians grows as $5023 \cdot k^i$ at layer $i$.

The GSN's core is a Graph Attention Network (GAT) with 4 attention heads. For layer $i$, Gaussian attributes (positions, opacities, rotations, scales, and colors) are first mapped to a 256-dimensional latent feature via a feature embedding module $\mathcal{E}$:

$$\mathcal{F}_i = \mathcal{E}(G_i)$$

The GNN processes these features using the edge topology of layer $i$, and the next-layer Gaussians are decoded via a decoder $\mathcal{D}$ with a residual connection to prevent gradient vanishing:

$$G_{i+1} = \lambda \, \mathcal{D}\!\left( N_\text{GSN}(\mathcal{F}_i, E^i) + \mathcal{F}_i \right)$$

Here $E^i$ is the adjacency matrix for layer $i$ and $\lambda$ is an attribute-specific scale factor. By setting gradually decreasing values for opacity and scale across layers, each layer naturally focuses on progressively finer details.

Mesh Topology Extension for Graph Consistency

When Gaussians split, the adjacency matrix must expand correspondingly. SplitAvatar resolves this with a mesh topology extension that dynamically constructs the graph at each layer from two components:

• Topological Inheritance Edges $E^i_\text{topo}$: Each of the $k^i$ copies of the FLAME mesh preserves the original FLAME edge connectivity within itself. A vertex $\mathbf{v}^i_{j,m}$ at layer $i$ is identified by its original FLAME index $j$ and its copy index $m \in \{1, \ldots, k^i\}$: $$V^i = \{ \mathbf{v}^i_{j,m} \mid \mathbf{v}^0_j \in V^0 \}$$ $$E^i_\text{topo} = \{ (\mathbf{v}^i_{j,m}, \mathbf{v}^i_{p,m}) \mid (\mathbf{v}^0_j, \mathbf{v}^0_p) \in E^0 \}$$
• Internal Connection Edges $E^i_\text{conn}$: All $k^i$ copies derived from the same parent vertex $\mathbf{v}^0_j$ are fully connected to each other, enabling information propagation among sibling split Gaussians: $$E^i_\text{conn} = \{ (\mathbf{v}^i_{j,m}, \mathbf{v}^i_{j,n}) \mid \mathbf{v}^0_j \in V^0,\; m,n \in \{1,\ldots,k^i\} \}$$

The total edge set for layer $i$ is: $E^i = E^i_\text{topo} \cup E^i_\text{conn}$, yielding a new graph $M^i = (V^i, E^i)$ that feeds into the GNN at each autoregressive step.

Gating Mechanism for Density Control

Soft Mask Network

Rather than using a fixed density control strategy (as in vanilla 3DGS), SplitAvatar introduces a learnable soft mask network $N_\text{mask}$ that predicts a per-Gaussian filter mask at each layer. The soft mask for layer $i+1$ is predicted from the latent features of layer $i$:

$$m_{i+1} = N_\text{mask}(\mathcal{F}_i)$$

This allows the network to adaptively allocate more Gaussians to high-complexity regions (eyes, mouth, teeth) and suppress Gaussians in smooth, low-complexity regions.

Delayed Filtering Strategy

Naively removing Gaussians during each splitting step would require expensive recomputation of graph edge indices and could cause batch size inconsistencies (since different samples would have different numbers of surviving Gaussians). SplitAvatar avoids this with a delayed filtering strategy: Gaussians are never physically removed during splitting. Instead, they are tagged with a binary mask and simply ignored during the final rendering step.

After each splitting operation, the mask from layer $i$ is duplicated $k$ times (one per split child) and combined with the new mask predicted by $N_\text{mask}$ via a logical AND:

$$q(x) = \begin{cases} 1 & x > 0.5 \\ 0 & \text{otherwise} \end{cases}$$

$$M_{i+1}^\text{logical} = q(m_{i+1}) \wedge q(M_i)$$

Gradient Propagation via Straight-Through Estimator

Both the quantization function $q(\cdot)$ and the logical AND are non-differentiable. SplitAvatar uses the Straight-Through Estimator (STE) — treating the gradient of $q(\cdot)$ as the identity — and introduces a stop-gradient operator $\operatorname{sg}(\cdot)$ to correctly route gradients. The differentiable mask update is:

$$M_{i+1} = m_{i+1} M_i + \operatorname{sg}\!\left(M_{i+1}^\text{logical} - m_{i+1} M_i\right)$$

This ensures that during the forward pass the mask is binary (hard thresholding via AND), while during the backward pass gradients flow through the soft product $m_{i+1} M_i$.

Final Rendering

The final Gaussian set used for rendering combines identity Gaussians with all masked expression Gaussians across every layer:

$$G = G_\text{identity} + \sum_{i=0}^{L} M_i \odot G_i$$

Training Objectives

Training supervises both a coarse rendered image $I_c$ (before neural rendering) and a fine rendered image $I_f$ (after neural rendering). The image reconstruction loss combines L1 pixel loss and perceptual loss:

$$\mathcal{L}_\text{image} = \|I_c - I_t\| + \|I_f - I_t\|$$

$$\mathcal{L}_\text{percep} = \|\varphi(I_c) - \varphi(I_t)\| + \|\varphi(I_f) - \varphi(I_t)\|$$

where $\varphi(\cdot)$ is a VGG-based perceptual feature extractor and $I_t$ is the target image.

A lifting loss $\mathcal{L}_\text{lifting}$ encourages the identity Gaussians to stay geometrically close to the tracked FLAME mesh vertices $P_\text{3dmm}$:

$$\mathcal{L}_\text{lifting} = \| P_\text{3dmm} - Q \|, \quad Q = \{ \arg\min_{q \in G_\text{identity}} \|p - q\| \mid p \in P_\text{3dmm} \}$$

A splitting Gaussian loss $\mathcal{L}_\text{split}$ similarly encourages the split expression Gaussians to align with the tracked mesh:

$$\mathcal{L}_\text{split} = \| P_\text{3dmm} - Q \|, \quad Q = \{ \arg\min_{q \in G_\text{split}} \|p - q\| \mid p \in P_\text{3dmm} \}$$

The overall training objective is:

$$\mathcal{L} = \mathcal{L}_\text{image} + \lambda_p \mathcal{L}_\text{percep} + \lambda_l \mathcal{L}_\text{lifting} + \lambda_s \mathcal{L}_\text{split}$$

with weights $\lambda_p = 0.5$, $\lambda_l = 1$, $\lambda_s = 0.5$.

Implementation Details

The system is implemented in PyTorch. Training uses the Adam optimizer with a learning rate of $1.0 \times 10^{-4}$ and batch size 4, running for 200,000 iterations consuming 77 GPU-hours on a single NVIDIA A100. Key hyperparameters for the autoregressive GSN: $L = 3$ layers and $k = 2$ split factor, meaning each expression Gaussian spawns 2 children per layer. At $k=2$, $L=3$, the expression Gaussian count grows from 5,023 to $5023 \cdot 2^3 = 40,184$ after all splits (before masking). The GAT has 4 attention heads and operates on 256-dimensional features. The mask network is an MLP with 256-dimensional input per Gaussian. Inference runs at 41 FPS with only 3.1 GB GPU memory.

Datasets and Evaluation Protocol

Datasets

• VFHQ (Video Faces High Quality): Training uses every 5th frame with detected faces, resulting in 636,359 frames from 14,513 clips. Camera poses and FLAME parameters are tracked. Backgrounds are removed and images resized to $512 \times 512$. Evaluation uses 50 test videos: first frame as source, 100 randomly sampled frames as driving/target.
• HDTF (High-Definition Talking Face): Evaluation follows the split used in prior work, sampling 230 random frames per clip.

Evaluation Metrics

• Image quality (self-reenactment): PSNR, SSIM, and LPIPS (perceptual similarity using VGG features).
• Identity similarity: Cosine similarity (CSIM) of ArcFace identity embeddings between source and generated images: $\text{CSIM} = \frac{A \cdot B}{\|A\|\|B\|}$.
• Expression accuracy: Average Expression Distance (AED) and Average Pose Distance (APD) — $L_1$ distances of 3DMM expression/pose parameters; Average Keypoint Distance (AKD) — average Euclidean distance of 68 facial landmarks.
• Cross-reenactment: Since no ground truth exists, only CSIM, APD, and AKD are reported.

Quantitative Results

VFHQ Benchmark

HDTF Benchmark

SplitAvatar achieves state-of-the-art results on almost all metrics across both datasets. The one area where it is not uniformly top is cross-reenactment CSIM on VFHQ (0.5896 vs. Portrait4D-v2's 0.6210), which the authors attribute to two factors: the 3DMM estimator not fully decoupling identity from expression, and increased network capacity prioritizing fine facial detail over strict identity preservation — a deliberate, acknowledged trade-off.

Qualitative Results

Ablation Studies

Component-Level Ablations

Effect of autoregressive splitting (w/o AR $N_\text{GSN}$): Replacing the full autoregressive GSN with a single layer and setting $k=1$ causes reconstruction quality to drop substantially (PSNR −0.27, AKD +0.76) while identity similarity slightly increases. This confirms the pattern that limited model capacity forces prioritization of identity at the expense of expression accuracy.

Effect of GNN (w/o GNN): Replacing the GAT with an equivalently sized MLP degrades AED, APD, and AKD, confirming that graph-topology-aware message passing over the FLAME mesh is crucial for accurate, detailed expression transfer. Identity similarity slightly increases (again the capacity-vs-expression trade-off pattern).

Effect of feature embedding (w/o feature embedding): Removing the embedding $\mathcal{E}$ and decoder $\mathcal{D}$ — feeding raw Gaussian attributes directly into the GSN — causes a slight drop across metrics. The embedding helps the network capture inter-attribute relationships and improves generalization.

Effect of mask network (w/o $N_\text{mask}$): Without masking, the model uses all predicted Gaussians and tends to produce a nearly static face during cross-reenactment. Cross-reenactment CSIM rises (to 0.6408) but this reflects identity leakage (the model retains the source appearance rather than transferring expression) rather than genuine identity preservation. AED (+0.04) and APD (+0.035) degrade significantly, confirming over-densification prevents correct expression transfer.

Ablation on Splitting Factor $k$ and Layers $L$

Medium settings ($2 \times L$ and $4 \times 2$) consistently outperform both the minimal $1 \times 1$ baseline and the larger $3 \times 3$ configuration. The authors observe that $k=2$, $L=3$ (the proposed default) achieves the best balance. Excessively large $k$ or $L$ values cause overfitting and information redundancy, degrading generalization.

Comparison with Scale-Matched Baseline

To validate that improvements come from the autoregressive GNN structure rather than simply from having more Gaussians, the authors compare against a GAGAvatar-style baseline whose expression output is expanded by $15\times$ to match the total Gaussian count of SplitAvatar ($15 \times 5023 \approx 75,345$ vs. SplitAvatar's $5023 \times 2^3 = 40,184$ before masking). Both are trained identically.

SplitAvatar outperforms the scale-matched baseline across nearly all metrics, confirming that the GNN-guided autoregressive splitting — not merely the increased Gaussian count — drives the performance gains.

Architectural Novelty and Contributions

1. Autoregressive Gaussian splitting framework: SplitAvatar introduces the first autoregressive paradigm for progressively expanding 3DMM-derived expression Gaussians in a coarse-to-fine manner, resolving the scale mismatch between 3DMM-driven (sparse) and image-based (dense) Gaussian branches.
2. Graph topology extension: A principled mesh topology extension rule dynamically constructs a valid GNN adjacency matrix as Gaussians proliferate through splitting, preserving FLAME mesh structure (topological inheritance edges) and enabling sibling communication (internal connection edges).
3. Learned gating mechanism with delayed filtering: A soft mask network predicts per-Gaussian filter weights at each autoregressive layer. The delayed filtering strategy avoids expensive graph recomputation during training while enabling dynamic density control at inference. Gradient flow is maintained via STE.
4. Practical efficiency: The resulting system runs at 41 FPS with 3.1 GB GPU memory, making it suitable for real-time conversational AI applications.

Relation to Prior Work

GAGAvatar is the closest predecessor, using a dual-lifting branch for identity and a FLAME-conditioned MLP for expression Gaussians. SplitAvatar replaces the fixed-scale expression MLP with the full autoregressive GSN, inheriting and extending the dual-lifting strategy.

GraphAvatar also combines GNNs with 3DGS on FLAME meshes, but requires per-identity training and does not generalize across identities. SplitAvatar is fully generalizable.

GSGAN proposes multi-level Gaussian hierarchies with coarse-to-fine generation but does not disentangle identity from expression, making direct expression transfer difficult and lacking active density filtering.

Visual Autoregressive Modeling (VAR) inspires the next-scale autoregressive paradigm applied here to 3D Gaussians rather than image token grids.

Limitations

The authors note that, due to the constraint of using a single source image, reconstruction quality may degrade under large camera angle changes or dramatic pose variations — a fundamental limitation shared by all one-shot methods. The authors suggest this could be addressed by integrating SplitAvatar with generative priors trained on large portrait datasets to compensate for missing viewpoint information. Additionally, the 3DMM estimator used does not fully decouple identity and expression, which contributes to the slightly reduced cross-reenactment identity similarity compared to some baselines.

Conclusion

SplitAvatar presents a principled solution to the scale mismatch and uncontrolled densification problems in 3DGS-based one-shot head avatar methods. By introducing an autoregressive GNN that progressively splits FLAME-derived Gaussians from coarse to fine — together with a topology extension rule, a learned soft mask network, and a delayed filtering training strategy — the system achieves state-of-the-art performance on both the VFHQ and HDTF benchmarks while running at real-time speed. The framework is highly configurable: the splitting factor $k$ and the number of autoregressive layers $L$ can be tuned to match the Gaussian density of any downstream identity reconstruction module, providing a modular, plug-and-play expression driving component for conversational AI systems.