FFAvatar

FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Images

A feed-forward method for reconstructing animatable 4D head avatars from sparse portrait images with incremental refinement. It uses alternating attention to disentangle identity from expression/viewpoint and combines sparse-to-dense learning with motion refinement for efficient, identity-consistent avatar animation.

• head-avatar
• gaussian-splatting
• portrait-animation
• face-animation
• few-shot
• audio-driven
• realtime

Authors: Jianjiang Yao, Ke Xian, Renxiang Dai, Robert Caiming Qiu

Categories: cs.CV, cs.AI

Published 2026-06-29 · Updated 2026-06-29

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3 stars0 watchers0 forks

Impact

• Semantic Scholar0 citations

Abstract

We present FFAvatar, a Transformer-based 3D Gaussian framework for fast construction of high-quality and animatable 4D head avatars from one or more reference portrait images. Unlike existing feed-forward approaches that require a fixed number of input views, FFAvatar supports incremental reconstruction, progressively refining the avatar representation as additional reference images become available. At the core of our method is an alternating attention mechanism that disentangles identity appearance from expression and viewpoint variations, enabling the reconstruction of a canonical 3D appearance that remains consistent across poses and facial expressions. To balance visual fidelity and computational efficiency, we introduce a sparse-to-dense learning paradigm. Coarse appearance features are first learned using sparse primitives anchored to the FLAME vertex level and are subsequently densified in the UV domain to capture fine-grained geometric and texture details. We further propose a plug-and-play motion refinement module that enables subject-specific dynamic personalization by modeling residual motion beyond parametric deformation. Extensive experiments demonstrate that FFAvatar efficiently produces high-fidelity and controllable 4D head avatars, achieving superior flexibility, driving efficiency, and identity-consistent rendering across diverse expressions and viewpoints.

---

Introduction and Motivation

High-quality 4D avatar head reconstruction is a foundational problem for applications in virtual reality, digital humans, telepresence, and immersive content creation. An ideal 4D avatar must faithfully preserve a person's identity while supporting realistic, temporally stable facial dynamics across diverse expressions, head poses, and viewpoints. Despite remarkable progress in neural rendering and avatar modeling, three key limitations continue to hamper existing approaches:

• Limited generalization under sparse observations. Many methods rely on dense multi-view captures and per-subject optimization, making them impractical in single- or few-shot settings. Existing feed-forward 4D avatar methods largely focus on single-reference scenarios, where ambiguous geometry and incomplete appearance remain significant challenges. Methods that extend to few-shot settings often construct multiple view-dependent canonical subspaces per input image, leading to redundant Gaussian representations and growing computational overhead as more views are added.
• Entanglement between identity appearance and motion. Identity features are often intertwined with expression- and pose-related variations, leading to visual artifacts, identity drift, and rendering inconsistencies when driving avatars to novel expressions or viewpoints.
• Trade-off between representation fidelity and efficiency. High-fidelity rendering typically demands dense UV representations or large numbers of 3D Gaussians, incurring substantial memory and computational cost.

FFAvatar is proposed to address all three challenges simultaneously through a novel, incremental, feed-forward Transformer-based 3D Gaussian framework that reconstructs animatable 4D head avatars from one or more reference portrait images in a single forward pass.

Problem Formulation

Given a set of $N$ head images $\mathcal{I} = \{I_i\}_{i=1}^{N}$ captured under different viewpoints and expressions (where $N$ can range from 1 to hundreds), the goal is to reconstruct a viewpoint- and expression-invariant canonical 3D head appearance represented as a set of 3D Gaussian primitives $G$, and to animate this representation under arbitrary target expressions and poses.

Unlike existing methods that fix the number of input views or create view-dependent canonical subspaces per input, FFAvatar builds a single unified global canonical Gaussian field anchored to the FLAME parametric head model. This means that the number of Gaussians, animation speed, and memory consumption are all independent of the number of input images.

Method Overview

The FFAvatar framework consists of two main stages:

1. Static Appearance Canonical Field Generation: extract a global appearance representation from multi-view and multi-expression images, perform sparse-to-dense cross-modal alignment, and decode an expression- and viewpoint-invariant static 3D Gaussian appearance.
2. 3D Head Avatar Animation: animate the static appearance using the FLAME model, followed by motion refinement through a Motion-Aware Refinement Module (MARM) to capture fine-grained dynamic details.

Stage 1: Static Appearance Canonical Field Generation

Feature Extraction and Global Appearance Aggregation

For each input image $I_i$, dense visual features are extracted using a frozen pretrained DINOv3 ViT-L/16 backbone:

The backbone operates at $512 \times 512$ resolution, producing a $32 \times 32$ patch grid (1024 visual tokens). Multi-scale features from four intermediate layers $\{4, 11, 17, 23\}$ are fused by a DPT-style head, yielding 1024-dimensional dense features per patch.

Each feature map is augmented with camera embeddings $\mathbf{Z}^{\text{cam}}$ and expression embeddings $\mathbf{Z}^{\text{exp}}$. Expression and pose parameters are projected to 1024-dimensional tokens and concatenated with visual tokens before aggregation. These augmented tokens are then processed by an alternating attention mechanism that interleaves intra-image (frame-level) and inter-image (global) attention blocks. This design, inspired by VGGT, aggregates identity-consistent appearance cues across all $N$ input images while suppressing viewpoint- and expression-specific variations:

The alternating attention aggregator uses token dimension 1024, depth 24, 16 attention heads, MLP ratio 4.0, alternating order [frame, global] with block size 1, QK normalization, 4 register tokens, and 2D rotary positional encoding with frequency 100. The result is a single global appearance representation that consolidates information from all input images into a unified identity-aware representation.

Sparse-to-Dense Cross-modal Alignment

To balance computational efficiency with representation fidelity, FFAvatar adopts a hierarchical sparse-to-dense alignment strategy. The key insight is that FLAME template vertices (5,023 points) provide semantically stable structural anchors, while UV-space densification enables recovery of high-frequency geometric and texture details.

Sparse alignment: Sparse Gaussian centers are initialized using FLAME template vertices $\mathbf{V}_0 \in \mathbb{R}^{5023 \times 3}$ and aligned with the global appearance representation via cross-attention with 8 sparse Transformer layers:

UV densification: The FLAME mesh is rasterized into a planar UV grid of size $\text{uv\_size} \times \text{uv\_size}$ (default 128). Each valid UV location $(u,v)$ corresponds to a triangle with vertex indices $(i,j,k)$ and barycentric coefficients $(\alpha, \beta, \gamma)$. Dense UV features are obtained by barycentric interpolation of the sparse features:

Flattening all valid UV samples yields dense features $\mathbf{T}_{\text{uv}} \in \mathbb{R}^{M \times C}$, where $M < \text{uv\_size}^2$ is the number of valid UV locations (approximately 15,000 for UV resolution 128).

Second-stage dense alignment: The combined sparse and UV-dense features are further refined through 2 additional dense cross-attention layers against the global appearance tokens to enforce cross-view consistency:

This hierarchical alignment progressively transfers identity-consistent appearance from global image observations to both sparse structural anchors and dense surface samples. The refined feature set $\mathbf{T}_{d}$ captures semantically stable coarse geometry together with high-frequency surface detail.

Canonical 3D Gaussian Field Decoding

The refined feature set $\mathbf{T}_d$ is decoded into a canonical 3D Gaussian head representation by a feed-forward GS decoder $\mathcal{D}_{\text{static}}$. Each Gaussian primitive has the following attributes:

where $\mathbf{f}_n \in \mathbb{R}^3$ is the positional offset relative to the underlying FLAME template (bounded to $[-0.1, 0.1]$ via scaled sigmoid), $o_n \in \mathbb{R}$ is opacity (sigmoid with initial bias $\sigma^{-1}(0.1)$), $\mathbf{c}_n \in \mathbb{R}^3$ is RGB color, $\mathbf{s}_n \in \mathbb{R}^3$ is anisotropic scaling (truncated exponential, clipped at 0.2), and $\mathbf{r}_n \in \mathbb{R}^4$ is the rotation quaternion ($\ell_2$-normalized). The GS decoder uses independent linear projection layers $\text{Linear}(1024, d_{\text{out}})$ for each attribute.

The resulting $M + 5023$ Gaussian primitives define a viewpoint- and expression-invariant canonical appearance field serving as the static structural basis for animation.

Stage 2: 3D Head Avatar Animation

FLAME-Based Mesh Deformation

FLAME parameterizes head geometry with pose and expression coefficients $\boldsymbol{\theta} = \{\boldsymbol{\theta}^{\text{pose}}, \boldsymbol{\theta}^{\text{exp}}\}$, where $\boldsymbol{\theta}^{\text{pose}}$ models rigid head motion and jaw articulation, and $\boldsymbol{\theta}^{\text{exp}}$ controls non-rigid facial expressions. The posed mesh is obtained via linear blend skinning with corrective blendshapes:

Following FLAME deformation, vertex-level motion is propagated to the dense UV-space surface points via the same UV rasterization. Each UV pixel $(u,v)$ is mapped to a barycentric combination over the deformed mesh:

Both the densified surface points and the original deformed FLAME vertices are retained, giving a pose-dependent surface with $M + 5023$ points:

Each Gaussian primitive is anchored to a corresponding point in $\tilde{\mathbf{X}}(\boldsymbol{\theta})$. During animation, the learned canonical offsets are preserved to retain identity-specific geometry, while coarse rigid motion is provided by FLAME.

Motion-Aware Refinement Module (MARM)

FLAME provides physically plausible coarse deformation but cannot capture identity-specific nonlinear motion patterns such as subtle muscle dynamics or personalized expression styles. The MARM addresses this by predicting residual Gaussian attribute updates conditioned on the current animation state. A lightweight residual Transformer $\mathcal{R}_{\text{motion}}$ takes as input the positional encoding of canonical FLAME vertices, the expression and pose parameters $\boldsymbol{\theta}$, and camera parameters $\boldsymbol{\theta}^{\text{cam}}$:

The module predicts motion-dependent corrections beyond template-driven deformation, refining geometry and appearance in a view-aware manner. This avoids directly regressing full Gaussian parameters, focusing instead on compact residual updates. The final animated Gaussian field is:

where $G'$ denotes the FLAME-driven coarse Gaussian field and $\oplus$ denotes attribute-wise updates. The MARM uses hidden dimension 1024, 4 Transformer layers, and 8 attention heads. It is a plug-and-play module, trained in a separate second stage after the rest of the framework is frozen, enabling subject-specific dynamic personalization.

Training Objective

The model is trained with photometric supervision and geometric regularization. The overall loss is:

RGB Loss combines pixel-level L1 reconstruction with LPIPS perceptual similarity:

Mask Loss enforces foreground alignment:

The mask weight gradually increases during training via an exponential schedule:

with $t = \min(\text{step}/T, 1)$, $\lambda_{\text{start}} = 0$, $\lambda_{\text{end}} = 1.0$, $\alpha = 0.5$, $T = 10{,}000$.

Offset Regularization stabilizes Gaussian deformation by penalizing large center offsets:

Loss weights are set as $\lambda_{\text{lpips}} = 1.0$ and $\lambda_{\text{off}} = 0.1$.

Architecture Summary

Implementation Details

FFAvatar is implemented in PyTorch and trained on 8 NVIDIA Tesla A800 GPUs using a two-stage training strategy: all modules except MARM are trained first, then the rest of the network is frozen while only MARM is optimized. Total training takes approximately one week. Optimization uses Adam with learning rate $4.0 \times 10^{-5}$ for $300{,}000$ iterations. For each batch, 1–8 frames are randomly sampled from a monocular video. GAGAvatar tracker is used to extract camera-pose and FLAME facial-expression conditions. All images are resized to $512 \times 512$ with background removal. Evaluation is performed on a single RTX 4090D GPU (24 GB memory).

Datasets and Evaluation Protocol

Training: The VFHQ dataset (15,204 monocular video clips, ~3M frames) is used exclusively for training. Facial regions are detected, bounding boxes enlarged for context, and images cropped and resized to $512 \times 512$. Camera pose estimation and FLAME parameter tracking are performed per frame following the GAGAvatar pipeline.

Evaluation datasets:

• VFHQ test set (50 identities): 8 expressions randomly selected as inputs, remaining frames used for evaluation under novel expression synthesis.
• NeRSemble multi-view dataset (4 identity sequences): evaluated under novel view synthesis (8 input views, 8 unseen views) and novel expression synthesis (16 input expressions, unseen poses and expressions). The NeRSemble dataset is not used during training.
• Ava-256 dataset: used for qualitative comparisons with Avat3r and FastGHA using color-calibrated inputs.

Metrics: PSNR, SSIM, LPIPS (rendering quality); CSIM/ArcFace (identity consistency); AED (average expression distance); APD (average pose distance); creation time (seconds); animation FPS.

Baselines

In the one-shot setting: Portrait4D-v2, Real3D-Portrait, GAGAvatar, LAM, FastAvatar.

In the few-shot setting: FlashAvatar, GHA, GaussianAvatars (optimization-based); GPAvatar, FastAvatar (feed-forward). Avat3r and FastGHA are compared qualitatively on Ava-256 since their implementations are not publicly available.

Quantitative Results

Novel Expression Synthesis on VFHQ

The following table reports results on the VFHQ monocular test set. FFAvatar (few-shot, 8 inputs) achieves the best performance across all quality and motion metrics in the few-shot setting, and the fast variant achieves 468 FPS — substantially outperforming all baselines.

Novel View and Expression Synthesis on NeRSemble

The model is trained exclusively on VFHQ; NeRSemble is used only for evaluation. In the one-shot setting, FFAvatar achieves competitive performance, outperforming most baselines in LPIPS and SSIM. In the few-shot setting, FFAvatar achieves the best LPIPS and SSIM for both novel-view and novel-expression tasks, demonstrating superior perceptual fidelity and structural consistency.

Qualitative Comparisons on Ava-256

Extended Expression and Viewpoint Comparisons

Ablation Studies

Incremental Reconstruction with Varying Input Count

FFAvatar supports incremental reconstruction from 1 to 32+ reference images. The table below shows progressive improvement as more input frames are added. The most significant gain occurs from 1 to 8 frames, suggesting multi-view and multi-expression cues are especially helpful for identity stabilization. Gains from 16 to 32 frames are smaller but consistent.

Sparse-to-Dense Learning Paradigm

The table below compares sparse-only (S-5K: 5,023 FLAME vertices), sparse-to-dense (S2D-20K at UV 128, S2D-64K at UV 256), and dense-only (D-20K, D-64K) strategies. Key findings:

• S-5K shows clear quality degradation, lacking representational capacity for fine structures.
• D-20K improves metrics but triples training time (120 min vs. 47 min) and uses 38.2 GB GPU memory.
• S2D-20K achieves nearly identical quality to D-20K at roughly 2.5× speedup and significantly lower memory (25.8 GB vs. 38.2 GB).
• S2D-64K achieves the best quality overall (23.42 PSNR), while D-64K is infeasible (OOM on 40 GB GPU).

Motion Refinement Module

Comparing Ours (with MARM) vs. Ours (fast) (without MARM) in the novel-expression VFHQ setting: enabling MARM consistently improves perceptual quality (PSNR 23.35 vs. 23.20, SSIM 0.864 vs. 0.862, LPIPS 0.081 vs. 0.088), identity consistency (CSIM 0.861 vs. 0.852), and motion accuracy (AED 0.079 vs. 0.084, APD 0.114 vs. 0.117) with no change in creation time (2.1s). The trade-off is a reduction in animation speed from 468 FPS to 31 FPS.

Additional Ablations: UV-only and Camera Conditioning

A UV-only representation (without combining original FLAME point features) yields PSNR 23.05 / SSIM 0.861 / LPIPS 0.092, vs. the full model's 23.35 / 0.864 / 0.081. This demonstrates that combining UV-space features with original FLAME point features provides a denser and geometrically more consistent 3D Gaussian representation. Removing camera conditioning yields comparable image-level metrics (23.33 / 0.864 / 0.080) but reduces the ability to disentangle head pose from facial motion; camera conditioning is therefore retained as auxiliary information.

Scalability and Runtime Analysis

The table below (tested on RTX 4090D unless noted) demonstrates that FFAvatar's animation speed is independent of the number of reference images — a key advantage over methods such as FastAvatar that fuse per-view Gaussian representations:

As more reference images are provided, creation time grows moderately but animation speed remains constant. The framework can scale to more than 100 input frames on NVIDIA A800 GPUs.

Robustness Analysis

FFAvatar's alternating attention mechanism demonstrates strong robustness to corrupted or irrelevant inputs. Two stress tests are reported:

• Degraded inputs: Using 32 reference views where 16 are severely degraded (missing facial regions, occlusions, geometric distortions, color shifts) — the reconstructed avatar remains stable and identity-consistent.
• Unrelated identities: Using 32 reference views where 16 correspond to completely different identities — the model still correctly reconstructs the target identity, effectively filtering out irrelevant information.

Applications

Text-to-4D Avatar Generation

Reference images generated from text prompts using a text-to-image model can be directly fed into FFAvatar to reconstruct a dynamic 4D avatar. Diffusion-based data augmentation (using MMDM/CAP4D) and incremental reconstruction improve view consistency and motion realism.

Open-Scenario Reconstruction from Stylized Portraits

Speech-Driven Talking Head, Image Editing, and Background Replacement

FFAvatar can be combined with audio-to-3DMM methods to enable speech-driven animation. Because FLAME expression/pose parameters are the driving signal, any method that produces FLAME parameters from audio can directly animate FFAvatar reconstructions. The framework also supports appearance editing by modifying 2D reference images (e.g., style/color changes), and background replacement by manipulating Gaussian opacity values — enabled by the explicit 3D representation without needing post-processing networks.

Limitations

Despite strong performance, several limitations remain:

• Tongue modeling: The tongue is not explicitly modeled, leading to inaccurate mouth interiors for large mouth-opening expressions.
• Unseen regions: Regions rarely observed in reference images (e.g., back of the head under front-facing inputs) have limited detail under extreme viewpoints.
• Single-view ambiguity: With only one reference image, unseen geometry and appearance can cause view-dependent artifacts under large pose changes. Multi-view inputs alleviate this significantly.
• FLAME tracking dependency: Animation accuracy is bounded by FLAME parameter estimation quality; inaccurate tracking degrades expression transfer for extreme expressions, rapid motion, or occluded faces.
• Training data bias: VFHQ is collected mainly from in-the-wild interview videos and may be imbalanced in identity, skin tone, age, and appearance distribution, potentially affecting reconstruction for underrepresented groups.

Future work will explore more balanced training data, improved modeling of dynamic facial components such as the tongue, more robust handling of unseen regions, and extensions to dynamic hair and full-body avatar reconstruction.

Contributions Summary

FFAvatar makes three primary technical contributions:

1. Incremental feed-forward framework with alternating attention for flexible, identity-disentangled multi-view feature aggregation into a unified global canonical Gaussian field — avoiding redundant multi-canonical representations and enabling scalable single-forward-pass 4D avatar reconstruction from 1 to 100+ images.
2. Sparse-to-dense hierarchical feature learning paradigm — coarse features at the FLAME vertex level provide stable structural anchors, UV-space densification recovers high-frequency geometric and texture details, yielding a substantially better efficiency-vs-fidelity trade-off than dense-only approaches and enabling configurations (S2D-64K) that are infeasible with direct dense optimization.
3. Motion-Aware Refinement Module (MARM) — a plug-and-play residual Transformer that captures subject-specific, identity-dependent nonlinear motion beyond the FLAME parametric model, improving perceptual quality, identity consistency, and motion accuracy with negligible creation time overhead.