JacobianAvatar
JacobianAvatar: Temporally Consistent Semi-rigid Avatar Reconstruction from a Monocular Video
Reconstructs temporally consistent 3D human avatars from monocular video using neural Jacobian fields to model semi-rigid deformations. A constrained Poisson solver and deformation-guided temporal consistency loss address monocular occlusions and clothing dynamics better than prior work.
Links
Paper & demos
Impact
Abstract
Generating realistic human avatars in complex motions--such as clothing dynamics--requires modeling of global and local deformations which remains challenging in monocular settings. We address this problem by leveraging neural Jacobian fields (NJFs) for representing semi-rigid deformations. We train self-supervised neural networks for predicting Jacobian matrices that give the pose-dependent deformations, by solving a Poisson equation. However, monocular input presents several difficulties such as self-occluded regions and invisible surfaces. To address these issues, we introduce three key components: a constrained Poisson solver, signed distance-based Jacobian regularization, and a deformation-guided residual flow loss, which together suppress boundary artifacts, recover frequently occluded regions such as armpits and thighs, and enforce temporal consistency during motion. Experiments on benchmark and in-the-wild videos demonstrate that our method generates temporally stable and geometrically coherent avatars, outperforming state-of-the-art approaches.
Introduction and Motivation
Reconstructing photorealistic, animatable human avatars from a single monocular video is a long-standing challenge in computer vision and graphics. Recent progress has improved the fidelity of clothed human modeling through a semi-rigid deformation paradigm, which combines articulated skeletal motion governed by Linear Blend Skinning (LBS) with non-rigid local deformations to capture clothing wrinkles and pose-dependent surface details.
The semi-rigid reconstruction problem naturally decomposes into two coupled sub-problems. First, recovering a canonical model that captures the subject's intrinsic shape and appearance in a resting pose. Second, predicting non-rigid local deformations — such as clothing wrinkles and pose-dependent correctives — which are crucial for realistic in-motion rendering. Most prior work assumes body pose is the dominant cause of local deformations (pose-driven approaches), regressing surface offsets conditioned on estimated SMPL-X pose parameters. While effective for pose-induced effects, these methods overlook history-dependent deformations from factors like cloth inertia, contact, or motion velocity. Time-driven methods partially address this by incorporating temporal context.
A critical limitation of existing approaches is their heavy reliance on photometric reconstruction loss, which is fundamentally ill-posed under monocular conditions due to appearance-geometry ambiguities (textureless regions, self-occlusions). From a dynamic object reconstruction perspective, temporal correspondences between frames can be used, parameterized as 6-DoF rigid transforms, optical flow, or learnable motion bases — but these strategies are suboptimal for avatar reconstruction due to frequent self-occlusions and complex non-rigid deformations.
This paper introduces JacobianAvatar, a framework that addresses these challenges through three key innovations: (1) hierarchical neural Jacobian fields for semi-rigid deformation representation, (2) a screened Poisson solver to handle sparse monocular observations, and (3) a deformation-guided residual flow loss for temporal consistency. Together, these components produce temporally stable, geometrically coherent avatars from a single video, outperforming state-of-the-art methods on benchmark and in-the-wild videos.
Related Work
Pose-driven and Time-driven Methods
SCANimate learns pose-conditioned clothed avatar networks from raw scans via implicit occupancy fields. SCALE represents clothed humans as articulated local elements. SNARF proposes differentiable forward skinning bridging neural implicit representations and LBS. NeuMan, SelfRecon, and Vid2Avatar use monocular videos with NeRF-based canonical spaces. LSAvatar uses graph neural networks for locality-sensitive deformation; FacAvatar decouples coarse and fine deformations via frequency control on positional encodings. These NeRF-based methods suffer from slow rendering and high-frequency artifacts.
3D Gaussian Splatting-based Methods
ExAvatar anchors 3D Gaussians to a canonical human template mesh for expressive whole-body reconstruction. GaussianAvatar integrates forward skinning with learned pose-dependent deformation fields. Animatable Gaussians learns pose-conditioned Gaussian attributes via CNNs. GomAvatar and SplattingAvatar anchor Gaussians to a mesh for real-time rendering. These methods are constrained by the fixed topology and intrinsic rigidity of LBS, struggling with complex non-rigid dynamics.
Diffusion-guided Generative Models
HumanGaussian, DreamWaltz, PSHuman, SiTH, HumanRef, and ReconFusion use diffusion priors to resolve geometric and textural ambiguity in occluded regions, though they have limitations in dynamic deformation and temporal consistency.
Method Overview
JacobianAvatar predicts neural Jacobian fields that capture semi-rigid deformations conditioned on body pose. The pipeline proceeds through several stages:
- Pre-processing: Initialize and refine the canonical mesh and body poses via differentiable rendering.
- Coarse NJF training: Train a coarse Jacobian field to capture global deformations.
- Fine NJF training: Train a fine Jacobian field on a subdivided mesh, guided by coarse features.
- 3DGS refinement: Train Gaussians anchored to mesh faces for high-fidelity appearance.
Pre-processing of Canonical Mesh and Body Poses
The canonical mesh is initialized from the Momentum Human Rig (MHR) template at Level of Detail 3. Due to imperfections of the initial mesh and body pose estimates, these are refined jointly using a differentiable mesh renderer. Specifically, each vertex is transformed by:
$$V_i^\theta = \text{LBS}(V_i^c, \theta) = \sum_{j=1}^{K} w_{i,j}\, G_j(\theta)\, V_i^c$$where $V_i^c$ is the $i$-th canonical vertex position, $\theta$ are the body pose parameters for $K$ joints, $G_j(\theta) \in \mathbb{R}^{4 \times 4}$ is the bone transformation matrix for joint $j$, and $w_{i,j}$ is the skinning weight. Vertex positions and body poses are jointly optimized by minimizing errors between rendered and predicted normal maps and segmentation masks from the Sapiens foundation model.
After convergence, the updated canonical mesh is denoted $\mathcal{M}^c$. It is subdivided twice to produce the fine mesh $\mathcal{M}^f$ for high-resolution detail recovery. LBS weights are adopted from SMPL-X and propagated using robust skin weight inpainting.
Semi-rigid Deformation with Neural Jacobian Fields
Let the canonical mesh be $\mathcal{M} = \{\mathcal{V}, \mathcal{F}\}$ with $\mathcal{V} \in \mathbb{R}^{N \times 3}$ vertex positions and $\mathcal{F} \in \mathbb{Z}^{M \times 3}$ faces. Following Neural Jacobian Fields (NJF), a mapping function $\Phi$ transforms $\mathcal{V}$ to new positions by minimizing:
$$\Phi^* = \min_{\Phi} \sum_{a_i \in \mathcal{F}} |a_i| \| \nabla_i(\Phi) - J_i \|_2^2$$where $\nabla_i$ is the gradient operator per face, $|a_i|$ is the face area, and $J_i$ is the per-face Jacobian matrix. The Jacobian field $\mathbf{J}$ encodes the full deformation; solving the Poisson equation given $\mathbf{J}$ yields the optimal vertex positions.
Constrained (Screened) Poisson Solver
Standard NJF suffers from unbounded expansion artifacts when Jacobians are noisy, which is especially problematic in monocular settings with incomplete observations. The paper introduces a screening term that anchors vertex positions to the initial mesh:
$$\Phi^* = \min_{\Phi} \left( \sum_{a_i \in \mathcal{F}} |a_i| \| \nabla_i(\Phi) - J_i \|_2^2 + \lambda_\Phi \| \Phi - \mathcal{V} \|_2^2 \right)$$The screening weight $\lambda_\Phi$ is set to 5 for the coarse mesh and 1 for the fine mesh. This constrained formulation has a closed-form linear system solution:
$$(L + \lambda_\Phi I)\, \Phi^* = \mathcal{A} \nabla^T \mathbf{J} + \lambda_\Phi \mathcal{V}$$where $L$ is the cotangent Laplacian, $I$ is the identity matrix, and $\mathcal{A}$ is the mass matrix. This is solved via differentiable Cholesky decomposition, allowing gradient flow through the solver.
Self-supervised Training of Neural Jacobian Fields
Five MLP networks are trained:
- $\psi^c$: coarse Jacobian field (coarse mesh $\mathcal{M}^c$)
- $\psi^f$: fine Jacobian field (fine mesh $\mathcal{M}^f$)
- $\psi^b$: isotropic (base) color component
- $\psi^r$: anisotropic (normal-conditioned) color component
- $\psi^g$: Gaussian offset and scale parameters
Coarse-to-Fine Jacobian Prediction
For the coarse level, a spatial latent code is extracted from a tri-plane encoder $\mathcal{E}^c$ at the face center $x_i^c$:
$$\mathbf{z}_i^c = \mathcal{E}^c(x_i^c), \quad J_i^c = \psi^c(\mathbf{z}_i^c,\, \theta)$$For the fine level, the coarse feature is fused with the fine tri-plane feature:
$$\mathbf{z}_i^f = \mathcal{E}^f(x_j^f), \quad J_i^f = \psi^f(\mathbf{z}_i^f + \mathbf{z}_i^c,\, \theta)$$The fine encoder has a tri-plane resolution of 128 (versus 64 for coarse), each with 32-dimensional features per plane (concatenated from all three planes). Both MLPs have their last layer initialized to zero weights and biases for training stability, following NJF.
Appearance Representation
Avatar colors are decomposed into isotropic and anisotropic components:
$$\mathbf{c}_i^b = \psi^b(\mathbf{z}_i), \quad \mathbf{c}_i^r = \psi^r(\mathbf{z}_i,\, \mathbf{n}_i), \quad \mathbf{c}_i = \mathbf{c}_i^b + \mathbf{c}_i^r$$where $\mathbf{n}_i$ is the face normal of the posed avatar. This decomposition (inspired by ExAvatar) separates view-independent and normal-dependent appearance, avoiding spherical harmonics.
3D Gaussian Splatting Representation
Each 3D Gaussian is defined as:
$$\mathcal{G}_i = \{ \boldsymbol{\mu}_i,\, \delta\boldsymbol{\mu}_i,\, \mathbf{s}_i,\, \mathbf{q}_i,\, o_i,\, \mathbf{c}_i \}$$where $\boldsymbol{\mu}_i$ is the deformed face center $\Phi_\theta(x_i)$, $\delta\boldsymbol{\mu}_i$ is an offset, $\mathbf{s}_i$ is scale, $\mathbf{q}_i$ is quaternion, $o_i$ is opacity, and $\mathbf{c}_i$ is face color. The offset and scale are predicted by:
$$\delta\boldsymbol{\mu}_i,\, \mathbf{s}_i = \psi^{\mathcal{G}}(\mathbf{z}_i,\, \theta)$$Rendering proceeds in two stages. First, the mesh renderer produces image $\hat{\mathbf{I}}^t$, normal map $\hat{\mathbf{N}}^t$, and foreground mask $\hat{\mathbf{A}}^t$ from the posed deformed mesh. Second, the GS rasterizer produces the final high-fidelity image from LBS-warped Gaussians.
Training Objectives
Mesh Renderer Loss
The mesh-stage training loss combines photometric, geometric, temporal, and regularization terms:
$$\mathcal{L}_{\text{mesh}} = \lambda_{L1} \mathcal{L}_{L1} + \lambda_{\text{lpips}} \mathcal{L}_{\text{lpips}} + \lambda_{\text{ssim}} \mathcal{L}_{\text{ssim}} + \lambda_{\text{mask}} \mathcal{L}_{\text{mask}} + \lambda_{\text{normal}} \mathcal{L}_{\text{normal}} + \lambda_{\text{residual}} \mathcal{L}_{\text{residual}} + \lambda_{\text{SDF}} \mathcal{L}_{\text{SDF}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}} + \lambda_{\text{lap}_\mathcal{V}} \mathcal{L}_{\text{lap}} + \lambda_{\mathbf{C}^b} \mathcal{L}_{\text{dual-lap}} + \lambda_{\mathbf{C}} \mathcal{L}_{\text{dual-lap}}$$where normal and mask pseudo-ground-truths come from Sapiens; $\mathcal{L}_{\text{reg}}$ is an L1 regularizer on pose-dependent deformation maps; $\mathcal{L}_{\text{lap}}$ is a Laplacian regularizer on vertices; and $\mathcal{L}_{\text{dual-lap}}$ is a dual Laplacian regularizer on face colors. This loss is applied sequentially to coarse then fine meshes.
GS Renderer Loss
$$\mathcal{L}_{\text{gs}} = \lambda_{L1} L_{L1} + \lambda_{\text{lpips}} L_{\text{lpips}} + \lambda_{\text{ssim}} L_{\text{ssim}} + \lambda_{\text{scale}} L_{\text{scale}}(\mathbf{s}) + \lambda_{\text{offset}} L_{\text{offset}}(\delta\boldsymbol{\mu}) + \lambda_{\text{lap}\mathbf{s}} L_{\text{lap}}(\mathbf{s}) + \lambda_{\text{lap}\delta\boldsymbol{\mu}} L_{\text{lap}}(\delta\boldsymbol{\mu}) + \lambda_{\text{residual}} L_{\text{residual}}$$Scale and offset L2 regularizers prevent unbounded Gaussian growth. Laplacian regularizers on scale and offset encourage spatial smoothness. The residual flow loss is also applied at this stage using a 2D flow map rendered via alpha-blending of per-Gaussian displacement vectors.
Deformation-guided Residual Flow Loss
Per-frame observations alone are insufficient for capturing fine-grained local deformations. To address temporal consistency, the paper proposes a deformation-guided residual flow loss. For frames at time $t$ and $t-1$, the projected 2D position of vertex $V_i$ at time $t$ is:
$$\mathbf{u}_i^t = \pi\bigl(\text{LBS}(\Phi_{\theta_t}(V_i),\, \theta_t),\, P_t\bigr)$$where $\pi$ is the camera projection with matrix $P_t$. The 2D displacement vector is:
$$\delta\mathbf{u}_i^t = \mathbf{u}_i^t - \mathbf{u}_i^{t-1}$$This displacement is assigned as a vertex attribute and rasterized to produce a dense 2D flow map $\mathbf{W}^t$. Instead of directly comparing $\mathbf{W}^t$ to the optical flow between the two frames, the method uses a pre-trained optical flow network $\mathcal{F}$ (WAFT) that searches for correspondences given an initial displacement map:
$$\Delta\mathbf{W}^t = \mathcal{F}(\mathbf{I}^t,\, \mathbf{I}^{t-1},\, \mathbf{W}^t)$$The residual $\Delta\mathbf{W}^t$ becomes zero if $\mathbf{W}^t$ is perfect. The loss minimizes this residual:
$$L_{\text{residual}} = \| \Delta\mathbf{W}^t \|_1$$This is a key design insight: the avatar's predicted 3D flow serves as the initialization for the optical flow network, which then refines it. As training progresses, the initialization improves, guiding optimization toward progressively finer-grained deformations. This is more robust than naively comparing with optical flow, especially for complex human motions where optical flow networks struggle.
Signed Distance-based Jacobian Regularization
Sparse monocular observations cause unwanted deformations (mesh stretching, flipped faces) in self-occluded regions like armpits and the back. The SDF-based regularizer maintains global avatar shape by computing a voxel grid of discrete SDF values from the initial coarse mesh $\mathcal{M}^c$. The spatial gradient of the SDF at any point approximates the surface normal (the gradient is anti-parallel to the normal at the surface).
Using these SDF-derived normals as a geometric prior, face and vertex normals of the deformed avatar are regularized via cosine similarity:
$$\mathcal{L}_{\text{SDF}} = \lambda_{\text{face}} \sum_{a_i \in \mathcal{F}} \left(1 - \hat{\mathbf{n}}_i^a \cdot \mathbf{n}_{\text{SDF}}(x_i)\right) + \lambda_{\text{vertex}} \sum_{v_j \in \mathcal{V}} \left(1 - \hat{\mathbf{n}}_j^v \cdot \mathbf{n}_{\text{SDF}}(v_j)\right)$$where $x_i$ is the center of the $i$-th face, $\hat{\mathbf{n}}_i^a$ and $\hat{\mathbf{n}}_j^v$ are the deformed face and vertex normals, and $\mathbf{n}_{\text{SDF}}(x)$ is the normalized spatial gradient of the SDF volume at position $x$. This regularization ensures Jacobian matrices evolve along neighboring iso-surfaces rather than undergoing abrupt positional changes, effectively preventing the back and other infrequently observed regions from developing noisy deformations.
Network Architecture Details (Supplementary)
Rather than predicting the $3 \times 3$ Jacobian matrix as a flat 9-channel output (as in the original NJF), three separate 3-layer MLPs are used to decompose the Jacobian into interpretable components:
- Rotation MLP: Outputs a 6D continuous rotation representation (converted to a $3 \times 3$ rotation matrix).
- Scale MLP: Outputs a 3D scale vector placed on the diagonal of an upper triangular matrix.
- Shear MLP: Outputs a 3D shear vector placed on the upper off-diagonal elements of the same upper triangular matrix.
The final Jacobian is obtained by multiplying the rotation matrix with the upper triangular matrix. This decomposition reduces non-linearity in the output space and makes optimization more stable.
Datasets and Evaluation
Datasets
MonoPerfCap: In-the-wild indoor/outdoor captures with natural poses and non-loose clothing under stable lighting. High frame rate provides temporally dense, continuous motion. The first 80% of frames are used for training, the remainder for testing — enabling evaluation of generalization to novel views (view extrapolation).
SynWild: A synthetic dataset with ground-truth meshes for quantitative geometry evaluation. Contains clothing deformations suitable for assessing semi-rigid deformations. ICP registration is used to align reconstructed avatars to ground-truth meshes.
NeuMan: Outdoor videos with repetitive motions and less cloth deformation than MonoPerfCap. The subject moves over longer distances, causing lighting variations. The official split uses interval views for testing (novel view interpolation evaluation).
DNA-Rendering: Multi-view studio captures with various clothing types including loose garments with challenging non-rigid deformations. Adapted to monocular by randomly sampling one camera per frame. Used for novel view synthesis and qualitative mesh reconstruction comparison.
Evaluation Metrics
- Rendering quality: PSNR, SSIM, LPIPS (×100)
- Geometry accuracy: Chamfer Distance (CD, ×1000), Normal Error (NE), F1-score at 1 cm and 2 cm thresholds
Normal Error is computed by rendering normals from four orthogonal views (front, back, left, right) and measuring cosine similarity in overlapping regions between reconstructed and ground-truth meshes. For most datasets, body poses are additionally optimized on the test set to reduce dependency on pose accuracy.
Experimental Results
Comparison Methods
Comparisons are made against: Vid2Avatar (SDF-based canonical avatar), LSAvatar (SDF + graph neural network locality-sensitive deformation), FacAvatar (SDF + frequency-controlled coarse-fine deformation), ExAvatar (3DGS anchored to template mesh), and GoMAvatar (3DGS with mesh-Laplacian regularizer). Additional comparisons on NeuMan include HumanNeRF, InstantAvatar, NeuMan, 3DGS-Avatar, and GaussianAvatar.
Rendering Quality — MonoPerfCap
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| GoMAvatar | 28.35 | 0.975 | 1.95 |
| Vid2Avatar | 28.74 | 0.977 | 2.06 |
| LSAvatar | 27.96 | 0.975 | 2.15 |
| FacAvatar | 28.06 | 0.975 | 1.99 |
| ExAvatar | 30.47 | 0.980 | 2.00 |
| Ours | 31.83 | 0.978 | 1.67 |
JacobianAvatar outperforms all comparison methods on MonoPerfCap. The improvement is attributed to the hybrid mesh-3DGS representation providing coherent geometry and high-fidelity appearance in unseen views, and the high frame rate benefiting the residual flow loss.
Rendering Quality — DNA-Rendering
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| Vid2Avatar | 29.46 | 0.972 | 2.88 |
| LSAvatar | 27.94 | 0.968 | 2.37 |
| ExAvatar | 28.54 | 0.970 | 2.85 |
| Ours | 29.90 | 0.971 | 2.19 |
On the challenging DNA-Rendering dataset with loose garments, the pipeline first optimizes global shape via initial mesh optimization, then recovers fine-level geometry with the network, achieving best rendering quality by providing more accurate geometry than baselines.
Rendering Quality — NeuMan
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| HumanNeRF | 27.06 | 0.967 | 1.92 |
| InstantAvatar | 28.47 | 0.972 | 2.77 |
| NeuMan | 25.48 | 0.966 | 2.87 |
| Vid2Avatar | 26.87 | 0.969 | 2.41 |
| 3DGS-Avatar | 29.75 | 0.975 | 1.75 |
| GaussianAvatar | 29.94 | 0.980 | 1.24 |
| ExAvatar | 29.37 | 0.980 | 1.87 |
| Ours | 29.75 | 0.974 | 1.94 |
Despite severe lighting changes in NeuMan scenes (not explicitly handled), JacobianAvatar achieves results comparable to state-of-the-art.
Geometry Accuracy — SynWild
| Method | Type | CD ↓ | NE ↓ | F1@1cm ↑ | F1@2cm ↑ |
|---|---|---|---|---|---|
| GoMAvatar | Explicit | 9.30 | 0.140 | 0.276 | 0.544 |
| ExAvatar | Explicit | 9.39 | — | 0.307 | 0.515 |
| Vid2Avatar | Implicit | 2.59 | 0.091 | 0.346 | 0.647 |
| LSAvatar | Implicit | 2.55 | 0.095 | 0.343 | 0.646 |
| FacAvatar | Implicit | 2.91 | 0.106 | 0.329 | 0.623 |
| Ours | Explicit | 2.46 | 0.091 | 0.397 | 0.681 |
JacobianAvatar is the only explicit method to match or beat implicit (SDF-based) methods on all geometry metrics — a significant result, since explicit mesh methods traditionally struggle with geometric precision. The key advantage is direct mesh deformation via NJF, coupled with the deformation-guided residual loss for topological integrity, free from Marching Cubes extraction artifacts.
Qualitative Results
A notable qualitative finding is the texture-copying artifact observed in competing methods: when a subject wears highly textured clothing, 2D textures are incorrectly "baked" into the 3D geometry of SDF-based approaches. JacobianAvatar independently captures semi-rigid deformations without optimizing texture-related variables, eliminating this entanglement between geometry and appearance.
Ablation Study
| Method | CD (×1000) ↓ | NE ↓ |
|---|---|---|
| w/o $\mathcal{L}_{\text{SDF}}$ | 3.51 | 0.113 |
| w/o $\mathcal{L}_{\text{residual}}$ | 3.01 | 0.106 |
| w/o Screened Poisson solver | 3.28 | 0.106 |
| w/o Coarse-to-fine Strategy | 2.96 | 0.107 |
| Ours (full) | 3.01 | 0.102 |
Coarse-to-fine Strategy: Without the coarse Jacobian field, optimizing only the fine field introduces high-frequency, noisy deformations and fails to capture prominent wrinkles. The ablation achieves better CD (insensitive to small noise) but worse NE (which penalizes noisy local normals).
Screened Poisson Solver: Without the screening term, the standard Poisson solver causes large mesh stretching artifacts due to limited monocular observations. These severe distortions significantly degrade CD.
Deformation-guided Residual Flow ($\mathcal{L}_{\text{residual}}$): Beyond temporal consistency, this loss is critical for suppressing artifacts in heavily self-occluded areas like armpits. It mitigates noisy flow predictions from pretrained models in such regions. Since self-occluded regions are a small fraction of the body, CD is only mildly affected, but local artifacts strongly degrade NE.
SDF-based Jacobian Regularization ($\mathcal{L}_{\text{SDF}}$): Without this regularization, rarely visible areas such as the woman's back suffer severe noisy deformations. The model becomes susceptible to viewpoint overfitting, causing large drops in both CD and NE — the largest degradation of all ablated components.
Animation and Generalization
Semi-rigid deformations are formulated as pose-dependent functions. Avatar animation uses SMPL-X body poses sampled from the AIST++ Dance Motion Dataset. The mesh-based representation maintains shape in unseen body poses, and 3DGS anchored to the deformable mesh provides a strong geometric prior for stable rendering in novel views.
Implementation Details
- Canonical mesh: MHR template at LoD 3
- Screened Poisson $\lambda_\Phi$: 5 (coarse), 1 (fine)
- Tri-plane resolutions: 64 (coarse), 128 (fine); feature dimension 32 per plane (concatenated from 3 planes)
- Last MLP layers initialized to zero weights and biases
- Hardware: single NVIDIA RTX 6000 Ada GPU
- Training time: ~10 hours including preprocessing (depends on video length)
- Jacobian MLPs decompose the Jacobian into rotation (6D representation), scale, and shear — preventing high non-linearity in the direct $3 \times 3$ parameterization
Key hyperparameters across the three training stages (Coarse / Fine / GS):
| Loss Term | Coarse | Fine | GS |
|---|---|---|---|
| RGB L1 ($\lambda_{L1}$) | 10.0 | 10.0 | 10.0 |
| LPIPS ($\lambda_{\text{lpips}}$) | 2.0 | 2.0 | 2.0 |
| SSIM ($\lambda_{\text{ssim}}$) | 2.0 | 2.0 | 2.0 |
| Residual flow ($\lambda_{\text{residual}}$) | 10.0 | 20.0 | 20.0 |
| Deformation L1 reg ($\lambda_{\text{reg}}$) | 1.0 | 1.0 | — |
| SDF regularization ($\lambda_{\text{SDF}}$) | 1.0 | 0.1 | — |
| Laplacian vertices ($\lambda_{\text{lap}_\mathcal{V}}$) | 5,000 | 30,000 | — |
| Scale L2 ($\lambda_{\text{scale}}$) | — | — | 10.0 |
| Offset L2 ($\lambda_{\text{offset}}$) | — | — | 1.0 |
| Laplacian scale ($\lambda_{\text{lap}\mathbf{s}}$) | — | — | 5,000 |
| Laplacian offset ($\lambda_{\text{lap}\delta\boldsymbol{\mu}}$) | — | — | 10.0 |
Contributions and Novelty
The paper makes the following distinct contributions to monocular avatar reconstruction:
- Hierarchical Neural Jacobian Fields: A coarse-to-fine NJF framework for topology-preserving semi-rigid deformation, where the fine network is conditioned on coarse latent features for stable, multi-scale deformation capture.
- Screened (Constrained) Poisson Solver: Modification of the standard NJF Poisson formulation with a screening term that anchors vertex positions to the canonical mesh, preventing expansion artifacts in regions with limited monocular observations.
- Signed Distance-based Jacobian Regularization: A voxel-grid SDF prior that constrains face and vertex normals to match the global shape of the canonical avatar, preventing overfitting to visible viewpoints and suppressing artifacts in occluded regions.
- Deformation-guided Residual Flow Loss: A temporal consistency loss that uses the avatar's predicted 3D vertex flow as an initialization for a pre-trained optical flow network, then minimizes the residual. This progressively refines temporal supervision as training improves the flow initialization, enabling robust correspondence in complex human motions.
- First explicit method to outperform implicit SDF methods on geometry metrics, while also achieving top rendering quality — demonstrated on MonoPerfCap, DNA-Rendering, NeuMan, and SynWild datasets.
Limitations and Future Work
The paper identifies two main limitations:
- Loose clothing: The method relies on LBS weights from a naked body template. Loose garments that behave differently from the inner body (e.g., skirts, dresses) cannot be fully modeled, as their dynamics are not well-explained by the body skeleton's LBS. The coarse-to-fine pipeline still achieves reasonable results (demonstrated on SCARF dataset), but this is outside the primary design scope.
- Facial expressions: The current model does not encompass facial expression modeling. The authors note that existing facial reconstruction methods could be integrated into the framework in future work.
These limitations point toward promising extensions: incorporating garment-aware topology adaptation for loose clothing, and integrating expression models (e.g., EMOCA, FLAME) for full expressive avatar reconstruction from monocular video.