HiReFF

HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video

A feed-forward method for streaming 360° human reconstruction at 2K resolution from uncalibrated sparse-view videos. It achieves real-time performance with temporal consistency by efficiently decomposing Gaussian reconstruction and high-resolution synthesis via a lightweight side-tuning architecture.

• gaussian-splatting
• 3d-avatar
• full-body
• streaming
• realtime

Authors: Yiming Jiang, Hanzhang Tu, Wenfeng Song, Siyou Lin, Liang An, Shuai Li, Aimin Hao, Yebin Liu

Categories: cs.CV

Published 2026-06-28 · Updated 2026-06-28

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub7 stars0 watchers0 forksMIT

Impact

• Semantic Scholar0 citations

Abstract

Uncalibrated volumetric video streaming for human reconstruction is essential for holographic communication and AR/VR, yet remains challenging due to the need for temporal consistency and computational efficiency from sparse-view inputs. Existing methods rely on per-scene optimization or calibrated cameras, while recent feed-forward models are limited to low-resolution (0.5K) single-frame synthesis. We present HiReFF, a feed-forward method for 2K-resolution 360° human video reconstruction from uncalibrated sparse-view videos. Our framework decomposes the problem into two key tasks: foreground 3D Gaussian reconstruction from sparse-view videos (four views separated by 90°) and computationally efficient high-resolution synthesis. To enable the former, we propose Scale-synchronized Camera Calibration to resolve scale ambiguity for multi-view supervision, and Gaussian-wise Foreground Masking to reconstruct clean foregrounds by modulating Gaussian parameters. For efficient high-resolution synthesis, our High-resolution Side-tuning achieves 2K rendering by augmenting the Gaussian head with supplementary features while keeping the backbone at 0.5K, drastically reducing computational overhead. Experiments demonstrate that HiReFF significantly outperforms existing methods in high-resolution streaming volumetric video reconstruction. https://iridescentjiang.github.io/HiReFF

---

Introduction and Motivation

Volumetric video streaming of human subjects — delivering photorealistic, 360° renderable representations in real-time — is a key enabling technology for holographic communication, AR/VR, and live sports broadcasting. The core challenge is performing this reconstruction from sparse, uncalibrated camera inputs while simultaneously achieving temporal consistency, computational efficiency, and high-resolution output.

Existing approaches fall into two broad categories, each with critical shortcomings. Per-scene optimization methods (e.g., NeRF-based or Gaussian-based iterative fitting) require long training times per sequence and cannot operate in a streaming fashion. Calibrated feed-forward methods (e.g., GPS-Gaussian) deliver faster inference but require precise, pre-computed camera parameters — a strong assumption that limits real-world deployment. The recent wave of uncalibrated feed-forward 3D reconstruction models (VGGT, AnySplat, Pi3, MapAnything) has shown that camera poses, depth, and even 3D Gaussians can be predicted directly from images, but these models are designed for single-frame inference, struggle with wide-baseline (90°) sparse views, and are computationally constrained to approximately 0.5K (518×518) output resolution.

HiReFF (High-Resolution FeedForward) directly addresses this gap. Given four uncalibrated RGB videos captured at 90° angular separation (providing full 360° coverage), HiReFF reconstructs a temporally consistent 3D Gaussian Splatting (3DGS) volumetric representation and renders novel views at 2K resolution (2072×2072) in a streaming, feed-forward manner — achieving 3.01 FPS on a single RTX 4090 GPU with only 34% additional VRAM compared to 0.5K training.

Problem Formulation

The input to HiReFF is a set of four uncalibrated high-resolution RGB videos $\{V_i\}_{i=1}^{4}$, where each video $V_i$ consists of frames $I_i^t \in \mathbb{R}^{H_\text{HR} \times W_\text{HR} \times 3}$, captured at viewpoints separated by approximately 90°. The goal is to estimate, for each time step $t$:

• A set of $G = 4 \times H_\text{HR} \times W_\text{HR}$ anisotropic 3D Gaussians $\{(\mu_g^t, \sigma_g^t, r_g^t, s_g^t, c_g^t)\}_{g=1}^{G}$, parameterized by position $\mu \in \mathbb{R}^3$, opacity $\sigma \in \mathbb{R}^+$, rotation $r \in \mathbb{R}^4$, scale $s \in \mathbb{R}^3$, and spherical harmonic coefficients $c \in \mathbb{R}^{3 \times (k+1)^2}$ of degree $k$.
• Temporally smooth per-view camera parameters $\{p_i^t \in \mathbb{R}^9\}_{i=1}^{4}$ estimated with inter-frame continuity constraints.

The full mapping is: $$f_\theta: \{V_i^\text{HR}\}_{i=1}^{4} \longmapsto \left\{ (\mu_g^t, \Theta_g^t)_{g=1}^{G} \cup (p_i^t)_{i=1}^{4} \right\}_{t=1}^{T},$$ where $\Theta_g^t = (\sigma_g^t, r_g^t, s_g^t, c_g^t)$ represents the non-positional 3DGS parameters. The model also produces auxiliary per-frame depth maps and smooth camera trajectories.

Method Overview

HiReFF builds on top of the pretrained VGGT backbone, which uses an Alternating-Attention (AA) Transformer to extract multi-view features and decode them into camera parameters, depth, and point clouds. HiReFF augments this backbone with three principal innovations:

1. Side-tuning Gaussian Prediction Head — a DPT-based Gaussian head that predicts 3DGS parameters from AA Transformer features, augmented with a lightweight side network for image-level details.
2. Scale-synchronized Camera Calibration — resolves the scale ambiguity between predicted and ground-truth camera parameters to enable supervision from additional novel viewpoints.
3. Gaussian-wise Foreground Masking — a mask head that selectively suppresses background Gaussians while preserving accurate camera estimation.

For efficient high-resolution rendering, HiReFF additionally employs High-resolution Side-tuning, which keeps the AA Transformer running at 518×518 while injecting high-resolution image features into the Gaussian head for 2K-resolution output.

Core Architecture Components

Side-tuning Gaussian Prediction

Inspired by AnySplat, HiReFF introduces a DPT-based Gaussian head $F_G$ to decode AA Transformer features into 3DGS parameters. A supplementary MLP $F_a$ extracts fine-grained, low-level features directly from the input images. These are combined with intermediate Gaussian head features: $$f_{A}^t = F_{AA}(I^t),$$ $$\Theta^t = F_D\!\left(F_a(I^t) \oplus F_G^{\text{mid}}(f_{A}^t)\right),$$ where $F_{AA}$ is the AA Transformer, $F_G^{\text{mid}}(\cdot)$ denotes intermediate features of the Gaussian head, $\oplus$ indicates addition, and $F_D(\cdot)$ is a final MLP that generates the Gaussian attributes $\{\sigma, r, s, c\}$.

The 3D position $\mu$ of each Gaussian is obtained by back-projecting the corresponding pixel using predicted depth $d$, intrinsics $\mathbf{K}$, and extrinsics $[\mathbf{R} \mid \mathbf{t}]$: $$\mu = \mathbf{R}^T \left(d\,\mathbf{K}^{-1}\mathbf{p} - \mathbf{t}\right),$$ where $\mathbf{p} = [u, v, 1]^T$ is the pixel in homogeneous coordinates.

Gaussian-wise Foreground Masking

A naive approach to human reconstruction would mask the input images to only foreground regions before feeding them into the network. However, with four views spaced 90° apart, the authors empirically find that using the full images — including background — produces significantly more accurate camera parameter estimates. Masking the images degrades camera prediction quality in this wide-baseline setting.

To reconcile foreground-only reconstruction with accurate camera estimation, HiReFF introduces a dedicated mask head $F_m$ that operates on Gaussian parameters rather than on input images. The mask head predicts per-Gaussian foreground probabilities and applies them to selectively suppress background Gaussians: $$(\Theta^t_{g}, \mu_g^t)_{g=1}^{H} = F_m\!\left((\Theta^t_{g}, \mu_g^t)_{g=1}^{G}\right) \odot (\Theta^t_{g}, \mu_g^t)_{g=1}^{G},$$ where $\odot$ denotes Gaussian-wise masking and $H \leq G$ indexes only the Gaussians belonging to the foreground human. Because the four input views are approximately orthogonal, a small number of extraneous background Gaussians may survive masking, but the authors observe these vanish during training, yielding clean foreground reconstructions.

Scale-synchronized Camera Calibration

VGGT-based models predict camera parameters only up to an unknown metric scale — the predicted scene (including camera translations) is consistent internally but may be far from physical scale. This creates a fundamental mismatch: to supervise novel-view renderings from additional viewpoints (beyond the four inputs), ground-truth camera parameters are needed, but they cannot be directly used to render Gaussians whose positions are aligned with the predicted (unscaled) cameras.

A second problem is that for 90°-separated inputs, direct supervision of the camera head causes large gradient fluctuations during training, destabilizing convergence. HiReFF addresses both issues simultaneously:

• The camera head is frozen during training; the AA Transformer is kept active so that its output indirectly steers the camera head. This eliminates fluctuations.
• Ground-truth camera parameters are dynamically scale-adjusted to match the predicted camera's scale before rendering. For each non-reference view $i$ (with reference view $i=1$), the per-view scale factor between predicted translation $\mathbf{t}_i$ and ground-truth translation $\hat{\mathbf{t}}_i$ is computed, averaged across all non-reference views, and then applied to normalize ground-truth translations: $$\bar{s} = \frac{1}{V-1}\sum_{i=2}^{V}\frac{\hat{\mathbf{t}}_i}{\mathbf{t}_i}, \qquad \hat{\mathbf{t}}'_i = \frac{\hat{\mathbf{t}}_i}{\bar{s}}, \quad i = 2, \dots, V.$$

Rendering then uses the true intrinsics $\hat{\mathbf{K}}^t$, true rotations $\hat{\mathbf{R}}^t$, and scale-adjusted translations $\hat{\mathbf{t}}^{t\prime}$, while Gaussian positions are back-projected using the predicted camera parameters: $$\mu_g^t = {\mathbf{R}^t}^T\left(d^t\,{\mathbf{K}^t}^{-1}\mathbf{p} - \mathbf{t}^t\right),$$ $$I^t = \mathcal{R}\!\left(\{\mu_g^t, \Theta_g^t\}_{g=1}^{H},\, \hat{\mathbf{K}}^t,\, \hat{\mathbf{R}}^t,\, \hat{\mathbf{t}}^{t\prime}\right).$$ This design introduces additional novel viewpoints for supervision, avoids camera-head instability, and also helps inter-frame scale remain stable across the video sequence.

Computationally Efficient High-Resolution Synthesis

High-resolution Side-tuning

Feed-forward 3D reconstruction backbones (VGGT, AnySplat) operate at 518×518 input resolution because the quadratic complexity of the AA Transformer makes higher resolutions computationally prohibitive. Directly feeding 2K images into the transformer would exceed GPU memory (the paper reports out-of-memory for 2K input even without side-tuning).

The key insight is that high-resolution rendering quality depends primarily on the high-dimensional Gaussian attributes $\Theta_g$ output by the Gaussian head, not on the 3D positions $\mu_g$ or camera poses $p_i$. Therefore, positions and depths can be obtained at low resolution and interpolated, while the Gaussian appearance attributes need high-resolution detail. HiReFF achieves this via a supplementary "side" network $F_a$ (based on EdgeNeXt) that processes high-resolution images $I_\text{HR}$ and injects fine-grained features into the mid-level of the Gaussian head: $$\Theta^t = F_D\!\left(F_a\!\left(I_\text{HR}^t\right) \oplus \mathcal{U}\!\left(F_G^{\text{mid}}(I_\text{LR}^t)\right)\right),$$ where $\mathcal{U}(\cdot)$ denotes bilinear upsampling of the intermediate Gaussian head features to high resolution, and $I_\text{LR}^t$ is the 518×518 version of the input fed to the AA Transformer. After predicting high-resolution Gaussian attributes, the mask and depth head outputs are also upsampled, and rendering proceeds at full 2K resolution for supervision against 2K ground truth.

This design means the AA Transformer backbone never sees images larger than 518×518, keeping its computational cost fixed, while the supplementary pathway operates on high-resolution images with a lightweight architecture (EdgeNeXt-based MLP).

Training Objectives

Rendering Loss

During training, in addition to the four input views, $V_a = 4$ additional novel viewpoints are sampled from frontal, top, and bottom positions outside the input viewing sphere, yielding $4 + V_a = 8$ supervision viewpoints in total. The rendering loss combines L1 loss and VGG-based perceptual loss: $$\mathcal{L}_\text{render} = \sum_{i=1}^{4+V_a}\|\hat{I}_i - I_i\|_1 + \lambda_\text{P}\sum_{i=1}^{4+V_a}\text{Perceptual}(\hat{I}_i, I_i),$$ where $\hat{I}_i$ and $I_i$ are the rendered and ground-truth images for view $i$. Computation graph surgery is used to reduce the memory overhead of the perceptual loss at 2K resolution.

Mask Loss

An L1 loss is applied between predicted and ground-truth foreground masks on the four input views: $$\mathcal{L}_\text{mask} = \sum_{i=1}^{4}\|\hat{M}_i - M_i\|_1.$$

Depth Distillation Loss

To prevent the active depth head from overfitting to the input viewpoints, a frozen reference depth head (initialized with pretrained VGGT weights) is maintained and an MSE consistency loss is applied: $$\mathcal{L}_\text{depth} = \sum_{i=1}^{4}\|\hat{D}_i^\text{a} - \hat{D}_i^\text{f}\|_2^2,$$ where $\hat{D}_i^\text{a}$ and $\hat{D}_i^\text{f}$ are depth predictions from the active and frozen heads.

Combined Objective

The total loss is: $$\mathcal{L}_\text{total} = \lambda_\text{render}\,\mathcal{L}_\text{render} + \lambda_\text{mask}\,\mathcal{L}_\text{mask} + \lambda_\text{depth}\,\mathcal{L}_\text{depth},$$ with weights $\lambda_\text{P} = 0.1$, $\lambda_\text{render} = 1.0$, $\lambda_\text{mask} = 5 \times 10^{-2}$, and $\lambda_\text{depth} = 10$.

Experimental Setup

Datasets

Training is conducted primarily on the DNA-Rendering dataset, which contains 153 actors across 439 distinct motion sequences, each comprising 48-view synchronized RGB video streams at 2448×2048 resolution with corresponding camera parameters. To improve generalization, ZJU-MoCap and MVHumanNet are also used. For evaluation, 20 motion sequences with distinct identities are reserved as the test set from within DNA-Rendering.

Implementation Details

The AA Transformer, camera head, and both active and frozen depth heads are initialized from pretrained VGGT weights. The Gaussian prediction head, side-tuning supplementary network, and mask head are zero-initialized. The Gaussian splatting renderer is based on gsplat. HR resolution is defined as 2072×2072 and LR as 518×518. Training is performed on 8 A800 GPUs with automatic mixed precision.

Evaluation Metrics and Protocol

Novel-view rendering quality is assessed using PSNR, SSIM, and LPIPS at 2072×2072 resolution. Methods that render at lower resolutions (e.g., GPS-Gaussian, AnySplat) have their outputs bilinearly upsampled to 2K before metric computation. All methods are evaluated in a 4-view input setting with ~90° angular separation. For calibrated-camera methods, ground-truth camera parameters are supplied (clearly indicated in results). Since comparison methods lack built-in foreground masks, an open-source video portrait segmentation algorithm (RVM) is used to generate masks; all predicted results are multiplied by a consistent mask before metric computation for fair comparison.

Baselines

• AnySplat — uncalibrated feed-forward 3DGS; 4-view input.
• NoPoSplat — uncalibrated feed-forward 3DGS; 2-view input.
• 4DGT — monocular volumetric video reconstruction; 1-view input.
• GPS-Gaussian — calibrated camera; 8 input views; receives pre-segmented foreground images. Represents a stronger but more constrained baseline.

Results

Qualitative Results

HiReFF consistently outperforms all baselines in global shape, garment details, and facial fidelity. AnySplat and NoPoSplat struggle to estimate accurate camera parameters at the 90° baseline, producing misaligned multi-view reconstructions. GPS-Gaussian, despite receiving 8 calibrated views and pre-segmented foreground images, still exhibits leg artifacts. 4DGT fails to produce plausible results from single-frame input.

Additional viewpoints — including top-down and bottom-up perspectives — show that HiReFF correctly reconstructs geometry and surface color. Complex clothing patterns are accurately reproduced.

HiReFF demonstrates excellent temporal consistency across frames, including for subjects with complex patterned garments — a particularly challenging case for feed-forward methods without per-scene optimization.

Quantitative Results

HiReFF outperforms all uncalibrated camera methods across all three metrics. Compared to the most directly comparable baseline (AnySplat, unoptimized), HiReFF improves PSNR by +2.73 dB, SSIM by +0.0124, and reduces LPIPS by 0.046. HiReFF also outperforms AnySplat even after the latter undergoes 200 steps of camera alignment optimization, while HiReFF requires no post-hoc optimization at all.

Compared to GPS-Gaussian — which receives ground-truth cameras and 8 pre-segmented input views — HiReFF achieves higher PSNR and better LPIPS with only 4 uncalibrated inputs. The paper notes that GPS-Gaussian's metric advantage on SSIM partly reflects a favorable evaluation artifact: GPS-Gaussian uses masked (foreground-only) inputs, so its extra-feet artifact is suppressed when evaluation masks are applied, while HiReFF's complete reconstruction of background objects (e.g., instruments) is penalized despite being visually correct.

Ablation Studies

Computational Efficiency of High-resolution Side-tuning

The following tables summarize training and inference VRAM/speed under different resolution configurations.

With HiReFF's proposed side-tuning strategy (518 input, 2072 supervision), VRAM is only 44,125 MiB — a 33.9% increase over the 518/518 baseline. Direct 2K input without side-tuning causes out-of-memory errors on 80 GiB A100 GPUs.

At 2K rendering resolution, HiReFF achieves 3.01 FPS with 2K side input — only 24% slower than the 4.40 FPS at 0.5K side input. When both configurations render at 2K, using 1K image input reduces speed by merely 8.6% compared to 0.5K input. The dominant cost is the Gaussian renderer at 2K resolution, not the backbone or side network.

Gaussian-wise Foreground Masking

As shown in the ablation figure, the mask head effectively eliminates background Gaussians. The small residual in the unmasked variant demonstrates that training with the Gaussian-wise masking loss encourages the Gaussian decoder to avoid reconstructing out-of-mask regions, further enhancing foreground reconstruction quality.

Limitations

The primary limitation of HiReFF is the appearance of isolated spurious Gaussian points in regions occluded from most input views. For example, if a body region is captured only by the frontal camera and not by any side cameras, the limited geometric cues lead to a handful of discrete floating points in that area. The authors plan to incorporate human body priors (e.g., for hands and faces) to enhance geometric and topological accuracy in these challenging regions.

Related Work Context

Feed-Forward 3D Reconstruction

HiReFF builds on a line of work that predicts 3D Gaussians directly from images: MVSplat, GGN, FreeSplat, and DepthSplat operate on calibrated inputs, while DUSt3R, VGGT, and MapAnything predict camera poses along with 3D point clouds. Uncalibrated Gaussian prediction methods include NoPosplat and AnySplat. FastAvatar and Human3R extend uncalibrated reconstruction to human subjects but remain limited to low-resolution single-frame inference.

Sparse-View Human Reconstruction

Parametric avatar methods (IDOL, LHM using SMPL priors) achieve fast reconstruction but struggle with loose clothing. Per-scene optimization methods (HumanNeRF, 3DGS-avatar) require calibrated rigs. Feed-forward calibrated methods (GPS-Gaussian, DoubleField, DiffuStereo, RoGSplat, Diffuman4D) require precise camera calibration. Forge4D first achieved frontal human novel-view synthesis from uncalibrated video but does not address 360° coverage, streaming operation, or high resolution simultaneously. HiReFF is the first method to address all four requirements jointly.

High-Resolution Novel View Synthesis

NeRF-based HRNVS methods (NeRF-SR, RefSR-NeRF, Super-NeRF) optimize high-resolution neural radiance fields. 3DGS-based methods (SRGS, GaussianSR) achieve high-quality HRNVS but are not compatible with modern feed-forward reconstruction backbones. HiReFF's side-tuning adaptation strategy bridges this gap within a feed-forward 3DGS framework.

Summary of Contributions

• HiReFF framework: the first feed-forward method for 2K-resolution 360° streaming volumetric video reconstruction from four uncalibrated 90°-spaced input videos, achieving 3.01 FPS on a single RTX 4090.
• Scale-synchronized Camera Calibration: resolves metric scale ambiguity by dynamically aligning predicted and ground-truth camera translations, enabling multi-view novel-view supervision without destabilizing the camera head.
• Gaussian-wise Foreground Masking: a mask head that applies foreground filtering at the Gaussian level rather than the image level, preserving camera estimation accuracy while achieving clean foreground reconstruction.
• High-resolution Side-tuning: a supplementary EdgeNeXt-based network that injects high-resolution image features into the Gaussian head's mid-level, enabling 2K rendering with only ~34% additional VRAM over 0.5K training and no modification to the AA Transformer backbone.
• State-of-the-art performance: significant improvements over all uncalibrated baselines (e.g., +2.73 dB PSNR vs. AnySplat) and favorable comparison to calibrated baselines with far stronger input assumptions.