PointSplat

PointSplat: Compact Gaussian Splatting via Human-Centric Prediction

A method for compact 3D human reconstruction that directly predicts Gaussian primitives in 3D space rather than per-view, eliminating inter-view redundancy. This human-centric approach produces naturally compact representations ideal for real-time streaming while maintaining high fidelity.

• gaussian-splatting
• 3d-avatar
• talking-human
• streaming
• low-latency
• avatar

Authors: Yujie Guo, Yudong Jin, Lingteng Qiu, Zehong Shen, Zhen Xu, Jing Zhang, Xianchao Shen, Hujun Bao, Sida Peng, Xiaowei Zhou

Lab: State Key Lab of CAD&CG, Zhejiang University

Categories: cs.CV

Comment: Project Page: https://zju3dv.github.io/pointsplat

Published 2026-06-30 · Updated 2026-06-30

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub37 stars2 watchers1 forks

Impact

• Semantic Scholar0 citations

Abstract

Producing 3D human representations from input views on the fly is essential for immersive live streaming systems, where representation compactness is as critical as high fidelity given limited computational power and transmission bandwidth. Although recent feed-forward reconstruction methods achieve impressive quality through the view-centric prediction of 3D representations, they repeatedly encode the same subject content across multiple views, leading to significant inter-view redundancy. Our key insight is to perform predictions directly in 3D space, enabling the network to learn and produce a highly compact representation. To this end, we propose PointSplat, a novel human-centric approach that directly infers Gaussian primitives from an input point set. The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points and establish explicit 2D--3D correspondences. Subsequently, it employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass. This design restricts predictions to foreground regions of interest, substantially reducing the total number of Gaussians while improving novel-view rendering quality. Extensive experiments demonstrate that PointSplat achieves higher efficiency and quality while exhibiting strong robustness to variations in view count and image resolution across multiple datasets.

---

Introduction and Motivation

Producing compact, high-fidelity 3D representations of humans in real time is a fundamental requirement for immersive live streaming, holographic communication, and related interactive applications. These systems must balance reconstruction quality against computational budget and transmission bandwidth — constraints that are especially severe on user devices.

Traditional light-field methods using dense camera arrays can achieve high-quality, low-latency rendering, but their reliance on costly capture hardware severely limits practical deployment. More recently, neural feed-forward reconstruction methods have dramatically lowered hardware requirements by predicting 3D representations directly from sparse views. The dominant paradigm within this family is the view-centric approach: for each input image, the network predicts a pixel-aligned Gaussian map or feature volume, and all per-view predictions are unprojected into a shared 3D space.

The central problem with view-centric representations is inter-view redundancy: the same region of the subject is encoded multiple times — once per camera that observes it. As the number of input views or the rendering resolution grows, the total Gaussian count, memory footprint, and transmission cost scale accordingly, undermining scalability. Post-processing strategies such as Gaussian pruning or feed-forward compression have been proposed to mitigate this, but they introduce ambiguities in which view should carry the "dominant" opacity signal, creating learning instabilities and blurry results.

PointSplat addresses redundancy at its root by shifting from a view-centric to a human-centric (object-centric) prediction paradigm. Instead of predicting Gaussians independently per view, PointSplat infers a single, unified set of Gaussian primitives directly in 3D space, drawing on multi-view appearance evidence. This design ensures each surface point is represented exactly once regardless of how many views observe it, yielding a naturally compact representation.

Related Work

Feed-Forward 3D Reconstruction

Feed-forward methods span implicit approaches — which map observations to target views via continuous functions (e.g., IBRNet, ENeRF) — and explicit approaches that directly regress geometry. LVSM learns to render novel views with minimal 3D bias but is slow to render. DUSt3R unifies monocular and stereo reconstruction via pairwise pointmap regression; follow-up works (VGGT, PI3, Depth Anything 3) extend this to full 3D attribute inference in a single forward pass, but retain a view-centric design that accumulates inter-view redundancy.

View-Centric Representations

Methods such as PixelSplat, MVSplat, DepthSplat, and GPS-Gaussian predict pixel-aligned Gaussian maps per view and unproject them. The LRM family (GS-LRM, GRM, LGM) uses transformer architectures to predict per-view Gaussian maps end-to-end. All share the common deficiency that overlapping predictions cover the same surface regions multiple times, and post-hoc pruning (Long-LRM, GGN) or compression (FCGS) fails to cleanly resolve the ambiguity.

Native 3D Representations

Voxel-based methods (Plenoxels, ScaffoldGS, OctreeGS) discretize scenes into grid structures. AnySplat employs differentiable voxelization to aggregate Gaussians in a feed-forward pass but relies on external geometric supervision from a pre-trained depth model (VGGT). Human-specific approaches like LHM and LHM++ exploit the SMPL parametric body model as a strong shape prior, enabling precise Gaussian prediction for typical human bodies, but struggle with loose clothing and occluded objects. Generative 3D methods (HunyuanD, CLAY, Direct3D) learn object-centric VAE latent spaces over point clouds but are not designed for multi-view reconstruction and cannot guarantee 2D consistency.

Method

Given a set of calibrated RGB images with corresponding object masks, PointSplat reconstructs a compact 3D Gaussian Splatting (3DGS) representation in a single feed-forward pass. The pipeline has four main stages: (1) appearance feature extraction, (2) geometry proxy construction and ray-casting-based point selection, (3) Point-Image Transformer fusion, and (4) Gaussian parameter decoding. The whole system is trained end-to-end from RGB supervision only.

Appearance Feature Extraction

Plücker Ray Embedding. Each pixel is described by its Plücker ray coordinates, which encode both the ray direction $\mathbf{d}$ and its moment $\mathbf{o} \times \mathbf{d}$ relative to the camera origin $\mathbf{o}$: $$\mathbf{f}_{\text{ray}} = [\mathbf{d},\; \mathbf{o} \times \mathbf{d}] \in \mathbb{R}^6$$ This 6-dimensional ray descriptor is concatenated with the pixel color $\mathbf{c} \in \mathbb{R}^3$ to produce a 9-dimensional pixel feature: $$\mathbf{f}_{\text{pixel}} = [\mathbf{c},\; \mathbf{f}_{\text{ray}}] \in \mathbb{R}^9$$ Images are then patchified (patch size $4 \times 4$) into a token sequence: $$\mathbf{t}_{\text{images}} = \operatorname{Patchify}(\mathbf{f}_{\text{pixel}}) \in \mathbb{R}^{N \times C}$$

Masked Token Sampling. Rather than processing all image patches, PointSplat applies mask-guided top-$n$ sampling to focus computation on foreground regions. Each token is assigned a score equal to the sum of binary mask values within its spatial footprint: $$s_i = \sum_{j \in \operatorname{Patchify}(i)} m_j$$ The top-$n$ tokens are selected and projected to the model's hidden dimension $C$ via a linear layer: $$\mathbf{T}_{\text{appearance}} = \operatorname{Linear}\!\left(\operatorname{SelectTop\text{-}}n(\{t_i\}, \{s_i\})\right) \in \mathbb{R}^{n \times C}$$ This step both reduces computation and ensures the network focuses on the human subject of interest.

Geometry Feature Extraction

Geometric Proxy via Space Carving. A coarse but visibility-consistent geometric proxy of the subject is constructed using a classical space carving algorithm. Given calibrated binary masks from all input views, the algorithm iteratively discards 3D points that are not visible (i.e., inside the silhouette) in every view, producing a dense visual hull $\mathbf{p}$ that tightly bounds the subject's surface. Unlike depth estimation, which can be noisy and view-inconsistent, the visual hull provides a deterministic, multi-view-consistent structure.

The visual hull is compared in ablation against VGGT-predicted depth, simple bounding-box fill, and frustum sampling. The visual hull consistently outperforms all alternatives, achieving 27.18 dB PSNR versus 25.95 dB (VGGT depth), 23.89 dB (bounding box), and 22.92 dB (frustum sample).

In supplementary experiments, visual-hull initialization is also compared with SMPL-based initialization. SMPL anchors are constrained by mocap accuracy and the template body geometry, making them less reliable for loose clothing or human–object interactions, whereas visual-hull anchors flexibly cover all observed foreground geometry.

Voxelization and Ray Casting. The visual hull point cloud $\mathbf{p}$ is voxelized into a regular grid (voxel size 0.005 in a normalized $2 \times 2 \times 2$ cube) to obtain structured voxel-level points $\mathbf{p}_v$. This grid then supports an efficient ray-casting step that serves two purposes: (i) removing redundant interior points that do not contribute to rendering, and (ii) establishing explicit pixel-to-point correspondences for cross-modal feature fusion.

For each pixel ray $\mathbf{r}(t) = \mathbf{c} + t\mathbf{d}$, a Digital Differential Analyzer (DDA) traverses the voxel grid. In each intersected voxel, the point closest to the ray is identified by minimizing the point-to-ray distance: $$d(\mathbf{p}_v, \mathbf{r}) = \frac{\|(\mathbf{p}_v - \mathbf{c}) \times \mathbf{d}\|}{\|\mathbf{d}\|}, \qquad \mathbf{p}_v^* = \arg\min_{\mathbf{p}_v \in v} d(\mathbf{p}_v, \mathbf{r})$$ A fixed number $s$ of points is retained per intersected voxel; voxels not intersected by any ray are discarded. The surviving set $\mathbf{p}_s^*$ represents only surface-visible points. This mechanism is implemented with a vectorized DDA that runs all rays in parallel via compact offset/count hash tables, enabling efficient large-scale use.

Point Encoding. Each occupied voxel (with its subset of retained surface points) is encoded as a geometry token. A sinusoidal positional encoding $\gamma : \mathbb{R}^3 \to \mathbb{R}^{3L}$ is applied to the point coordinates. The encoded features are concatenated with the Plücker ray embedding $\mathbf{f}_{\text{ray}} \in \mathbb{R}^6$ from the anchor view that cast the ray, then projected through a linear layer followed by layer normalization: $$\mathbf{T}_{\text{geometry}} = \operatorname{LN}\!\left(\operatorname{Linear}\!\left([\gamma(\mathbf{p}_s^*);\; \mathbf{f}_{\text{ray}}]\right)\right) \in \mathbb{R}^{M \times C}$$ where $M$ is the number of occupied voxels. This produces a unified geometry token that jointly encodes 3D position and the viewing direction from which that surface point was observed.

3DGS Prediction via Point-Image Transformer

Point-Image Transformer Architecture. The core of PointSplat is a transformer that takes the appearance tokens $\mathbf{T}_{\text{appearance}}$ and geometry tokens $\mathbf{T}_{\text{geometry}}$ as joint input and outputs fused Gaussian tokens $\mathbf{T}_{\text{Gaussians}}$: $$\mathbf{T}_{\text{Gaussians}} = \operatorname{Network}(\mathbf{T}_{\text{appearance}},\; \mathbf{T}_{\text{geometry}})$$

The transformer comprises 4 blocks, each containing three types of attention layers arranged in alternating fashion:

• Global Attention: Full self-attention over all tokens (both appearance and geometry) to capture holistic scene context.
• Point-wise Attention: Full self-attention restricted to geometry (point) tokens to enhance local geometric coherence.
• Image-wise Attention: Self-attention within each view's tokens (frame-wise) followed by cross-view attention across views, enabling multi-view feature aggregation.

Each attention module uses 16 heads with RMS normalization applied to query and key vectors (QK-Norm) for training stability. Layer normalization precedes each attention and MLP layer, with residual connections after each block. MLPs use two hidden layers with GELU activations. The total model has approximately 190M trainable parameters. This alternating attention scheme is markedly more efficient than full self-attention over all tokens: ablation shows it achieves 27.18/0.891/0.071 (PSNR/SSIM/LPIPS) at 0.4 s inference vs. 27.09/0.887/0.076 at 1.1 s for full self-attention.

Gaussian Parameter Decoding. Each of the $M$ geometry tokens is decoded into $K = 16$ Gaussian primitives. The decoding maps each fused token $g_i$ to a tuple of Gaussian attributes: $$\{g_i\}_{i=1}^M \;\to\; \left\{\left\{(o_i^k,\; c_i^k,\; s_i^k,\; \alpha_i^k,\; r_i^k)\right\}_{k=1}^K\right\}_{i=1}^M$$ where $o$ is a position offset, $c$ is a color (spherical harmonics coefficients), $s$ is scale, $\alpha$ is opacity, and $r$ is a rotation quaternion. Each property $v \in \{o, c, s, \alpha, r\}$ is predicted by a dedicated linear head with a property-specific activation function: $$v = f_v\!\left(\operatorname{Linear}_v(\mathbf{T}_{\text{Gaussians}})\right) \in \mathbb{R}^{K \times D_v}$$ Final Gaussian centers are obtained by adding the predicted offset to the anchor point positions: $$\mathbf{p}_{\text{final}} = \mathbf{p}_s^* + o$$ This offset regression design makes PointSplat robust to moderate visual-hull inaccuracies: even when the hull is noisy (e.g., under sparse-view input), the network learns to correct geometric errors.

Training Objective

The entire pipeline is trained end-to-end from RGB supervision only — no depth, normal, or other geometric priors are used. The training loss combines a pixel-wise L1 photometric term and a learned perceptual image patch similarity (LPIPS) term: $$\mathcal{L} = \lambda_{\text{L1}}\,\mathcal{L}_{\text{L1}} + \lambda_{\text{LPIPS}}\,\mathcal{L}_{\text{LPIPS}}$$ Both weights $\lambda_{\text{L1}}$ and $\lambda_{\text{LPIPS}}$ are set to 1.0. The L1 term enforces per-pixel photometric consistency; LPIPS encourages perceptual sharpness and structural fidelity.

Implementation Details

Scenes are normalized to a $2 \times 2 \times 2$ unit cube. The appearance branch uses $4 \times 4$ pixel patches. The geometry branch uses voxels of size 0.005. The Point-Image Transformer has 4 blocks and a hidden size of 1024. Each geometry token is decoded into $K = 16$ Gaussians. QK-Norm is applied to all attention layers. The learning rate is initialized at $2 \times 10^{-4}$ with a linear warmup over the first 10% of training. FlashAttention, gradient checkpointing, and mixed precision are used for training efficiency. Training runs for 300k iterations with a total batch size of 32 on NVIDIA H20 GPUs. All reported inference times are measured on a single NVIDIA A6000 (48 GB).

Experiments

Datasets and Metrics

PointSplat is trained on DNA-Rendering (~1,000 sequences for training, 16 for evaluation). Evaluation is also performed on:

• ActorsHQ — 12 dynamic human sequences; used for zero-shot generalization.
• PKU-DyMVHumans — multi-human dynamic scenes.
• THuman2.0 and RenderPeople — synthetic benchmarks with ground-truth geometry.

Metrics are PSNR, SSIM, and LPIPS computed on foreground bounding-box regions, following established protocols for human novel-view synthesis. Efficiency metrics include the total number of Gaussians (GS-Num) and per-frame inference time in seconds.

Comparison with State-of-the-Art (512×512, 8 views)

All trainable baselines are trained or fine-tuned on DNA-Rendering with a matched optimization budget. Evaluated at 512×512 resolution with 8 input views.

PointSplat achieves the best overall performance on both datasets while using only ~33% of the Gaussians of view-centric methods like DepthSplat (71k vs. 212k). GPS-Gaussian's GS count scales linearly with the number of target views ($N$), making it increasingly expensive for arbitrary viewpoint rendering. LVSM also scales with $N$ views and produces over-smoothed, mosaic-like outputs. GS-LRM and DepthSplat exhibit floating artifacts from inter-view redundancy.

Multi-Human Scenes (DyMVHumans)

High-Resolution Evaluation (1024×1024, Zero-Shot)

Models trained at 512×512 are evaluated zero-shot at 1024×1024 on the DNA-Rendering test set. LVSM cannot handle this resolution due to memory constraints.

PointSplat is the best feed-forward method at 1024×1024 and achieves comparable quality to Diffuman4D — a generative method that uses SMPL priors, generates 40 extra views, and then optimizes 3DGS on all 48 views — while being much faster. It also approaches dense-view (44-view) optimization-based 4DGS quality at a fraction of the cost.

Robustness to View Count (Zero-Shot)

PointSplat consistently outperforms all baselines across both 4-camera and 16-camera configurations, even though it was trained on 8-camera setups. GPS-Gaussian degrades sharply with fewer views due to its per-pair binocular design. This robustness stems directly from the object-centric 3D prediction: the visual hull scales naturally with the amount of available input, and the point cloud representation decouples reconstruction quality from a fixed number of views.

Dense Input View Comparison (32 Views, Zero-Shot)

Even with 32 input views (4× more than training), PointSplat maintains only ~76k Gaussians and achieves substantially higher quality than post-processing-based compression methods. Long-LRM's opacity-based pruning creates ambiguity in learning per-view opacity, leading to uniformly low-opacity (transparent) results. AnySplat's geometric inaccuracies from its VGGT depth backbone cause ghosting artifacts. Depth Anything 3's confidence pruning fails to capture high-frequency details.

Synthetic Dataset Results

On synthetic benchmarks (evaluated over full images following RoGSplat's protocol), PointSplat substantially outperforms all baselines while using fewer Gaussians than comparable methods. With 8 views, PointSplat achieves 34.30 dB PSNR on THuman2.0, exceeding GPS-Gaussian (31.08 dB) by over 3 dB.

Ablation Studies

Module-Level Ablations

Removing voxel encoding (point encoding module) increases Gaussian count by 2.25× and degrades all metrics, as the model loses the ability to ground predictions in explicit 3D geometry. Removing ray casting increases Gaussian count by 3.15× (from 71k to 224k), showing that ray casting is the primary mechanism for removing interior redundant points. Both removals also increase inference time, confirming efficiency and quality are coupled.

Transformer Architecture

The alternating attention design achieves slightly better quality than full self-attention while running 2.75× faster, demonstrating that the structured separation of global, point-wise, and image-wise attention is both effective and efficient.

Geometry Proxy Type

The visual hull geometry proxy consistently outperforms alternatives. VGGT-predicted depth provides some geometric structure but is noisier and view-inconsistent. Coarser proxies (bounding box, frustum sampling) include many non-surface points and degrade performance substantially.

Voxel Size

Voxel size 0.0050 achieves the best efficiency–quality trade-off: it reduces Gaussians by 3.3× relative to 0.0025 with negligible quality loss. Very fine voxels (0.0025, 0.0030) offer marginal quality gains at the cost of much higher Gaussian count and inference time. Very coarse voxels (0.0100) significantly degrade quality.

Mask Robustness

The model degrades gracefully under mask corruption: at a 10% random foreground-pixel drop rate, PSNR decreases by only ~2.2 dB while maintaining reasonable visual quality, demonstrating practical robustness to imperfect segmentation.

Camera Setup Robustness

Although trained with farthest-view sampling (FVS) on DNA-Rendering, PointSplat generalizes to randomly sampled views and uniformly distributed camera configurations without retraining, reflecting the strong view-agnostic properties of object-centric 3D prediction.

Contributions Summary

• Human-centric 3DGS prediction: PointSplat is the first feed-forward approach to directly infer a compact set of 3D Gaussian primitives from a point cloud in 3D space, eliminating the inter-view redundancy inherent in view-centric methods. This results in ~33% of the Gaussian count of comparable view-centric methods.
• Point-Image Transformer with ray casting: A novel transformer architecture that combines three types of attention (global, point-wise, image-wise) with an explicit ray-casting mechanism. Ray casting simultaneously prunes interior points and establishes 2D–3D correspondences that enrich point-level embeddings for cross-modal interaction between image and point features.
• State-of-the-art performance and robustness: PointSplat outperforms all feed-forward baselines on DNA-Rendering, ActorsHQ, THuman2.0, and RenderPeople benchmarks, and shows strong zero-shot generalization to varying view counts (4–32), image resolutions (512 to 1024), and camera configurations.

Limitations and Future Work

PointSplat has two main limitations acknowledged by the authors:

• Unbounded scenes: The current formulation normalizes scenes to a fixed cube, and extending the method to large unbounded environments is non-trivial due to memory constraints on the voxel grid and visual hull construction.
• Temporal consistency (4D): Extending PointSplat to produce temporally consistent and compact 4D representations remains an open challenge. Each frame is currently reconstructed independently.
• Severe self-occlusion: When large surface regions are entirely invisible across all input views, the reconstructed areas become blurry, as no image evidence is available to guide the Gaussian prediction in those regions.

The authors suggest that more memory-efficient voxel architectures and 3D/4D representations could address these limitations in future work.