EgoRelight

EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering

EgoRelight captures full-body motion and photoreal avatars with relightable appearance and HDR lighting from a single head-mounted display. It integrates egocentric stereo cameras and neural rendering to enable immersive telepresence with realistic motion and lighting, without studio setups at inference.

• avatar
• full-body
• 3d-avatar
• gaussian-splatting

Authors: Jianchun Chen, Yinda Zhang, Rohit Pandey, Thabo Beeler, Marc Habermann, Christian Theobalt

Categories: cs.CV

Published 2026-05-27 · Updated 2026-05-27

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Mixed Reality (MR) headsets promise a future of immersive telepresence where virtual humans blend indistinguishably into real or virtual surroundings. Achieving this vision requires a method for capturing a user's motion, estimating appearance under novel lighting, and understanding the environment - all from the constrained viewpoint of a head-mounted display (HMD). Existing approaches treat these as isolated problems: they either focus on driving avatars with baked-in lighting or rely on studio setups for relighting. In this paper, we present EgoRelight, a holistic framework for egocentric telepresence that simultaneously captures full-body human performance, synthesizes photorealistic and relightable appearance, and estimates high dynamic range (HDR) environment maps from a single HMD. First, to ensure motion and surface reconstruction, we propose an egocentric perception module that leverages stereo down-facing cameras to extract dense depth maps, which serve as geometric control signals to drive a mesh-based avatar. Second, we introduce a novel neural appearance model that learns to synthesize view-dependent specular and view-independent diffuse shading separately. By employing a specialized ray-sampling strategy, our model generalizes to unseen illumination without relying on restrictive analytical BRDF priors. Third, we enable seamless avatar integration into the physical world via a test-time inverse rendering process, which recovers an HDR environment map by matching the pre-trained avatar's appearance to live egocentric camera observations. We demonstrate our system through a social telepresence application, where remote users are coherently relit according to their physical environment. Extensive experiments show that our components and the integrated system significantly outperform state-of-the-art baselines in geometric accuracy and rendering as well as relighting fidelity.

---

1. Problem Setting and Main Idea

EgoRelight addresses a tightly coupled mixed-reality problem: from a single head-mounted display (HMD), it aims to recover (i) a person's full-body motion and deforming surface geometry, (ii) a photorealistic avatar whose appearance can be relit under novel illumination, and (iii) an HDR environment map for the surrounding scene. The paper argues that prior work typically solves these pieces in isolation: egocentric avatar systems usually bake in lighting, while relighting methods often rely on studio capture and do not operate from egocentric sensors.

The proposed system is a person-specific pipeline with two sensing modes: down-facing stereo cameras are used to capture the wearer's body, pose, and surface geometry; front-facing cameras are used to scan the environment for lighting estimation. The core claim is that these components can be learned and combined into a single framework that supports telepresence with coherent relighting.

At a high level, the pipeline has three stages. First, an egocentric perception module estimates 3D body joints and dense depth from stereo down-facing views, then converts them into motion and geometry signals for mesh animation. Second, a relightable appearance model learns to separate diffuse and specular transport for a person-specific avatar represented with 3D Gaussians in the UV domain. Third, at test time, an inverse-rendering procedure uses the relightable avatar itself as a calibration target to recover an HDR environment map from front-facing egocentric imagery.

The authors claim the paper is the first to holistically combine egocentric full-body capture, photoreal relightable rendering, and environment lighting recovery using only an HMD, without requiring a studio rig at inference time.

• Egocentric capture: stereo down-facing cameras provide pose and depth cues for body motion capture.
• Relightable appearance: diffuse and specular transport are modeled separately, with a learned ray-sampling strategy for specular effects.
• Environment estimation: front-facing cameras plus inverse rendering are used to estimate a color-calibrated HDR environment map.

2. Data Acquisition and Training Supervision

The paper introduces a new capture protocol built around a lightstage. The setup includes 331 individually controllable RGB light sources and 40 calibrated 4K HDR cameras; 37 cameras are used for training and 3 are held out for testing. The HMD is based on a Meta Quest 3, augmented with a pair of stereo down-facing cameras and a pair of stereo front-facing cameras. ArUco markers are attached to the headset to support accurate head tracking and coordinate alignment between egocentric and studio cameras.

The capture procedure is deliberately split into multiple sequences. One sequence is used for hand-eye calibration. Another is used for egocentric perception training under neutral white light, which is important because it provides reliable supervision for pose and depth without confounding illumination. A separate relighting training sequence alternates flat-lit and HDR-lit frames; the HMD is removed in that sequence to capture facial details. Finally, the evaluation sequences are recorded under unseen lighting, with the wearer again using the HMD.

Across four subjects, the dataset covers diverse outfits and skin tones. The supplement states that two subjects are also used for in-the-wild testing in indoor and outdoor environments, demonstrating that the HDR capture pipeline is not limited to the lightstage domain.

3. Human Representation: Mesh Geometry plus UV-Space 3D Gaussians

The avatar representation is explicitly two-layered. The geometry layer is a person-specific deformable mesh model based on a template mesh, a skeleton, and skinning weights. The appearance layer adds a dense field of 3D Gaussian primitives parameterized in the mesh's UV space. This choice allows the system to exploit the geometric structure of the body while still producing high-frequency photorealistic effects that are difficult to capture with a purely mesh-based texture model.

The mesh deformation is expressed as

$$ \mathbf{V} = \mathcal{W}(\mathcal{E}(\bar{\mathbf{V}}, \boldsymbol{\alpha}, \boldsymbol{\beta}, \mathbf{o}), \mathcal{K}(\boldsymbol{\theta}, \mathbf{B}), \mathbf{W}) $$

where $\bar{\mathbf{V}}$ is the canonical template mesh, $\mathcal{E}$ is embedded-graph deformation, $\mathcal{K}$ is forward kinematics, $\mathcal{W}$ is dual quaternion skinning, $\boldsymbol{\theta}$ is the pose, $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are embedded graph control parameters, and $\mathbf{o}$ is the per-vertex offset.

For appearance, each UV texel is associated with a Gaussian primitive carrying offset, color, rotation, scaling, and opacity. The Gaussians are rasterized in image space and alpha-composited along the ray. Importantly, unlike standard 3D Gaussian splatting, the paper does not rely on spherical harmonics for color. Instead, it predicts view-dependent color directly, which is better suited to relighting and high-frequency shading.

4. Egocentric Perception and Depth-Conditioned Geometry Capture

The egocentric perception module produces the driving signals used to animate the avatar. It has three pieces: a pose estimator, an inverse kinematics solver, and a depth estimator. Together they recover sparse joints, temporally consistent full-body motion, and dense surface depth from the down-facing stereo cameras.

4.1 Pose Estimation

The pose estimator adapts FRAME as the backbone and is fine-tuned per identity using ground-truth keypoints from a markerless motion-capture system. The network takes the left and right egocentric images plus head pose as input and predicts $57 \times 3$ 3D keypoints, including body and hand joints. To better support hand tracking, the final layer is modified so that hands and body are regressed jointly, with reweighted losses for the hand joints.

A practical detail that matters in the paper is illumination robustness: although training data for pose estimation is captured under uniform lighting, the input images are heavily augmented with brightness, contrast, and hue jitter so the pose detector generalizes to novel illuminations and in-the-wild sequences.

4.2 Inverse Kinematics

Given 3D keypoints, the method performs sequence-level inverse kinematics to obtain a smooth motion trajectory. The optimization solves for global 6D pose, body motion, and hand motion by minimizing a sum of data, temporal, degree-of-freedom limit, and regularization terms:

$$ \arg \min_{\{\boldsymbol{\theta}\}_1^T} E_{\mathrm{Data}} + E_{\mathrm{Temporal}} + E_{\mathrm{DoFLimit}} + E_{\mathrm{Reg}}. $$

A notable design choice is to constrain hand motion with a PCA subspace of 6 degrees of freedom per hand, instead of optimizing the full higher-dimensional hand pose directly. The temporal term is applied in the PCA space, which helps stabilize noisy hand estimates under self-occlusion and fast motion.

The ablation confirms that both color augmentation and hand-specific IK are important. Without augmentation, pose quality degrades sharply under the relit test conditions. Without hand IK, motion remains plausible but is noticeably worse for hands. The full model improves both full-body and hand accuracy.

4.3 Depth Estimation and Surface Encoding

The depth estimator begins from DepthAnythingV2 and is fine-tuned to predict metric depth for each stereo view. The network is supervised by metric depth generated from multi-view implicit reconstruction with NeuS2. At inference time, the two depth maps are unprojected using camera intrinsics and the hand-eye calibration, producing a point cloud in world space. Surface normals are recovered from this point cloud with a nearest-neighbor KD-tree search.

This depth signal is crucial because stereo egocentric views provide limited overlap, especially under extreme head poses, and the visible clothing often lacks strong correspondence cues. The paper positions the depth predictor as an off-the-shelf relative depth prior adapted to metric egocentric capture.

4.4 Depth-Conditioned Animatable Avatar

Instead of performing expensive test-time mesh registration against depth, the paper proposes a learned animation network, AnimationNet, that conditions on a UV-rasterized representation of the mesh. The key supervision signal is the point-to-surface distance from the unprojected depth point cloud to the template mesh, unwarped into canonical space via inverse skinning. This behaves like a learned, robust proxy for a point-to-surface loss and filters out correspondences with large normal or distance discrepancies.

The inputs include temporal mesh normals and positions plus the encoded depth guidance. These are rasterized into a UV map and fed to a U-Net with three downsampling and upsampling stages. The network predicts the embedded-graph rotation and translation parameters and the per-vertex offsets, which are then plugged back into the deformation equation to produce the final posed mesh.

Training of AnimationNet is done purely with 3D supervision: Chamfer losses against NeuS2 reconstructions, as well as ARAP, spatial, and isometry terms. The supplement notes that direct Chamfer supervision can produce artifacts around self-contact because NeuS2 reconstructs only the outer surface, so correspondences with large normal disparities are filtered out. The model is trained for 720K steps.

For relighting-stage training, where paired egocentric depth inputs are not available, the authors train a separate depth-agnostic student model and fine-tune it on interleaved flat-lit frames so that geometry remains stable in the more challenging relit sequences.

The geometry experiment shows that the proposed depth-conditioned model is the best feed-forward method among those compared and comes close to the optimization-heavy EgoAvatar variant, but at far better speed. The strongest gains are on surfaces visible to the egocentric cameras, especially the frontal and upper-body regions.

5. Relightable Avatar Appearance Modeling

The relighting component takes the tracked mesh sequence and predicts a photorealistic, relightable avatar. The authors explicitly separate geometry enhancement from material estimation to reduce the geometry-material ambiguity common in inverse rendering. The relightable representation consists of three learned modules: GeoLiftingNet, DiffuseNet, and SpecularNet.

5.1 GeoLiftingNet: Geometry Lifting in UV Space

GeoLiftingNet takes a stack of temporal mesh normals and predicts two UV-space outputs: a refined normal map and a flat-lit texture map that approximates albedo. The motivation is that the tracked mesh alone is not detailed enough to capture wrinkles and other high-frequency geometric effects, while also being susceptible to tracking drift. Predicting flat-lit appearance gives the model slack so that shading errors are not incorrectly absorbed by material parameters.

The supervision blends three signals: an $L_1$ loss on flat-lit RGB reconstruction, an $L_1$ loss to screen-space normals from Sapiens, and a perceptual loss between rendered normals and normals from a NeuS2 mesh reconstruction. This is designed to balance 3D consistency with perceptual detail. The network is trained first, for 400K iterations, before the rest of the relighting modules.

5.2 Diffuse and Specular Factorization

The paper models relighting with a rendering-equation decomposition into diffuse and specular transport. For each surface point, it computes physically informed features from the lightstage environment: a diffuse shadow-like response, a diffuse irradiance map, and a ray-sampled specular descriptor. The light visibility is traced with OptiX so that self-occlusion by the body is accounted for.

The diffuse term is view independent. The paper computes a diffuse map for flat-lit frames and a lighting-weighted map for HDR-lit frames, then feeds these together with albedo and normals into DiffuseNet. DiffuseNet predicts Gaussian offsets and rendering parameters as well as a diffuse color. This module is important because, for human avatars, much of the visible shading energy is diffuse and therefore structurally tied to the surface and illumination rather than to view-dependent effects.

The specular term is harder: instead of applying a closed-form BRDF such as Disney or a precomputed radiance transfer approximation, the paper samples the most relevant light directions for each primitive using a Blinn-Phong-inspired importance score, then encodes each ray with a 6D feature built from angular relationships among the normal, incoming direction, outgoing direction, and half-vector. SpecularNet processes these features with cross-attention so that the ray set is permutation invariant and the network can reason about multiple incident lights efficiently.

The cross-attention is implemented as a one-to-many attention block, which reduces redundant computation while still allowing the model to incorporate multiple rays. The final activation is softplus so that specular shading remains non-negative.

The networks are trained jointly on HDR-lit frames with a combination of photometric and perceptual losses:

$$ \mathcal{L}_{\mathrm{relight}} = \mathcal{L}_{1}(\mathbf{I}_{\mathrm{render}}, \mathbf{I}_{\mathrm{GT}}) + \mathcal{L}_{\mathrm{SSIM}}(\mathbf{I}_{\mathrm{render}}, \mathbf{I}_{\mathrm{GT}}) + \mathcal{L}_{\mathrm{IDMRF}}(\mathbf{I}_{\mathrm{render}}, \mathbf{I}_{\mathrm{GT}}) + \mathcal{L}_{\mathrm{Reg}} + \mathcal{L}_{1}(\mathbf{I}_{\mathrm{diff}}, \mathbf{I}_{\mathrm{GT}}). $$

The extra loss on the diffuse-only rendering discourages the specular branch from explaining away view-independent color. A regularizer constrains Gaussian offsets and scales, which is especially important because the HDR-lit frames do not have reliable background matting.

To improve robustness to out-of-distribution illuminations such as OLAT or high-contrast environment maps, the supplement applies a RAW-space data augmentation: after 100K iterations, with 40% probability, both the incident light and the image intensity are jointly rescaled by a factor sampled uniformly from $[0,1]$. This is one of the key tricks behind the method's qualitative robustness.

The ablation study shows three especially important findings. First, removing DiffuseNet causes a catastrophic drop in all metrics, indicating that diffuse transport is the backbone of the model. Second, removing GeoLiftingNet degrades fine detail and FID, confirming that the lifted normals and flat-lit texture help recover wrinkles and other high-frequency structure. Third, the specular modeling choices matter: the learned cross-attention design outperforms a specular-map baseline, and a moderate number of sampled rays gives the best perceptual balance. The paper also compares against an explicit DisneyBRDF parameterization; this baseline introduces noticeable color shifts and unrealistic metallic highlights, especially in hair.

The paper stresses that the model generalizes not only to standard relighting evaluation but also to challenging target illuminations such as OLAT and rotating high-contrast environment maps. In those cases, inverse-rendering-based baselines tend to become noisy, while the diffusion-based image relighting baseline often misses correct color tone. The authors highlight that their own model preserves sharp self-shadowing and highlights more convincingly.

5.3 Main Relighting Results

The main comparison is against two groups of baselines: physics-based relightable avatars and an image-based relighting pipeline. For fairness, the compared relightable-avatar baselines are given the ground-truth environment map and are only trained to learn intrinsic decomposition on relit frames. For EgoAvatar combined with NeuralGaffer, the motion and geometry are supplied by the proposed egocentric capture module so that the comparison isolates relighting quality. Quantitative evaluation uses PSNR, SSIM, LPIPS, and FID on novel testing sequences with novel view and novel illumination; the head region is excluded to avoid inconsistency between HMD and non-HMD rendering.

The proposed method is consistently the best or near-best across all subjects in PSNR, LPIPS, and FID, and achieves the best SSIM in most cases. Relative to the image-based baseline, the method preserves facial detail and produces more faithful shading. Relative to MeshAvatar and Relighting4D, it avoids the severe noise, color shift, and loss of fine structure that appear in the baselines. The paper's qualitative discussion emphasizes that the method recovers both photorealistic texture and plausible lighting-dependent color changes.

6. Affordable HDR Environment Map Capture from the HMD

The lighting-estimation component begins with an approximately 10-second in-place scan using the front-facing cameras of the HMD. Twenty images are sampled from the scan, used to reconstruct a 3D scene, and then unprojected into an axis-aligned LDR panorama. A diffusion-based inpainting step fills the unobserved regions of the panorama. This yields a low-dynamic-range environment estimate, but it still does not match the color space used to train the relightable avatar.

To bridge that gap, the paper performs inverse rendering and color calibration using the relightable avatar as a calibration target. The LDR environment map is converted to HDR via a second-order color-correction model with matrix parameters $\mathbf{A}$ and gamma parameters $\boldsymbol{\gamma}$:

$$ \mathbf{E}_h = \mathcal{C}(\mathbf{E}_l; \mathbf{A}, \boldsymbol{\gamma}). $$

The optimized objective compares the rendered avatar in the egocentric down-facing view against the actual egocentric image, after inverse warping with optical flow and masking by body-part segmentation. Part segmentation is important because the upper body dominates the egocentric view, and without it the optimization overfits the most visible skin region.

The environment map regularizer constrains pixel values to the $[0,1]$ range. In the supplement, the optimization is reported as a single 1,000-step Adam pass per scene.

The environment-map experiment compares against image harmonization baselines such as Photoshop harmonization and IC-Light. The paper argues that these methods are not designed to recover true scene illumination: Photoshop harmonization produces inconsistent color tone across a sequence, and IC-Light can destroy facial details. The proposed pipeline is more plausible because it uses actual 360-degree observations and then calibrates them through inverse rendering on the avatar.

The ablation analysis also shows why the calibration stage matters. Directly using an LDR panorama causes incorrect color tone, while removing the panorama prior can overfit the egocentric images and produce an incorrect greenish face. The part-segmentation mask improves brightness consistency and reduces overfitting to the upper body.

7. Training Setup, Baselines, and Runtime

The experimental section uses four subjects in total. All quantitative results are reported on the testing sequences. The authors also record two in-the-wild sequences with stereo down-facing cameras at 720p, stereo front-facing cameras at 1080p, and headset SLAM pose. Because the headset SLAM is unstable under the bright flickering lightstage conditions, head pose inside the dome is measured using triangulated ArUco markers instead.

The comparison set is deliberately split into two categories: relightable-avatar baselines and a hybrid egocentric-avatar-plus-image-relighting baseline. Specifically, the paper compares with Relighting4D, MeshAvatar, EgoAvatar, and EgoAvatar combined with NeuralGaffer. For relightable-avatar baselines, the authors supply the ground-truth environment map to isolate the relighting component. For the hybrid baseline, they supply the motion and geometry predicted by the proposed egocentric module.

The evaluation metrics are PSNR, SSIM, LPIPS, and FID. FID is included because the test sequences are novel and not all fine details can be reconstructed deterministically, so distributional similarity becomes informative in addition to per-pixel fidelity.

The runtime breakdown makes the main bottleneck clear: the pipeline is not yet real-time. In particular, the inverse kinematics stage is an offline sequence-level optimization, and the relightable appearance model is also expensive enough to keep the system above a real-time budget. The paper explicitly notes that the iterative IK solver is the major bottleneck and suggests more efficient second-order solvers as future work.

8. Limitations and Failure Modes

The paper is unusually explicit about limitations. First, the method is not optimized for real-time use. Second, the HMD cameras operate with fixed exposure, which makes very bright or very dark conditions difficult. This also creates color-synchronization issues across the different egocentric cameras. Third, outdoor relighting under strong sunlight remains limited by the capture pipeline; in the experiments, the authors clip test environment-map intensity to 1.0 when the real sunlight reaches roughly $1.6 \times 10^5$, which softens contrast and shadows.

There are also quality limitations: the system still shows limited facial expressiveness and occasional foot-ground contact artifacts. The authors suggest that these could be improved by adding dedicated face sensing modules and physics-based constraints. Finally, the method depends heavily on high-quality person-specific lightstage capture, which limits immediate public applicability and motivates future work on universal relightable avatars built from generative priors.

9. Bottom Line

EgoRelight combines egocentric motion capture, depth-conditioned mesh reconstruction, learned diffuse/specular relighting, and test-time HDR environment map recovery into one pipeline. Its main technical contributions are the depth-conditioned animatable avatar, the UV-space geometry lifting and relightable Gaussian appearance model, and the avatar-driven environment-map calibration strategy. The experiments show clear gains in reconstruction quality, relighting fidelity, and robustness to challenging illumination, while the reported limitations mostly concern runtime and reliance on person-specific lightstage capture.