ER-NeRF

Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis

ER-NeRF uses region-aware Neural Radiance Fields to efficiently generate high-fidelity talking portraits. It models spatial region contributions explicitly for fast, real-time rendering with precise audio-lip sync and small model size, outperforming prior work in efficiency and visual quality.

• talking-head
• audio-driven
• lip-sync
• nerf
• realtime
• head-pose
• portrait-animation

Demos

The demos showcase ER-NeRF's ability to synthesize high-fidelity 3D talking portraits with superior lip-sync accuracy, realistic details, and efficient training from brief video clips. Watch for fine surface reconstruction, the effectiveness of the region attention module in linking audio cues to spatial regions, and overall rendering quality compared to state-of-the-art baselines. The performance table highlights ER-NeRF's leading speed, accuracy, and model compactness.

Authors: Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, Lin Gu

Categories: cs.CV

Comment: Accepted by ICCV 2023. Project page: https://fictionarry.github.io/ER-NeRF/

Published 2023-07-18 · Updated 2023-08-24

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub1,258 stars17 watchers144 forksMIT

Abstract

This paper presents ER-NeRF, a novel conditional Neural Radiance Fields (NeRF) based architecture for talking portrait synthesis that can concurrently achieve fast convergence, real-time rendering, and state-of-the-art performance with small model size. Our idea is to explicitly exploit the unequal contribution of spatial regions to guide talking portrait modeling. Specifically, to improve the accuracy of dynamic head reconstruction, a compact and expressive NeRF-based Tri-Plane Hash Representation is introduced by pruning empty spatial regions with three planar hash encoders. For speech audio, we propose a Region Attention Module to generate region-aware condition feature via an attention mechanism. Different from existing methods that utilize an MLP-based encoder to learn the cross-modal relation implicitly, the attention mechanism builds an explicit connection between audio features and spatial regions to capture the priors of local motions. Moreover, a direct and fast Adaptive Pose Encoding is introduced to optimize the head-torso separation problem by mapping the complex transformation of the head pose into spatial coordinates. Extensive experiments demonstrate that our method renders better high-fidelity and audio-lips synchronized talking portrait videos, with realistic details and high efficiency compared to previous methods.

---

Introduction

ER-NeRF is a person-specific audio-driven talking portrait synthesis method built on Neural Radiance Fields (NeRF). The paper targets the usual tension in this area: prior NeRF-based talking-head systems can produce high-fidelity renderings, but they are often slow, memory-heavy, and difficult to optimize; meanwhile, efficient grid-based methods improve speed but can still struggle with dynamic facial details, audio-motion coupling, and head-torso consistency.

The central design idea is to exploit the fact that different spatial regions do not contribute equally. The authors use this observation in three places: (1) a compact head representation that prunes empty space and reduces hash collisions, (2) an explicit region-aware attention mechanism to connect audio and spatial regions, and (3) a lightweight adaptive pose encoding to stabilize torso rendering and reduce head-torso separation artifacts.

The paper positions itself against prior NeRF-based talking portrait systems such as AD-NeRF, SSP-NeRF, and RAD-NeRF, and also compares against non-NeRF baselines like Wav2Lip, PC-AVS, NVP, LSP, and SynObama. The key claim is that ER-NeRF can deliver real-time rendering, fast convergence, and small model size while improving lip synchronization and perceptual quality.

• Problem: audio-driven portrait synthesis with realistic motion, stable torso, and fast inference.
• Main hypothesis: spatial regions have unequal importance, and modeling that asymmetry explicitly improves both efficiency and fidelity.
• Core outputs: an explicit tri-plane hash head representation, region attention for audio/eye control, and adaptive pose encoding for torso rendering.

Method Overview

The overall pipeline splits the portrait into a head NeRF and a torso NeRF. For the head, a tri-plane hash encoder maps 3D coordinates to geometry features, then audio and eye-blinking conditions are reweighted by the region attention module, and an MLP decoder predicts color and density. For the torso, the method uses adaptive pose encoding to convert head pose into a set of normalized 2D coordinates that condition a 2D deformable neural field.

Rendering and conditioning formulation

As in standard NeRF, a ray is integrated along depth to form a pixel color. The paper writes this as

$$\hat{C}(\mathbf{r}) = \int_{t_n}^{t_f} \sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,T(t)\,dt,$$

with transmittance

$$T(t) = \exp\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s))\,ds\right).$$

For audio-driven portrait synthesis, the basic conditional field is written as $\mathcal{F}^A:(\mathbf{x}, \mathbf{d}, \mathbf{a}; \mathcal{H}) \rightarrow (\mathbf{c}, \sigma)$, where $\mathbf{x}$ is 3D position, $\mathbf{d}$ is view direction, $\mathbf{a}$ is the audio feature, and $\mathcal{H}$ is the hash encoder.

The paper follows the usual person-specific setting: a few minutes of single-subject video captured from a fixed frontal camera are used for training, head poses are estimated by a 3DMM model, audio features are extracted from a pretrained DeepSpeech model, and semantic parsing is used to separate head, torso, and background. Head and torso are trained and rendered separately for acceleration.

Tri-Plane Hash Representation

The head representation is the first major contribution. The paper argues that a full 3D hash grid treats all space equally, even though large parts of the portrait volume are empty. This creates unnecessary hash collisions and makes the decoder spend capacity on resolving conflict rather than on modeling audio-driven details.

ER-NeRF factorizes the 3D space into three orthogonal 2D hash grids: $\mathcal{H}^{XY}(x,y)$, $\mathcal{H}^{YZ}(y,z)$, and $\mathcal{H}^{XZ}(x,z)$. Their outputs are concatenated into a geometry feature:

$$\mathbf{f}_{\mathbf{x}} = \mathcal{H}^{XY}(x,y) \oplus \mathcal{H}^{YZ}(y,z) \oplus \mathcal{H}^{XZ}(x,z).$$

The resulting tri-plane hash encoder is denoted $\mathcal{H}^3$, and the head implicit function becomes $\mathcal{F}^H:(\mathbf{x}, \mathbf{d}, \mathcal{D}; \mathcal{H}^3) \rightarrow (\mathbf{c}, \sigma)$, where $\mathcal{D}$ includes the dynamic conditions.

The rationale is that hash collisions then occur in lower-dimensional subspaces rather than the entire 3D volume. The supplementary material gives the authors’ complexity argument that, for a typical near-frontal viewing setup, collision cost is reduced from $O(R^2 N)$ to $O(R^2 + 2RN)$, where $R^2$ is the image resolution and $N$ is the number of samples per ray. With the paper’s typical setting $N=16$ and $R \approx 256$, this is described as an idealized $5\times$ reduction in hash collisions for the same model size.

In the implementation, each 2D hash encoder uses $L=14$ levels and feature dimension $F=1$, with multiresolution spanning 64 to 512. The density MLP has 3 layers and the color MLP has 2 layers, each with 64 hidden units.

Region Attention Module

The second major contribution is an explicit region-aware conditioning mechanism. Instead of concatenating audio to geometry and hoping an MLP learns the spatial correspondence implicitly, ER-NeRF uses the spatial feature itself to generate an attention vector that gates audio or eye features according to location.

The paper adapts an external-attention-style formulation. If the encoded spatial feature is viewed as $F$, the attention is computed as

$$A = \mathrm{ReLU}(F M_k^T), \qquad V_{out} = A M_v,$$

and then $V_{out}$ is treated as a region attention vector $\mathbf{v}$ that reweights each channel of a dynamic condition $\mathbf{q}$:

$$\mathbf{q}_{out} = \mathbf{v} \odot \mathbf{q}.$$

For audio, the paper uses $\mathbf{v}_{a,\mathbf{x}} = \mathrm{MLP}_a(\mathcal{H}^3(\mathbf{x}))$ and then forms the region-aware audio feature $\mathbf{a}_{r,\mathbf{x}} = \mathbf{v}_{a,\mathbf{x}} \odot \mathbf{a}$. For eye blinking, a scalar control is processed similarly, with a sigmoid gate producing $\mathbf{e}_{r,\mathbf{x}} = \mathbf{e} \cdot \mathrm{Sigmoid}(\mathbf{v}_{e,\mathbf{x}})$.

This means the model can emphasize audio in dynamically moving mouth or cheek regions while suppressing it in static regions where audio is mostly noise. The same logic is used for blink control: the eye branch should be active near the eyes and near zero elsewhere.

The architecture uses a 2-layer MLP with 64 hidden dimensions for the speech-audio attention branch, and a 2-layer MLP with 16 hidden dimensions for eye blinking.

Adaptive Pose Encoding and Torso-NeRF

The third design is a lightweight pose encoding for torso synthesis. Rather than conditioning the torso renderer on a raw pose matrix or full image context, the method creates a small set of trainable key points in canonical 3D space, transforms them by the inverse head pose, and projects them to 2D coordinates. These coordinates provide a clearer relation between head pose and torso layout, helping the torso NeRF learn a stable implicit pose representation.

With $N=3$ trainable points, the canonical homogeneous coordinates are transformed as $\hat{\mathbf{X}}_{keys} = \mathbf{P}^{-1} \mathbf{X}_{keys}$, then projected to normalized image-plane coordinates $\bar{\mathbf{X}}_{keys}$. The torso renderer is then a 2D deformable neural field conditioned on $\bar{\mathbf{X}}_{keys}$ and a pixel coordinate $\mathbf{x}_{pixel}$, producing torso color and alpha.

The supplementary material notes that this torso branch uses only linear transformations in the forward pass, so it improves torso stability without adding much computational burden.

The paper reports that the torso branch is trained separately, and the final system uses the same basic 2D deformable field as RAD-NeRF but conditions it with the adaptive pose encoding.

Training Objective and Optimization

ER-NeRF is trained in a two-stage coarse-to-fine procedure. The coarse stage uses the standard pixel MSE loss:

$$\mathcal{L}_{coarse} = \sum_{i \in \mathcal{I}} \| C(i) - \hat{C}(i) \|_2^2.$$

The fine stage adds patch-wise LPIPS to sharpen details:

$$\mathcal{L}_{fine} = \sum_{i \in \mathcal{P}} \| C(i) - \hat{C}(i) \|_2^2 + \lambda\,\mathrm{LPIPS}(\hat{\mathcal{P}}, \mathcal{P}).$$

The training recipe is intentionally lightweight: the head is trained for 100,000 coarse iterations and 25,000 fine iterations, the torso is trained separately for another 100,000 iterations, and each iteration samples a batch of $256^2$ rays from one image. The optimizer is AdamW, with learning rate 0.01 for hash encoders and 0.001 for other modules. Training is reported to take about 2 hours on a single RTX 3080Ti GPU.

For blink control, the authors use AU45 to parameterize eye closing. The audio is extracted from pretrained DeepSpeech, and the paper also states that the audio feature output dimension is reduced from 64 to 32 in this implementation.

Experimental Setup

The main evaluation uses four high-definition speaking video clips from publicly released datasets related to AD-NeRF, SSP-NeRF, and RAD-NeRF. Each clip is about 6,500 frames long at 25 FPS. The videos are cropped to $512 \times 512$ around a centered portrait, except the AD-NeRF clip, which is at $450 \times 450$.

The paper evaluates two scenarios:

• Head reconstruction setting: each video is split into train and test segments to measure person-specific reconstruction quality.
• Lip synchronization setting: unseen audio clips from public demos of NVP and SynObama are used to drive the methods and compare lip sync.

The metrics are PSNR, LPIPS, FID, LMD, AUE, and SyncNet confidence. The main baselines are Wav2Lip, PC-AVS, NVP, LSP, SynObama, AD-NeRF, SSP-NeRF, and RAD-NeRF. The supplement additionally compares against GeneFace and DFRF.

Quantitative results: head reconstruction

The paper notes that Wav2Lip is not directly comparable on PSNR/LPIPS in this setting because it sees a ground-truth clip as image input during self-driven evaluation. The authors also report that RAD-NeRF^† uses AU45 and an overall LPIPS finetune.

The headline result here is that ER-NeRF achieves the best LPIPS, FID, LMD, and AUE, while keeping Sync strong and reducing the model size dramatically compared with RAD-NeRF. The reported FPS is 34, and the model size is 2.51 MB, which is substantially smaller than prior NeRF-based baselines in the table.

Quantitative results: lip synchronization

In the lip-sync setting, ER-NeRF does not always minimize landmark distance, but it obtains the best Sync score among the NeRF-based methods on both test sets and generally improves over earlier end-to-end NeRF systems. The one-shot methods Wav2Lip and PC-AVS still score higher on SyncNet confidence, which the authors interpret as a reminder that these methods optimize lip synchronization very aggressively but do not reconstruct a specific person as faithfully.

The supplement extends the comparison to DFRF and GeneFace. In that additional table, ER-NeRF remains the strongest overall method in the head-reconstruction metrics and also attains the best Sync scores among NeRF-based methods in the lip-sync setting.

Qualitative results and user study

Qualitatively, the paper emphasizes two recurring advantages: better preservation of fine facial details and better head-torso alignment. In the figure below, the method is shown against Wav2Lip, PC-AVS, AD-NeRF, and RAD-NeRF, along with the ground truth. The authors report that AD-NeRF suffers from severe head-torso separation, while RAD-NeRF occasionally mismatches torso and head. ER-NeRF is presented as more robust thanks to adaptive pose encoding.

The authors also test out-of-range poses. Even when the head rotation is more extreme than what appears in the training video, the tri-plane hash representation is reported to preserve structure better than the stronger baselines.

User study results support the perceptual gains. The authors sample 28 generated clips and ask 18 participants to rate lip-sync accuracy, video realness, and image quality on a 1–5 scale. ER-NeRF scores 4.14 for lip-sync accuracy, 4.08 for image quality, and 3.86 for video realness, which is best or second-best across the three dimensions.

The supplementary material also reports a torso-specific user study over stability and image quality. On that evaluation, ER-NeRF scores 3.89 for stability and 4.00 for image quality, outperforming AD-NeRF and RAD-NeRF.

Ablation Studies

The ablation section is important because it isolates the effect of the two main ideas: representation choice and region attention. The paper compares three backbones: an MLP backbone like AD-NeRF, a pure tri-plane backbone like EG3D, and an Instant-NGP-style 3D hash grid like RAD-NeRF. It also compares concatenation with the proposed region attention mechanism.

These results support two main conclusions. First, the proposed tri-hash backbone is the best overall representation: it improves reconstruction and lip-sync compared with an MLP and with pure tri-plane or pure 3D hash-grid baselines. Second, the attention module improves all downstream metrics relative to direct concatenation, and is especially helpful for synchronization.

The paper also compares attention types. Channel-wise attention is the default choice and performs better than feature-wise attention on the final synchronization metrics, even though feature-wise attention slightly edges out PSNR.

The supplementary material adds an eye-blink ablation. Without region attention, directly concatenating AU45 leads to jitter and unnatural mouth motion; with the proposed mechanism, blink control becomes more localized and more robust.

The supplementary discussion also highlights that LPIPS fine-tuning is harder to exploit in more complex architectures such as RAD-NeRF, while the smaller ER-NeRF architecture benefits more clearly from the LPIPS stage and can reproduce high-frequency details more effectively. The paper illustrates this with validation LPIPS curves on the Obama and May datasets.

Strengths, Limitations, and Ethics

The paper’s strengths are straightforward: it combines efficient rendering, strong reconstruction quality, and explicit region-aware conditioning in a way that preserves real-time performance. It also provides a simple, computationally light torso mechanism that improves head-torso alignment relative to prior NeRF-based systems.

• Strength: compact tri-plane hash representation reduces collision burden and improves dynamic head reconstruction.
• Strength: region attention explicitly links audio and spatial regions, rather than relying on a large MLP to learn the relation implicitly.
• Strength: adaptive pose encoding improves torso stability without a deep pose network.

The paper is also explicit about limitations. First, ER-NeRF remains a per-scene training method, unlike one-shot methods such as Wav2Lip. Second, because the model is trained on a small number of frames from a single video and relies on pretrained speech features, it is weaker on out-of-domain audio such as cross-lingual speech or singing voice. Third, while the torso branch is improved, blurry regions can still appear, likely because of both uncertain motion and the representation itself.

The ethics section warns that the high realism and lip synchronization can be misused to generate deceptive content. The authors recommend sharing generated results to help improve deepfake detectors, protecting real portrait speech videos, and thinking carefully about limits and regulation around deepfake use.

Conclusion

ER-NeRF proposes a focused, efficiency-oriented answer to audio-driven talking portrait synthesis: factorize the head into tri-plane hash grids, explicitly attend to region-dependent audio and blink conditions, and encode pose in a compact form for torso rendering. Across reconstruction, synchronization, user studies, and ablations, the paper argues that these region-aware design choices yield a strong quality-efficiency tradeoff, with real-time rendering, a very small model footprint, and improved visual fidelity over prior NeRF-based talking portrait methods.