AD-NeRF
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
AD-NeRF uses audio-driven neural radiance fields to generate high-fidelity talking-head videos by conditioning directly on audio features, bypassing intermediate facial representations. It models head and torso separately, allowing natural upper-body motion with flexible pose and background editing.
Demos
These demos showcase AD-NeRF's capability for high-fidelity, audio-driven talking head synthesis that directly maps audio features to dynamic neural radiance fields. Watch for photorealistic facial expressions synchronized with diverse audio inputs across different identities, genders, and languages. Additional highlights include flexible free-viewpoint rendering with background changes and a dual-branch neural field architecture modeling head and torso separately, enabling natural movements and high-resolution outputs beyond traditional methods.
Links
Paper & demos
Code & resources
Impact
Abstract
Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a high-fidelity talking-head video corresponding to the audio signal is synthesized using volume rendering. Another advantage of our framework is that not only the head (with hair) region is synthesized as previous methods did, but also the upper body is generated via two individual neural radiance fields. Experimental results demonstrate that our novel framework can (1) produce high-fidelity and natural results, and (2) support free adjustment of audio signals, viewing directions, and background images. Code is available at https://github.com/YudongGuo/AD-NeRF.
Introduction
AD-NeRF addresses audio-driven talking-head synthesis as a cross-modal generation problem: given a short portrait video of a target person and its audio track, the goal is to synthesize high-fidelity talking-head video that remains synchronized with speech while preserving natural appearance, realistic lip motion, and plausible upper-body motion. The central claim of the paper is that this can be done more directly by conditioning a neural radiance field on audio features, rather than first translating audio into an intermediate modality such as 2D landmarks, 3D face shape, or expression coefficients.
The paper argues that intermediate representations can lose semantic information and may limit final image quality. In contrast, AD-NeRF uses semantic audio features as direct conditioning for a conditional implicit radiance field and renders images with volume rendering. The authors also explicitly model the head and torso with two separate neural radiance fields, which is intended to address the mismatch between rigid face pose and the more complex motion of the upper body. This design further enables editing capabilities such as pose manipulation and background replacement.
The method is trained on a short video sequence of a target speaker. The paper states that the recording camera and background are assumed to be static during training, and that test-time audio can come from different identities, genders, and languages. The main practical outcome emphasized by the paper is that the model can generate full-frame talking-head imagery, not just a cropped face region.
Method
Overall pipeline
The framework consists of three conceptual stages. First, audio is converted into a semantic conditioning code. Second, a conditional implicit function predicts radiance field values for a talking-head scene. Third, rendered pixels are obtained through standard NeRF-style volume rendering. At inference time, only audio and the learned scene parameters are needed to synthesize a sequence.
The paper explicitly frames AD-NeRF as a dynamic NeRF for talking heads. Unlike prior audio-driven talking-head systems that rely on explicit face priors or intermediate supervision, AD-NeRF maps audio directly to a neural scene representation. The model is implemented as multi-layer perceptrons with positional encoding, following the standard NeRF design philosophy.
Audio-conditional neural radiance field
Let the audio feature be denoted by $\mathbf{a}$, the viewing direction by $\mathbf{d}$, and a 3D point by $\mathbf{x}$. The paper defines an audio-conditional implicit function
$$\mathcal{F}_{\theta} : (\mathbf{a}, \mathbf{d}, \mathbf{x}) \rightarrow (\mathbf{c}, \sigma),$$
where $\mathbf{c}$ is RGB color and $\sigma$ is density. The model uses the same basic positional encoding and network structure as NeRF. The key difference is the extra conditioning on audio, which is intended to control facial dynamics and speech-related motion directly.
For audio encoding, the authors use DeepSpeech to extract a 29-dimensional feature for each 20 ms audio clip. To improve temporal stability, they aggregate $16$ neighboring frames into a tensor $\mathbf{a} \in \mathbb{R}^{16 \times 29}$, and then process this sequence with a temporal convolutional network. The paper notes that a self-attention-inspired temporal filter, implemented using 1D convolution layers with softmax activation, is used to smooth the audio condition.
The motivation for using the audio feature directly is twofold: it removes the need for a separate audio-to-landmark or audio-to-expression predictor, and it avoids semantic mismatch caused by information loss in a low-dimensional intermediate representation.
Volume rendering
Given a camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ with near bound $t_n$ and far bound $t_f$, the rendered color is computed by integrating predicted color and density along the ray:
$$\mathcal{C}(\mathbf{r}; \theta, \Pi, \mathbf{a}) = \int_{t_n}^{t_f} \sigma_{\theta}(\mathbf{r}(t)) \, \mathbf{c}_{\theta}(\mathbf{r}(t), \mathbf{d}) \, T(t) \, dt,$$
with transmittance
$$T(t) = \exp\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s)) \, ds\right).$$
The pose parameters are represented as $\Pi = \{R, t\}$, where $R$ is a rotation matrix and $t$ is a translation vector. Following dynamic NeRF-style modeling, the sampled points are transformed to a canonical space using the estimated face pose during head rendering. The paper also uses the standard two-stage NeRF sampling strategy: a coarse network first estimates densities, and a fine network then samples more densely in regions of high density.
Two separate NeRFs for head and torso
A key architectural choice is to decompose the portrait scene into two individual radiance fields: one for the head and one for the torso. The motivation is that the head and upper body do not move rigidly together. Applying the same rigid face transformation to the torso causes visible artifacts, especially around the neck and shoulders.
To train the two components, the paper first uses an automatic face parsing method to split each frame into three regions: static background, head, and torso. The head NeRF is trained first. In this stage, the parsed head region is treated as foreground, and the rest is background. A background image is used so that the final sample on each ray corresponds to the background color for rays passing outside the head region.
After the head NeRF is trained, its rendered output is treated as the new background, and the torso region becomes foreground. The torso NeRF is then trained separately. Importantly, the torso model does not transform sample points using the head pose. Instead, the pose $\Pi$ is appended as an additional input condition together with $\mathbf{a}$, $\mathbf{x}$, and $\mathbf{d}$. This design lets the torso model learn the residual upper-body motion implicitly rather than forcing a rigid geometric alignment that does not fit the anatomy of the scene.
At inference time, both networks receive the same audio and pose inputs. The head model renders the foreground head over a static background, and the torso model fills in the remaining upper-body region. The authors argue that the two-branch design is important for naturalness because it explicitly handles the inconsistent motion between head pose and torso movement.
Editing capabilities
Because both the audio feature and the pose coefficients are explicit inputs to the model, the paper highlights three editing modes. First, speech content can be changed by swapping the audio input. Second, viewing direction can be manipulated by adjusting pose coefficients within the range covered by the training video. Third, the background can be replaced because the method decouples foreground and background through its rendering formulation and background-image handling.
Training, data processing, and objective
The training data for each subject is a short portrait video with audio. The paper reports an average video length of $3$ to $5$ minutes at $25$ fps. The training setup assumes a static camera and static background.
Preprocessing consists of three steps. First, a face parsing method is used to label semantic regions in each frame. Second, multi-frame optical flow is used to obtain dense correspondences in near-rigid areas such as forehead, ear, and hair, and bundle adjustment is used to estimate pose parameters. The paper explicitly states that these pose estimates are effective for the face but not for the full upper body, especially the neck and shoulders. Third, a clean background image is reconstructed by removing the human region from each frame and aggregating background evidence across the sequence; missing pixels are filled with Poisson blending.
The network is trained with two main constraints. One is temporal smoothing of the audio conditioning through the 16-frame temporal filter. The other is reconstruction of the target frame with a photometric loss:
$$\mathcal{L}_{\text{photo}}(\theta) = \sum_{w=0}^{W} \sum_{h=0}^{H} \| I_r(w,h) - I_g(w,h) \|^2,$$
where $I_r$ is the rendered image and $I_g$ is the ground-truth frame. The paper does not describe an adversarial loss or a perceptual loss in the main text; the objective given in the source is the per-pixel reconstruction loss above.
Implementation details reported by the paper are: PyTorch implementation, Adam optimization with initial learning rate $0.0005$, $400\,000$ iterations for each model, and a random batch of $2048$ rays per iteration. The authors report training on an RTX 3090. For a $5$-minute video at resolution $450 \times 450$, training both NeRFs takes about $36$ hours and rendering a frame takes about $12$ seconds.
Experiments and results
The experiments compare AD-NeRF against image-based, model-based, and video-driven talking-head methods. The paper emphasizes two major evaluation goals: audio-visual synchronization and visual fidelity. It also presents qualitative evidence that the method produces full-frame portraits rather than face crops and can render natural hair, teeth, background, and upper-body context.
For image-based comparisons, the paper argues that those methods are constrained by their fixed crop size and generally produce only face-region outputs. AD-NeRF, by contrast, generates larger full-frame imagery and can better preserve the portrait setting. For model-based comparisons, the authors note that prior systems typically require much larger amounts of training data, whereas AD-NeRF is trained on a short clip of $3$ to $5$ minutes for a target identity.
The paper also compares with a video-driven method. That comparison is used mainly to show that AD-NeRF can produce visually competitive results even though it is driven by audio rather than by a source motion video.
Quantitative evaluation
The paper evaluates on two test sets: testset A from the demos of Neural Voice Puppetry and testset B from the demos of Synthesizing Obama. Metrics include SyncNet score for audio-visual synchronization, Action Unit (AU) error using OpenFace for facial-action consistency, and a user study.
The table below reproduces the reported quantitative comparison values from the paper.
| Methods | SyncNet score A | SyncNet score B | AU error A | AU error B | Pose | Full-frame | Background |
|---|---|---|---|---|---|---|---|
| Chen et al. | 6.129 | 4.388 | 2.588 | 3.475 | static | no | no |
| Wiles et al. | 4.257 | 3.976 | 3.134 | 3.127 | static | no | no |
| Vougioukas et al. | 5.865 | 6.712 | 2.156 | 2.658 | static | no | no |
| Thies et al. | 4.932 | — | 1.976 | — | copied from source | yes | no |
| Suwajanakorn et al. | — | 5.836 | — | 2.176 | copied from source | yes | no |
| Ours | 5.239 | 5.411 | 2.133 | 2.287 | freely adjusted | yes | yes |
| Original | 5.895 | 6.178 | 0 | 0 | — | — | — |
The paper explains the metrics as follows. SyncNet score measures speech-video synchronization confidence, with higher being better. AU error measures the difference between the detected facial action units of source and generated results, with lower being better. The user study asks 30 participants to rate $100$ video clips on audio-visual synchronization, image fidelity, and realism, using scores from $1$ to $10$.
In the reported results, AD-NeRF achieves synchronization and AU scores that are competitive with the compared methods, while also supporting full-frame synthesis and background control. The authors emphasize that this is achieved with a much smaller training video than the model-based baselines.
Ablations and analysis
The paper contains two primary ablations: one on the conditioning signal and one on the head/torso decomposition. It also reports an additional SSIM comparison on a $500$-frame test sequence.
In the audio-conditioning ablation, the authors compare direct audio conditioning against an intermediate facial-expression representation estimated from audio. They report that direct audio conditioning better preserves lip synchronization and leads to more accurate mouth motion. This supports the paper's main argument that introducing an intermediate modality can reduce performance because it discards information.
In the head/torso ablation, the authors train a single NeRF for the whole human body and compare it to the two-branch design. They report visible mismatches around the upper-body boundary when the torso is forced to share the head pose transformation. The separate head and torso models reduce these artifacts and improve reconstruction quality. The paper reports SSIM values of $0.92$ for the proposed method, $0.88$ for the intermediate-expression setting, and $0.87$ for the single-NeRF setting on a whole test sequence of $500$ frames.
The analysis suggests that the two main design choices of AD-NeRF are both important: direct audio conditioning improves speech-related motion accuracy, and separate modeling of head and torso improves spatial consistency of the rendered portrait.
Limitations
The paper explicitly notes two failure modes. First, in cross-identity audio-driven generation, the synthesized mouth region can look unnatural when the driving language does not match the training language. Second, torso rendering can become blurry because the head pose and audio features do not fully determine upper-body motion. These observations indicate that the method still has difficulty modeling motion not directly explained by audio and face pose.
The training setup also depends on a short, relatively controlled capture sequence with a static camera and background. The paper does not claim robustness to highly unconstrained capture conditions in the main text.
Conclusion
AD-NeRF presents an audio-driven talking-head synthesis pipeline built around neural radiance fields. Its core contribution is to directly condition a dynamic NeRF on audio features, avoiding intermediate facial representations that may lose information. Combined with separate head and torso radiance fields, the method can synthesize full-frame talking-head videos with realistic appearance and natural motion from a short training clip. The paper also demonstrates practical editing abilities, including changing audio, adjusting viewing direction, and replacing the background.
Overall, the paper's technical message is that neural scene representation can serve as a strong bridge between speech and portrait video generation, especially when the model is structured to respect the different motion patterns of head and upper body.
Code & Implementation
This repository contains a PyTorch implementation for AD-NeRF, a method from the paper "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis." The code realizes the approach of synthesizing high-fidelity talking head videos driven directly by audio input via neural radiance fields (NeRFs).
The implementation is primarily structured around two main NeRF models corresponding to different parts of the scene: Head-NeRF and Torso-NeRF. These are found in the NeRFs/HeadNeRF/ and NeRFs/TorsoNeRF/ directories, where each contains the training and rendering scripts, such as run_nerf.py.
The key design follows the paper's method of conditioning the NeRF implicit function with audio features, directly linking audio input to dynamic 3D neural scene synthesis using volume rendering. The two NeRFs independently model the head (including hair) and the upper body regions, enabling high-fidelity and natural results.
Audio and face processing utilities, data preprocessing scripts, pretrained models, and configuration files are also provided to facilitate training and inference workflows described in the README.
Usage notes detailed in the README include commands for data preprocessing, training the two separate NeRF models, and rendering synthesized video frames driven by either original or new audio inputs.