GeneFace
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
GeneFace is a NeRF-based system that generates photorealistic 3D talking faces from arbitrary speech audio, uniquely addressing out-of-domain audio generalization and avoiding the common "mean face" averaging issue by using a variational motion generator and domain-adaptive refinement prior to high-fidelity rendering.
Demos
These demos showcase GeneFace's ability to generate high-fidelity, photo-realistic 3D talking faces driven by arbitrary speech audio, emphasizing natural lip synchronization and facial motion. Watch for improved image quality, video stability, and artifact reduction, especially compared to prior methods like Wav2Lip and LSP. The demos also highlight GeneFace's robust generalization to out-of-domain audio and advanced 3D landmark-based facial motion synthesis.
Links
Paper & demos
Abstract
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field in this task to improve 3D realness and image fidelity. However, the generalizability of previous NeRF-based methods to out-of-domain audio is limited by the small scale of training data. In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based talking face generation method, which can generate natural results corresponding to various out-of-domain audio. Specifically, we learn a variaitional motion generator on a large lip-reading corpus, and introduce a domain adaptative post-net to calibrate the result. Moreover, we learn a NeRF-based renderer conditioned on the predicted facial motion. A head-aware torso-NeRF is proposed to eliminate the head-torso separation problem. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared to previous methods.
1. Problem Setting and Core Motivation
GeneFace addresses audio-driven talking face synthesis: given an arbitrary speech waveform, synthesize a photorealistic portrait video whose lip motion, facial dynamics, head pose, and upper-body appearance remain consistent with the audio. The paper focuses on the regime that matters for conversational AI and avatar systems: generalization to out-of-domain audio while retaining high image fidelity.
The authors argue that prior NeRF-based talking-face systems improved 3D realism and visual detail, but still struggle with two coupled issues:
- Weak generalizability: end-to-end NeRF talking-head models are usually trained on only a few minutes of subject-specific video, so they degrade on cross-lingual speech, singing, or other audio far from the training distribution.
- The “mean face” problem: mapping audio to facial motion is inherently one-to-many. Deterministic regression tends to average over plausible motions, producing over-smoothed mouths and weak lip synchronization.
GeneFace is designed as a three-stage system that decouples general audio-to-motion learning from subject-specific rendering:
- a variational motion generator trained on a large lip-reading corpus to predict expressive 3D facial landmarks from HuBERT audio features;
- a domain-adaptive post-net that refines predicted landmarks into the target person’s landmark distribution;
- a NeRF-based renderer conditioned on landmarks to synthesize high-fidelity frames, including a head-aware torso model to reduce head-torso separation artifacts.
2. System Overview
The inference-time pipeline, shown above, is conceptually simple but structurally important. Speech audio is first encoded into HuBERT features. These features are passed through a variational motion generator to produce a sequence of 3D facial landmarks. A learned post-net then adapts those landmarks to the target identity. Finally, a conditional NeRF renders the head and torso in a consistent image space.
This design intentionally separates audio generalization from person-specific appearance synthesis. The large-scale corpus is used where scale matters most—learning robust audio-to-motion mappings—while the small target-person video is used where identity and appearance matter most—learning the NeRF renderer.
2.1 Why 3D landmarks as the motion representation?
Rather than using coarse 3DMM expression coefficients, GeneFace conditions the generator and renderer on 68 selected 3D landmarks extracted from a reconstructed head mesh. The paper motivates this choice as a practical compromise:
- Compared with 2D landmarks, 3D landmarks preserve depth and are less pose-entangled.
- Compared with 3DMM expression codes, landmarks retain more local geometric detail relevant to lips, cheeks, and eye motion.
- They are directly usable as conditioning signals for both sequence generation and NeRF rendering.
Formally, the landmark representation is
$$L_{3D} = \{(M - \bar{M})_i \mid i \in I\}, \qquad L_{3D} \in \mathbb{R}^{68 \times 3},$$
where $M$ is the reconstructed 3D head mesh, $\bar{M}$ is the mean mesh, and $I$ indexes the selected keypoints.
3. Variational Motion Generator
The first stage is the paper’s main response to the “mean face” problem. Instead of learning a deterministic mapping from audio to landmarks, GeneFace uses a variational auto-encoder (VAE) with a flow-based prior so that multiple plausible motion patterns can be represented. The aim is to generate landmarks that are both expressive and temporally coherent.
3.1 Inputs and outputs
The generator takes HuBERT features as the audio condition and outputs a sequence of 3D landmarks. The paper explicitly uses HuBERT as the acoustic front-end to capture semantic and phonetic information from the waveform. The output is a full landmark sequence, not frame-by-frame sliding-window chunks, which lets the model synthesize arbitrary-length sequences in one forward pass.
3.2 Encoder, decoder, and flow prior
The architecture follows the general structure of PortaSpeech-style variational sequence generation: a convolutional encoder, a convolutional decoder, and a normalizing-flow prior. The appendix specifies that the encoder uses 8 layers, the decoder uses 4 layers, the convolution kernel size is 5, the channel size is 192, the latent size is 16, and the prior flow contains 4 layers with kernel size 3 and channel size 64.
The model is described by the conditional latent-variable factorization $q_\phi(z\mid l,a)$, $p_\theta(l\mid z,a)$, and a flow prior $p_\epsilon(z\mid a)$, where $a$ denotes audio features and $l$ denotes the landmark sequence. The encoder and decoder are fully convolutional and use dilated convolutions, inspired by WaveNet, to enlarge the receptive field and model long-range temporal dependencies.
The paper emphasizes two reasons for using a flow prior rather than a standard Gaussian prior:
- frame-wise latent variables should not be independent because facial motion is temporally correlated;
- a simple Gaussian prior encourages posterior collapse toward the mean, which hurts diversity and expressive motion generation.
To improve temporal stability, the predicted landmark sequence is also smoothed by a Gaussian filter as a post-processing step.
The paper’s training objective for the variational motion generator combines reconstruction, KL divergence, and a synchronization term from a pretrained sync-expert:
$$ \mathcal{L}_{\mathrm{VG}} = -\mathbb{E}_{q_\phi(z\mid l,a)}[\log p_\theta(l\mid z,a)] + D_{\mathrm{KL}}\big(q_\phi(z\mid l,a)\,\|\,p_\epsilon(z\mid a)\big) - \mathbb{E}_{\hat{l} \sim p_\theta(l\mid z,a)}[\log D_{\mathrm{sync}}(\hat{l})]. $$
The synchronization loss is important because the generator is otherwise free to produce plausible but poorly aligned mouth movements. The sync-expert acts as a learned proxy for audio-landmark coherence.
3.3 Sync-expert
The sync-expert takes a temporal window of landmark frames and a corresponding audio feature clip, encodes them with two 1D-convolutional towers, and predicts whether they are synchronized. The paper trains it with cosine similarity followed by binary cross-entropy. Its role is not to directly generate images, but to provide a differentiable training signal that encourages the landmark generator to keep the audio and motion aligned.
In the appendix, the sync-expert uses 14 layers and a channel size of 512.
4. Domain-Adaptive Post-Net
Although the motion generator learns on a large multi-speaker corpus, the NeRF renderer is trained on a specific person’s video. This creates a domain gap: predicted landmarks from the lip-reading corpus are semantically correct but do not necessarily match the target identity’s landmark distribution. Without correction, this mismatch causes blur and identity drift when the renderer is applied to the target person.
GeneFace therefore introduces a domain-adaptive post-net that refines generated landmarks into the target person domain while preserving lip synchronization.
4.1 Architecture and training signals
The post-net is a 1D convolutional residual network. The appendix specifies 8 layers, kernel size 3, and channel size 256. The discriminator is an MLP with 5 layers, hidden size 256, and dropout rate 0.25.
The post-net is trained in a semi-supervised adversarial manner with three complementary objectives:
- adversarial realism / identity matching against a frame-level discriminator;
- lip-sync preservation via the pretrained sync-expert;
- weak supervised regression on the target person video by reconstructing the ground-truth landmarks.
The discriminator uses an LSGAN-style objective:
$$ \mathcal{L}_{D} = \mathbb{E}_{\hat{l}}\big[(D(PN(\hat{l})) - 0)^2\big] + \mathbb{E}_{l'}\big[(D(l') - 1)^2\big], $$
where $l'$ denotes ground-truth target-person landmarks and $\hat{l}$ denotes landmarks predicted by the variational generator from large-scale data. The post-net objective is:
$$ \mathcal{L}_{PN} = \mathbb{E}_{\hat{l}}\big[(D(PN(\hat{l})) - 1)^2\big] + \mathbb{E}_{\hat{l}}[D_{\mathrm{sync}}(\hat{l})] + \mathbb{E}_{\hat{l}'}\big[\|PN(\hat{l}') - l'\|^2\big]. $$
Here $\hat{l}'$ is the motion predicted from the target-person audio. The important practical point is that the post-net does not simply fine-tune the whole motion generator on the tiny target set; instead, it acts as a targeted adaptation layer that can correct domain mismatch while retaining the generality learned from LRS3.
4.2 What the T-SNE analysis shows
The paper uses T-SNE to visualize landmark distributions and demonstrates a clear gap between LRS3-landmark samples and target-person landmarks. The post-net moves predicted landmarks from the LRS3 region toward the target-person region, which supports the paper’s claim that the post-net is functioning as a genuine domain adapter rather than a cosmetic denoiser.
5. NeRF-Based Renderer
Once the landmarks are refined, GeneFace uses a conditional NeRF to render photorealistic frames. The renderer is trained on the target person’s video and is responsible for recovering details that landmarks alone cannot specify, including skin texture, hair, teeth, and fine image appearance.
5.1 Landmark-conditioned radiance field
The conditional NeRF is defined as
$$F_\theta : (x, d, l) \mapsto (c, \sigma),$$
where $x$ is a 3D location, $d$ is the viewing direction, $l$ is the landmark condition, $c$ is RGB color, and $\sigma$ is volume density. The paper uses the landmarks from three neighboring frames, so the landmark condition has shape $3 \times 204$. This choice is meant to improve temporal continuity between adjacent frames.
The authors also point-wise normalize the landmark inputs because some landmark coordinates vary only over a small range, which can make it harder for the NeRF to learn high-frequency details.
Standard volume rendering is used to compute pixel color along each camera ray $r(t) = o + td$:
$$ C(r,l;\theta) = \int_{t_n}^{t_f} \sigma_\theta(r(t), l)\, c_\theta(r(t), l, d)\, T(t)\, dt, $$
with transmittance
$$ T(t) = \exp\left(-\int_{t_n}^{t} \sigma_\theta(r(\tau), l)\, d\tau\right). $$
The renderer is trained with a simple photometric reconstruction loss:
$$ \mathcal{L}_{\mathrm{NeRF}} = \sum_{r \in \mathcal{R}} \|C_\theta(r,l) - C_g\|_2^2, $$
where $\mathcal{R}$ is the set of camera rays and $C_g$ is the ground-truth pixel color.
5.2 Head-aware torso-NeRF
A practical failure mode in two-part talking-head renderers is head-torso separation: if the torso model does not know exactly where the head renderer will place the head, the seam between the two parts becomes visible, especially under larger head motions. GeneFace addresses this by making the torso NeRF head-aware.
The authors first train a head NeRF, then train a torso NeRF using the rendered head image as background. The torso NeRF is conditioned not only on pose and landmarks, but also on the pixel-wise output color of the head NeRF. The implicit function is written as
$$F_{\mathrm{torso}} : (x, C_{\mathrm{head}}; d_0, \Pi, l) \mapsto (c, \sigma),$$
where $d_0$ is the canonical-space view direction and $\Pi$ is the head pose transform. This design gives the torso model a better perceptual cue about where the head actually is, which reduces seam artifacts.
6. Datasets, Preprocessing, and Implementation Details
The paper uses a two-dataset setup that cleanly matches the two main subproblems:
- LRS3-TED for large-scale audio-to-motion learning. The authors select a high-quality subset containing 19,775 short videos from 3,231 speakers, totaling about 120 hours.
- A target-person dataset used to train the NeRF renderer and the post-net. For comparison with prior work, the paper follows the setup of earlier NeRF talking-head systems and uses 5 videos with an average length of about 6,000 frames at 25 fps.
Preprocessing details are also clearly specified:
- Audio is downsampled to 16 kHz and encoded with a pretrained HuBERT model.
- Video frames are resampled to 25 fps.
- 3D head pose and 3D landmarks are extracted using the 3D face reconstruction method cited in the paper.
- Target-person frames are cropped to 512 × 512.
- An automatic parsing method is used to segment head and torso regions and to extract a clean background.
Training is done on a single NVIDIA RTX 3090. The variational generator and post-net converge in about 40k and 12k steps respectively, taking around 12 hours. The NeRF renderer is trained for 800k iterations total, split evenly across the head and torso NeRFs, and takes about 72 hours.
The appendix provides the following key architecture settings:
- Variational motion generator: 8 encoder layers, 4 decoder layers, latent size 16.
- Flow prior: 4 layers.
- Sync-expert: 14 layers.
- Post-net: 8 layers.
- Renderer: 11 layers for both head and torso NeRFs.
7. Quantitative Evaluation
The paper evaluates image quality with FID, lip alignment with LMD and Sync confidence, and out-of-domain robustness with the same metrics computed on an OOD audio set containing cross-lingual, cross-gender, and singing voice clips.
| Method | FID ↓ | LMD ↓ | Sync ↑ | FID (OOD) ↓ | Sync (OOD) ↑ |
|---|---|---|---|---|---|
| Wav2Lip | 71.40 | 3.988 | 9.212 | 68.05 | 9.645 |
| MakeItTalk | 57.96 | 4.848 | 4.981 | 53.33 | 4.933 |
| PC-AVS | 96.81 | 5.812 | 6.239 | 98.31 | 6.156 |
| LSP | 29.30 | 4.589 | 6.119 | 35.21 | 4.320 |
| AD-NeRF | 27.52 | 4.199 | 4.894 | 35.69 | 4.225 |
| Ground Truth | 0.00 | 0.000 | 8.733 | N/A | N/A |
| GeneFace | 22.88 | 3.933 | 6.987 | 27.38 | 6.212 |
The paper’s main quantitative takeaways are:
- Best overall image fidelity among the evaluated methods: GeneFace achieves the lowest FID.
- Best lip localization among the non-synthetic baselines: GeneFace achieves the lowest LMD.
- Strong OOD robustness: while person-specific methods such as LSP and AD-NeRF degrade noticeably on OOD audio, GeneFace remains relatively stable.
- SyncNet-based methods can inflate synchronization scores: Wav2Lip attains the highest sync score, even above ground truth, which the authors note in their discussion.
7.1 How the results compare to the baselines
The baseline set is useful because it spans both one-shot face animation methods and person-specific neural renderers:
- Wav2Lip: strong lip synchronization, but low image fidelity and blurry mouths.
- MakeItTalk: weaker identity preservation and lower visual quality.
- PC-AVS: underperforms strongly on FID and synchronization in this setup.
- LSP: better visual fidelity than most one-shot methods, but weaker OOD synchronization than GeneFace.
- AD-NeRF: the closest NeRF baseline, but weaker generalization and more artifacts.
7.2 Qualitative observations
The qualitative comparison above is particularly informative because it isolates the exact failure modes the method is designed to fix. The paper reports that compared with AD-NeRF, GeneFace reduces:
- head-torso separation,
- blurry mouth rendering, and
- lip desynchronization on difficult audio.
These failures are consistent with the method’s design: the post-net handles domain mismatch in motion space, while the head-aware torso NeRF handles seam consistency in image space.
The full qualitative panel compares GeneFace to all baselines on an in-domain clip. The paper’s narrative is that one-shot methods can synchronize lips but tend to look less realistic, while person-specific NeRF methods preserve identity better but still suffer from OOD degradation or motion artifacts. GeneFace is positioned as the method that best balances both criteria.
8. User Study
To complement the objective metrics, the paper runs a user study with 20 attendees evaluating 10 audio clips sampled from English, Chinese, and German. The participants rate each video on a 1–5 MOS scale along three axes:
- lip-sync accuracy,
- image quality,
- video realness.
| Method | Lip-sync Accuracy | Image Quality | Video Realness |
|---|---|---|---|
| Wav2Lip | 3.77 ± 0.25 | 3.38 ± 0.19 | 3.27 ± 0.26 |
| MakeItTalk | 2.86 ± 0.33 | 2.84 ± 0.20 | 2.52 ± 0.30 |
| PC-AVS | 3.11 ± 0.30 | 2.73 ± 0.25 | 2.46 ± 0.28 |
| LSP | 3.65 ± 0.20 | 3.92 ± 0.13 | 3.62 ± 0.24 |
| AD-NeRF | 3.05 ± 0.26 | 3.44 ± 0.22 | 3.31 ± 0.24 |
| GeneFace | 3.82 ± 0.24 | 3.87 ± 0.16 | 3.87 ± 0.16 |
The user study supports the paper’s quantitative story: GeneFace is rated highest for lip-sync accuracy and video realness, while LSP slightly edges it on image quality. The authors interpret this as evidence that GeneFace better balances synchronization and realism than prior systems.
9. Ablation Study
The ablation study is important because it isolates the contribution of each design choice. The reported results are:
| Setting | FID ↓ | LMD ↓ | Sync ↑ | FID (OOD) ↓ | Sync (OOD) ↑ |
|---|---|---|---|---|---|
| GeneFace | 22.88 | 3.933 | 6.987 | 27.38 | 6.212 |
| w/o prior flow | 24.71 | 4.063 | 6.404 | 29.55 | 5.831 |
| w/o sync-expert | 24.02 | 4.151 | 5.972 | 30.77 | 5.549 |
| w/o post-net | 30.26 | 4.532 | 5.085 | 35.58 | 5.248 |
| w. fine-tune | 25.75 | 4.227 | 6.875 | 29.30 | 5.966 |
| w/o head-aware | 26.34 | 3.948 | 6.899 | 28.89 | 6.167 |
9.1 Variational generator ablations
Removing the flow prior hurts both FID and synchronization, consistent with the claim that a richer latent prior is needed for temporally structured motion. Removing the sync-expert also degrades synchronization more sharply, showing that the learned audio-motion alignment constraint is not redundant.
9.2 Post-net ablations
Removing the post-net causes the largest degradation among the main components. This supports the paper’s diagnosis that a domain gap exists between LRS3 and the target identity and must be corrected explicitly. The authors also compare against direct fine-tuning on the target-person video: fine-tuning improves in-domain scores somewhat, but is worse on OOD audio, which is exactly where GeneFace is supposed to be robust.
9.3 Head-aware torso-NeRF ablation
When the torso NeRF is not conditioned on the head renderer’s output, the FID score drops and head-torso separation artifacts reappear. The improvement is not huge numerically, but it is visually meaningful because seam artifacts are easy to notice in portrait video.
9.4 Landmark quality ablation
The appendix also reports landmark reconstruction error. GeneFace with VAE + flow + landmark NeRF achieves an L2 landmark error of 0.0371, compared with 0.0385 for vanilla VAE + landmark NeRF and 0.0424 for a regression model + landmark NeRF. This reinforces the paper’s claim that the variational formulation is better suited to the one-to-many audio-to-motion mapping than direct regression.
10. Limitations and Future Work
The paper is unusually explicit about its limitations:
- Tiny motion fluctuations: the variational motion generator and post-net occasionally produce slight landmark jitter, which can lead to artifacts such as shaking hair. The current workaround is a heuristic Gaussian filter.
- NeRF runtime cost: the renderer is still based on a vanilla NeRF formulation, so training and inference are relatively slow. The authors propose exploring accelerated or lightweight NeRF variants in future work.
These limitations are important for deployment: GeneFace improves realism and generalization, but it is not yet optimized for low-latency conversational use cases without further engineering.
11. Takeaway for a Talking-Head / Conversational-AI Team
GeneFace’s core contribution is architectural: it breaks the talking-face problem into three stages that each handle a different failure mode. The variational motion generator learns a diverse audio-to-motion prior from large-scale data, the post-net performs identity-specific domain alignment, and the NeRF renderer supplies photorealistic appearance synthesis with a head-aware torso model. The paper’s experiments show that this decomposition improves both out-of-domain robustness and visual fidelity relative to prior GAN-based and NeRF-based baselines.