Face-vid2vid

One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

Face-vid2vid is a one-shot neural talking-head synthesis method that uses a compact 3D keypoint representation to disentangle identity, pose, and expression. It enables realistic motion transfer and free-view head pose control, supporting immersive video conferencing with 10x bandwidth savings.

• talking-head
• face-reenactment
• head-pose
• one-shot
• 3d-avatar
• realtime
• streaming
• low-latency

Demos

The demos showcase one-shot free-view neural talking-head synthesis from a single source image driven by video motion, highlighting natural and accurate facial motion transfer and head rotation. Viewers should evaluate the visual quality of facial reenactment, the fidelity of head pose changes, and seamless synthesis of expressions that enable realistic video conferencing avatars at low bandwidth.

Authors: Ting-Chun Wang, Arun Mallya, Ming-Yu Liu

Categories: cs.CV

Comment: CVPR 2021 camera ready (oral). Our project page can be found at https://nvlabs.github.io/face-vid2vid

Published 2020-11-30 · Updated 2021-04-02

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar648 citations129 influential citations

Abstract

We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.

---

Introduction

This paper studies one-shot talking-head synthesis: given a single source image of a person and a driving video, generate a target video whose identity comes from the source image and whose motion comes from the driving sequence. The paper’s central claim is that a purely neural renderer can do more than conventional one-shot reenactment. By learning a structured 3D keypoint representation, the model can both faithfully reconstruct or transfer head motion and also locally change the viewpoint of the head during synthesis, enabling a face-to-face style experience for video conferencing.

The method is built around three themes that recur throughout the paper:

• Disentanglement: canonical, person-specific keypoints are separated from pose and expression.
• Compactness: only pose and keypoint-deformation metadata need to be transmitted for conferencing, rather than full frames.
• 3D controllability: because the representation is 3D, the user can modify head rotation and translation directly.

The paper positions itself against prior 2D talking-head systems that can synthesize realistic motion but remain tied to the original viewpoint. In contrast, the proposed 3D decomposition supports local free-view synthesis, meaning the head can be rotated within a neighborhood of the original pose without requiring an explicit 3D face model.

The headline contributions reported by the paper are:

• a one-shot neural talking-head synthesis framework with state-of-the-art reconstruction and motion-transfer quality on the reported benchmarks;
• a learned unsupervised 3D keypoint decomposition into identity, head pose, and expression components;
• a video-conferencing compression scheme that transmits only compact motion metadata and achieves approximately $10\times$ bandwidth reduction relative to H.264 at comparable visual quality.

Method Overview

Let $s$ denote the source image and $\{d_1, d_2, \dots, d_N\}$ the driving video. The goal is to synthesize output frames $\{y_1, y_2, \dots, y_N\}$ such that the target identity matches $s$ while the motion follows the driving frames. The same framework covers both video reconstruction when $s$ is a frame from the driving video and motion transfer when $s$ and the driving video are from different identities.

The architecture has six learned components: a source appearance encoder $F$, a canonical keypoint detector $L$, a head-pose estimator $H$, an expression deformation estimator $\Delta$, a motion-field estimation network $M$, and a generator $G$. The model is trained end-to-end on talking-head videos containing a single person per clip.

Canonical 3D keypoints and decomposition

The core representation is a set of $K$ unsupervisedly learned canonical 3D keypoints $x_{c,k} \in \mathbb{R}^3$, with $K=20$ in the default setting. These canonical keypoints are intended to encode a person’s neutral geometric signature and to be independent of pose and expression.

For a given image, the model predicts:

• head pose, parameterized by a rotation matrix $R$ and translation vector $t$;
• expression-aware keypoint deformations $\delta_k$;
• canonical keypoints $x_{c,k}$.

The image-specific keypoints are then obtained by the explicit decomposition

$$x_k = R\,x_{c,k} + t + \delta_k.$$

The source image is encoded into appearance features and source-specific keypoints using the source pose and expression estimates $(R_s, t_s, \delta_{s,k})$. For the driving frame, the same canonical keypoints are reused and transformed using driving pose and expression estimates $(R_d, t_d, \delta_{d,k})$. This reuse is what makes the per-frame representation compact enough for conferencing.

A particularly important detail is that the paper does not estimate local Jacobians, unlike First Order Motion Model-style approaches. Instead, it assumes the head is largely rigid and derives the local transformation directly from the estimated head rotation. The authors note this simplification is one reason the conferencing representation becomes smaller.

Driving-frame representation and free-view control

For each driving frame, the encoder outputs the compact set of pose and deformation variables. Because the target identity is determined by the source image, the canonical keypoints are not recomputed for the driving frame; they are reused from the source. This makes a driving frame representable by the parameters $(R_d, t_d, \{\delta_{d,k}\})$.

The paper also exposes a direct editing interface for viewpoint control. If the user wants to inject an additional pose change specified by $R_u$ and $t_u$, the final pose is updated as

$$R_d \leftarrow R_u R_d, \qquad t_d \leftarrow t_u + t_d.$$

This is the mechanism that enables local free-view synthesis during talking-head generation and is one of the main architectural differences from prior 2D reenactment systems.

Video synthesis by 3D warping

After keypoint extraction, the source appearance volume is warped toward the driving configuration. The model first computes per-keypoint warp fields $w_k$ using a first-order approximation around each keypoint. These flows warp the source 3D feature volume, and the warped results are combined by a motion-field estimation network $M$ to produce a flow-composition mask $m$. The mask determines which local warp to trust at each 3D location, yielding the final flow field $w$. The generator $G$ then converts the warped feature volume into the output image.

The paper emphasizes that this is a pure neural rendering pipeline rather than a graphics pipeline: it does not require a 3D morphable model, explicit mesh fitting, or subject-specific 3D reconstruction.

Training Objective and Supervision

Training samples consist of pairs of frames drawn from the same talking-head video. One frame is used as the source image and another as the driving image. The model is trained jointly by minimizing a sum of reconstruction, adversarial, geometric-consistency, and prior terms:

$$\mathcal{L}_P + \mathcal{L}_G + \mathcal{L}_E + \mathcal{L}_L + \mathcal{L}_H + \mathcal{L}_\Delta.$$

The paper describes each term as follows:

• Perceptual loss $\mathcal{L}_P$: encourages feature-level similarity between the generated image and the driving image, helping produce sharp outputs.
• GAN loss $\mathcal{L}_G$: uses a multi-resolution patch discriminator and includes discriminator feature matching.
• Equivariance loss $\mathcal{L}_E$: enforces that keypoints behave consistently under image transformations; because the model uses 3D keypoints, these are projected orthographically before applying the loss.
• Keypoint prior loss $\mathcal{L}_L$: encourages keypoints to spread across the face and keeps their mean depth near a preset value.
• Head pose loss $\mathcal{L}_H$: compares predicted head rotation with an approximate ground-truth pose from a pretrained pose estimator.
• Deformation prior loss $\mathcal{L}_\Delta$: penalizes large expression deformations, since they should represent small deviations from the canonical geometry.

The paper’s training recipe is therefore a combination of image reconstruction, realism, pose consistency, and structural priors that collectively make the unsupervised keypoint decomposition identifiable enough to support rendering, motion transfer, and compression.

Experimental Setup

The paper evaluates on two main talking-head datasets:

• VoxCeleb2: about $1$M talking-head videos overall; the paper uses $280$K high bit-rate videos for training and reports results on a validation set of about $36$K videos.
• TalkingHead-1KH: a newly collected dataset composed from multiple sources, including YouTube videos under Creative Commons licenses, the Ryerson audiovisual dataset, and videos recorded by the authors with permission. The paper describes it as roughly $1000$ hours of videos and reports a collection of about $180$K videos in the experiment section. The videos are higher resolution and higher quality than VoxCeleb2.

For synthesis quality, the paper reports $L_1$, PSNR, SSIM, MS-SSIM, FID, and average keypoint distance (AKD). $L_1$, PSNR, SSIM, and MS-SSIM are used to measure fidelity; FID measures distributional realism; AKD measures how closely the synthesized landmarks match the ground truth landmarks extracted by an off-the-shelf detector.

Talking-Head Reconstruction and Motion Transfer

The main synthesis evaluation compares the proposed method with fs-vid2vid, FOMM, a larger FOMM-L variant, and bi-layer neural avatars. The key takeaway is that the proposed model is consistently strongest or tied for strongest across the reported metrics on both VoxCeleb2 and TalkingHead-1KH.

On same-identity reconstruction, the proposed method achieves the best numerical scores on both datasets. On VoxCeleb2 it reports $L_1=10.74$, PSNR $=24.37$, SSIM $=0.80$, MS-SSIM $=0.85$, FID $=69.13$, and AKD $=2.07$. On TalkingHead-1KH it reports $L_1=10.67$, PSNR $=24.20$, SSIM $=0.81$, MS-SSIM $=0.84$, FID $=52.08$, and AKD $=3.74$.

The paper also notes that simply increasing model size does not guarantee better performance: the larger FOMM-L variant is actually worse than the standard FOMM on the reported TalkingHead-1KH evaluation, which supports the claim that the proposed structured representation matters more than raw capacity.

Cross-identity motion transfer

For motion transfer, the source and driving identities differ. The paper reports that the proposed method achieves lower FID and higher identity-preserving similarity than the baselines on both VoxCeleb2 and TalkingHead-1KH. It also reports a supplementary relative-motion variant, but the main qualitative conclusion is unchanged: the model preserves identity while better matching the driving motion.

The paper also includes a supplementary relative-motion evaluation, where the proposed method again ranks best:

Face redirection / frontalization

Because the model operates in 3D keypoint space, the same learned representation can be used for face redirection. The paper compares against pSp and Rotate-and-Render (RaR). It measures whether the rotated result preserves identity and whether the head pose reaches the desired angle. The proposed method achieves the best combined tradeoff and the lowest FID.

Ablations and Representation Analysis

The appendix contains several ablations that clarify which design choices matter most. The paper compares direct keypoint prediction, 2D warping, and different keypoint counts. The preferred setting is the full model with $20$ keypoints and the 3D decomposition.

The ablations support three main conclusions. First, the two-step decomposition into canonical keypoints plus pose/deformation is better than direct final keypoint prediction, even though direct prediction is only slightly worse numerically. Importantly, the paper says direct prediction loses pose control, so the slight numerical gap understates the qualitative downside. Second, the 3D warping formulation is better than a 2D warp alternative. Third, more keypoints help, with $20$ keypoints outperforming $15$ and $10$ keypoints across all metrics.

The authors also analyze the learned canonical keypoints as a representation. Using canonical keypoints from $384$ identities in VoxCeleb2, a nearest-neighbor face-recognition experiment yields accuracy $0.070$, compared with $0.0026$ for random guessing and $0.013$ when using dlib facial landmarks. The interpretation is that the canonical keypoints retain strong identity/geometry information while being less entangled with pose and expression than standard landmarks.

The supplementary also reports that the canonical keypoints are similar across different poses of the same person, but differ across different identities in the same pose, which matches the intended decomposition into person-specific geometry and motion-specific transformations.

Video Conferencing Compression

The paper’s video conferencing application treats the receiver as an observer of an animated version of the sender’s face. Instead of sending full frames, the sender transmits the compact driving representation: the head pose and keypoint deformations. Since the source image can be sent once at the beginning of a session and reused later, it does not consume additional bandwidth for each frame.

For a driving frame, the representation consists of $K$ keypoint deformations plus pose parameters. When rotation is parameterized by Euler angles, the paper states the frame can be encoded with $3K+6$ numbers: $3K$ deformation values, $3$ rotation values, and $3$ translation values. These values are then further compressed using entropy coding.

The paper highlights two practical refinements:

• Adaptive number of keypoints: keypoint dropout during training allows the model to work with fewer transmitted keypoints at inference. On average, the number of sent keypoints drops from $20$ to $11.52$.
• Binary residual encoding: for hard cases such as new objects, background changes, or extreme poses, a binary residual encoder can transmit a residual code instead of sending a fresh source image. Each residual binary latent code is reported to require about $13.40$ KB.

The conferencing benchmark consists of $222$ manually selected high-quality talking-head videos of resolution $512\times512$ and length up to $1024$ frames. The paper compares against H.264 using settings suitable for real-time streaming, explicitly disabling bidirectional B-frames, and also compares with FOMM and fs-vid2vid after compressing their keypoints/Jacobians with the same entropy coding scheme.

Compression quality is evaluated both automatically and via human preference. The paper argues that pixel-alignment metrics are not ideal for this use case because the goal is faithful facial motion and gesture reproduction, not exact frame reconstruction. It therefore uses LPIPS for perceptual quality and bits-per-pixel for bandwidth.

Its key bandwidth claim is that, at comparable perceived quality, the adaptive scheme and the fixed $20$-keypoint scheme achieve $10.37\times$ and $6.5\times$ bandwidth reduction, respectively, relative to H.264 at CRF $36$.

This table makes the representation advantage concrete: the proposed method starts from a much smaller uncompressed metadata payload than the baselines and benefits further from entropy coding and adaptive keypoint selection.

Limitations and Failure Modes

The paper is explicit about a major weakness: the system still struggles with large occlusions. When hands or other objects cover the face, synthesis quality degrades. The appendix presents example failure cases and notes that image degradation can also hurt robustness.

A second limitation is conceptual rather than failure-specific: although the method supports local viewpoint changes, it is still a learned synthesis model and does not guarantee pixel-aligned output. This is acceptable for conferencing and motion realism, but it is not a general-purpose codec in the strict traditional sense.

Takeaways

• The paper’s main technical move is to factor talking-head motion into canonical identity keypoints, head pose, and expression deformation in 3D.
• This factorization makes the model both controllable and compressible.
• Across reconstruction, cross-identity motion transfer, and face redirection, the method reports the strongest results among the compared baselines in the paper.
• For conferencing, the representation is small enough to yield roughly an order-of-magnitude bandwidth reduction versus H.264 at comparable quality, while also enabling viewpoint manipulation that standard codecs do not provide.

Overall, the paper’s message is that an unsupervised 3D keypoint decomposition can serve simultaneously as a synthesis prior, a viewpoint-control mechanism, and a compact motion code for live talking-head communication.