PC-AVS

Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation

PC-AVS generates pose-controllable talking faces by learning modular audio-visual embeddings for speech and head pose. It avoids explicit landmarks or 3D models, using a compact latent pose code to achieve robust lip sync and flexible pose control under challenging conditions.

• talking-face
• audio-driven
• lip-sync
• head-pose
• face-animation

Demos

This demo visualizes the pose-controllable talking face generation capability of the proposed framework, highlighting its strength in synthesizing natural lip synchronization driven by audio with accurate and personalized head pose movements. Watch for the seamless integration of speech-driven lip movements and independently controllable head poses from a single reference photo without relying on pre-estimated landmarks or 3D parameters, showcasing the model's robustness and modularized audio-visual representation.

Authors: Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, Ziwei Liu

Categories: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS, eess.IV

Comment: Accepted to IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Code and models are available at https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS

Published 2021-04-22 · Updated 2021-04-22

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub961 stars22 watchers172 forksCC-BY-4.0

Abstract

While accurate lip synchronization has been achieved for arbitrary-subject audio-driven talking face generation, the problem of how to efficiently drive the head pose remains. Previous methods rely on pre-estimated structural information such as landmarks and 3D parameters, aiming to generate personalized rhythmic movements. However, the inaccuracy of such estimated information under extreme conditions would lead to degradation problems. In this paper, we propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. The key is to modularize audio-visual representations by devising an implicit low-dimension pose code. Substantially, both speech content and head pose information lie in a joint non-identity embedding space. While speech content information can be defined by learning the intrinsic synchronization between audio-visual modalities, we identify that a pose code will be complementarily learned in a modulated convolution-based reconstruction framework. Extensive experiments show that our method generates accurately lip-synced talking faces whose poses are controllable by other videos. Moreover, our model has multiple advanced capabilities including extreme view robustness and talking face frontalization. Code, models, and demo videos are available at https://hangz-nju-cuhk.github.io/projects/PC-AVS.

---

Overview

This paper proposes Pose-Controllable Audio-Visual System (PC-AVS), a framework for arbitrary-subject talking-face generation that keeps lip synchronization accurate while allowing the head pose to be controlled by a separate pose source video. The central idea is to avoid explicit structural intermediates such as landmarks or 3D fitting parameters, and instead learn an implicit low-dimensional pose code jointly with an audio-visual speech-content embedding and an identity embedding. The authors argue that pose and speech content both live in a non-identity latent space, and that a reconstruction-based training signal can force these factors to become modularized without direct supervision from pose annotations.

The method is designed for non-aligned raw face images and uses only a single photo as an identity reference at inference time. It is positioned as a clean alternative to landmark-based or 3D-model-based pipelines, especially when those structural estimates become unreliable under large poses, low light, blur, or other extreme viewing conditions.

The overall pipeline of our Pose-Controllable Audio-Visual System (PC-AVS) framework. The identity reference $I_{(ref)}$ is encoded by $E_i$ to the identity space (red). Encoder $E_n$ encodes video clip $V$ to $F_n$ in the non-identity space (grey). Then it is mapped to $F^v_c$ in the speech content space (blue), which it shares with $F^a_c$ encoded by $E^a_c$ from audio spectrograms $A$. We also draw encodings $F^{v-}_{c(j)}$, $F^{v-}_{c(k)}$ from two negative examples. Specifically, we map $F_n$ to pose features $F_p=f_{p(1:k)}$ in the pose space (yellow). Though motivated by 3D priors, the pose features are not supervised by or necessarily represent the traditional $[R, t, s]$ 3D parameters. Finally, a pair of features $\{f_{i(ref)}, f_{p(k)}, f^a_{c(k)}\}$ are assembled together and sent to generator $G$.

Problem Setting and Core Motivation

Prior audio-driven talking-face systems have largely optimized for lip sync and identity preservation, but most of them either keep the head pose fixed or depend on explicit structural inputs. The paper highlights two core difficulties:

• Audio alone does not provide reliable information about absolute head pose, so pose cannot be inferred in the same way as mouth motion.
• Structural intermediates, especially 2D/3D estimates, degrade under hard cases such as large viewpoint changes, blur, and low light.

PC-AVS addresses these issues by separating the generation problem into three latent factors: identity, speech content, and head pose. The speech-content factor is learned by audio-visual synchronization; the pose factor is learned implicitly from reconstruction; and the identity factor is learned from celebrity labels in VoxCeleb2.

The key design choice is to identify a non-identity space from augmented video frames, and then map that space into two complementary subspaces: a synchronized speech-content space and a compact pose space. The paper motivates a 12-dimensional pose code by analogy with minimal 3D pose parameterization, namely a rotation matrix $R \in \mathbb{R}^{3 \times 3}$, a 2D translation vector $t$, and a scale scalar $s$.

Method

1. Identifying a Non-Identity Feature Space

Training is framed as reconstructing a target frame $I_{(k)}$ from an identity reference image $I_{(ref)}$ and the corresponding audio segment. Let $V=\{I_{(1)},\dots,I_{(K)}\}$ be a $K$-frame clip, and let $A=\{S_{(1)},\dots,S_{(K)}\}$ be the associated mel-spectrogram slices.

To isolate information that is not tied to identity, the target frames are augmented with three operations:

• Color transfer by random RGB alteration to suppress texture cues.
• Perspective transformation to account for facial deformation and geometric variation.
• Centered random crop to reduce sensitivity to facial scale changes and detector noise.

The resulting augmented frames $V'$ are encoded by $E_n$ into a non-identity latent representation $F_n$. The paper presents the perspective transform via source points $P_s$ and target points $P_t$, linked by a $3\times 3$ transformation matrix $M$.

2. Modularizing Audio-Visual Representations

Once the non-identity space is available, the model splits it into three modular latent spaces.

Speech Content Space

The speech-content branch maps $F_n$ to a visual speech feature $F^v_c = \operatorname{mp}_c(F_n)$, while a separate audio encoder $E^a_c$ maps the audio spectrograms to $F^a_c = E^a_c(A)$. The two are trained to lie in the same embedding space through a contrastive objective based on InfoNCE.

With $N^-$ negative audio samples and cosine similarity $\mathcal{D}(\cdot,\cdot)$, the visual-to-audio synchronization loss is:

$$ \mathcal{L}^{v2a}_{c} = -\log \frac{\exp\left(\mathcal{D}(F^v_c, F^a_c)\right)}{\exp\left(\mathcal{D}(F^v_c, F^a_c)\right) + \sum_{j=1}^{N^-} \exp\left(\mathcal{D}(F^v_c, F^{a-}_{c(j)})\right)}. $$

The audio-to-visual direction is formulated symmetrically, and the total synchronization loss is $\mathcal{L}_c = \mathcal{L}^{v2a}_c + \mathcal{L}^{a2v}_c$.

A major point emphasized in the paper is that this contrastive pretraining is not just for lip sync in isolation: by establishing a clean speech-content subspace, it also makes the later pose modularization more stable.

Pose Space

The pose branch maps $F_n$ to a 12-dimensional pose code $F_p = \operatorname{mp}_p(F_n)$. The paper does not claim that these features are literal 3D pose parameters; instead, the dimension choice is inspired by the minimal information needed to represent pose in a 3D parameterization. The authors argue that making the pose code low-dimensional discourages it from absorbing mouth-shape or identity information.

Identity Space

Identity is learned by a dedicated encoder $E_i$ with a softmax cross-entropy classification loss $\mathcal{L}_i$. Since VoxCeleb2 contains celebrity identities, the model can use speaker labels to supervise this branch directly.

3. Talking Face Generation with Modulated Convolutions

At generation time, the model concatenates three latent codes: the identity feature from $I_{(ref)}$, the audio speech-content feature for the target time step, and the pose code from the pose source frame. The concatenated vector $f_{cat(k)} = \{f_{i(ref)}, f^a_{c(k)}, f_{p(k)}\}$ is used to modulate the convolution kernels in the generator.

This choice is important. Unlike skip-connection-based reconstruction models, which preserve too much low-level geometry and make pose change difficult, PC-AVS uses a style-like generator with modulated convolutions. For each block, an MLP produces a modulation vector $\mathcal{M}$, and the convolution weights are normalized after modulation:

$$ w^m_{xyz} = \frac{\mathcal{M}_x \cdot w_{xyz}}{\sqrt{\sum_{x,z} (\mathcal{M}_x \cdot w_{xyz})^2 + \epsilon}}. $$

The generator uses 6 blocks of modulated convolutions. The paper notes that a traditional skip-connection design does not work well in this setting, while the modulation-based design gives the generator enough flexibility to recover details and control pose.

4. Joint Training Objective

Training combines adversarial, reconstruction, perceptual, synchronization, and identity losses. The generated image and the target frame are sent to a multi-scale discriminator $D$ with $N_D$ layers. The paper uses an adversarial loss, an $L_1$ loss on discriminator features, and a VGG perceptual loss:

$$ \mathcal{L}_{GAN} = \min_G \max_D \sum_{n=1}^{N_D} \Big( \mathbb{E}_{I_{(k)}}[\log D_n(I_{(k)})] + \mathbb{E}_{f_{cat(k)}}[\log(1 - D_n(G(f_{cat(k)})))] \Big), $$

$$ \mathcal{L}_{L_1} = \sum_{n=1}^{N_D} \left\| D_n(I_{(k)}) - D_n(G(f_{cat(k)})) \right\|_1, $$

$$ \mathcal{L}_{vgg} = \sum_{n=1}^{N_P} \left\| \operatorname{VGG}_n(I_{(k)}) - \operatorname{VGG}_n(G(f_{cat(k)})) \right\|_1. $$

The total objective is:

$$ \mathcal{L}_{total} = \mathcal{L}_{GAN} + \lambda_1 \mathcal{L}_{L_1} + \lambda_v \mathcal{L}_{vgg} + \lambda_c \mathcal{L}_{c} + \lambda_i \mathcal{L}_{i}. $$

The paper states that the $\lambda$ coefficients are set empirically to $1$.

Implementation Details

The authors use ResNeXt-50 for $E_i$, borrow $E_n$ from a prior talking-face work, and use a ResNetSE-34 for the audio encoder $E^a_c$. All images are resized to $224 \times 224$. Audio is preprocessed to $16$ kHz, then converted to mel-spectrograms with FFT window size $1280$, hop length $160$, and $80$ Mel filter banks. For each frame, a $0.2$ s mel segment centered on the target time step is sampled as the condition.

The identity encoder is first pretrained on VoxCeleb2 identity labels using $\mathcal{L}_i$. The speech-content space is pretrained first with $\mathcal{L}_c$, after which these modules are loaded into the overall framework for pose-space and generator learning. The models are implemented in PyTorch on eight $16$ GB Tesla V100 GPUs. The paper does not use facial keypoint alignment during training or inference.

Datasets and Experimental Protocol

Experiments are conducted on two widely used in-the-wild audio-visual datasets: VoxCeleb2 and LRW. VoxCeleb2 contains $6{,}112$ celebrities and over $1$ million utterances, with $5{,}994$ speakers for training and $118$ for testing. The test identities are unseen during training, and the videos include difficult cases such as large pose changes, low light, and blur. LRW contains over $1000$ utterances for each of $500$ different words in one-second clips; it is generally cleaner, with near-frontal BBC news faces, and many identities in the test set are seen during training.

Evaluation follows a self-driven setting in which the first frame of each test video serves as the identity reference, and the audio drives the full output video. For PC-AVS, the paper considers two settings: Ours-Fix Pose, where the pose code is fixed to match the input pose and the head remains still; and the full pose-controlled setting, where an additional pose source video supplies pose features.

To make pose evaluation fair, the paper also constructs a pose-source video by taking target frames and re-driving them with a different audio and identity, so that the pose source shares the target pose while changing mouth shapes and identity.

Compared Methods

The paper compares against representative arbitrary-subject talking-face methods that support audio-driven generation:

• ATVG, a 2D landmark-based method.
• Wav2Lip, a reconstruction-based method focused on lip synchronization.
• MakeitTalk, which uses 3D landmarks and can generate personalized head motions.
• Rhythmic Head, which generates rhythmic head motion under a different setting; the authors note that the code was not runnable at the time, so only provided samples were evaluated.
• Ground Truth, reported as an upper reference in the tables.

Quantitative Results

The main metrics are SSIM for overall image quality, CPBD for sharpness, mouth-region landmark distance (LMD) for mouth accuracy, and SyncNet confidence $\text{Sync}_{conf}$ for lip synchronization. The authors also report PSNR and CSIM in the supplementary material, but those are not the main focus of the paper body.

The paper’s main conclusions from this table are:

• On LRW, Wav2Lip achieves the best SSIM and SyncNet confidence, but PC-AVS improves mouth accuracy and preserves controllable motion.
• On VoxCeleb2, PC-AVS achieves the best SSIM among methods, very strong SyncNet confidence, and substantially better mouth LMD than Wav2Lip and MakeitTalk.
• PC-AVS-Fix Pose is consistently weaker than the pose-controlled version, showing that the pose source contributes meaningful motion diversity and quality.
• The authors note that Wav2Lip’s strong LRW performance is partly due to the dataset’s clean, mostly frontal faces and the model’s tendency to leave most of the input unchanged.

User Study

A user study with $15$ participants evaluates $30$ generated videos: $20$ from the VoxCeleb2 test set and $10$ from LRW. Participants rate each video on a $1$--$5$ Mean Opinion Score (MOS) scale for lip sync quality, head-movement naturalness, and overall realness.

The user study shows that PC-AVS matches Wav2Lip on lip-sync quality while substantially outperforming the baselines on motion naturalness and perceptual realness. The paper emphasizes that this is important because many competing systems can make the mouth move correctly but still look static or unnatural overall.

Qualitative Findings

The qualitative comparison highlights the specific trade-offs among baselines. ATVG can be accurate in easy cases, but cropped faces look less realistic and the face detector fails on some hard examples. Wav2Lip usually produces correct mouth openings, but the outputs are basically static. MakeitTalk introduces some head motion, yet the mouth shapes are not always accurate. Rhythmic Head can produce diverse head motion, but the identity preservation is weaker.

The paper’s narrative is that PC-AVS improves both motion diversity and realism because the pose source is explicitly disentangled from speech content, so the system can keep the lips synchronized while borrowing head motion from another video.

Ablation Studies

The ablation study examines three design choices: removing the contrastive speech-content loss, changing the pose-code dimension, and replacing the modulated-convolution generator with an AdaIN-based generator. The authors also introduce an additional Pose LMD metric that measures only facial-contour landmarks to capture pose accuracy.

The ablation results support the paper’s design claims:

• Removing the contrastive loss hurts speech-content alignment and collapses the modularization process, which in turn damages pose learning.
• Increasing the pose-code size from the intended low-dimensional setting to $36$ makes training harder and degrades both pose and mouth metrics.
• Switching to an AdaIN-style generator reduces reconstruction quality and weakens synchronization, confirming that the modulated-convolution design is better suited to the modular latent representation.

The text also notes that training without the target-frame data augmentation causes pose learning to fail in a similar way to the no-contrastive-loss setting, reinforcing the importance of the non-identity space construction.

Robustness and Frontalization

One of the paper’s distinctive claims is robustness under extreme viewpoints. In the additional visual analysis, ATVG fails when the reference input is hard, Wav2Lip produces mouth artifacts, and PC-AVS still generates accurate lip motion. The authors further show that setting the pose code to all zeros can frontalize the talking face while preserving identity.

What the Paper Claims as Its Main Contributions

• It introduces a framework that modularizes talking-face generation into identity, speech content, and head pose spaces without explicit structural intermediates.
• It proposes an implicit low-dimensional pose code learned from a non-identity embedding rather than from landmarks or 3D fitting.
• It uses a contrastive audio-visual synchronization objective to strengthen the speech-content representation and improve lip sync.
• It relies on a modulated-convolution generator to combine identity, content, and pose in a way that supports pose control, realism, and robustness.
• It demonstrates free pose control from another video, along with extreme-view robustness and frontalization.

Limitations and Caveats Stated or Implied by the Paper

The paper is optimistic about the approach, but several caveats are explicit in the experimental discussion:

• The pose code is learned implicitly and is not guaranteed to correspond numerically to true 3D pose parameters; it is a learned control representation rather than an exact geometric estimator.
• Performance depends on the quality of the learned modularization; when the contrastive loss is removed or the pose code is made too large, training degrades noticeably.
• Compared with clean frontal datasets such as LRW, the method’s advantage is more visible on harder in-the-wild cases, while Wav2Lip can be very competitive on lip-sync metrics in easier settings.
• The authors rely on a pose-source video for explicit pose control; the system is not presented as inferring pose directly from audio.

In other words, the paper’s contribution is not a universal solution to pose inference from speech, but a practical decoupling strategy that lets the user supply pose separately while preserving audio-driven mouth articulation.

Conclusion

PC-AVS demonstrates that talking-face generation can be organized around a modular latent representation instead of explicit structure pipelines. By learning a compact pose code from a non-identity space, aligning audio and visual speech content with InfoNCE, and reconstructing through a modulated-convolution generator, the system achieves lip-synced talking faces whose head motion can be controlled by another video. The empirical results show strong lip-sync quality, improved pose naturalness, and better robustness to challenging viewpoints than several prior arbitrary-subject baselines.