SadTalker

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

SadTalker generates realistic talking-head videos from a single image and audio by modeling 3D motion coefficients for facial expression and head pose separately. It distinguishes lip-sync and head movements with 3DMM coefficients, then uses a 3D-aware renderer for natural, identity-preserving video animation.

• talking-head
• audio-driven
• face-animation
• head-pose
• 3d-avatar
• lip-sync
• portrait-animation

Demos

These demos highlight SadTalker's capability to generate realistic 3D motion coefficients for talking face animation from audio, emphasizing natural head movements, expressive facial expressions, and identity preservation. Key evaluation points include the quality of motion and expression fidelity across languages, singing, and varied speaking styles, as well as demonstrated control over eye blinking. Comparisons with other methods validate its superior video quality and motion realism.

Authors: Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, Fei Wang

Categories: cs.CV

Comment: Accepted by CVPR 2023, Project page: https://sadtalker.github.io, Code: https://github.com/Winfredy/SadTalker

Published 2022-11-22 · Updated 2023-03-13

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub13,928 stars163 watchers2,661 forksApache-2.0
• Hugging Face0 downloads1,428 likes

Abstract

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. ie, unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly because of learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.

---

Introduction and Problem Setting

SadTalker addresses audio-driven single-image talking face animation: given one reference portrait and a speech clip, synthesize a realistic talking-head video. The paper argues that many of the failure modes seen in prior work—unnatural head movement, distorted facial expression, identity drift, and blur—stem from learning motions in coupled 2D motion-field or latent warping spaces. In contrast, the authors propose to explicitly represent motion using 3D Morphable Model (3DMM) coefficients, then use those coefficients to drive a 3D-aware face renderer.

The core idea is to separate the motion problem into two parts that have different relationships with audio:

• Facial expression, especially lip motion, is strongly coupled to speech and should be learned with supervision that emphasizes mouth-related motion.
• Head pose is only weakly linked to audio and is better modeled as a diverse, style-dependent latent process rather than a deterministic regression target.

The method is built around two learned motion generators—ExpNet for expression and PoseVAE for head pose—and a FaceRender module that maps the predicted 3DMM motion into the implicit motion space used by a face-vid2vid-style renderer.

Method Overview

SadTalker decomposes the problem into motion coefficient generation and video rendering. First, the source image is analyzed by a pretrained 3D face reconstruction model to obtain its initial coefficients. Then, audio drives two separate predictors: ExpNet outputs expression coefficients frame by frame, and PoseVAE synthesizes head-pose trajectories with controllable style diversity. Finally, these 3DMM motion coefficients are mapped into the unsupervised 3D keypoint space of a face animation network to synthesize the final video.

The paper emphasizes two design principles:

• Disentangle motion types so the network does not have to learn incompatible audio-to-motion mappings jointly.
• Keep rendering identity-preserving by using a 3D-aware warping-based renderer rather than directly regressing pixels from audio.

3DMM as the intermediate representation

The 3D face shape is written as

$$ \mathbf{S} = \bar{\mathbf{S}} + \mathbf{\alpha} \mathbf{U}_{id} + \mathbf{\beta} \mathbf{U}_{exp}, $$

where $\bar{\mathbf{S}}$ is the mean face, $\mathbf{U}_{id}$ and $\mathbf{U}_{exp}$ are the identity and expression bases, and $\mathbf{\alpha} \in \mathbb{R}^{80}$, $\mathbf{\beta} \in \mathbb{R}^{64}$ are identity and expression coefficients. Head pose is represented by rotation $\mathbf{r} \in SO(3)$ and translation $\mathbf{t} \in \mathbb{R}^3$. In SadTalker, identity coefficients are not generated; only motion-related parameters are modeled: expression $\mathbf{\beta}$ and pose $\rho = [\mathbf{r}, \mathbf{t}]$.

ExpNet: Audio-to-Expression Generation

ExpNet predicts expression coefficients from a short audio window while conditioning on the source identity’s initial expression. The paper’s motivation is that audio-to-expression is not one-to-one across identities, and raw expression coefficients contain motions irrelevant to speech. To reduce uncertainty, ExpNet uses a reference expression $\beta_0$ from the first frame and targets a lip-motion-only expression signal distilled from Wav2Lip and 3D face reconstruction.

The model takes an audio segment $a_{\{1,\dots,t\}}$ (each frame uses a 0.2 s mel-spectrogram), processes it with a ResNet-style audio encoder $\Phi_A$, and maps it with a linear layer $\Phi_M$ to expression coefficients. The prediction is conditioned on the reference expression and a blink control signal:

$$ \beta_{\{1,\dots,t\}} = \Phi_M\big(\Phi_A(a_{\{1,\dots,t\}}), z_{blink}, \beta_0\big). $$

The blink signal is included because eye blinks are weakly present in the training signal and can otherwise be washed out by the model. The paper uses an explicit landmark-based loss to control eye opening/closing and maintain other facial landmarks close to the lip-only target.

ExpNet losses

ExpNet is trained with three losses:

• Distillation loss $\mathcal{L}_{distill}$: mean-squared error between predicted expression coefficients and lip-only coefficients extracted from Wav2Lip outputs.
• Landmark loss $\mathcal{L}_{lks}$: constrains eye blinking and keeps the rest of the face close to the lip-only target in 2D landmark space.
• Lip-reading loss $\mathcal{L}_{read}$: compares character logits predicted from rendered mouth crops using a pretrained lip-reading model.

The paper defines the total expression loss as

$$ \mathcal{L}_{exp} = \lambda_{distill}\mathcal{L}_{distill} + \lambda_{read}\mathcal{L}_{read} + \lambda_{lks}\mathcal{L}_{lks}, $$

with weights $\lambda_{distill}=2$, $\lambda_{read}=0.01$, and $\lambda_{lks}=0.01$.

The appendix gives the distillation loss explicitly as

$$ \mathcal{L}_{distill} = \frac{1}{T}\sum_{t=1}^{T} (\beta_t^g - \beta_t^{lip})^2, $$

where $\beta_t^{lip}$ is the lip-only target coefficient and $\beta_t^g$ is the generated expression coefficient. For blinking, the paper computes the eye width and height from landmark pairs, defines the eye ratio $R_t$, and uses $\mathcal{L}_{eye}=\sum_t \lVert R_t - Z_t^{blink} \rVert_1$ plus a landmark regularizer on non-eye landmarks.

PoseVAE: Audio-Conditioned Stylized Head Motion

Unlike expression, head pose is not tightly determined by speech. The paper therefore uses a conditional VAE to generate a diverse head-motion residual relative to the first-frame pose. PoseVAE learns the residual motion $\Delta \rho_{\{1,\dots,T\}} = \rho_{\{1,\dots,T\}} - \rho_0$, where $\rho_0$ is the reference pose from the first frame. Modeling the residual helps generate longer, smoother, and more stable head-motion sequences during inference.

PoseVAE conditions on three inputs:

• the initial head pose $\rho_0$ from the reference image,
• the audio segment $a_{\{1,\dots,t\}}$, and
• a style identity code $Z_{style}$, implemented as a one-hot identity vector over the training identities.

The encoder and decoder are both two-layer MLPs. The encoder maps the motion residuals and conditions to a Gaussian latent distribution, and the decoder samples from it to reconstruct head-pose trajectories. The paper uses the terms “rhythm awareness” and “identity style” to describe the effect of the audio and style conditions.

PoseVAE losses

PoseVAE is trained with three losses:

• Reconstruction loss $\mathcal{L}_{MSE}$ between predicted and true pose residuals.
• KL divergence $\mathcal{L}_{KL}$ to match the latent distribution to a Gaussian prior.
• Adversarial loss $\mathcal{L}_{GAN}$ from a PatchGAN-style discriminator over motion sequences.

The total loss is

$$ \mathcal{L}_{pose} = \lambda_{MSE}\mathcal{L}_{MSE} + \lambda_{KL}\mathcal{L}_{KL} + \lambda_{GAN}\mathcal{L}_{GAN}, $$

with weights $\lambda_{MSE}=1$, $\lambda_{KL}=1$, and $\lambda_{GAN}=0.7$.

3D-aware FaceRender

After predicting motion coefficients, SadTalker renders the final image sequence with a face animation module inspired by face-vid2vid. The renderer does not take a real driving video; instead, it takes the predicted 3DMM coefficients and maps them into the unsupervised 3D keypoint space used by face-vid2vid.

The mapping network is a stack of 1D convolution layers. It consumes a temporal window of coefficients, using $t-2$ through $t+2$ frames, and predicts the motion representation of face-vid2vid for frame $t$ (the paper names these parameters as yaw, pitch, roll, translation, and a deformation term). The renderer is trained in two stages: first the base face-vid2vid model is trained self-supervised; then all of its appearance-related components are frozen and the mapping network is trained to align 3DMM coefficients with the unsupervised keypoint domain.

The paper also notes an important design choice: it removes alignment coefficients used by PIRenderer. These translation/scaling coefficients help crop aligned faces for 3D reconstruction, but the authors find that including them in audio-driven synthesis makes the generated head motion look unnaturally aligned and hurts realism.

FaceRender loss

Besides the original face-vid2vid losses, the mapping network is trained with an $L_1$ regularization between keypoints predicted from the 3DMM coefficients and the corresponding keypoints produced by face-vid2vid:

$$ \mathcal{L}_{1} = \frac{1}{N} \sum_{n=1}^{N} \lVert K'_n - K_n \rVert_1. $$

The paper sets the weight of this loss to 20. The rest of the rendering losses are kept as in face-vid2vid and are applied to the final image output.

Training Setup, Data, and Evaluation

Training data. The main training dataset is VoxCeleb, used to train FaceRender. The paper reports using over 100k videos from 1,251 subjects, cropping them following prior image-animation methods and resizing frames to $256\times256$. For ExpNet and PoseVAE, the authors select a subset of 1,890 aligned audio-video clips from 46 subjects because not all VoxCeleb samples are audio-video aligned. For evaluation, they use 346 videos from the HDTF dataset, taking the first 8 seconds of each video, for roughly 70k frames total, and resize to $256\times256$.

Audio preprocessing. Audio is downsampled to 16 kHz and converted to mel-spectrograms using the same settings as Wav2Lip: FFT window size 800, hop length 200, and 80 mel filter banks. Each frame receives a 0.2 s mel feature of shape $16\times80$.

Optimization. All modules are trained separately with Adam on 8 A100 GPUs. The learning rates are $2\times10^{-5}$ for ExpNet, $10^{-4}$ for PoseVAE, and $2\times10^{-4}$ for FaceRender. Temporal windows are 5 frames for ExpNet, 32 frames for PoseVAE, and 5 continuous frames for FaceRender. At inference, the method runs end-to-end without manual intervention.

Metrics. The paper evaluates:

• FID for visual realism.
• CPBD for sharpness.
• CSIM using ArcFace for identity preservation.
• LSE-C and LSE-D from Wav2Lip for lip synchronization.
• Diversity of head motion using Hopenet embeddings.
• Beat Align for alignment between audio rhythm and head motion.

Main Quantitative Results

The main comparison is performed on HDTF. The baselines include Wav2Lip, PC-AVS, MakeItTalk, Audio2Head, and Wang et al. The paper notes that Wav2Lip and PC-AVS are evaluated in one-shot settings and that Wav2Lip scores especially well on image quality because it primarily changes the mouth region while leaving the rest of the face intact. PC-AVS uses a fixed reference pose and fails on some samples in the cross-identity setting.

The reported numbers show a favorable trade-off: SadTalker is not the best on the lip-only metrics that can be artificially inflated by freezing most of the face, but it produces much stronger head-motion diversity and overall visual quality than motion-rich baselines. Its identity similarity is competitive with the strongest baselines and much better than methods that suffer from distortions or identity drift.

User study

The authors ran a user study on 20 test videos with 20 participants. Each participant judged lip synchronization, motion diversity, video sharpness, and overall naturalness. SadTalker was selected most often across all categories.

Cross-identity evaluation

The appendix also reports cross-identity evaluation on HDTF and VoxCeleb2, where the driving audio comes from a different video. This setting follows common video-reenactment evaluation practice and is more demanding because the audio, identity, and pose statistics no longer come from the same clip.

On VoxCeleb2 cross-identity, SadTalker again leads on a balanced combination of lip synchronization, head-motion diversity, and overall quality, while Wav2Lip remains strongest in the narrow sense of image quality because it changes the least of the frame.

Ablation Studies

ExpNet ablation

The paper ablates the factors used by ExpNet to show that each design choice matters. The strongest improvement comes from conditioning on the reference expression $\beta_0$ and adding the lip-reading loss. The authors also show that using the full real coefficients instead of lip-only targets hurts lip synchronization, which supports their claim that speech-related expression should be disentangled from other facial motions.

The accompanying figure shows that the reference expression stabilizes identity, while the lip-only target improves lip synchronization.

PoseVAE ablation

For PoseVAE, the authors measure head-motion diversity and beat alignment. The full model with a fixed style already outperforms the baseline, and the mixed-style setting gives the best values. Removing the GAN loss, the initial pose, or the audio/style conditions all degrades the results, confirming that each condition helps generate more realistic and controllable head motion.

FaceRender ablation

The renderer ablation compares SadTalker’s FaceRender with PIRenderer. On the reported numbers, SadTalker improves FID from 26.521 to 19.646 and CSIM from 0.857 to 0.880, while CPBD drops slightly from 0.363 to 0.334. The visual ablation also shows that the proposed mapping into unsupervised 3D keypoints models expressions more faithfully and avoids the overly aligned motion that results from including face-alignment coefficients.

Limitations

The paper explicitly states two main limitations. First, because 3DMMs do not model eye and teeth variation well, the renderer can produce artifacts around the teeth in some cases. The authors note that a face restoration method such as GFPGAN can help. Second, the current system focuses mainly on lip motion and eye blinking, so it does not explicitly model richer facial dynamics such as emotion or gaze direction. As a result, the generated videos can have a relatively fixed emotional state.

The conclusion also highlights a broader concern: because the system can generate highly realistic talking-head videos from a single image, misuse is possible. The authors say that both visible and invisible watermarks should be inserted for content identification.

Takeaways

• SadTalker’s central contribution is the use of 3DMM motion coefficients as a better intermediate representation for audio-driven portrait animation.
• ExpNet focuses on speech-driven expression, especially lip motion, and uses reference expression, blink control, coefficient distillation, landmark supervision, and lip-reading supervision to stabilize prediction.
• PoseVAE models head pose as a conditional, style-aware residual process to generate more diverse and natural motion.
• FaceRender maps the predicted motion coefficients into a 3D-aware unsupervised keypoint renderer, improving the final video’s realism and identity preservation.
• Across HDTF and VoxCeleb2, including cross-identity settings, the method reports strong results on motion diversity, beat alignment, identity similarity, and overall perceptual quality.