SyncTalk

SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis

SyncTalk is a NeRF-based method addressing synchronization in talking-head synthesis by coordinating subject identity, lip movement, facial expression, and head pose. It uniquely integrates modules for lip-sync, expression control, and pose stabilization to create realistic, stable speech-driven talking head videos.

• talking-head
• lip-sync
• speech-driven
• face-animation
• head-pose
• blendshape
• nerf
• realtime

Demos

The demos showcase SyncTalk's ability to generate highly synchronized and realistic talking head videos driven by speech. Key evaluation points include the precise lip synchronization with audio, natural and accurate facial expressions, stable and coherent head poses, and high-fidelity restoration of hair and background details. Together, these demonstrate SyncTalk's effectiveness in overcoming common synchronization issues found in existing methods, resulting in visually and temporally consistent avatars.

Authors: Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, Zhaoxin Fan

Categories: cs.CV

Comment: Accepted by CVPR 2024

Published 2023-11-29 · Updated 2024-04-28

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads7 likes

Abstract

Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. Traditional Generative Adversarial Networks (GAN) struggle to maintain consistent facial identity, while Neural Radiance Fields (NeRF) methods, although they can address this issue, often produce mismatched lip movements, inadequate facial expressions, and unstable head poses. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic and artificial outcomes. To address the critical issue of synchronization, identified as the "devil" in creating realistic talking heads, we introduce SyncTalk. This NeRF-based method effectively maintains subject identity, enhancing synchronization and realism in talking head synthesis. SyncTalk employs a Face-Sync Controller to align lip movements with speech and innovatively uses a 3D facial blendshape model to capture accurate facial expressions. Our Head-Sync Stabilizer optimizes head poses, achieving more natural head movements. The Portrait-Sync Generator restores hair details and blends the generated head with the torso for a seamless visual experience. Extensive experiments and user studies demonstrate that SyncTalk outperforms state-of-the-art methods in synchronization and realism. We recommend watching the supplementary video: https://ziqiaopeng.github.io/synctalk

---

Introduction

SyncTalk addresses a central weakness in speech-driven talking-head synthesis: synchronization. The paper argues that realism depends on coordinated control of four coupled factors—subject identity, lip motion, facial expression, and head pose—and that existing approaches usually optimize only a subset of these factors. GAN-based methods can drive lip motion but often destabilize identity across frames. NeRF-based methods preserve identity better, but commonly suffer from inaccurate lip shapes, weak expression control, and jittery or detached head motion.

The authors frame synchronization as the key failure mode, or the "devil" in talking-head generation, and propose a NeRF-based system that explicitly disentangles and stabilizes the components needed for plausible speech animation. In their formulation, the model takes a cropped reference video of a talking head and the corresponding speech, then extracts lip features, expression features, and head pose before rendering a rough video and refining it into a high-resolution portrait video.

The paper's main claim is that better synchronization yields better perceptual quality. SyncTalk is designed around three dedicated modules: a Face-Sync Controller for lip and expression synchronization, a Head-Sync Stabilizer for pose stability, and a Portrait-Sync Generator for restoring hair and torso blending. The method reports strong improvements in both objective metrics and user study ratings, while maintaining real-time rendering speed.

Method

Overall pipeline

SyncTalk uses a tri-plane hash NeRF representation conditioned not only on spatial coordinates and view direction, but also on synchronized speech-driven latent codes. The pipeline first extracts a lip feature $f_l$, an expression feature $f_e$, and head pose parameters $(R,T)$; these are then used by the renderer to synthesize a rough talking-head video. A post-processing portrait generator then restores finer details such as hair and background, producing a higher-resolution final result.

Face-Sync Controller

The Face-Sync Controller has two parts: an Audio-Visual Encoder for lip synchronization and a Facial Animation Capturer for expression control. The motivation is that generic audio encoders from automatic speech recognition, such as DeepSpeech, Wav2Vec 2.0, or HuBERT, are trained for audio-to-text discrimination and therefore do not necessarily align well with audio-to-lip motion. SyncTalk instead pre-trains an audio-visual encoder on the 2D audiovisual dataset LRS2 so that the learned audio features share a feature distribution with lip movements.

The encoder is supervised by a lip synchronization discriminator. Given a continuous face window $F$ and the corresponding audio segment $A$, the discriminator computes cosine similarity,

$$ \operatorname{sim}(F,A) = \frac{F \cdot A}{\|F\|_2\,\|A\|_2} $$

and is trained with a binary cross-entropy synchronization loss:

$$ \mathcal{L}_{\text{sync}} = -\left(y\log\big(\operatorname{sim}(F,A)\big) + (1-y)\log\big(1-\operatorname{sim}(F,A)\big)\right). $$

Synchronized pairs are labeled $y=1$ and non-synchronized pairs $y=0$. Under this supervision, the paper pre-trains a facial generation network with a reconstruction objective

$$ \mathcal{L}_{\text{recon}} = \left\|F - \operatorname{Dec}(\operatorname{Conv}(A) \oplus \operatorname{Conv}(F))\right\|_1, $$

where audio and face encoders are concatenated and decoded back to a face window. After training, the audio branch output $\operatorname{Conv}(A)$ is used as the lip-space representation. The key point is that lip features are learned from audiovisual synchronization itself rather than from a speech-recognition proxy task.

For facial expression control, SyncTalk introduces a 3D facial prior using 52 semantically meaningful blendshape coefficients. The paper uses a facial animation capturer to extract expression code $E(B)$, then focuses on seven core coefficients that control the eyebrow, forehead, and eye regions. This design is intended to model expressions such as squinting, eyebrow raising, and frowning without collapsing facial structure or entangling expression with lip motion.

To reduce interference between lip and expression control, the paper adds a Facial-Aware Disentangle Attention mechanism. Starting from a region-attention vector $V$, it applies separate masks for the lip and expression regions:

$$ V_{\text{lip}} = V \odot M_{\text{lip}}, \qquad V_{\text{exp}} = V \odot M_{\text{exp}}. $$

The disentangled features are then formed as $f_l = f_{\text{lip}} \odot V_{\text{lip}}$ and $f_e = f_{\text{exp}} \odot V_{\text{exp}}$. The intended effect is to prevent lip motion from leaking into unrelated regions such as blinking, eyebrow motion, or hair volume, and vice versa.

Head-Sync Stabilizer

The Head-Sync Stabilizer targets the head-pose instability common in NeRF talking-head systems. The head pose is represented by a rotation $R$ and translation $T$. The paper first estimates a rough pose by searching over candidate focal lengths and minimizing the projection error between 3D Morphable Model landmarks and observed 2D landmarks. The optimal focal length is chosen as

$$ f_{\text{opt}} = \arg\min_{f_i} E_i\big(L_{2D}, L_{3D}(f_i, R_i, T_i)\big), $$

and then rotation and translation are refined with the optimal focal length:

$$ (R_{\text{opt}}, T_{\text{opt}}) = \arg\min_{R,T} E\big(L_{2D}, L_{3D}(f_{\text{opt}}, R, T)\big). $$

Here $E$ is mean squared error between projected 3D landmarks and detected 2D landmarks. The paper emphasizes that this coarse fit is not yet stable enough on its own, so it adds a second keypoint-based tracking stage inspired by SLAM-style bundle adjustment.

In the Head Motion Tracker, an optical-flow model is used to estimate facial keypoints $K$. A Laplacian filter selects keypoints where the flow changes the most, and these keypoints are tracked across the sequence. In the subsequent Bundle Adjustment stage, the method first initializes 3D keypoint coordinates and minimizes

$$ \mathcal{L}_{\text{init}} = \sum_j \lVert P_j - K_j \rVert_2, $$

where $P_j$ are projected keypoints. In a second optimization stage, it jointly refines keypoint positions, rotation, and translation by minimizing

$$ \mathcal{L}_{\text{sec}} = \sum_j \lVert P_j(R,T) - K_j \rVert_2. $$

The paper reports that this two-stage optimization yields smoother, more continuous head motion and reduces the head-to-torso separation artifact that often appears in prior methods.

Dynamic Portrait Renderer

The renderer is a tri-plane hash NeRF conditioned on geometry, view direction, lip features, and expression features. NeRF is written as an implicit function $\mathcal{F}:(\mathbf{x},\mathbf{d}) \rightarrow (\mathbf{c},\sigma)$, where $\mathbf{x}=(x,y,z)$ is a 3D point, $\mathbf{d}$ is the viewing direction, $\mathbf{c}=(r,g,b)$ is color, and $\sigma$ is density. Pixel colors are computed by volume rendering along rays $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$:

$$ \hat{C}(\mathbf{r}) = \int_{t_n}^{t_f} \sigma(\mathbf{r}(t))\, \mathbf{c}(\mathbf{r}(t),\mathbf{d})\, T(t)\, dt, $$

where $t_n$ and $t_f$ are near and far bounds and $T(t)$ is accumulated transmittance. To make the representation efficient, SyncTalk uses three oriented 2D multiresolution hash grids corresponding to the $XY$, $YZ$, and $XZ$ planes:

$$ \mathcal{H}^{AB}:(a,b) \rightarrow f_{ab}^{AB}, $$

and concatenates them into

$$ \mathbf{f}_\mathbf{x} = \mathcal{H}^{XY}(x,y) \oplus \mathcal{H}^{YZ}(y,z) \oplus \mathcal{H}^{XZ}(x,z). $$

The final implicit function becomes $\mathcal{F}^H:(\mathbf{x},\mathbf{d},f_l,f_e;\mathcal{H}^3) \rightarrow (\mathbf{c},\sigma)$. In other words, the renderer is not driven by audio alone; it is conditioned on the synchronized lip and expression codes produced by the earlier modules.

Training follows a coarse-to-fine strategy. The coarse stage uses a mean squared error loss between the rendered and target colors, while the fine stage adds LPIPS on random image patches to better preserve detail:

$$ \mathcal{L}_{\text{total}} = \sum_{\mathbf{r}} \lVert C(\mathbf{r}) - \hat{C}(\mathbf{r}) \rVert_2 + \lambda\, \mathcal{L}_{\text{LPIPS}}(\hat{\mathcal{P}}, \mathcal{P}). $$

The authors explicitly note that the LPIPS refinement is used to recover fine visual structure that pure MSE tends to miss.

Portrait-Sync Generator

The Portrait-Sync Generator is responsible for the last-mile visual cleanup. The paper identifies two recurring issues in NeRF rendering: loss of hair detail and visible seams where the generated head meets the torso. To address the first issue, the rendered face region $F_r$ is blurred to obtain $G(F_r)$ and then merged with the original image $F_o$ using the synchronized head pose. This improves the fidelity of hair and related boundary detail.

To address the second issue, if the generated face and the original torso leave a dark gap in the neck area, the model fills that region with the average neck color $C_n$. The result is a more seamless head-to-body composition and higher perceived realism.

Experimental setup

The paper evaluates SyncTalk on the same well-edited video sequences used by prior talking-head NeRF work. The dataset includes English and French videos, with an average length of about 8,843 frames. Videos are recorded at 25 FPS. All clips except the AD-NeRF video are at $512 \times 512$ resolution; the AD-NeRF video is $450 \times 450$.

The comparison set includes five GAN-based methods: Wav2Lip, VideoReTalking, DINet, TalkLip, and IP-LAP; and four NeRF-based methods: AD-NeRF, RAD-NeRF, GeneFace, and ER-NeRF. The paper also evaluates two SyncTalk variants: one without the Portrait-Sync Generator and one with it.

Implementation details are specific and practical. The coarse stage trains the portrait head for 100,000 iterations, followed by 25,000 iterations in the fine stage. Each iteration samples $256^2$ rays. The model uses a 2D hash encoder with $L=14$ and $F=1$. Optimization uses AdamW with a learning rate of 0.01 for the hash encoder and 0.001 for the other modules. Total training time is about 2 hours on an NVIDIA RTX 3090 GPU.

Rendering speed is reported separately from training. On the same GPU, the system reaches 52 FPS for head rendering at $512 \times 512$ resolution when data is already loaded on the GPU, and 50 FPS in Portrait mode with CUDA acceleration. This is higher than the 25 FPS input rate, so the authors argue that the system can support real-time generation.

Quantitative results

The paper evaluates both reconstruction quality and synchronization quality. Full-reference image metrics are PSNR, LPIPS, MS-SSIM, and FID. No-reference perceptual metrics are NIQE and BRISQUE. Synchronization is measured by landmark distance (LMD), action units error (AUE), and lip-sync confidence (LSE-C). For the OOD lip-sync study, the paper also reports LSE-D.

Main head reconstruction benchmark

In the main benchmark, SyncTalk is reported in two configurations. The full Portrait version achieves the best PSNR, LPIPS, MS-SSIM, FID, and LMD among all methods in the table, while the no-Portrait variant gives the best NIQE, BRISQUE, and AUE, and the second-best LSE-C. On this benchmark, Wav2Lip still has the highest LSE-C value, so the paper’s strongest synchronization claim is better supported by the OOD audio experiment below than by this in-distribution table alone.

The most important trend is that SyncTalk substantially improves reconstruction fidelity over prior NeRF systems, especially on LPIPS and FID. The paper highlights that LPIPS improves by roughly a factor of three relative to ER-NeRF. At the same time, the portrait generator trades some no-reference scores and AUE for sharper detail restoration and stronger overall visual fidelity.

Lip synchronization under out-of-distribution audio

To test generalization, the paper drives the same subject with two different audio samples and reports LSE-D and LSE-C. In this setting SyncTalk is best on all four reported numbers, showing that the audiovisual pretraining helps the model generalize better to unseen speech patterns.

The authors interpret this table as evidence that SyncTalk’s pre-trained audio-visual encoder overcomes the small-sample limitations of prior NeRF systems and gives more robust lip-audio alignment under unseen speech conditions.

User study and qualitative analysis

The qualitative comparison emphasizes three visually salient improvements: more accurate lip shapes, more expressive eyebrows and eye-region motion, and better integration between the generated head and the torso. The paper contrasts SyncTalk with Wav2Lip, IP-LAP, GeneFace, and ER-NeRF to argue that the proposed synchronization modules improve not just numerical metrics but also the artifact profile perceived by humans.

The user study used 24 video clips of more than 10 seconds each, with 35 participants rating five dimensions on a 1-5 mean opinion score scale. The average completion time was 19 minutes and the standardized Cronbach $\alpha$ was 0.96, suggesting strong internal consistency in the questionnaire. SyncTalk is best in every category, with the strongest relative margin in video realness.

Ablation study

The ablation study evaluates one subject, named “May,” using PSNR, LPIPS, and LMD. The results show that each designed component contributes meaningfully to the final performance, and that the biggest degradations come from removing the Head-Sync Stabilizer or replacing the audiovisual encoder.

The paper’s textual ablation analysis is very clear: replacing the audiovisual encoder weakens lip synchronization, replacing the facial animation module hurts eyebrow and expression motion, removing the masked-attention mechanism slightly increases cross-region entanglement, removing the Head-Sync Stabilizer causes obvious pose jitter and head-torso separation, and removing the portrait generator makes hair restoration and seam handling worse. The figure accompanying the ablation highlights the failure modes caused by dropping the first two synchronization modules.

Limitations and ethical considerations

The paper does not provide a separate formal limitations section, but it does include an explicit ethical warning. Because SyncTalk can generate highly realistic talking-head videos, the authors note the risk of misuse for misleading deepfakes. They state that they intend to share their results to support the development of deepfake detection tools.

The only other caveat stated in the paper is that they observed some possible artifacts during fusion, especially around facial edges. These artifacts are hard to perceive visually but can be detected by models as discontinuous facial changes. Beyond that, the paper does not enumerate additional failure cases, domain limits, or dataset biases.

Conclusion

SyncTalk is a NeRF-based talking-head system built around the thesis that synchronization is the critical missing ingredient in realistic speech-driven video synthesis. Its design explicitly separates and stabilizes lip motion, expression control, head pose, and portrait refinement. Empirically, the method outperforms a strong set of GAN-based and NeRF-based baselines on reconstruction quality, synchronization, and user preference, while also running at real-time speeds. The core technical novelty is not a new backbone alone, but a coordinated set of synchronization modules that attack the main sources of visible artifact in talking-head generation.