Wav2Lip

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

Wav2Lip generates accurate lip-sync for arbitrary talking-face videos by learning from a strong pre-trained lip-sync expert, enabling natural and precise mouth movements in dynamic and unconstrained settings. This surpasses earlier methods limited by weaker supervision and static-image focus.

• lip-sync
• talking-face
• audio-driven
• dubbing
• speech-to-speech

Demos

The demos showcase Wav2Lip's superior lip-sync accuracy on dynamic, unconstrained talking face videos of arbitrary identities. Watch for the precise alignment of lip movements with speech, showing natural and accurate lip shapes that outperform previous models. The comparison clearly highlights Wav2Lip's ability to avoid out-of-sync lip motion, a key limitation of earlier methods.

Authors: K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C V Jawahar

Categories: cs.CV, cs.LG, cs.SD, eess.AS

Comment: 9 pages (including references), 3 figures, Accepted in ACM Multimedia, 2020

Published 2020-08-23 · Updated 2020-08-23

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub13,073 stars182 watchers2,832 forks
• Hugging Face0 downloads73 likes

Impact

• Semantic Scholar1,188 citations284 influential citations

Abstract

In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model and evaluation benchmarks on our website: https://cvit.iiit.ac.in/research/projects/cvit-projects/a-lip-sync-expert-is-all-you-need-for-speech-to-lip-generation-in-the-wild. The code and models are released at this GitHub repository: https://github.com/Rudrabha/Wav2Lip. You can also try out the interactive demo at this link: https://bhaasha.iiit.ac.in/lipsync.

---

Introduction

This paper addresses speaker-independent speech-to-lip generation in unconstrained videos: given a source talking-face video and a target speech segment, synthesize a version of the video whose mouth motion matches the new audio while preserving the rest of the visual content as much as possible. The problem is practically important for multilingual video translation, lecture dubbing, public-address translation, and other talking-head applications where viewers are highly sensitive to audio-visual mismatch.

The authors argue that prior methods were successful mainly in two narrower settings: either they generated lip motion for a single known identity, or they worked well only on static images of arbitrary identities. When applied to dynamic, unconstrained videos in the wild, these methods often leave substantial segments out of sync because they do not adequately handle the diversity of poses, scales, illumination changes, and mouth shapes present in real footage. The central claim of the paper is that accurate lip-sync in the wild can be achieved by learning from a strong, pre-trained lip-sync expert rather than relying only on pixel reconstruction or a weak discriminator.

The proposed model is called Wav2Lip. It combines a frame generator with a frozen, pre-trained lip-sync discriminator and an auxiliary visual-quality discriminator. The paper also argues that evaluation protocols from earlier work are not well aligned with real usage and introduces new metrics and benchmarks based on SyncNet-style audio-visual correspondence scoring.

Problem Setting and Why Prior Methods Fail

The task studied here is: given a short speech segment and a face video of arbitrary identity, generate a new version of the video that lip-syncs to the target speech. Unlike constrained talking-face synthesis methods, the model should not require speaker-specific training data or a narrow vocabulary. The paper focuses on the speaker-independent regime where the system must generalize to identities and videos it has never seen before.

The key observation is that existing speaker-independent approaches were designed and evaluated in ways that work well on static images, but degrade on full videos. The authors identify two main failure modes:

• Weak reconstruction supervision. Pixel-level reconstruction losses are dominated by the full face image, while the mouth region occupies only a small fraction of the pixels. As a result, the model learns coarse appearance and pose first, and mouth correction can be delayed or underweighted.
• Weak synchronization discrimination. Earlier discriminators were not accurate enough at spotting off-sync audio-video pairs, and when trained in a GAN setup they could focus on artifacts instead of true audio-lip correspondence.

The paper emphasizes that even tiny errors matter: humans can detect mismatches on the order of about $0.05$ to $0.1$ seconds. This makes unconstrained lip-syncing especially challenging, because the model must be correct not only in average appearance, but also at a fine temporal scale.

Method: Wav2Lip Architecture

The core design principle is: generate accurate lip motion by learning from an already well-trained lip-sync expert. The system has three components: a generator, a frozen expert lip-sync discriminator, and an auxiliary visual-quality discriminator.

1) Expert lip-sync discriminator

The authors adapt SyncNet-style audio-visual matching, but make it significantly stronger for the task at hand. The expert discriminator takes a short window of $T_v = 5$ consecutive face frames and a matching audio segment. Instead of gray-scale inputs concatenated along channels, it uses color images, is deeper, and includes residual skip connections. The output is a probability that the audio and video are in sync.

The probability is computed from the normalized dot product between the learned audio and video embeddings $v$ and $s$:

$$ P_{\mathrm{sync}} = \frac{v \cdot s}{\max(\lVert v \rVert_2 \cdot \lVert s \rVert_2, \epsilon)} $$

The discriminator is trained on the LRS2 training split using a batch size of 64, Adam with learning rate $10^{-3}$, and a 5-frame temporal window. The paper reports that this expert reaches about 91% off-sync detection accuracy on the LRS2 test set, compared with only 56% for the discriminator used in LipGAN. This difference is central to the paper’s argument.

2) Generator

The generator is broadly similar to the LipGAN generator, but the crucial difference is how it is trained. It contains three blocks:

• Identity encoder for a reference frame $R$ concatenated with a pose prior $P$.
• Speech encoder for the target speech segment $S$.
• Face decoder that produces the final frame.

The pose prior is the target face crop with the lower half masked. This helps preserve pose and appearance while the model focuses on mouth motion. During inference, the model operates frame-by-frame: the current face crop is used as the reference, and the same crop with the lower half masked is used as the pose prior. The corresponding audio segment is passed through the speech encoder, and the decoder outputs the same face with the mouth region morphed to match the speech.

The generator itself is a 2D CNN encoder-decoder that generates each frame independently. The temporal window is needed only because the expert lip-sync discriminator consumes multiple consecutive frames.

3) Frozen expert supervision

During generator training, the expert discriminator is kept frozen. The generator is penalized when the discriminator believes the generated audio-video pair is not in sync. This is the paper’s main novelty: instead of adversarially training the sync expert on noisy synthetic frames, the generator is explicitly guided by a strong pre-trained expert that already understands the notion of true sync from real videos.

The expert sync loss is:

$$ E_{\mathrm{sync}} = \frac{1}{N} \sum_{i=1}^{N} -\log\left(P_{\mathrm{sync}}^{i}\right) $$

4) Reconstruction and visual-quality adversarial loss

The reconstruction loss is standard $L_1$ over generated and ground-truth frames:

$$ L_{ \mathrm{recon}} = \frac{1}{N} \sum_{i=1}^{N} \lVert L_g - L_G \rVert_1 $$

To recover photo-realism when the strong sync discriminator makes the mouth region a bit blurry, the paper adds a separate visual-quality discriminator trained in a GAN setup. Importantly, this discriminator is not responsible for synchronization; it only penalizes unrealistic facial appearance.

The generator’s total objective is a weighted combination of reconstruction, sync, and adversarial terms:

$$ L_{\mathrm{total}} = (1 - s_w - s_g) \cdot L_{\mathrm{recon}} + s_w \cdot E_{\mathrm{sync}} + s_g \cdot L_{g} $$

where the paper uses $s_w = 0.03$ for synchronization and $s_g = 0.07$ for adversarial quality. The visual discriminator is trained with the usual real-versus-generated objective, while the sync discriminator remains frozen.

Training Details

The full Wav2Lip generator and visual-quality discriminator are trained only on the LRS2 train split, which the paper states is about 29 hours of data. The generator and visual discriminator use Adam with learning rate $10^{-4}$ and betas $\beta_1 = 0.5$, $\beta_2 = 0.999$. The batch size for this stage is 80.

The expert lip-sync discriminator is trained separately, also on LRS2, with batch size 64 and learning rate $10^{-3}$. Since the discriminator is not fine-tuned on generated frames, its weights remain fixed while training the generator. The temporal input window is $T_v = 5$ frames, which the paper later shows is superior to shorter windows.

A subtle but important implementation detail is how the $T_v$-frame window is handled. The generator produces frames independently, but the discriminator expects contiguous temporal context. The authors therefore sample contiguous windows during training, stack time steps appropriately for the generator, and concatenate the generated lower-half frames along the channel dimension before feeding them to the sync expert. This allows the generator to receive a temporally informed lip-sync signal without becoming a recurrent model itself.

Evaluation Framework and Benchmarks

A major contribution of the paper is a rethinking of how speaker-independent lip-sync models should be evaluated. The authors argue that the earlier protocol used for static-image lip-syncing is not representative of real usage for talking-face videos:

• it evaluates the model with random reference frames rather than the current frame used in practice;
• it is inconsistent across methods because random frame selection changes the test set each time;
• it cannot naturally assess temporal consistency;
• standard image-quality metrics such as SSIM and PSNR do not directly measure sync.

To address this, the paper proposes two SyncNet-based metrics:

• LSE-D (Lip-Sync Error - Distance): lower is better and indicates tighter audio-visual alignment.
• LSE-C (Lip-Sync Error - Confidence): higher is better and indicates stronger confidence of sync.

The paper uses a publicly available SyncNet model trained on a different dataset from the authors’ expert discriminator. This is important because evaluation should be independent of the generator’s training objective. The paper also reports Fréchet Inception Distance (FID) to assess visual quality.

Consistent benchmarks from standard test sets

Using the new metric, the authors build consistent evaluation sets from the test splits of three standard datasets:

• LRS2: about 14K audio-video pairs.
• LRW: about 28K pairs, emphasizing frontal and near-frontal videos.
• LRS3: about 14K pairs, including profile views.

The consistency comes from pairing each source video with a pseudo-random speech segment from another video, rather than pairing with a randomly chosen frame. This makes the benchmark repeatable and better aligned with real deployment.

ReSyncED: a real-world evaluation dataset

The authors also collect and release ReSyncED, a real-world evaluation set sourced from YouTube to evaluate lip-sync in the wild. It contains three categories of videos:

• Dubbed: videos whose audio is naturally out of sync, such as dubbed movie clips or live-translated public addresses.
• Random: arbitrary videos paired with random audio.
• TTS: videos paired with synthetic speech from text-to-speech systems, intended to model video translation scenarios.

The human evaluation protocol uses 14 evaluators and asks them to score Sync Accuracy, Visual Quality, and Overall Experience on a 1--5 scale, plus a single-choice Preference vote.

Main Quantitative Results

The paper reports results on the consistent benchmarks derived from LRW, LRS2, and LRS3. The central result is that Wav2Lip’s sync quality is much closer to real videos than previous methods, while Wav2Lip+GAN improves visual quality further at a small cost in synchronization accuracy.

The paper’s interpretation is straightforward: Wav2Lip is essentially at the level of real videos on sync metrics, which is a strong result for a speaker-independent lip-syncing system. Wav2Lip+GAN slightly improves FID, confirming that the extra visual discriminator helps appearance quality, but it does not always improve sync metrics as much as the pure Wav2Lip objective.

Real-world evaluation on ReSyncED

The real-world results are especially important because they reflect the paper’s intended deployment scenario. The following table summarizes the main numbers reported on ReSyncED.

These numbers support several claims made in the paper. First, Wav2Lip sharply improves sync metrics over Speech2Vid and LipGAN in every real-world category. Second, Wav2Lip+GAN typically improves FID and human-rated visual quality, but the pure Wav2Lip model is often stronger on sync. Third, the model’s outputs are preferred substantially more often than older methods, and even the unsynced originals remain visually attractive enough that weakly synced generations are sometimes not preferred over them—underscoring why true synchronization matters.

Ablation Study: Why the Sync Expert Design Matters

The paper includes an ablation that directly tests two design choices: the temporal window size $T_v$ and whether the sync discriminator is fine-tuned on generated faces. The results are consistent and important.

The ablation reveals two clear conclusions. First, larger temporal context helps: $T_v = 5$ is better than $T_v = 3$, which is better than $T_v = 1$. Second, fine-tuning the sync discriminator on generated faces hurts. The authors argue that this happens because the discriminator begins to focus on synthetic artifacts rather than true audio-lip correspondence, which reduces its usefulness as a lip-sync expert. This is one of the paper’s strongest empirical findings, because it directly justifies the frozen expert design.

Applications, Fair Use, and Limitations

The paper highlights several applications enabled by accurate in-the-wild lip-syncing: translation of lecture videos into other languages, dubbing of movies and public addresses, correction of live-translated speeches, and animation of CGI or game characters from recorded voice acting. In all of these cases, the key requirement is that the generated mouth motion should be good enough for real deployment, not just for static-image demos.

The authors also explicitly discuss the ethical risks. Near-realistic lip-sync generation can be misused for manipulated media, so they recommend that any synthetic output be clearly identified as such. At the same time, they argue that releasing the model and code can also accelerate work on detection and forensics for manipulated video.

The main technical trade-off reported in the paper is between sync accuracy and visual quality. The pure Wav2Lip model is strongest on synchronization, while Wav2Lip+GAN improves FID and human visual judgments but can slightly reduce sync scores. The paper also notes that synthetic speech remains difficult: the human evaluations show there is still room for improvement when the input audio comes from TTS. Finally, the generator is still frame-based rather than fully temporally generative; temporal coherence is encouraged through the discriminator window and careful reference sampling, but the synthesis itself is not recurrent.

Conclusion

The paper’s central contribution is a simple but effective insight: for lip-syncing videos in the wild, a strong frozen lip-sync expert is more useful than a weak jointly trained discriminator. By training Wav2Lip against a discriminator that is already good at detecting real sync errors, the authors obtain a generator that produces much more accurate mouth motion for arbitrary identities and arbitrary speech. Their new evaluation protocol—consistent test pairs, SyncNet-based LSE metrics, and the ReSyncED real-world benchmark—also corrects several weaknesses in prior evaluation practice.

Across standard datasets and real-world videos, the paper reports that Wav2Lip’s lip-sync accuracy is close to real synced videos and substantially better than previous speaker-independent methods. The main practical lesson is that, in this task, getting the synchronization signal right matters more than optimizing pixel reconstruction alone. The paper closes by suggesting that similar ideas could extend beyond lip motion to other aspects of talking-face synthesis such as expressions and head pose.