MakeItTalk
MakeItTalk: Speaker-Aware Talking-Head Animation
MakeItTalk animates talking heads from a single image and audio by separating speech content and speaker identity to control lip sync, expressions, and head motion. It supports natural and stylized portraits, enabling personalized and high-quality animations using speaker-aware facial landmarks.
Demos
These demos illustrate MakeItTalk’s ability to generate realistic talking-head animations that sync lip movements precisely to audio and reflect dynamic facial expressions. Watch for the natural variation in expressions and seamless lip-sync, demonstrating personalized speaker-aware animation from audio input alone.
Links
Paper & demos
Impact
Abstract
We present a method that generates expressive talking heads from a single facial image with audio as the only input. In contrast to previous approaches that attempt to learn direct mappings from audio to raw pixels or points for creating talking faces, our method first disentangles the content and speaker information in the input audio signal. The audio content robustly controls the motion of lips and nearby facial regions, while the speaker information determines the specifics of facial expressions and the rest of the talking head dynamics. Another key component of our method is the prediction of facial landmarks reflecting speaker-aware dynamics. Based on this intermediate representation, our method is able to synthesize photorealistic videos of entire talking heads with full range of motion and also animate artistic paintings, sketches, 2D cartoon characters, Japanese mangas, stylized caricatures in a single unified framework. We present extensive quantitative and qualitative evaluation of our method, in addition to user studies, demonstrating generated talking heads of significantly higher quality compared to prior state-of-the-art.
Introduction
MakeItTalk is a single-image talking-head animation system that takes audio plus one portrait image and generates an expressive video of a talking face. The central problem it addresses is that audio-to-face animation is not one-to-one: the same speech content can produce different lip shapes, facial expressions, and head motions depending on the speaker. The paper’s key design choice is to avoid directly regressing raw pixels from audio. Instead, it first predicts intermediate facial landmarks and explicitly separates the audio into two factors:
- Content, which drives speech-synchronized motion of the lips, jaw, and nearby regions.
- Speaker identity, which modulates the rest of the facial dynamics, including expression style and head motion.
This decomposition is used to build a unified framework that can animate both natural human faces and non-photorealistic portraits such as paintings, sketches, cartoon characters, manga, and stylized caricatures. The paper emphasizes two claims throughout: first, that disentangling content and speaker identity improves lip synchronization and personalized motion; second, that landmarks provide a compact and editable representation that generalizes better than direct pixel prediction, especially for unseen target images.
Method Overview
The overall system has three stages:
- Voice conversion / audio disentanglement to obtain speaker-agnostic content and a separate speaker embedding.
- Landmark prediction in two stages: a speech-content branch and a speaker-aware branch.
- Single-image animation that converts the predicted landmark trajectory into output frames, using different rendering strategies for cartoons and natural photographs.
During training, the system uses off-the-shelf 3D facial landmark detection to preprocess video frames. The paper’s architectural motivation is that a low-dimensional landmark sequence is easier to learn than direct image synthesis, yet still expressive enough to capture lip motion, face shape changes, and head pose. Compared with prior methods that focus on only lip motion or require source videos / rigged models, MakeItTalk is designed for single-image, zero-shot target animation with unseen identities at test time.
Core factorization of audio
The method follows the voice-conversion literature and uses AutoVC to extract a speaker-agnostic content representation $A \in \mathbb{R}^{T \times D}$ from the input audio, while a separate speaker embedding is obtained from a speaker-verification model. The paper’s main premise is that the same phonetic content should determine the mouth and jaw motion, whereas the speaker embedding should determine the stylistic parts of motion that are not uniquely specified by the phoneme sequence.
Speech Content Animation
The first landmark predictor focuses on content-driven lip synchronization. It maps the audio content sequence to a neutral-style landmark displacement field. The paper reports that recurrent models outperform feedforward alternatives for this task because the mapping from audio to mouth movement is sequential. The chosen model is a three-layer LSTM operating over a local audio window of length $\tau = 18$ frames, corresponding to about $0.3$ seconds.
The content branch can be summarized as:
$$c_t = \operatorname{LSTM}_c(A_{t:t+\tau})$$
$$\Delta q_t = \operatorname{MLP}_c(c_t, q)$$
$$p_t = q + \Delta q_t$$
where $q \in \mathbb{R}^{68 \times 3}$ denotes the input static facial landmarks extracted from the target portrait, and $p_t$ is the predicted landmark position at frame $t$. The LSTM hidden size is 256 per layer, and the decoder MLP uses hidden sizes 512, 256, and 204 (for the $68 \times 3$ output).
A notable detail is that the paper does not train this branch on many speakers. Instead, it uses the Obama Weekly Address dataset, a single consistently framed speaker with about 6 hours of video, because the goal is to learn a generic mapping from speech content to lip motion after head pose has been factored out by landmark registration to a front-facing template.
Speaker-Aware Animation
The second landmark predictor captures the harder part of the problem: personalized facial dynamics and head motion. The paper argues that these are speaker-specific and operate over much longer time scales than phonemes. To model this, the method uses a separate LSTM for content encoding, then combines the result with the speaker embedding using a self-attention network.
The speaker-aware branch is written as:
$$\tilde c_t = \operatorname{LSTM}_s(A_{t:t+\tau})$$
$$h_t = \operatorname{Attn}_s(\tilde c_{t:t+\tau'}, s)$$
$$\Delta p_t = \operatorname{MLP}_s(h_t, q)$$
$$y_t = p_t + \Delta p_t$$
where $s$ is the speaker embedding, $\tau' = 256$ frames, or roughly 4 seconds, and $y_t$ is the final landmark sequence. The paper notes that head motions such as swings and nods last much longer than individual phonemes, so the longer temporal window is important. This branch is intended to model the correlation between speech and the speaker’s characteristic expression and pose style, not merely the mouth opening pattern.
The speaker embedding is extracted by a speaker verification model and then reduced from 256 to 128 dimensions with a one-layer MLP, which the authors found improved generalization to unseen speakers. The attention module follows a Transformer-style encoder block.
To encourage plausible speaker-specific dynamics during training, this branch is paired with a discriminator and optimized adversarially with an LSGAN objective. The key training intuition is that the generated landmark sequences should look like real trajectories for the same speaker and audio context, not just match per-frame positions.
Single-Image Animation
Once the final landmark sequence is available, the system renders image frames from a single portrait. The rendering path differs for cartoon-like images and natural photographs.
Cartoon and non-photorealistic portraits
For cartoons, sketches, manga, paintings, and similar stylized images, the paper uses a morphing / warping approach based on the predicted landmarks. Facial landmarks are extracted from the input portrait, Delaunay triangulation is applied, and the resulting triangles are warped according to the displaced landmarks. This preserves sharp edges and line art better than pixel-level synthesis. The implementation is described as a real-time GLSL-based pipeline.
Natural face images
For human faces, the system rasterizes the predicted landmarks into an image representation and feeds it, together with the input portrait, to a U-Net-like image-to-image translation network. The landmarks are drawn as connected line segments in a 6-channel input of size $256 \times 256$. The encoder-decoder uses six convolutional stages, residual blocks, and skip connections. The architecture is trained to reconstruct paired frames, and the paper reports that it can generate plausible natural talking-head videos with head motion, although background distortion may appear because the network synthesizes the whole image rather than separating foreground and background.
Training Procedure and Losses
The paper trains the three modules separately, with different datasets and losses.
Voice conversion pretraining
The content encoder and speaker embedding pipeline is trained following AutoVC, using the VCTK corpus with 109 native English speakers. Training minimizes self-reconstruction of source speech spectrograms. The speaker embedding is initialized from a pretrained speaker-verification model.
Content animation training
The content branch is trained on the Obama Weekly Address dataset, with facial landmarks extracted by a 3D landmark detector and then registered to a front-facing template. The loss combines a landmark position term and a graph-Laplacian term:
$$L_c = \sum_{t=1}^{T} \sum_{i=1}^{N} \|p_{i,t} - \hat p_{i,t}\|_2^2 + \lambda_c \sum_{t=1}^{T} \sum_{i=1}^{N} \|\mathcal{L}(p_{i,t}) - \mathcal{L}(\hat p_{i,t})\|_2^2$$
with $\lambda_c = 1$. The Laplacian coordinate for landmark $i$ is
$$\mathcal{L}(p_{i,t}) = p_{i,t} - \frac{1}{|\mathcal{N}(p_i)|} \sum_{p_j \in \mathcal{N}(p_i)} p_{j,t}$$
where the neighborhood $\mathcal{N}(p_i)$ is defined within one of eight facial parts. This loss encourages both absolute position accuracy and local shape preservation.
Speaker-aware training
For speaker-aware motion, the paper uses a subset of VoxCeleb2 with 67 speakers and 1,232 video clips, selected by manual landmark-quality verification and Poisson disk sampling in speaker-representation space. The split is 60% training, 20% validation, and 20% test. Unlike the content branch, landmarks are not front-facing registered here because the model is supposed to learn head pose variation.
The generator is optimized with a combination of landmark position loss, Laplacian loss, and adversarial realism loss:
$$L_s = \sum_{t=1}^{T} \sum_{i=1}^{N} \|y_{i,t} - \hat y_{i,t}\|_2^2 + \lambda_s \sum_{t=1}^{T} \sum_{i=1}^{N} \|\mathcal{L}(y_{i,t}) - \mathcal{L}(\hat y_{i,t})\|_2^2 + \mu_s \sum_{t=1}^{T} (r_t - 1)^2$$
with $\lambda_s = 1$ and $\mu_s = 0.001$. The discriminator is trained with an LSGAN loss:
$$L_{gan} = \sum_{t=1}^{T} (\hat r_t - 1)^2 + r_t^2$$
where $\hat r_t$ is the discriminator output on real landmarks and $r_t$ is the output on generated landmarks. The paper alternates generator and discriminator training.
Image-to-image translation training
The image synthesis module for natural faces is trained on paired frames from VoxCeleb2 and then fine-tuned on a high-resolution cropped video set from Siarohin et al. The input is a source frame and a target landmark image; the objective is a sum of per-pixel $L^1$ loss and VGG19 perceptual loss:
$$L_a = \sum_{\{src,trg\}} \|Q_{trg} - \hat Q_{trg}\|_1 + \lambda_a \sum_{\{src,trg\}} \|\phi(Q_{trg}) - \phi(\hat Q_{trg})\|_1$$
with $\lambda_a = 1$. Here $\phi$ concatenates activations from pretrained VGG19 layers.
Implementation details
- Landmarks are converted to 62.5 fps and audio is sampled at 16 kHz.
- Optimization uses Adam with learning rate $10^{-4}$ and weight decay $10^{-6}$.
- The content animation module has 1.9M parameters and trains in about 12 hours on one NVIDIA 1080 Ti GPU.
- The speaker-aware module has 3.8M parameters and trains in about 30 hours on one NVIDIA 1080 Ti GPU.
- The natural-face image translation module has 30.7M parameters, trains in about 20 hours on 8 NVIDIA 1080 Ti GPUs, and runs at about 22 FPS for natural faces and 28 FPS for cartoons at test time.
Datasets and Evaluation Protocol
The paper evaluates on a test split derived from VoxCeleb2 containing 268 video segments from 67 speakers. Speaker identities are seen during training, but the speech and video segments are distinct from the training clips. Reference landmarks are extracted with a 3D landmark detector and manually checked for quality.
The paper uses two groups of metrics:
- Jaw and lip metrics: $D$-LL (jaw-lips landmark distance), $D$-VL (jaw-lips landmark velocity difference), and $D$-A (difference in open mouth area).
- Global motion metrics: $D$-L (all-landmark distance), $D$-V (all-landmark velocity difference), and $D$-Rot/Pos (head rotation and position difference).
The key design of these metrics is that the paper evaluates not only local articulation, but also motion dynamics and pose consistency, since those are the parts that distinguish a purely lip-syncing method from a truly speaker-aware talking-head system.
Quantitative Results
Landmark-based content animation
The paper compares its content branch to prior landmark-driven methods by evaluating lip synchronization under a neutral head pose. Baselines include VisemeNet, Eskimez et al. (2018), and Chen et al. (2019). The full method achieves the best values across all reported lip metrics.
| Method | $D$-LL | $D$-VL | $D$-A |
|---|---|---|---|
| VisemeNet | 6.2% | 0.63% | 15.2% |
| Eskimez et al. 2018 | 4.0% | 0.42% | 7.5% |
| Chen et al. 2019 | 5.0% | 0.41% | 5.0% |
| Ours (no separation) | 2.9% | 0.64% | 17.1% |
| Ours (no speaker branch) | 2.2% | 0.29% | 5.9% |
| Ours (no content branch) | 3.1% | 0.38% | 10.2% |
| Ours (full) | 2.0% | 0.27% | 4.2% |
Qualitatively, the paper reports that VisemeNet mostly predicts the lower face and struggles with closed mouths, while Eskimez et al. and Chen et al. tend to produce conservative mouth openings. The full model is reported to better capture subtle expression details such as lip-corner lifting.
Speaker-aware head motion evaluation
To evaluate head motion, the paper compares against two retrieval baselines that copy head pose and position from training videos: one retrieved from the same speaker and one from a random speaker. The paper also evaluates reduced variants of its own model, including a random speaker-ID injection setting.
| Method | $D$-L | $D$-V | $D$-Rot/Pos |
|---|---|---|---|
| Retrieve-same ID | 17.1% | 1.2% | 10.3 / 8.1% |
| Retrieve-random ID | 20.8% | 1.1% | 21.4 / 9.2% |
| Ours (no separation) | 12.4% | 1.1% | 8.8 / 5.4% |
| Ours (random ID) | 33.0% | 2.4% | 28.7 / 12.3% |
| Ours (no speaker branch) | 13.8% | 1.2% | 12.6 / 6.9% |
| Ours (no content branch) | 12.5% | 0.9% | 8.6 / 5.7% |
| Ours (full) | 12.3% | 0.8% | 8.0 / 5.4% |
The reported pattern is important: copying head motion from another video, even from the same speaker, is not enough because the pose dynamics still do not align with the current audio. The full model is better than both retrieval baselines and the ablated variants. The paper also shows a t-SNE visualization over action-unit, head-pose, and position variance, indicating that predicted speaker dynamics lie closer to the reference speaker clusters than the baselines.
Ablation Study
The ablations are central to the paper’s claim that both disentanglement and two-branch modeling matter.
- Ours (no separation): removes the voice-conversion disentanglement and feeds raw audio features directly to the speaker-aware branch. It performs noticeably worse on lip dynamics, showing that content and speaker information need to be separated before animation.
- Ours (no speaker branch): keeps disentanglement but removes the speaker-aware branch. It does fairly well on lip synchronization but loses head-motion accuracy.
- Ours (no content branch): keeps disentanglement but removes the content branch. It retains better head motion but loses detailed mouth articulation.
- Ours (random ID): uses the full model but injects the wrong speaker embedding. This is the worst head-motion variant and confirms that the speaker embedding controls personalized dynamics rather than just adding noise.
The ablation table makes the tradeoff explicit: the content branch is responsible for accurate lip motion, while the speaker branch is responsible for head pose and expressive style. The full model is best because it composes both sources of information.
User Studies
The paper reports two Amazon Mechanical Turk studies with a total of 6,480 query responses from 324 participants.
- Speaker-awareness study: 300 queries comparing the full model with variants such as random-ID and no-speaker-ID settings. The task asked which cartoon animation better matched the reference speaker’s facial expressions and head motion. Each query got three votes from reliable participants, and the full model was preferred more often.
- Natural-face realism study: 780 queries comparing the full model against Chen et al. (2019) or Vougioukas et al. (2019), asking which animation looked more realistic and plausible. Again, the full model was favored by a large majority.
The study design included repeated queries to screen out inconsistent responses, and the paper explicitly filtered out unreliable workers.
Qualitative Results and Applications
The paper demonstrates two broad application families: non-photorealistic portrait animation and natural human video generation. For cartoons, the warping approach generalizes to a wide range of styles: paintings, sketches, manga, stylized caricatures, and even casual photos. For natural images, the image-to-image module can synthesize full-face talking-head videos with head pose changes, not just mouth motion.
The paper specifically highlights that the system can be useful for video dubbing, bandwidth-limited video conferencing, privacy-preserving telepresence, and text-to-video workflows when audio is produced by a speech synthesizer. It also mentions that the landmark representation allows manual editing and pose adjustment in principle.
Limitations and Future Work
The authors are explicit about several limitations. First, the method does not always capture bilabial and fricative sounds well, especially $/b/$, $/m/$, $/p/$, $/f/$, and $/v/$, and the paper suggests this may be caused by the voice-conversion module missing short phoneme patterns during spectrum reconstruction. Second, the current image translation module can introduce background distortion because it warps the entire image rather than separating foreground from background. Third, large head motion remains difficult because a single source image does not contain enough information to hallucinate unseen regions such as the neck, shoulders, and hair when the head turns too far.
The paper also notes broader avenues for future work: incorporating sentiment or mood signals to better model expressive head motion; adding phoneme- or viseme-aware features to improve articulation; foreground/background separation or portrait matting to reduce artifacts; longer-range temporal modeling at the pixel level; and more structured, editable control over the animation process. The conclusion also mentions that user interaction and landmark editing would be valuable future capabilities.
Ethical Note
The paper includes a brief ethical discussion on misuse potential. Because talking-head synthesis can be used for misinformation or other malicious deepfake applications, the authors state that their main intention is to demystify the technology and raise awareness. They also note that their released code includes a watermark to make generated videos clearly synthetic.
Takeaway
MakeItTalk’s main contribution is the combination of audio disentanglement, landmark-based animation, and speaker-aware motion modeling into a single system that can animate a single portrait image without target-specific fine-tuning. Empirically, the paper argues that the content branch captures synchronized articulation, the speaker branch captures expressive style and pose, and the landmark intermediate representation provides a robust bridge to both natural and stylized portrait synthesis. Across its metrics, ablations, and user studies, the full model consistently outperforms the paper’s ablated variants and the reported baselines on the evaluated tasks.