MeshTalk

MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

MeshTalk generates realistic 3D face animations from speech by disentangling lip motion and upper-face movements like blinks using a cross-modality latent space. It works for any identity without person-specific training, enabling natural full-face animation driven by audio alone.

• audio-driven
• speech-driven
• face-animation
• talking-face
• 3d-avatar
• autoregressive

Demos

The demo showcases MeshTalk's ability to generate accurate 3D face animations directly from speech audio using a cross-modality disentanglement approach. Viewers should watch for the naturalness of lip movements, synchronization with speech, and preservation of facial expressions in 3D mesh animations. The rendering differs slightly from the paper due to a different open-source engine but highlights MeshTalk's robustness and fidelity in speech-driven face animation.

Authors: Alexander Richard, Michael Zollhoefer, Yandong Wen, Fernando de la Torre, Yaser Sheikh

Categories: cs.CV

Comment: updated link to github repository and supplemental video

Published 2021-04-16 · Updated 2022-05-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub403 stars2 watchers59 forksAttribution-NonCommercial 4.0 International (Creative Commons)

Abstract

This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and audio-uncorrelated information based on a novel cross-modality loss. Our approach ensures highly accurate lip motion, while also synthesizing plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion. We demonstrate that our approach outperforms several baselines and obtains state-of-the-art quality both qualitatively and quantitatively. A perceptual user study demonstrates that our approach is deemed more realistic than the current state-of-the-art in over 75% of cases. We recommend watching the supplemental video before reading the paper: https://github.com/facebookresearch/meshtalk

---

Introduction

MeshTalk tackles the problem of generating a full 3D facial animation sequence from speech alone, with the explicit goal of making the output look natural across the entire face rather than only around the mouth. The paper is motivated by two recurring failures in audio-driven facial animation: methods either produce uncanny or static upper-face motion, or they rely on person-specific training data that limits scalability and practical deployment.

The central claim is that speech contains enough information to drive lip motion accurately, but not enough to fully determine all facial motion. As a result, any useful system must model a one-to-many mapping from audio to face motion, while still separating the parts of facial behavior that are strongly correlated with speech from those that are not. MeshTalk addresses this with a categorical latent expression space and a cross-modality loss that explicitly disentangles audio-correlated and audio-uncorrelated facial motion.

The paper’s main practical contribution is a non-personalized, audio-driven 3D face animator that can be conditioned on a neutral template mesh of an arbitrary identity. It is designed to synthesize realistic lip motion as well as plausible upper-face motion such as eye blinks and eyebrow raises, and the authors also demonstrate qualitative use cases for retargeting and mesh dubbing.

Problem Formulation and High-Level Idea

The input to the system is a time-aligned sequence of facial meshes and speech snippets. If $ \mathbf{x}_{1:T} = (\mathbf{x}_1, \dots, \mathbf{x}_T)$ denotes the mesh sequence, then each frame is a mesh in $\mathbb{R}^{V \times 3}$ with $V$ vertices. The aligned audio sequence is $\mathbf{a}_{1:T} = (\mathbf{a}_1, \dots, \mathbf{a}_T)$, where each audio snippet is a vector in $\mathbb{R}^D$. The system also takes a neutral template mesh $\mathbf{h}$ as the target identity.

The method proceeds in two stages. First, a learned encoder maps the speech-expression pair into a discrete latent representation of facial motion. Second, a decoder animates the template mesh from that latent code. During inference from audio alone, the latent codes are generated by an autoregressive model conditioned on speech, so that missing information not directly encoded in the audio can still be sampled in a plausible way.

The key design requirements for the latent space are spelled out clearly in the paper:

• Categorical: the latent representation should be discrete so that a powerful autoregressive model can operate over it.
• Expressive: it must represent diverse facial behaviors, including sparse events such as blinks.
• Semantically disentangled: audio-correlated motion and audio-uncorrelated motion should be separable to avoid collapsing everything into a mouth-only solution.

Method

Categorical latent expression space

MeshTalk does not use a single categorical variable. Instead, it uses $H$ latent classification heads, each with $C$ categories, so that the total number of possible latent configurations is $C^H$. The paper uses $C = 128$ and $H = 64$, which provides a large combinatorial space without requiring a single enormous vocabulary of states.

The encoder produces a continuous tensor over time and latent heads, and this is converted into a discrete code with a Gumbel-softmax operation. In standard notation, the encoder outputs a tensor that can be viewed as a set of logits over categories for each time step and each head, and the sampled categorical code is then used by the decoder to reconstruct facial motion.

This design is important for two reasons. First, discretization makes the latent space compatible with a PixelCNN-like autoregressive predictor. Second, the categorical space empirically preserves richer and more diverse upper-face motion than the continuous alternative explored in the ablations.

Cross-modality disentanglement loss

A naive reconstruction loss would encourage the model to copy whatever is easiest from the input sequence, which in this setting would allow the expression input to dominate and the audio stream to be ignored. The paper argues that this is especially problematic because the task is a one-to-many mapping: audio alone does not determine all facial motion, so the model must learn to distribute responsibility across modalities rather than collapse to a trivial reconstruction path.

To force the latent space to use both audio and expression information, MeshTalk introduces a cross-modality reconstruction objective. Given an expression sequence and an audio sequence, the model constructs two reconstructions by swapping one modality with a randomly sampled sequence from the training set:

$$ \hat{\mathbf{h}}^{(\text{audio})}_{1:T} = D\big(\mathbf{h}_x, E(\tilde{\mathbf{x}}_{1:T}, \mathbf{a}_{1:T})\big), \qquad \hat{\mathbf{h}}^{(\text{expr})}_{1:T} = D\big(\mathbf{h}_x, E(\mathbf{x}_{1:T}, \tilde{\mathbf{a}}_{1:T})\big). $$

These are supervised with region-specific masks so that the upper face is encouraged to be correct even when audio is swapped, while the mouth region is encouraged to be correct even when the expression sequence is swapped. The resulting loss is:

$$ \mathcal{L}_{\text{xMod}} = \sum_{t=1}^{T} \sum_{v=1}^{V} M_v^{(\text{upper})}\|\hat{h}^{(\text{expr})}_{t,v} - x_{t,v}\|^2 + \sum_{t=1}^{T} \sum_{v=1}^{V} M_v^{(\text{mouth})}\|\hat{h}^{(\text{audio})}_{t,v} - x_{t,v}\|^2. $$

The authors further add an eyelid-specific loss because blinks are sparse, brief, and affect relatively few vertices. The final objective is the sum of the cross-modality loss and the eyelid loss, with the paper noting that equal weighting works well in practice.

The intended effect is semantic factorization: audio should primarily determine lip shape and jaw movement, while the expression stream should carry upper-face behavior such as eye closure. At the same time, the system is not forced into a rigid separation; the paper later shows that audio can still influence eyebrow motion, which is consistent with speech emphasis.

Network architecture

The architecture is intentionally modular:

• Audio encoder: a 4-layer 1D temporal convolutional network, similar in spirit to the one used in the authors’ earlier audio-gaze work.
• Expression encoder: three fully connected layers followed by a single LSTM to capture temporal dynamics.
• Fusion module: a 3-layer MLP.
• Decoder: a UNet-style decoder with additive skip connections, followed by two LSTM layers and then three fully connected layers mapping back to vertex space.

The UNet-style skip connections are an important inductive bias: they help preserve the identity-specific geometry of the template mesh and reduce the chance that the animation drifts away from the target face. In the bottleneck, the latent expression code is concatenated with the encoded template mesh, so the decoder receives both the target identity and the facial-motion state.

Audio-conditioned autoregressive sampling

At inference time, speech is available but the input expression sequence is not. MeshTalk therefore learns an autoregressive model over the discrete latent codes. The conditional distribution factorizes as:

$$ p(\mathbf{c}_{1:T,1:H} \mid \mathbf{a}_{1:T}) = \prod_{t=1}^{T} \prod_{h=1}^{H} p(c_{t,h} \mid \mathbf{c}_{<t,1:H}, \mathbf{c}_{t,<h}, \mathbf{a}_{\le t}). $$

This factorization enforces temporal causality: when predicting a code at time $t$, the model only uses current and past audio rather than future audio. The latent predictor is implemented as a masked autoregressive temporal CNN with four convolutional layers and increasing temporal dilation, in the style of PixelCNN. The masking ensures that the prediction for each category only depends on previously generated heads and previous time steps.

Training the autoregressive model is done by first encoding the training data into categorical latent sequences using the pretrained encoder, and then optimizing the predictor with teacher forcing and a cross-entropy loss over the latent labels. At test time, the latent sequence is sampled sequentially one code at a time.

Evaluation Setup

The paper evaluates on an in-house dataset built specifically for high-fidelity full-face animation. It contains 250 subjects, each reading 50 phonetically balanced sentences. The sequences are captured at 30 fps using 80 synchronized cameras surrounding the subject’s head. The tracked face model has 6,172 vertices and includes detailed structures such as eyelids, upper-face geometry, and different hair styles. In total, the dataset contains about 13 hours of paired audio-visual data or approximately 1.4 million frames of tracked 3D meshes.

The split is by subject and sentence: the model is trained on the first 40 sentences of 200 subjects, while the remaining 50 subjects are used for validation and test, with 10 subjects for validation and 40 for test. The paper emphasizes that all qualitative and quantitative numbers reported are obtained on held-out subjects and held-out sentences.

Audio features are computed from 16 kHz recordings. For each visual frame, the model uses a 600 ms audio snippet that starts 500 ms before and ends 100 ms after the frame. The audio is represented by 80-dimensional Mel spectrogram features extracted every 10 ms, with 1,024 frequency bins and an 800-sample Fourier window.

Ablations and Analysis of the Latent Space

The paper’s ablations are focused on answering four questions: whether audio must be part of latent-space learning, how to prevent modality collapse, whether the learned representation disentangles mouth and upper-face motion, and whether a categorical space is preferable to a continuous one.

Why audio is needed during latent-space learning

One ablation removes audio from the encoder entirely and learns the latent space from expression alone. This can still reconstruct training sequences well, because the decoder receives enough information through the expression input and the skip-connected architecture. However, the resulting latent space is poorly structured for audio-conditioned generation. In particular, the paper reports that the autoregressive model becomes less confident and lip motion is less accurate, because nothing forces the latent representation to align with audio in a way that separates speech-correlated from speech-uncorrelated facial behavior.

The paper quantifies this with reconstruction error and autoregressive perplexity.

The important takeaway is that a plain $\ell_2$ decoder objective gives slightly lower reconstruction error, but the cross-modality loss produces a much better latent space for speech-conditioned generation, as reflected in the lower perplexity of the autoregressive model. The paper interprets this as evidence that the cross-modality objective improves factorization without sacrificing much reconstruction quality.

How the cross-modality loss changes the latent structure

The authors visualize latent codes generated by fixing one modality and varying the other. Latent codes induced by changing audio cluster separately from those induced by changing expression, with only limited leakage between the two. The paper also visualizes which vertices are most affected by each latent cluster: audio mainly drives mouth motion, while expression mainly affects the upper face, including eyelids.

The heatmap analysis is especially noteworthy because it shows that the learned factorization is not a hard separation. Audio still influences the eyebrow area, and the paper explicitly notes that this is consistent with speech emphasis, for example when certain words are stressed. So the model captures a more nuanced relation: the upper face is mostly expression-driven, but some upper-face motion is still speech-correlated.

A companion visualization shows the vertices most affected by expression input.

Categorical vs. continuous latent space

The paper directly compares the proposed categorical latent representation with a continuous latent variable model that predicts a mean and variance. In the continuous version, the autoregressive model also predicts mean and variance and samples the next latent embedding from them. The result is substantially worse lip accuracy and overall vertex error, and visually the continuous space tends to collapse upper-face motion toward mean expressions.

The conclusion from this ablation is that discreteness is not just a modeling convenience; it is part of what enables rich, diverse upper-face motion. The categorical representation appears better suited to a multimodal autoregressive generator than the tested continuous alternative.

Audio-Driven Results

Lip-sync evaluation against VOCA

The main quantitative comparison is against VOCA, which the paper treats as the state of the art for animating arbitrary neutral face meshes from audio. For fairness, the authors also evaluate a VOCA variant where the original DeepSpeech features are replaced with Mel spectrogram features and the paper’s own audio encoder. Lip error is defined per frame as the maximum $\ell_2$ error over lip vertices, then averaged over the test set; the authors explain that this better captures visually noticeable failures than an average over all lip vertices.

MeshTalk achieves the best lip error among the compared methods. The paper also notes that VOCA’s output quality depends strongly on the chosen conditioning identity, and that this issue appears amplified on the authors’ large, high-detail dataset, where VOCA tends to produce more muted lip motion and often misses lip closures.

Perceptual user study

The paper reports a human preference study with 100 participants. Each participant saw side-by-side clips and judged them on three subtasks: full-face realism, lip sync, and upper-face realism. For each row in the table below, 400 pairs of short clips were ranked. Participants could choose either clip or mark them as equally good.

The study supports the paper’s main claims: compared with VOCA, MeshTalk is preferred or tied in more than 75% of cases for full-face realism and lip sync, and in 66.4% of cases for upper-face motion. Against ground truth, the model is still often judged quite favorably for full-face and lip-sync realism, though naturally the real tracked face remains stronger for upper-face realism.

Qualitative behavior and applications

The qualitative examples emphasize two recurring behaviors: lip shapes remain aligned with speech across identities, while upper-face motion such as eyebrow raises and blinks varies across sequences. The authors argue that this diversity is a consequence of the latent disentanglement rather than a side effect.

The paper also shows two downstream uses that benefit from the same disentangled representation:

• Retargeting: encode motion from a source identity and decode it onto a different target template mesh. The paper states that no autoregressive model is needed for this task.
• Mesh dubbing: re-synthesize the facial motion for a new audio track, adapting the lip motion while preserving upper-face behaviors such as blinks from the original clip.

Limitations

The paper is explicit about several limitations. First, the model uses audio that extends 100 ms beyond the current visual frame, which introduces an inherent 100 ms latency. This improves lip-sync quality, particularly for closure-heavy sounds such as /p/, but makes the system unsuitable for online applications.

Second, the model is not real-time on low-cost commodity hardware such as a laptop CPU or VR devices. The authors suggest that the computational cost could be improved in future work, but they do not claim real-time operation.

Third, the method depends on the quality of the face tracker. If hair occludes eyebrows or eyes and the tracker fails on those regions, the model cannot correctly learn their correlation with audio. This is especially relevant for the very regions that the paper wants to model realistically: upper-face motion and eyelid dynamics.

Conclusion

MeshTalk’s main contribution is a practical recipe for full-face speech-driven 3D animation that combines a discrete latent code, cross-modality disentanglement, and an audio-conditioned autoregressive generator. The paper shows that these design choices are not merely architectural preferences: they materially improve latent structure, speech conditioning, lip accuracy, and perceived realism relative to the tested alternatives. The strongest empirical story is that a categorical latent space trained with a cross-modality objective can preserve expressiveness while making the audio-conditioned generator more confident and more realistic.

In the authors’ framing, the method is a step toward more natural avatar animation in settings such as dubbing, games, e-commerce, and telepresence, especially where accurate facial motion must be synthesized from speech rather than captured directly.