FaceFormer

Introduction and Problem Setting

FaceFormer addresses speech-driven 3D facial animation: generating a time sequence of 3D face meshes from raw audio. The paper focuses on the difficult case where the goal is not a 2D talking-head video, but a 3D geometry sequence suitable for VR, games, and other applications that directly manipulate 3D avatars. The authors argue that existing 3D methods are limited by two core issues:

• Short audio context: many prior methods process short windows of audio, which makes lip motion ambiguous and can miss longer-range phonetic/coarticulatory cues.
• Data scarcity: high-quality 3D audio-visual motion capture is expensive and limited, so end-to-end models can be under-trained without stronger priors or pretraining.

The proposed solution is FaceFormer, an autoregressive Transformer encoder-decoder that models long audio context and predicts a facial motion sequence one step at a time. It explicitly conditions on both the input speech and the history of generated facial motions, which the paper frames as necessary for temporally stable animation and more accurate mouth movements. A key design choice is to integrate self-supervised pre-trained speech representations from wav2vec 2.0 to mitigate the data shortage problem.

The main conceptual contributions are: (1) an autoregressive Transformer for 3D facial motion synthesis; (2) biased attention mechanisms that help align audio and motion while generalizing to longer sequences; and (3) use of pre-trained speech features for data efficiency. The paper reports that these choices improve lip synchronization, visual realism, and robustness on both seen and unseen speakers.

Method Overview

FaceFormer is formulated as a sequence-to-sequence model. Given raw audio $\mathcal{X}$ and a target mesh sequence $\mathbf{Y}_T = (y_1, \dots, y_T)$, the model predicts an output sequence $\hat{\mathbf{Y}}_T = (\hat{y}_1, \dots, \hat{y}_T)$ autoregressively. At step $t$, the prediction depends on the past generated motion and a speaker/style embedding:

$$ \hat{y}_t = \mathrm{FaceFormer}_\theta(\hat{y}_{<t}, s_n, \mathcal{X}). $$

The architecture has two major parts:

• Encoder: converts raw audio into contextualized speech features using a wav2vec 2.0-style stack.
• Decoder: predicts face motion autoregressively using motion history, a speaker identity embedding, a periodic positional encoding, biased causal self-attention, and biased cross-modal attention.

The overall design is specifically shaped for facial animation rather than generic seq2seq generation. The paper emphasizes three properties: long-range audio context, explicit audio-motion alignment, and temporal stability from motion history.

Encoder: wav2vec 2.0 Features and Audio Resampling

The encoder follows the general structure of wav2vec 2.0. Raw waveform input is passed through a temporal convolutional feature extractor, then through a multi-layer Transformer encoder. The paper’s rationale is that self-supervised pretraining on large speech corpora provides richer phonetic information than training from scratch on limited 3D facial animation data.

Concretely, the audio front end produces features at frequency $f_a$, while the motion data may be captured at a different frame rate $f_m$. To align these two modalities, the paper inserts a linear interpolation layer after the temporal convolutions. This resamples the audio sequence to length $kT$, where $k = \lceil f_a / f_m \rceil$ and $T$ is the number of motion frames. The resulting speech feature sequence is written as $\mathbf{A}_{kT} = (a_1, \dots, a_{kT})$.

The encoder is initialized from pretrained wav2vec 2.0 weights, and a new linear projection layer is added on top. During training, the temporal convolutional network (TCN) parameters are frozen, while the rest of the model is learnable. The paper reports that this initialization is important: removing it harms lip synchronization and increases jitter.

The role of the encoder is not merely feature extraction, but contextualization. Because the Transformer encoder uses self-attention over the full resampled audio stream, it can exploit both short- and long-range phonetic context, which the authors argue is necessary for realistic full-face motion, not only mouth articulation.

Decoder: Autoregressive Motion Generation with Biased Attention

The decoder predicts facial motion one frame at a time from previous motion, speaker identity, and audio features. The paper uses a single decoder layer with two custom attention modules: biased causal multi-head self-attention over motion history, and biased cross-modal multi-head attention to connect motion queries to audio keys and values.

Periodic Positional Encoding

The authors found that standard sinusoidal positional encoding does not generalize well to longer sequences, while a direct ALiBi-style replacement can cause the decoder to collapse toward static expressions because it injects bias into attention scores without adding explicit position information to the input representations. Their solution is a periodic positional encoding (PPE), which reuses sinusoidal encodings modulo a period $p$:

$$ \operatorname{PPE}(t, 2i) = \sin\left((t \bmod p)/10000^{2i/d}\right), \qquad \operatorname{PPE}(t, 2i+1) = \cos\left((t \bmod p)/10000^{2i/d}\right). $$

Here $t$ is the current time step, $i$ the channel index, $d$ the model dimension, and $p$ the chosen period. The design injects temporal order information repeatedly within each period, which the paper argues is better aligned with facial motion sequences that often exhibit smooth local continuity.

Before PPE is added, the previous motion prediction is mapped through a motion encoder and combined with a style embedding for the speaker identity. At the first step, the decoder starts from the style embedding alone:

$$ f_t = \begin{cases} (W^f \hat{y}_{t-1} + b^f) + s_n, & 1 < t \le T, \\ s_n, & t = 1. \end{cases} $$

Then the temporally encoded state is $\hat{f}_t = f_t + \operatorname{PPE}(t)$.

Biased Causal Multi-Head Self-Attention

For motion history, the decoder applies a causal attention mask so that frame $t$ cannot see future frames. The paper additionally introduces a periodic temporal bias inspired by ALiBi, but extended with a period $p$:

$$ B^{\hat{F}}(i,j) = \begin{cases} \lfloor (i-j)/p \rfloor, & j \le i, \\ -\infty, & \text{otherwise}. \end{cases} $$

The idea is that closer periods of past motion should receive higher attention weight than farther ones. The authors explicitly interpret this as a generalization of ALiBi, with ALiBi recovered when $p = 1$. For each attention head, a fixed head-specific slope is used, following the ALiBi family of designs.

The scaled dot-product attention used here is:

$$ \operatorname{Att}(Q, K, V, B) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + B\right)V. $$

The multi-head output is produced by concatenating the heads and projecting them with a learned matrix.

Biased Cross-Modal Multi-Head Attention

To align speech and motion, FaceFormer uses a second biased attention module in which the motion representation is the query and the audio representation provides keys and values. The alignment bias constrains each motion step to attend to a corresponding local audio segment:

$$ B^A(i,j) = \begin{cases} 0, & ki \le j < k(i+1), \\ -\infty, & \text{otherwise}. \end{cases} $$

This means motion frame $i$ can only attend to the block of audio features aligned with that frame. The paper argues that this is a simple but effective way to improve audio-motion synchronization. The same multi-head formulation is extended to this cross-modal setting.

The final hidden state is projected back to the 3D mesh vertex space with a motion decoder, producing the next predicted mesh frame $\hat{y}_t$.

Training Objective and Inference

Training uses an autoregressive scheme rather than teacher forcing. The authors state that this less guided setup performs better in their experiments. The model is optimized with mean squared error between the predicted and ground-truth vertex sequences:

$$ \mathcal{L}_{\mathrm{MSE}} = \sum_{t=1}^{T} \sum_{v=1}^{V} \|\hat{y}_{t,v} - y_{t,v}\|^2, $$

where $V$ is the number of vertices in the 3D mesh representation. At inference time the model autoregressively rolls out a full sequence from the input audio and the speaker identity. The paper notes that changing the one-hot identity embedding can alter the style of the generated motion.

Implementation Details Reported in the Paper

The supplement clarifies that the decoder’s periodic positional encodings have the same dimensionality as the motion encoder so they can be summed. Residual connections and layer normalization are used around the attention and feed-forward sublayers, consistent with standard Transformer blocks.

Datasets and Experimental Protocol

The paper evaluates FaceFormer on two public 3D audio-visual datasets: BIWI and VOCASET. Both contain English speech paired with dense 3D face motion, but BIWI is described as more challenging for lip synchronization because it covers fewer phonemes.

BIWI

BIWI contains 14 subjects reading 40 English sentences, each recorded twice, once in a neutral and once in an emotional context. The 3D face geometry is captured at 25 fps with 23,370 vertices per mesh. The paper uses the emotional subset. The split is:

• BIWI-Train: 192 sentences from 6 subjects.
• BIWI-Val: 24 sentences from 6 subjects.
• BIWI-Test-A: 24 sentences from 6 seen subjects.
• BIWI-Test-B: 32 sentences from 8 unseen subjects.

VOCASET

VOCASET consists of 480 facial motion sequences from 12 subjects. Each sequence is 3 to 4 seconds long, captured at 60 fps, with 5,023 vertices per mesh. The authors follow the same train/validation/test split as the original VOCA paper and refer to them as VOCA-Train, VOCA-Val, and VOCA-Test.

Baselines and Evaluation

FaceFormer is compared against VOCA and MeshTalk. The paper notes that VOCA and FaceFormer require a speaker identity during inference; for unseen subjects, predictions are obtained by conditioning on all training identities. The primary quantitative metric is the average lip vertex error used by MeshTalk, defined as the maximum $L_2$ error over lip vertices per frame, averaged across sequences. In addition, the authors perform user studies on Amazon Mechanical Turk to assess realism and lip synchronization.

Quantitative Results

On BIWI-Test-A, FaceFormer achieves the best lip synchronization score among the compared methods.

The paper interprets the lower lip error as evidence that the model better captures mouth closures and fine-grained articulation, especially for difficult phonemes such as /b/, /m/, and /p/.

Perceptual User Study Results

The authors emphasize that user study results are important because facial animation quality, especially for the upper face, is not fully captured by a single numerical metric. They therefore conduct pairwise A/B preference tests on Amazon Mechanical Turk.

BIWI-Test-B

For BIWI, each method is conditioned on all six training identities, yielding 192 videos per method and 576 A/B pairs. Three Turkers evaluate each pair. The results below report the percentage of times the first item in the pair is preferred over the second:

The paper notes that Turkers prefer FaceFormer over both baselines on realism and lip sync, while the ground truth is still preferred over FaceFormer, as expected.

VOCA-Test

For VOCA, the authors sample 10 sentences from VOCA-Test and again condition on all training identities, producing 80 videos per method and 240 A/B pairs. The reported preferences are:

Again, FaceFormer is preferred over the baselines, but the ground truth retains the advantage in both realism and lip sync.

Attention Visualization and Behavioral Analysis

The paper includes a visualization of attention weights to interpret what the model learns. For a 100-frame BIWI test sequence, the encoder self-attention attends both locally and globally: there is a strong diagonal pattern, but also nontrivial attention to farther past and future audio frames. This supports the claim that the encoder captures long-term speech context rather than only short phonetic windows.

For the decoder’s biased causal self-attention, the visualization shows that the most recent motion periods receive the highest weights, matching the design of the periodic temporal bias. This is consistent with the intuition that facial expressions are temporally smooth and that near-past expressions are informative for current motion.

The authors also compare positional encoding strategies. The paper reports that the original sinusoidal positional encoding can produce correct mouth closures but may jitter during silent intervals, especially when test sequences are longer than those seen in training. Pure ALiBi can cause the motion to freeze into a static expression on VOCASET because it lacks explicit positional information in the input state. The proposed TB + PPE combination is therefore presented as the best compromise between sequence-length generalization and temporal expressiveness.

Ablation Findings

The paper’s ablation discussion is qualitative, but it is detailed enough to isolate the role of each design choice.

• Removing the encoder self-attention and keeping only the pre-trained TCN plus the decoder leads to poor mouth closure and temporal jitter, indicating that the Transformer encoder is needed for contextual audio modeling.
• Removing wav2vec initialization degrades facial motion quality and lip synchronization. The model can converge to a poorer solution when trained from random initialization, reinforcing the value of self-supervised pretraining.
• Replacing the decoder with FC or LSTM yields less stable and less accurate lip motion than the autoregressive Transformer decoder.
• Removing the alignment bias in cross-modal attention causes muted facial expressions, showing that explicit audio-motion alignment is important.
• Positional encoding choice matters: original sinusoidal PE works reasonably on BIWI but jitters on longer or more difficult sequences; ALiBi can freeze on VOCASET; the proposed TB + PPE generalizes better to longer audio clips.

For long audio clips from TED videos, the user study reported that FaceFormer with TB + PPE is preferred over the original positional encoding variant, with 57.78% ± 16.78% preference for realism and 62.22% ± 10.18% for lip sync. The paper interprets this as evidence that the proposed positional strategy improves robustness to sequence lengths longer than those seen during training.

Qualitative Behavior Reported by the Authors

The supplementary video and discussion claim that FaceFormer produces more realistic and natural-looking facial animation than the compared methods, with especially improved mouth closures on bilabial phonemes such as /b/, /m/, and /p/. The model is also reported to animate both upper and lower face regions more convincingly than VOCA, which tends to focus motion mostly on the lower face. On VOCASET, where upper-face motion is scarce, accurate lip synchronization becomes especially important, and FaceFormer is reported to do better in that setting as well.

The authors also state that the model can generate different talking styles and even different languages when driven by the corresponding audio, although the core experiments are on English speech datasets.

Limitations, Ethics, and Conclusion

The main limitation acknowledged in the paper is computational: self-attention has quadratic memory and time complexity, which makes FaceFormer unsuitable for real-time applications in its current form. The authors suggest that more efficient attention mechanisms such as Linformer- or Longformer-style methods could be useful future directions.

The paper also includes an ethics note: collecting 3D scan data requires actor consent, and realistic face animation can be misused to create embarrassing or deceptive synthetic content. The authors explicitly encourage responsible use and continued research on misuse mitigation.

Overall, FaceFormer’s technical contribution is the combination of: a wav2vec-initialized audio encoder; autoregressive motion prediction; speaker/style conditioning; periodic positional encoding; periodic causal bias in motion self-attention; and a strict alignment bias in cross-modal attention. The empirical results on BIWI and VOCASET show consistent improvements in lip synchronization and perceptual realism over VOCA and MeshTalk, with the strongest gains appearing in cases where audio context, long-sequence generalization, or subtle mouth closure matters most.