CodeTalker

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

CodeTalker creates realistic 3D facial animations from speech by mapping audio to discrete motion codes learned from real facial motions. This reduces over-smoothing, improves expression vividness, and achieves accurate lip-sync and natural upper-face dynamics, surpassing prior regression-based methods.

• speech-driven
• face-animation
• 3d-avatar
• lip-sync
• autoregressive
• audio-driven
• talking-face

Demos

The demos illustrate CodeTalker's novel approach to speech-driven 3D facial animation, showcasing its ability to generate vivid, realistic facial motions with accurate lip-sync and expressive dynamics from audio input. Watch for comparisons against prior methods highlighting smoother yet more lifelike animation sequences, as well as visualizations of the discrete motion codebook and the autoregressive synthesis process that underlie these improvements.

Authors: Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, Tien-Tsin Wong

Categories: cs.CV

Comment: CVPR2023 Camera-Ready. Project Page: https://doubiiu.github.io/projects/codetalker/, Code: https://github.com/Doubiiu/CodeTalker

Published 2023-01-06 · Updated 2023-04-03

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub619 stars21 watchers68 forksMIT

Impact

• Semantic Scholar244 citations84 influential citations

Abstract

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality.

---

1. Problem Setting and Core Idea

CodeTalker addresses speech-driven 3D facial animation: given an audio sequence, generate a temporally coherent sequence of 3D face motions that can animate an arbitrary neutral mesh with accurate lip synchronization and plausible upper-face expression. The paper frames the central difficulty as an ill-posed cross-modal mapping problem. A single speech segment can correspond to multiple valid facial motion sequences, so direct regression tends to average over plausible outputs and produce over-smoothed motions, especially in the weakly correlated upper face.

The main technical move is to replace direct continuous regression with a discrete motion prior. The authors first learn a codebook of motion primitives from real facial motion sequences using a vector-quantized autoencoder, then cast animation as a code-query problem in that discrete space. A speech-conditioned autoregressive transformer predicts motion features sequentially, and these are quantized against the learned codebook before being decoded back to mesh motions. The claim is that the finite proxy space reduces ambiguity, preserves realism, and still allows expressive synthesis because the codebook and decoder jointly encode realistic motion priors.

2. Method Overview

The system has two stages. In stage one, a transformer-based VQ-VAE learns a discrete facial motion space by self-reconstruction of ground-truth motion sequences. In stage two, a speech encoder plus cross-modal autoregressive decoder predicts the sequence of motion tokens conditioned on audio, past generated motion, and a style vector. The pre-trained codebook and motion decoder are frozen in stage two.

Let $\mathbf{M}_{1:T}=(\mathbf{m}_1,\dots,\mathbf{m}_T)$ be the motion sequence, where each frame $\mathbf{m}_t \in \mathbb{R}^{V\times 3}$ is the 3D vertex displacement relative to a neutral mesh template. Let $\mathbf{A}_{1:T}=(\mathbf{a}_1,\dots,\mathbf{a}_T)$ be the aligned speech snippets. The goal is to synthesize $\mathbf{M}_{1:T}$ from $\mathbf{A}_{1:T}$ so that a neutral face can be animated as $\mathbf{H}_{1:T}=\{\mathbf{m}_t+\mathbf{h}\}_{t=1}^T$.

3. Discrete Facial Motion Space

The discrete prior is learned with a codebook $\mathcal{Z}=\{\mathbf{z}_k\in\mathbb{R}^C\}_{k=1}^N$ that represents reusable motion primitives. The motion encoder maps a sequence to latent features $\hat{\mathbf{Z}}=E(\mathbf{M}_{1:T})\in\mathbb{R}^{T'\times H\times C}$, where $T'$ is the number of temporal units and $H$ is the number of face components. Each latent token is quantized to its nearest codebook entry:

$$ \mathbf{Z}_{q}=Q(\hat{\mathbf{Z}})=\arg\min_{\mathbf{z}_k\in\mathcal{Z}} \|\hat{\mathbf{z}}_t-\mathbf{z}_k\|_2. $$

The decoder reconstructs the motion as $\hat{\mathbf{M}}_{1:T}=D(\mathbf{Z}_q)$, and the representation is intentionally discrete to reduce mapping uncertainty. The authors emphasize that this discreteness does not eliminate expressiveness because the codebook is context-rich and the decoder reconstructs motion in a high-capacity latent space.

The paper contrasts this design with prior discrete facial-expression approaches such as Learn2Listen, whose codebook is speaker-specific and sequence-based in a 3DMM coefficient space. CodeTalker instead learns a speaker-agnostic generic motion prior directly in vertex/motion space, so each code corresponds to motion primitives of facial components rather than to a fixed speaker’s expression sequence.

3.1 Stage-one training objective

The VQ-VAE is trained with a reconstruction term and two codebook-related terms:

$$ \mathcal{L}_{\mathrm{VQ}} = \|\mathbf{M}_{1:T}-\hat{\mathbf{M}}_{1:T}\|_1 + \|\operatorname{sg}(\hat{\mathbf{Z}})-\mathbf{Z}_q\|_2^2 + \beta \|\hat{\mathbf{Z}}-\operatorname{sg}(\mathbf{Z}_q)\|_2^2. $$

Here $\operatorname{sg}(\cdot)$ denotes stop-gradient, and the straight-through estimator is used because quantization is not differentiable. The first term supervises motion reconstruction; the latter two update the codebook and encoder toward each other.

4. Speech-Driven Motion Synthesis

The second stage learns a cross-modal mapping from audio to motion codes. The speech branch uses the architecture of wav2vec 2.0: a temporal convolutional feature extractor followed by a transformer encoder. The motion branch uses an autoregressive transformer decoder with causal self-attention and cross-attention to the encoded speech.

A style vector $\mathbf{s}\in\mathbb{R}_+^M\cup\{0\}$ modulates speaking style. The embedding block combines past motions and style through

$$ \mathbf{F}_{\mathrm{emb}}^{1:t-1}=\mathcal{P}_\theta(\hat{\mathbf{M}}_{1:t-1}) + \mathbf{B}\cdot\frac{\mathbf{s}}{\|\mathbf{s}\|_1}, $$

where $\mathbf{B}=[\mathbf{b}_1,\dots,\mathbf{b}_M]$ is a learned basis spanning the style space. The decoder predicts a latent motion feature sequence, quantizes it with the frozen codebook, decodes it into a motion sequence, and recursively feeds the newly generated motion back as history for the next step.

Formally, the recursive synthesis is written as

$$ \hat{\mathbf{m}}_t = D_{\mathrm{cross\text{-}modal}}\big(E_{\mathrm{speech}}(\mathbf{A}_{1:T}),\mathbf{s},\hat{\mathbf{M}}_{1:t-1}\big). $$

4.1 Stage-two objective

Training uses teacher forcing with two losses: a feature regularity term that pulls predicted latent features toward the quantized target codes, and a motion reconstruction term that matches predicted and ground-truth motions.

$$ \mathcal{L}_{\mathrm{syn}} = \|\hat{\mathbf{Z}}^{1:T}-\operatorname{sg}(\mathbf{Z}_q^{1:T})\|_2^2 + \|\hat{\mathbf{M}}_{1:T}-\mathbf{M}_{1:T}\|_2^2. $$

The frozen codebook and decoder make the stage-two problem easier: the model only needs to predict into a learned finite proxy space instead of directly regressing dense vertex offsets.

4.2 Architecture and implementation details

The appendix specifies a transformer-based implementation. In stage one, motion is reshaped from $T\times V\times 3$ to $T\times (3V)$, passed through linear and 1D convolution layers, instance normalization, and a transformer encoder, then quantized into $T'\times H\times C$. The decoder mirrors this process. In stage two, the speech encoder uses multiple 1D convolution layers with group normalization and GeLU, followed by interpolation to align audio features with visual frames and a transformer encoder. The cross-modal decoder uses a transformer decoder with cross-attention to the speech features, then quantizes and decodes the result with the frozen VQ-VAE decoder.

The reported codebook settings are $N=256$ items, with $P=1$ frame per temporal unit. On BIWI, the authors use $H=8$ and $C=128$; on VOCASET, they use $H=16$ and $C=64$. The speech encoder is initialized from pre-trained wav2vec 2.0 weights.

5. Datasets, Evaluation Protocol, and Baselines

The paper evaluates on two widely used paired audio-visual 3D face datasets: BIWI and VOCASET. Both contain 4D face scans and English utterances. BIWI includes 40 unique sentences shared across speakers and 14 subjects in total. The emotional subset is used, with the data split following FaceFormer: 192 training sentences, 24 validation sentences, BIWI-Test-A with 24 sentences from six seen subjects, and BIWI-Test-B with 32 sentences from eight unseen subjects. VOCASET contains 480 paired sequences from 12 subjects, with 255 unique sentences, and the same train/validation/test split as VOCA and FaceFormer is used.

The compared methods are VOCA, MeshTalk, and FaceFormer. VOCA is trained on BIWI with the official codebase and directly tested with the released model on VOCASET. MeshTalk is trained and tested with the official implementation on both datasets. FaceFormer is evaluated with pre-trained weights. For VOCA, FaceFormer, and CodeTalker, the test-time protocol conditions on a training speaking style; for unseen subjects, animations are generated by conditioning on all training styles.

5.1 Metrics

The paper uses two quantitative measures.

• Lip vertex error: the maximal $\ell_2$ error across all lip vertices in each frame, averaged over frames; lower is better.
• Upper-face dynamics deviation (FDD): a sequence-level measure of how much the temporal variation in the upper face deviates from ground truth. For vertex $v$ in the upper-face set $\mathcal{S}_U$, $\operatorname{dyn}(\cdot)$ is the standard deviation of the per-frame $\ell_2$ norm over time, and FDD averages the difference over all upper-face vertices.

$$ \operatorname{FDD}(\mathbf{M}_{1:T},\hat{\mathbf{M}}_{1:T}) = \frac{1}{|\mathcal{S}_U|}\sum_{v\in\mathcal{S}_U}\left(\operatorname{dyn}(\mathbf{M}^v_{1:T})-\operatorname{dyn}(\hat{\mathbf{M}}^v_{1:T})\right). $$

6. Main Quantitative Results

On BIWI-Test-A, CodeTalker achieves the best values on both reported metrics, confirming improved lip synchronization and better preservation of upper-face motion statistics.

The paper also notes that CodeTalker qualitatively improves bilabial closures and challenging pouting mouth shapes, where baseline methods often remain over-smoothed. In the visual comparison figure, the authors point to examples such as /b/ in “bedside,” and speech segments such as “waterproof” and “shaving,” where CodeTalker produces more accurate lip articulation than VOCA and MeshTalk.

The lower-part temporal statistics in the figure are used to argue that CodeTalker yields stronger motions and richer dynamics than the baselines, especially for non-lip regions. This is consistent with the authors’ explanation that the discrete motion space encourages more varied yet plausible expressions instead of collapsing to average facial states.

7. Qualitative Style Control

The style space is not restricted to one-hot speaker IDs. Instead, a style vector is modeled as coefficients over a learned basis $\mathbf{B}$. The paper demonstrates linear interpolation between two style vectors $\mathbf{e}_i$ and $\mathbf{e}_j$ as

$$ \mathbf{s}_{\mathrm{new}} = \mathbf{B}\,[\omega \mathbf{e}_i + (1-\omega)\mathbf{e}_j], $$

and reports a smooth transition of mouth amplitudes as $\omega$ changes. This is presented as evidence that the model can synthesize new talking styles and, in principle, adapt to unseen speaking performances by interpolating in the style space.

8. User Study

Because the task is ultimately perceptual, the paper performs A/B user studies on BIWI-Test-B and VOCA-Test. For each dataset, 30 samples are randomly selected, producing 120 comparison pairs across four conditions. Each pair is judged by at least three participants, and 31 participants with normal vision and hearing complete the study. The authors filter out responses completed in less than two minutes.

The interface asks two questions for each pair: which face is more lip-synced with the audio, and which full face looks more realistic. The paper reports the percentage of trials in which the first sample in the pair is preferred; in the stated comparisons this corresponds to CodeTalker against each competitor.

The strongest preferences are against VOCA and MeshTalk, while the margins over FaceFormer are more modest on BIWI-Test-B. Against ground truth, preference remains below 50%, which the authors treat as evidence that perceptual quality is still below real recordings even though the gap has been reduced.

9. Ablation Studies

The ablation analysis focuses on three questions: what representation space is used for the codebook, how codebook hyperparameters affect the learned prior, and whether the particular token-query formulation matters relative to a direct code-classification setup.

9.1 Motion-space vs. shape-entangled codebook

The baseline learns a shape-entangled codebook, where the code represents a full shape state $\mathbf{m}_t+\mathbf{h}$ rather than a motion offset $\mathbf{m}_t$. This significantly hurts reconstruction and downstream lip synchronization because the codes become more speaker-specific and less reusable. In contrast, the proposed motion-based representation is speaker-agnostic and better captures generic motion priors.

9.2 Codebook construction hyperparameters

The paper varies the temporal unit length $P$ and the number of face components $H$. The observed trend is that increasing $P$ makes reconstruction harder because each token must represent a longer and more complex motion primitive. Increasing $H$ helps reconstruction up to a point, but too many components risk overfitting; the authors report that performance degrades when $H\geq 8$ in their setting. The same trend appears in lip vertex error, which supports the claim that overly complex primitives reduce code reusability and increase ambiguity in code querying.

9.3 Instance normalization in stage one

The appendix reports that adding instance normalization improves self-reconstruction and stabilizes codebook learning. The authors suggest that normalizing temporal statistics within channels helps encapsulate motions of different magnitudes into the discrete prior. They explicitly note that batch normalization is not suitable because the minibatch size is very small. The ablation reports reconstruction errors of $0.12$ and $3.27$ without instance normalization on VOCA-Test and BIWI-Test-A, versus $0.08$ and $2.83$ with instance normalization.

9.4 Alternative data flow and supervision

The appendix tests a different formulation where the decoder autoregressively predicts discrete code indices with categorical cross-entropy, then decodes them through the frozen codebook and decoder. This setup performs worse, especially when trained with cross-entropy alone, because direct code classification is still difficult under the audio-to-motion ambiguity. Adding the regularization and motion losses helps but does not close the gap to the proposed feature-query formulation.

The reported lip vertex errors on BIWI-Test-A are $9.6356$ for cross-entropy alone, $5.1138$ for $\lambda \mathcal{L}_{\mathrm{ce}} + \mathcal{L}_{\mathrm{reg}}$, $5.0254$ for $\lambda \mathcal{L}_{\mathrm{ce}} + \mathcal{L}_{\mathrm{reg}} + \mathcal{L}_{\mathrm{motion}}$, and $4.7914$ for CodeTalker.

10. Main Takeaways

The paper’s central empirical conclusion is that learning a discrete motion prior is more effective than directly regressing facial geometry for speech-driven animation. The codebook reduces the uncertainty of the cross-modal mapping, and the autoregressive decoder preserves temporal coherence while remaining controllable through style embeddings. In the reported experiments, this yields the best lip-sync metrics, stronger upper-face dynamics, and better user preference than the baselines.

The authors also argue that the discrete prior is not merely a compression trick: because it is learned from real motions, it captures realistic facial motion priors and provides reusable motion primitives that generalize better than shape-entangled representations.

11. Limitations and Discussion

The conclusion is unusually explicit about limitations. First, the method assumes facial motions are independent of face shape, and the authors note that this assumption may deserve further study. Second, the overall perceptual quality still lags behind ground truth, which they attribute primarily to the scarcity of paired audio-visual training data. Third, the learned generic motion prior is bounded by the training distribution and may therefore deviate from real-world facial motions.

The stated future direction is to leverage priors from large-scale talking-head videos to guide 3D facial animation, which the authors expect would improve realism beyond what is achievable from the limited paired 3D datasets used here.

12. Concise Conclusion

CodeTalker reframes speech-driven 3D facial animation as discrete code querying over a learned motion prior. Its two-stage design combines VQ-VAE motion-space learning with an autoregressive audio-conditioned predictor, and the reported experiments on BIWI and VOCASET show consistent improvements in lip synchronization, expression dynamics, and subjective preference over VOCA, MeshTalk, and FaceFormer.