CapTalk

CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation

CapTalk is a framework for audio-driven 3D head animation that enables separate, text-guided control of speaking style and emotion. It generates real-time, expressive 3D facial animations with synchronized lip movements, adapting styles dynamically from natural language and audio inputs.

• talking-head
• speech-driven
• 3d-avatar
• emotion
• head-pose
• face-animation
• autoregressive
• realtime

Authors: Xuangeng Chu, Yuan Gan, Ziteng Cui, Shuhong Liu, Jian Wang, Bing Zhou, Tatsuya Harada

Categories: cs.CV

Published 2026-05-28 · Updated 2026-05-28

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips. While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users' ability to freely control speaking styles. Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio. To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion. Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions. Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.

---

Introduction

CapTalk addresses a core limitation in speech-driven 3D head animation: most prior systems can synchronize lip motion with speech, but they usually expose only fixed identity/style controls or require reference videos to extract style embeddings. As a result, they are awkward to use for conversational AI workflows where a user may want to specify how a character speaks in natural language, and where the desired speaking style may need to change over the course of a longer utterance.

The paper’s key idea is to separate speaking style from character emotion and control both with text captions, alongside the driving audio. The authors explicitly define style in terms of three factors: mouth movement amplitude, head movement amplitude, and emotion. This framing is important because it moves style control away from latent vectors derived from specific identities or reference motions, and toward a more user-facing caption interface.

To support this interface, the authors build a large-scale in-the-wild dataset with paired style and emotion annotations and then train a two-stage model: first a multi-scale discrete motion codec over FLAME parameters, then a time-windowed autoregressive generator that predicts motion codes from audio plus text. The overall system is designed for real-time generation and supports dynamic emotion/style changes across windows during inference.

The main contributions, as stated in the paper, are:

• the first model in this setting that directly exposes textual control over both speaking style and emotion for 3D head motion generation;
• a large-scale 3D facial motion dataset with both style and emotion captions, collected from in-the-wild video;
• a multi-scale codec and caption-conditioned autoregressive generator that preserves lip synchronization while improving controllability of expressions and head motion.

Dataset: CapTalkingHead

The dataset is central to the paper’s framing. Rather than relying on identity-specific capture sessions, the authors derive data from the TalkingHead1KH corpus of roughly 1,000 hours of raw YouTube videos released under a Creative Commons license. They process this source through a multi-stage pipeline: detect, track, and crop face sequences longer than 8 seconds; filter clips with poor audio-visual synchronization using SyncNet; extract FLAME parameters using a hybrid MICA+EMOCA-based pipeline; and annotate each segment with both style and emotion captions using language models.

Style captions are generated from the video stream using a finetuned Qwen2.5-VL 7B model that is prompted to focus on human appearance, especially head shape, mouth opening size, and head movement amplitude, while ignoring background context. Emotion captions are generated from the audio stream using Qwen-Audio-Chat, which outputs one label from $ {\text{angry}, \text{disgust}, \text{contempt}, \text{fear}, \text{happy}, \text{sad}, \text{surprised}, \text{neutral}} $. The paper emphasizes that audio and video are complementary here: audio reveals prosody and emotional tone, while video reveals visible motion style.

After processing, CapTalkingHead contains 24,441 clips, about 200.8 hours of footage, and 18,074,445 frames at 25 fps with 16 kHz audio. The average clip length is about 29 seconds. Every clip is paired with a style description, an emotion annotation, and FLAME motion parameters.

The supplementary material positions this dataset as the largest dataset in the paper’s comparison table with both 3D motion annotations and style text annotations.

Method

CapTalk uses a two-stage pipeline. First, it learns a compact discrete representation of motion with a multi-scale codec over FLAME motion parameters. Second, it trains an autoregressive Transformer to generate those discrete motion codes from audio, style captions, and emotion captions. The design choices are motivated by two constraints: speech-driven facial motion is temporally dense, and style control needs to be fine-grained without losing lip synchronization.

Motion representation and FLAME preprocessing

The paper represents a face with FLAME, where the motion sequence $M$ is formed by concatenating expression and pose parameters over $N$ frames. The supplementary explicitly writes the FLAME head model as:

$$T_P(\hat{\beta}, \hat{\theta}, \hat{\psi}) = \bar{T} + BS(\hat{\beta}; S) + BP(\hat{\theta}; P) + BE(\hat{\psi}; E).$$

Here, $\bar{T}$ is the template mesh, $BS$ captures identity-dependent shape variation, $BP$ captures pose corrections such as jaw and neck deformation, and $BE$ captures expression variation such as smiling and eye closure. The paper’s practical motivation is that FLAME reduces the complexity of direct mesh prediction while staying compatible with many downstream head-animation pipelines.

Multi-scale binary spherical codec

To turn continuous motion into a modeling-friendly sequence, CapTalk trains a multi-scale binary codec. Given a motion window of $N$ frames, the encoder maps it to a latent sequence, then quantizes that latent sequence into discrete codes at multiple temporal resolutions using binary spherical quantization. The code hierarchy is built by resizing the latent sequence to each target temporal length, quantizing it, and subtracting the residual before moving to the next scale. During decoding, each scale is upsampled back to the full window length and summed, after which a Transformer decoder reconstructs the motion.

The model uses a 32-bit code dimension and a window length of 100 frames, which corresponds to 4 seconds at the dataset’s 25 fps sampling rate. The multi-scale code lengths are $[1, 5, 25, 50, 100]$, so the hierarchy captures both coarse window-level context and frame-level detail.

The codec is trained with a hybrid loss:

$$L_{\text{codec}} = \|\hat{M}_n - M_n\|_1 + w_{\text{full}} \|\hat{V} - V\|_2^2 + w_{\text{lips}} \|\hat{V}_{\text{lips}} - V_{\text{lips}}\|_2^2 + L_{\text{vq}}.$$

In this expression, $M_n$ and $\hat{M}_n$ are the input and reconstructed FLAME motion sequences, $V$ and $\hat{V}$ are the full head meshes, $V_{\text{lips}}$ and $\hat{V}_{\text{lips}}$ are the lip-region vertices, and $L_{\text{vq}}$ stabilizes the binary quantization. The use of both full-mesh and lip-region terms reflects the paper’s emphasis on preserving both global motion fidelity and lip detail.

Speech-to-motion autoregressive model with captions

Once the codec is trained, CapTalk autoregressively predicts motion codes in the discrete space. The generator operates within and across time windows, so it can preserve continuity across longer speeches rather than treating each window independently. The paper argues that this windowed design is important for synthesizing coherent motions with sufficient temporal receptive field.

Audio features are extracted using a frozen multilingual wav2vec 2.0 encoder. Text features for style and emotion are extracted with a frozen T5 encoder. The generator combines modalities using self-attention and cross-attention layers. A key design choice is how positional information is handled: audio is injected with Rotary Position Embedding so that each code position aligns precisely with its corresponding audio features, whereas the text captions are treated as time-insensitive global conditioning and therefore are not aligned position-by-position with the code sequence.

In effect, audio provides the time-synchronized content stream, while style and emotion captions provide a less temporally specific control signal that can influence all code levels in the current window. The model also conditions on previous action information, so it can maintain motion continuity even if the text description changes between windows. During inference, a new window starts from a start token $S$ after the current window is completed.

Training uses cross-entropy supervision on the discrete codes. To improve robustness and controllability, the authors introduce label perturbations: binary codes are flipped with probability $0.1$, and previous actions, style captions, or emotion captions are independently dropped with probability $0.1$.

Optimization and implementation details

The training procedure is two-stage. In stage 1, the codec is trained for 100,000 iterations with AdamW, learning rate $10^{-4}$, batch size 64, and 28 total GPU hours across both stages reported on a single NVIDIA Tesla A100. In stage 2, the autoregressive generator is trained with the same optimizer, learning rate, batch size, and iteration count. The paper states that the codec stage takes about 8 GPU hours and the autoregressive stage about 20 GPU hours.

The authors also state that they plan to release the core code for the codec and generative model, and that the code is intended to be reproducible from the supplementary details.

Experimental Setup

Evaluation is performed on the test split of the new CapTalkingHead dataset, and for generalization the authors also report results on the test split of the MEAD dataset. Importantly, the model is not trained or fine-tuned on MEAD; at inference, it uses the original MEAD emotion labels and does not receive additional style captions. This makes the MEAD setting a stronger test of out-of-domain generalization but also a weaker test of the full style-control interface.

The paper compares CapTalk against a mix of mesh-based and stylized baselines: FaceFormer, CodeTalker, SelfTalk, MultiTalk, UniTalker, DiffPoseTalk, and ARTalk. For methods that require a reference video or style clip, the authors follow the baseline’s native protocol and provide the first few seconds of the ground-truth motion clip as a style reference. For FaceFormer, CodeTalker, and SelfTalk, the corresponding mesh and the first speaker identity are used. For MultiTalk, the English style setting is used. For UniTalker, metrics are computed using meshes generated from generalized pivot identities.

The paper uses five objective metrics on CapTalkingHead: lip vertex error (LVE), mean head distance (MHD), upper-face dynamic deviation (FFD in the paper’s table notation), lip open dynamic deviation (LODD), and head pose dynamic deviation (HPDD). LVE measures the maximum per-frame lip-vertex error, MHD measures average full-head mesh difference, FFD measures temporal consistency of upper-face motion, LODD measures mouth-opening dynamics, and HPDD measures head-rotation dynamics. On MEAD, head-pose-related metrics are not reported because the dataset’s heads are often close to motionless.

Quantitative results on CapTalkingHead

On the CapTalkingHead test split, CapTalk is reported as the best method across all five metrics. Relative to the strongest baselines, it improves both synchronization and style control. The main takeaway is that the style captions are not just decorative metadata: they materially improve motion expressiveness, especially for mouth dynamics, upper-face motion, and head motion.

Note: the paper marks HPDD for methods without head-pose generation with an asterisk and computes it using zero head pose.

Generalization results on MEAD

On MEAD, CapTalk remains competitive despite not being trained on the dataset and despite receiving only emotion labels rather than full style captions. It achieves the best LVE among the listed methods and strong results on the other reported metrics, although ARTalk remains better on MHD and FFD, and DiffPoseTalk is better on LODD. The paper interprets this as evidence that CapTalk can generalize reasonably well, but also that emotion labels alone provide less style control than the full text interface used on CapTalkingHead.

Qualitative findings

The qualitative comparisons support the quantitative claims. The paper shows that CapTalk better matches the ground-truth expression style, mouth dynamics, and lip synchronization than prior methods when head pose is fixed. A separate qualitative figure demonstrates that, when certain words are stressed, the model generates plausible head movements aligned with the style description. The style-control figure further shows that keeping speech fixed while changing only the text prompts can alter the generated motion in a controlled way.

User study

The authors also conduct a pairwise user study with 13 respondents, each answering 132 questions across 38 comparison trials. Twenty comparisons were against fixed-head-pose baselines, and 18 were against dynamic-head-pose baselines (DiffPoseTalk and ARTalk). For each trial, the order of the two videos was randomized to reduce bias, and participants judged lip sync, style consistency, expression consistency, and, when applicable, pose consistency. The paper reports the fraction of participants who preferred CapTalk over each baseline.

Overall, the user study supports the paper’s claim that CapTalk improves not only objective synchronization but also perceived style naturalness, expression consistency, and head-pose consistency.

Ablation Study

The ablations focus on two questions: whether the T5 text encoder matters, and whether the model really needs both style and emotion captions. The paper compares the default T5-based design against a CLIP text encoder, and also removes captions selectively.

The main conclusions are straightforward:

• T5 slightly outperforms CLIP for text-driven style injection, although the margin is small.
• Removing all captions hurts the most, especially for style-sensitive metrics such as FFD, LODD, and HPDD, even though lip synchronization remains partially intact from audio alone.
• Style captions matter more than emotion captions for expressive control. The style-only variant is consistently stronger than the emotion-only variant on style-related metrics.

This matches the paper’s conceptual claim that mouth-opening and head-motion amplitude carry crucial style information that is not recoverable from audio alone, while emotion labels contribute useful but comparatively weaker guidance.

Limitations and Future Work

The paper is explicit about two limitations. First, the window-based generation scheme is not fully streamlined or perfectly continuous across long utterances, because motion is modeled in 4-second windows rather than as one uninterrupted sequence. Second, the model’s semantic understanding is limited, which means it may struggle with culturally specific or strongly context-dependent facial motions.

These limitations point to future work in two directions: more coherent long-form motion synthesis and richer semantic conditioning so that the generated talking head can better adapt to context and culture.

The supplementary also includes an ethics statement acknowledging the dual-use nature of synthetic talking-head generation and recommending clear disclosure such as watermarks or tags for generated media. That note is important for deployment, but the paper does not frame it as a technical limitation.

Conclusion

CapTalk’s central technical contribution is to make speech-driven 3D head animation directly controllable from text descriptions of style and emotion, rather than from identity latents or reference videos. The combination of a large annotated in-the-wild dataset, a multi-scale discrete motion codec, and a caption-conditioned autoregressive generator yields strong quantitative and qualitative gains on the authors’ new dataset, while remaining competitive on MEAD under reduced text conditioning. For talking-head and conversational-AI systems, the most relevant takeaway is that natural-language style control can be made practical without sacrificing synchronization, provided that the model is trained on rich motion-text supervision and uses a temporally structured discrete representation.