FantasyTalking

FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis

FantasyTalking creates realistic talking portraits from a single static image by synthesizing coherent facial, body, and background motion driven by audio. It uniquely uses dual-stage audio-visual alignment and motion intensity control to achieve natural, expressive animations beyond simple lip-sync.

• talking-head
• audio-driven
• face-animation
• portrait-animation
• lip-sync
• emotion
• head-pose

Authors: Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu

Categories: cs.CV

Published 2025-04-07 · Updated 2025-04-07

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads41 likes

Impact

• Semantic Scholar75 citations8 influential citations

Abstract

Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.

---

Introduction

FantasyTalking addresses the problem of generating a realistic, animatable talking portrait from a single static reference image, driven by audio and text. The paper focuses on a central limitation of prior talking-head systems: they often produce acceptable lip synchronization, but they under-model the rest of the scene. In particular, the authors argue that existing methods tend to neglect subtle facial expressions, global body motion, and background/context dynamics, resulting in videos that look constrained or static even when the mouth motion is synchronized.

The proposed system is built on a pretrained video diffusion transformer and is designed to generate talking portraits with coherent motion synthesis across the entire scene. The key idea is that audio should not only control the lips, but should also influence broader dynamics such as eyebrows, eyes, shoulders, and even contextual objects and background. To support this, the paper introduces a dual-stage audio-visual alignment strategy, a facial-focused identity preservation module, and a motion intensity modulation network for explicit control over expression and body-motion amplitude.

The authors position the method against both earlier 3D-intermediate approaches and recent end-to-end diffusion-based talking-head systems. Their argument is that video diffusion transformers, when adapted carefully, can simultaneously improve realism, identity consistency, and motion diversity, particularly in more challenging “wild” talking-head scenarios where the background and subject motion are less constrained.

Core Idea and Contributions

• Dual-stage audio-visual alignment: first learn coarse, clip-level global motion coherence; then refine lip motion with frame-level alignment and lip-focused supervision.
• Facial-focused identity preservation: replace the conventional full-image reference network with a face-centric cross-attention pathway built from cropped facial features and ArcFace/Q-Former embeddings.
• Motion intensity modulation: explicitly control facial expression and body-motion amplitude using motion coefficients extracted from pose and facial landmark variance.
• Strong empirical performance: the paper reports improvements in image/video quality, synchronization, identity preservation, motion diversity, and user preference over state-of-the-art baselines.

A notable design choice is that the model does not treat the talking head as just a lip-synchronization problem. Instead, it is formulated as a scene-level generation problem in which audio conditions a temporally coherent scene with facial, body, and background motion.

Background: Diffusion and Talking-Head Generation

The method is grounded in a latent diffusion formulation. Videos are encoded into latent tokens using a pretrained VAE encoder $E$, perturbed with Gaussian noise, and denoised by a network $\epsilon_\theta$ conditioned on side information such as audio, text, and images. The paper writes the standard diffusion objective as

$$ L = \mathbb{E}_{t, z_t, c, \epsilon \sim \mathcal{N}(0,1)} \left[ \|\epsilon_\theta(\mathbf{z}_t, t, c) - \epsilon\|_2^2 \right], $$

where $c$ denotes the conditioning inputs and the denoised latent is decoded back to video by the VAE decoder $D$.

The backbone adopted in the paper is Wan2.1-I2V-14B, a DiT-based video generation model that uses a causal 3D VAE for spatiotemporal compression, with text encoded by UMT5 and injected by cross-attention. The authors build their talking-head adaptation on this pretrained foundation rather than training a video generator from scratch.

The paper’s related-work discussion highlights two limitations of prior audio-driven talking-head systems:

• Many methods rely on 3D intermediate representations such as 3DMM or FLAME, which are limited in expressive capacity.
• More recent diffusion-based methods often focus narrowly on lips and identity, while leaving head motion, body motion, and background dynamics under-modeled.

Method Overview

FantasyTalking takes as input a single reference image, a driving audio, and a prompt, and generates a video in which the subject remains identifiable while moving in response to audio. The architecture has three main components:

• Dual-stage audio-visual alignment (DAVA) to learn scene-level and lip-level synchronization.
• Identity preservation through facial cross-attention rather than a full-frame reference network.
• Motion intensity modulation to control the magnitude of facial and body motion.

The paper emphasizes that these components are trained end-to-end with the pretrained DiT backbone so that identity and motion control can coexist rather than compete.

Pretrained video diffusion transformer setup

The authors adopt a latent-video Diffusion Transformer design rather than a UNet-based architecture. Text is encoded with UMT5 and used in cross-attention layers, while timestep information is injected through six modulation parameters. Audio is represented by Wav2Vec tokens containing multi-scale acoustic features. The audio-token length is generally different from the video-token length, so the model learns a mapping between audio and visual token sequences.

Dual-Stage Audio-Visual Alignment

The central technical contribution is the dual-stage audio-visual alignment strategy. The authors motivate it by arguing that “wild” talking heads require more than lip sync: nonverbal motion such as eyebrow movement, eye movement, shoulder motion, and background changes contribute to realism, but these are only weakly correlated with audio and are not strictly aligned frame by frame.

Stage 1: Clip-level training for global coherence

In the first stage, the model computes 3D full attention across the entire audio-visual token sequence at the clip level. The goal is to learn broad dependencies between audio and video over a whole clip, allowing the model to jointly capture:

• strongly correlated lip motion,
• weakly correlated facial expressions,
• body motion such as shoulders and head movement, and
• scene-level dynamics including contextual objects and background.

The paper presents this stage as important for generating globally coherent motion, but it also notes a limitation: because lips occupy only a small part of the frame, clip-level supervision alone is not sufficient to learn precise mouth articulation.

Stage 2: Frame-level refinement with lip tracing

The second stage focuses on precise lip synchronization. Here the authors segment the audio and video according to a one-to-one temporal mapping and reshape the tokens into frame-wise tensors: video tokens into shape $f \times (h \times w) \times c$ and audio tokens into shape $f \times l' \times c$, where $f$ is the number of frames, $h$ and $w$ are the latent spatial dimensions, $l'$ is the per-frame audio token length, and $c$ is the channel dimension.

They then compute 3D full attention only between corresponding frame-aligned audio and visual tokens. To localize the supervision further, MediaPipe is used to obtain lip masks in pixel space, which are projected to latent space via trilinear interpolation, producing a lip-focused constraint mask $M$. The frame-level loss is reweighted as

$$ L_c = M \odot L, $$

where $\odot$ denotes element-wise multiplication. To avoid over-constraining the model and suppressing natural head or background motion, the paper uses a Bernoulli-style selection with probability $\eta$:

$$ L' = \begin{cases} L_c, & \text{if } p > \eta \\ L, & \text{otherwise} \end{cases} $$

This design intentionally balances accurate mouth synchronization against the need for unconstrained scene dynamics.

Identity Preservation

Instead of using the full-image reference network common in prior work, FantasyTalking adopts a facial-focused identity preservation pathway. The authors argue that full-frame reference networks tend to bias generation toward static or overly constrained outputs because they process the whole scene rather than concentrating on the identity-bearing face region.

The pipeline is:

1. crop the facial region from the reference image,
2. extract facial features using ArcFace,
3. align these features with Q-Former, producing an identity embedding $F_{id}$, and
4. inject identity conditioning through dedicated cross-attention layers in each DiT block.

The hidden state update for block $i$ is written as

$$ Z_i' = Z_i + \lambda_1 \cdot \operatorname{Attention}(Q_i, K_i^a, V_i^a) + \lambda_2 \cdot \operatorname{Attention}(Q_i, K_i^{id}, V_i^{id}), $$

where $Q_i$ are the queries, $K_i^a$ and $V_i^a$ are the audio keys and values, and $K_i^{id}$ and $V_i^{id}$ are the identity keys and values. The coefficients $\lambda_1$ and $\lambda_2$ control the relative contribution of audio and identity conditioning.

The main claim is that facial-focused identity conditioning preserves who the subject is without imposing the strong motion restrictions that a full-frame reference network can induce.

Motion Intensity Modulation

To control not just whether a person speaks, but how strongly they express themselves, the paper introduces a motion intensity modulation network. This module addresses the fact that people have different speaking styles and that wild talking-head videos require more flexible control over body and facial movement amplitude.

The authors compute two normalized coefficients in $[0,1]$:

• $\omega_l$: facial-expression movement coefficient, derived from the variance of facial landmark sequences extracted by MediaPipe.
• $\omega_b$: body movement coefficient, derived from the variance of body joint sequences extracted by DWPose.

The motion modulation network is implemented with MLP layers, a ResNet layer, and average pooling. The resulting motion embeddings are added to the timestep embeddings, so that motion intensity is integrated into the diffusion process. At inference time, users can set $\omega_l$ and $\omega_b$ to control facial and body-motion intensity explicitly.

The paper’s examples use a neutral setting of $\omega_l = 0.5$ and $\omega_b = 0.5$; more intense motion is obtained by increasing these coefficients, while subtle motion is obtained by lowering them.

Training Details

The implementation is based on Wan2.1-I2V-14B. Training is performed in two stages:

• Clip-level stage: approximately 80,000 steps.
• Frame-level stage: approximately 20,000 steps.

Both the identity network and the motion network are trained end-to-end during all training phases. The optimization uses Flow Matching, and training is run on 64 A100 GPUs with a learning rate of $10^{-4}$.

The paper reports the following hyperparameters:

• $\lambda_1 = 1$
• $\lambda_2 = 0.5$
• $\eta = 0.2$

To improve diversity, the reference image, driving audio, and prompt are each independently dropped with probability $0.1$ during training. At inference, the model uses 30 sampling steps and an audio classifier-free guidance scale of 4.5. The motion intensity parameters default to a neutral setting of 0.5 unless the user wants to emphasize or reduce motion amplitude.

Datasets

The training corpus combines three sources:

• Hallo3
• Celebv-HQ
• Internet-collected data

The data are filtered with InsightFace, removing clips with facial confidence below 0.9 and clips where speech and mouth motion are not synchronized. After filtering, the training set contains approximately 150,000 clips.

Evaluation is performed on two settings:

• Tame talking-head evaluation: 50 clips from HDTF.
• Wild talking-head evaluation: a collected dataset with 80 different individuals.

The wild setting is intended to test foreground, background, and motion variability more rigorously than the tame setting.

Evaluation Protocol

The paper evaluates with eight metrics:

• FID: image quality.
• FVD: video quality.
• Sync-C: audio-lip synchronization.
• Sync-D: lip-motion fluidity.
• ES: expression similarity, measuring consistency of facial features across frames.
• IDC: identity consistency using DINO similarity on facial regions.
• SD: subject dynamics from optical flow in the foreground.
• BD: background dynamics from optical flow in the background.
• Aesthetic: LAION aesthetic predictor score.

Baselines include Aniportrait, EchoMimic, Sonic, and Hallo3. The paper reports fair comparison by using an empty prompt at inference for their method.

Quantitative Results

On the tame HDTF subset, FantasyTalking achieves the best or near-best scores on most metrics. The strongest gains are in FID, FVD, ID consistency, expression similarity, and aesthetic quality. The method also obtains the best Sync-C and near-best Sync-D, indicating that the dual-stage alignment learns precise synchronization while preserving richer motion.

On the wild dataset, the method is reported to outperform all compared baselines on every listed metric, including the newly introduced background dynamics metric. The paper attributes this to the combination of scene-level audiovisual alignment and face-centric identity conditioning, which together allow more natural foreground and background motion while preserving subject identity.

Interpretation of the quantitative results

On tame data, the paper reports that FantasyTalking achieves the best FID, FVD, IDC, ES, and Aesthetic score, while also producing competitive synchronization. On wild data, it reaches the best score on every reported metric. This is important because wild data stress not only lip synchronization but also the ability to preserve identity under larger motion ranges and to animate the broader scene.

Motion-Intensity Control Study

The paper compares its motion intensity controller against Sonic, which also exposes a motion-control parameter. The authors evaluate three levels of motion intensity:

• Subtle: $\beta = 0.5$, $\omega_l = 0.1$, $\omega_b = 0.1$
• Natural: $\beta = 1.0$, $\omega_l = 0.5$, $\omega_b = 0.5$
• Intense: $\beta = 2.0$, $\omega_l = 1.0$, $\omega_b = 1.0$

The reported results show that both methods handle subtle and natural motion reasonably well, but FantasyTalking is stronger in the intense-motion regime because its control module covers the full body motion rather than just head motion.

User Study

The paper also reports a user study on the wild talking-head dataset with 24 participants. Each participant rates Lip Synchronization (LS), Video Quality (VQ), Identity Preservation (IP), and Motion Diversity (MD) on a 0–10 scale. FantasyTalking receives the highest score in every category.

Ablation Studies

The ablations isolate the contributions of DAVA, identity preservation, and motion modulation. The DAVA ablation shows that clip-level and frame-level alignment each have different strengths: clip-level alignment better supports dynamics and global coherence, while frame-level alignment improves audio-lip synchronization. The full DAVA design combines them to obtain a better overall trade-off.

The identity ablation shows a large drop in IDC when identity preservation is removed, and the paper reports visible facial artifacts and distortions in that setting. This supports the claim that a face-centered identity pathway is necessary for stable long-form talking portraits with stronger motion.

The motion-intensity ablation indicates that a natural setting around $\omega_l = 0.5$ and $\omega_b = 0.5$ gives the best FVD, while larger values increase subject dynamics. The authors interpret this as a trade-off: overly weak or overly strong motion can reduce realism, but the controller gives users explicit control over expressiveness.

Limitations and Future Work

The paper explicitly notes one main limitation: because diffusion inference relies on iterative sampling, runtime remains relatively slow. This makes real-time or latency-sensitive applications such as live streaming more challenging. The authors suggest that future work should study acceleration strategies and interactive portrait-dialogue systems with real-time feedback.

Importantly, the paper does not present a separate failure analysis beyond this speed limitation. Its stated future direction is therefore focused on making the current architecture practical for interactive digital-human scenarios rather than changing the modeling formulation.

Conclusion

FantasyTalking is presented as a diffusion-transformer-based talking-portrait generator that goes beyond lip synchronization. Its main technical contributions are a two-stage alignment strategy, a facial-centric identity module, and explicit motion-intensity control. The reported experiments show strong gains on both tame and wild datasets, particularly for motion diversity, identity consistency, and overall realism. The work’s main message is that high-quality talking portraits require coordinated control of lips, facial expressions, body movement, and background dynamics, not just mouth motion alone.