EMO2

EMO2: End-Effector Guided Audio-Driven Avatar Video Generation

EMO2 is a two-stage approach for audio-driven talking-head video generation that uses hand gestures as key controls. It first generates hand motions from audio, then guides a diffusion model to synthesize realistic face and body movements, improving expressiveness and synchronization over prior full-body methods.

• audio-driven
• talking-head
• gesture
• co-speech
• avatar
• face-animation

Authors: Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, Liefeng Bo

Categories: cs.CV

Published 2025-01-18 · Updated 2025-01-18

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub7,616 stars328 watchers931 forks

Impact

• Semantic Scholar35 citations

Abstract

In this paper, we propose a novel audio-driven talking head method capable of simultaneously generating highly expressive facial expressions and hand gestures. Unlike existing methods that focus on generating full-body or half-body poses, we investigate the challenges of co-speech gesture generation and identify the weak correspondence between audio features and full-body gestures as a key limitation. To address this, we redefine the task as a two-stage process. In the first stage, we generate hand poses directly from audio input, leveraging the strong correlation between audio signals and hand movements. In the second stage, we employ a diffusion model to synthesize video frames, incorporating the hand poses generated in the first stage to produce realistic facial expressions and body movements. Our experimental results demonstrate that the proposed method outperforms state-of-the-art approaches, such as CyberHost and Vlogger, in terms of both visual quality and synchronization accuracy. This work provides a new perspective on audio-driven gesture generation and a robust framework for creating expressive and natural talking head animations.

---

Overview and Core Idea

EMO2 reframes audio-driven talking-head / upper-body video generation as a two-stage pipeline: first generate hand motion from audio, then use that motion as a control signal inside a diffusion-based video generator. The central claim is that the audio-to-gesture mapping is much easier and more stable when expressed at the level of the human end-effector—the hands—than when trying to predict full-body or half-body poses directly. The paper argues that audio correlates more strongly with hand movement than with the rest of the body, and that a pretrained pixel-space video generator can implicitly resolve the remaining body articulation through human-body priors, which the authors describe as “pixels prior IK”.

This design is motivated by a robotics analogy: in robot control, the end-effector pose is often the most intuitive target, and inverse kinematics resolves the rest of the chain. EMO2 applies that intuition to co-speech generation by treating the hands as the primary controllable signal, then letting the video model synthesize the face, torso, arms, and temporal coherence around it.

Problem Formulation and Contributions

The paper is positioned against prior work that predicts full-body or half-body parameters such as SMPL pose sequences or pixel-space body motion directly from audio. The authors argue that such formulations are hard because the relationship between audio and complete body pose is weak and many-to-many. EMO2’s main contribution is to split the task into a simpler intermediate prediction problem—audio to hand motion—and then use those generated motion signals to guide a diffusion model that produces the final talking video.

• Stage 1: audio-driven hand motion generation using a diffusion transformer.
• Stage 2: a reference-image-conditioned diffusion video generator that consumes the generated hand motion, audio, and auxiliary motion guidance.
• Motion representation: direct hand control via MANO, with an additional SMPL-based variant used for comparison.
• Empirical claim: improved motion diversity, hand expressiveness, synchronization, and overall video quality compared with the reported baselines and demo-based comparisons.

Method: Two-Stage Pipeline

The system consists of a motion-generation stage and a video-generation stage. In both stages, the model operates under a diffusion objective of the form

$$\mathcal{L} = \mathbb{E}_{t,c,x_t,\epsilon}\left[\|\epsilon - \epsilon_\theta(x_t,t,c)\|^2\right],$$

where $x_t$ is the noisy sample at diffusion timestep $t$, $\epsilon$ is the Gaussian noise, and $c$ denotes conditioning signals such as audio, motion maps, keypoints, or reference-image features depending on the stage.

Stage 1: Partial Body Motion Generation

Stage 1 predicts hand motion directly from audio instead of full-body motion. The paper emphasizes that the hands are the most useful intermediate control signal because they are strongly linked to intent and tend to vary more meaningfully with speech. The implementation uses a Diffusion Transformer (DiT) backbone with 24 DiT blocks and hidden size 512.

The hand representation is based on the MANO model. Each hand is represented by 48 joint rotation values in axis-angle form plus 3 translation values. To reduce ambiguity in rotation, the paper converts the 48 axis angles into 64 quaternion parameters, yielding a total of 134 parameters for both hands. Motion sequences are padded to 300 frames to support arbitrary-length generation, and the previous 12 frames are concatenated to the current clip to create smooth transitions across clips.

Conditioning in the DiT blocks includes:

• Audio features extracted using Wav2vec, injected via cross-attention.
• Timestep embeddings augmented with AdaLN-single, following the PixArt-$\alpha$ design.
• Style embeddings to control speaking, singing, or gesture-dance styles.
• Speed / amplitude embeddings to modulate hand motion intensity; the authors note that using variance of hand translation can provide stronger control than a pure speed bucketization.
• Reference-image embeddings from CLIP vision features, used optionally to adapt motion to context such as holding a microphone or guitar.
• Hand motion masks to mark valid annotated frames and padding regions.
• Hand offset embeddings to disentangle hand motion from root body posture differences in the dataset.

The paper notes a practical issue: some dataset frames have inaccurate or missing MANO annotations because the hands are occluded or out of view. To reduce the impact of these cases, EMO2 uses hand confidence information and motion masks, so that noisy or unreliable hand annotations contribute less to training. Stage 1 is trained from scratch on an A100 GPU with batch size 8 for 400k steps.

Stage 2: Co-Speech Video Generation

Stage 2 uses a diffusion video generator based on ReferenceNet and the EMO-style architecture. The model is described as a denoising 2D U-Net with temporal modules from AnimateDiff. It receives multi-frame latent noise and denoises it into a sequence of video frames. The architecture has four main parts:

• Denoising backbone: a 2D U-Net augmented with temporal modules.
• Frame reference: a parallel ReferenceNet that encodes the reference image and motion frames to preserve identity.
• Audio-driven control: audio features, shared with Stage 1, are injected through cross-attention.
• Motion guidance: the MANO maps and keypoint maps produced in Stage 1 are concatenated channel-wise and used to modulate the generated upper-body motion.

Stage 2 further introduces two important control mechanisms. First, hand confidence embeddings are learned from MANO detection confidence: low-confidence cases usually correspond to occlusion or motion blur, and the confidence scalar modulates a trainable embedding before it is added to the latent features. At inference time, higher confidence can be used to encourage clearer hand structure. Second, keypoint guidance adds 2D joint maps for arms and legs. The paper explicitly states that Stage 1 keypoints are only 2D and do not fully preserve 3D geometry such as exact arm length, so they are temporally median-filtered with a large kernel during Stage 2 training to intentionally allow some misalignment and thereby give the generator creative freedom while still guiding body dynamics.

To improve structural plausibility, the authors also train a pose discriminator. At each diffusion timestep, they perform one-step sampling to obtain a latent prediction $z_{t0}$ from the model output and feed it to the discriminator, which predicts body pose keypoints and limb heatmaps $\hat{H}$. The pose loss is

$$L_{pd} = \|H - \hat{H}\|_2,$$

where $H$ is the ground-truth heatmap. This loss is added to the denoising objective. The pose discriminator is a pretrained ResNet operating in latent space.

Training Setup and Data

The paper trains the two stages separately. For Stage 2, training is itself split into two phases:

• Image training: two frames are sampled and cropped to $704 \times 512$ pixels, serving as the reference frame and target frame. The ReferenceNet, motion guidance layer, and basic backbone modules are optimized.
• Audio-video training: temporal modules, audio attention layers, and additional components are activated; the ReferenceNet is frozen for memory efficiency.

Stage 2 uses 4 A100 GPUs. Batch sizes are 32 for image training and 4 for audio-video training. Each phase runs for 100k steps with learning rate $1 \times 10^{-5}$. Video clips contain 24 frames at $704 \times 512$ resolution, and the number of motion frames is set to 12.

The training data is drawn from MOSEI and AVSPEECH, both of which contain half-body speech scenarios. The authors also add internet-collected videos, resulting in a dataset of approximately 275 hours total duration. The paper does not provide further dataset curation details in the excerpt, but it does emphasize that the training data covers speech-driven upper-body motion rather than full-body locomotion.

Experimental Protocol

The experiments are organized around two questions: (1) how well does the Stage 1 motion generator model co-speech hand motion, and (2) how effectively does the full two-stage system generate realistic audio-driven videos.

Hand-motion evaluation: conducted on the Talkshow test set. The authors sample 100 input audios and generate 50 results per audio. Metrics are:

• DIV: diversity of generated motions.
• BA: beat alignment between audio and motion.
• PCK: closeness to ground-truth motion.
• FGD: Fréchet Gesture Distance between generated and GT motion distributions.

Video evaluation: conducted on the EMTD dataset. Metrics include:

• FID, SSIM, and PSNR for frame-level quality.
• FVD for video coherence.
• CSIM for identity consistency.
• Sync-C for lip-audio synchronization.
• HKC for hand keypoint confidence.
• HKV for hand keypoint variance / motion richness.
• EFID for expression divergence from ground truth.

Hand Motion Generation Results

The hand-motion comparison table contrasts Talkshow, DiffSHEG, EMO2 trained on SMPL poses, and EMO2 trained on MANO poses. The MANO version achieves the best reported DIV and BA, which the authors interpret as the strongest evidence that directly modeling hands produces more vivid and more synchronized co-speech gestures than SMPL-based approaches. The SMPL-based EMO2 variant also improves diversity over prior SMPL methods, but the paper’s qualitative interpretation is that SMPL kinematics can limit expressiveness and lead to more monotonous motion.

The authors explicitly note an important tradeoff: the MANO-based method has lower PCK and FGD than SMPL-based variants on the ground-truth-aligned metrics because it is less constrained by SMPL forward kinematics and therefore can deviate more from the exact ground truth while still remaining plausible. In other words, the paper favors expressive freedom and beat alignment over strict geometric closeness to the dataset motion. This is consistent with the qualitative result that the MANO model produces a wider range of 2D hand locations and more diverse movement patterns.

Video Generation Results

On EMTD, EMO2 is compared against EchoMimicV2 and MimicMotion. The paper also reports two extra comparisons against demo videos from CyberHost and Vlogger, since their pretrained models were not open-sourced. The authors’ main message is that EMO2 generates more coherent upper-body motion and preserves identity while improving synchronization and hand quality.

One key ablation is “w/o motion gen”, where the model uses ground-truth pose sequences rather than the Stage 1 motion generator output. As expected, this variant produces better image/video fidelity metrics because it is driven by cleaner motion input, but it is not the complete end-to-end system. Another ablation is “w/o hand confidence”, which removes the learned hand-confidence conditioning and therefore tests how much the confidence signal helps with hand quality.

Several patterns are worth highlighting from the reported numbers. Compared with EchoMimicV2 and MimicMotion, the full EMO2 system achieves a substantially lower FVD than both baselines, indicating better temporal coherence. It also improves Sync-C over both baselines, suggesting better lip-audio alignment. The paper reports stronger identity preservation as well, as reflected by CSIM. Hand-specific metrics improve as well, especially HKC and HKV, which the authors use to argue that the generated hands are both clearer and more diverse.

The demo-based comparisons provide a different perspective. Against CyberHost, EMO2 reaches higher Sync-C and higher CSIM while also increasing HKV, which the authors interpret as stronger motion diversity. Against Vlogger, EMO2 again shows much higher Sync-C and substantially higher HKV, although the reported HKC is slightly lower. The authors’ qualitative conclusion is that EMO2 better balances identity preservation, realistic body motion, and expressive hands.

Ablation and Design Analysis

The paper’s ablations focus on the components that make the two-stage design work:

• Hand representation choice: MANO directly improves motion diversity and beat alignment relative to SMPL-based gesture generation.
• Motion generation vs. ground-truth motion: using GT pose as the driver boosts image/video fidelity, showing that the stage-2 generator benefits from better motion control, but it is not a fair end-to-end comparison.
• Hand confidence: confidence-aware conditioning improves the quality and structure of hand rendering, especially in cases with occlusion or motion blur.
• Keypoint guidance: 2D keypoints help prevent unrealistic torso-hand mismatches when hand motion is large.
• Pose discriminator: the discriminator is used to encourage body structure plausibility in latent space.

Overall, the ablation narrative is that each added control signal narrows a specific failure mode: hand motion provides high-level intent, keypoints constrain body trends, confidence suppresses bad hand annotations, and the pose discriminator adds structural regularization. The architecture is deliberately permissive with respect to exact body geometry so that the diffusion model can exploit its learned pixel-space prior to produce natural upper-body motions.

Interpretation of the Main Claims

The paper’s main empirical claim is not simply that the generated videos are visually attractive, but that the hand-first factorization is a more effective way to model co-speech motion. This claim is supported by the Stage 1 results: directly modeling hands yields more diverse and better-aligned gestures than SMPL-based full-body motion generation. In Stage 2, those hand motions provide a better motion scaffold than raw audio alone, helping the video generator synthesize more coherent motion, clearer hands, and improved synchronization.

A second conceptual claim is that human video diffusion models can serve as a kind of implicit kinematic prior. Rather than explicitly solving full inverse kinematics for every joint, EMO2 lets a pretrained video generator fill in plausible body structure from the hand control signal and the reference image. The authors present this as a practical compromise between control and realism: hands are controlled directly, and the remaining body motion emerges from pixel-space priors.

Stated Caveats and Practical Limitations

The paper does not include a dedicated limitations section, but several caveats are explicit in the body of the work and the reported results:

• Reliance on imperfect hand annotations: the authors note that MANO annotations can be missing or inaccurate when hands are occluded or out of frame, which motivates hand masks and confidence conditioning.
• 2D keypoint supervision is incomplete: the stage-2 keypoint maps do not fully determine 3D structure, so the model relies on the diffusion prior to recover plausible geometry.
• Motion fidelity tradeoff: the strongest hand-diversity setting can be less tightly matched to ground truth motion in metrics such as PCK and FGD, indicating a controllable tradeoff between diversity and exact motion reproduction.
• Limited comparison availability: some competing systems are compared only via demo videos because pretrained models were not released, so those comparisons are less controlled than the main benchmark tables.
• Training-domain scope: the training data is focused on half-body speech scenarios, so the method is designed primarily for talking-head / upper-body animation rather than unconstrained full-body choreography.

Takeaway for Talking-Head / Conversational AI Systems

For a conversational avatar pipeline, EMO2’s most useful design lesson is to separate semantic motion planning from pixel-level synthesis. The first stage learns a compact, expressive control signal tied to speech rhythm; the second stage uses that signal to animate a high-fidelity character while preserving identity and lip synchronization. The result is a practical architecture for audio-driven avatars where expressive hands, facial motion, and upper-body dynamics are generated in a coordinated way without forcing the model to solve the full complexity of body pose prediction end-to-end.