EchoMimicV2

EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation

EchoMimicV2 presents a novel approach to striking half-body human animation by dynamically harmonizing audio and pose inputs. It progressively reduces pose dependency while enhancing audio-driven expressiveness, enabling synchronized, vivid facial and upper body animation with fewer control complexities.

• audio-driven
• gesture
• co-speech
• full-body
• face-animation
• lip-sync
• speech-driven

Demos

These demos showcase EchoMimicV2's ability to generate striking, simplified, and semi-body human animations driven by audio and pose inputs. Watch for the model's enhanced facial expressiveness, natural and accurate gestural motions, and the impact of the novel Phase-specific Denoising Loss on animation quality and motion phase fidelity. The videos reflect improvements in detail, motion accuracy, and overall animation realism, aligning with the claims of faster, more vivid, and lifelike human animation synthesis.

Authors: Rang Meng, Xingyu Zhang, Yuming Li, Chenguang Ma

Categories: cs.GR, cs.CV

Comment: CVPR2025

Published 2024-11-15 · Updated 2026-03-02

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub4,601 stars47 watchers541 forksApache-2.0
• Hugging Face0 downloads135 likes

Impact

• Semantic Scholar71 citations11 influential citations

Abstract

Recent work on human animation usually involves audio, pose, or movement maps conditions, thereby achieves vivid animation quality. However, these methods often face practical challenges due to extra control conditions, cumbersome condition injection modules, or limitation to head region driving. Hence, we ask if it is possible to achieve striking half-body human animation while simplifying unnecessary conditions. To this end, we propose a half-body human animation method, dubbed EchoMimicV2, that leverages a novel Audio-Pose Dynamic Harmonization strategy, including Pose Sampling and Audio Diffusion, to enhance half-body details, facial and gestural expressiveness, and meanwhile reduce conditions redundancy. To compensate for the scarcity of half-body data, we utilize Head Partial Attention to seamlessly accommodate headshot data into our training framework, which can be omitted during inference, providing a free lunch for animation. Furthermore, we design the Phase-specific Denoising Loss to guide motion, detail, and low-level quality for animation in specific phases, respectively. Besides, we also present a novel benchmark for evaluating the effectiveness of half-body human animation. Extensive experiments and analyses demonstrate that EchoMimicV2 surpasses existing methods in both quantitative and qualitative evaluations.

---

Introduction and Problem Setting

EchoMimicV2 targets half-body human animation: generating a realistic human-centric video from a reference image, speech audio, and a small amount of pose guidance, with the goal of animating both the face and the upper body in a synchronized and visually convincing way. The paper frames the problem as a practical extension of prior human animation work, which has often focused either on talking heads or on body motion with relatively heavy control stacks. The authors argue that existing systems commonly rely on many auxiliary conditions such as text, audio, pose, optical flow, or movement maps, along with complex condition-injection modules. This can make training less stable, inference slower, and deployment more cumbersome.

The central question is whether striking half-body animation can be achieved while simplifying unnecessary conditions. EchoMimicV2 answers this with a design that progressively reduces pose dependency while increasing the role of audio, instead of treating all control signals as permanently required. The paper’s core idea is called Audio-Pose Dynamic Harmonization, and it is paired with a phase-aware training loss. The method is explicitly positioned as an end-to-end extension of EchoMimic, but adapted to the harder semi-body setting.

The paper also introduces a new evaluation benchmark for half-body animation, EMTD, because the authors state that public datasets mostly cover talking heads or pose-driven human animation rather than audio-driven half-body performance. In the reported experiments, EchoMimicV2 is evaluated on a mix of qualitative comparisons, quantitative metrics, and ablations designed to isolate the contributions of the training schedule, partial attention, data augmentation, and phase-specific loss.

High-Level Architecture

EchoMimicV2 uses a ReferenceNet-based latent diffusion backbone. A pretrained 2D U-Net is duplicated to form a ReferenceNet that extracts appearance features from the input reference image. Those features are injected into the denoising U-Net via cross-attention so that the generated frames preserve identity and clothing/appearance consistency. The system also includes an audio encoder, specifically a pretrained Wav2Vec model, and a pose encoder for keypoint maps. Temporal attention blocks are inserted into the denoising U-Net to model inter-frame dependencies and improve motion smoothness.

The paper’s overall pipeline is shown below. The figure is important because it reflects the simplification strategy: the method does not require a dense stack of auxiliary control modules at inference, but instead stages the condition interaction during training and progressively shifts control from pose to audio.

In the latent diffusion objective, an image $I$ is encoded into a latent $z_0 = E(I)$, noise is added over timesteps $t \in [1, \ldots, T]$, and the denoiser learns to predict the injected noise. The paper writes the standard latent loss as

$$ L_{\text{latent}} = \mathbb{E}_{z_t, t, c, \epsilon \sim \mathcal{N}(0,1)}\left[\|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2\right], $$

where $c$ denotes conditions such as audio or pose, and $\epsilon_\theta$ is the denoising U-Net. EchoMimicV2 builds on this standard formulation, but modifies how the conditions are used during training so that the model learns to rely less on redundant pose signals and more on the audio signal where appropriate.

Audio-Pose Dynamic Harmonization

The main contribution is Audio-Pose Dynamic Harmonization (APDH), a training strategy that progressively modulates pose and audio control in a staged, waltz-like manner. The authors describe the relationship between audio and pose as complementary: pose provides structure early on, while audio increasingly takes over control of lip motion, facial expression, and eventually broader body motion. APDH contains two subcomponents: Pose Sampling and Audio Diffusion.

Pose Sampling

Pose Sampling begins with an Initial Pose Phase, where the model is trained with complete pose keypoints for the half-body figure. During this stage, audio cross-attention is muted, and the denoising U-Net and pose encoder are trained to learn a robust motion prior from full pose supervision. The goal is to let the backbone first understand the spatial layout and coarse motion of the body.

Next comes an Iterative Pose Sampling Phase. Here, pose dropout is applied progressively at the iteration level, with dropout probability increasing over training. This discourages over-reliance on pose and forces the model to remain stable when pose information becomes incomplete.

The third component is Spatial Pose Sampling, where keypoints are removed in a staged order: lips first, head second, and body-related points last. In the paper’s notation, the sampled pose conditions are $\mathcal{P}^{-\text{lips}}$, $\mathcal{P}^{-\text{head}}$, and $\mathcal{P}^{\text{hands}}$ (hands-only pose), respectively. The key idea is that the pose signal is gradually stripped of regions that overlap with audio-driven expressions, creating room for the audio condition to become dominant in those areas.

Conceptually, this is the mechanism by which the model moves from a pose-heavy training regime to a simpler half-body regime. In the final form, pose is reduced to hand guidance, while audio is responsible for mouth, facial expression, breathing rhythm, and some audio-gesture correlation.

Audio Diffusion

Audio Diffusion is introduced progressively as pose is reduced. During the initial and iterative pose phases, the audio cross-attention blocks are frozen. Once the model enters the spatial sampling stage, audio control is gradually expanded using partial attention masks.

First, after lip keypoints are removed, the model uses Lips Partial Attention so the audio condition only controls lip motion. This improves audio-to-lip synchronization.

Second, when head keypoints are removed, the attention mask is expanded into a Head Partial Attention region. This lets audio dominate the full facial region, improving synchronization between speech and facial expression.

Third, when the pose condition is reduced to hands-only, the audio mask is diffused globally, producing an essentially audio-driven half-body animation with hand emphasis. The hands become the intersection between audio and pose, which the authors use to encourage gesture generation that is correlated with speech rhythm and emphasis.

Head Partial Attention as Data Augmentation

The paper highlights data scarcity as a major issue for half-body animation, especially for facial expressiveness. To address this, EchoMimicV2 introduces Head Partial Attention as a form of data augmentation that allows headshot data to be incorporated into the half-body training pipeline without additional injection modules.

The procedure is simple but important: headshot data is padded so that its spatial dimensions and head location match those of half-body images. The same head partial attention mask is then used to exclude the padded regions. The authors emphasize that no new cross-attention blocks are needed. They also state that this augmentation is applied before half-body data is introduced in the audio-face synchronization phase, which is intended to mitigate distribution shift caused by padding.

The reported effect of this augmentation is especially visible in synchronization metrics, where the ablation without headshot augmentation is substantially worse than the full model. The paper treats this as a “free lunch” in the sense that headshot data can be reused to improve facial expression without adding inference-time complexity.

Temporal Modules Optimization

Temporal coherence is handled by temporal cross-attention blocks inserted into the denoising U-Net. Following EchoMimic, these temporal modules are initialized from the earlier model and then optimized in a second stage. The paper uses 24-frame clips for this stage and freezes other modules to stabilize training. This separates spatial/condition learning from temporal refinement, reducing optimization difficulty.

Phase-Specific Denoising Loss

In addition to condition scheduling, the paper proposes Phase-specific Denoising Loss (PhD Loss). The key idea is that different denoising timesteps should optimize different aspects of the generated video:

• Pose-dominant phase for early timesteps, where the model should learn motion structure and human contours.
• Detail-dominant phase for middle timesteps, where the model should refine edges and character-specific detail.
• Quality-dominant phase for late timesteps, where the model should improve color, texture, and low-level image quality.

The paper expresses the total loss as a phase-dependent combination of an auxiliary objective and the standard latent diffusion loss:

$$ L_{\text{PhD}} = \begin{cases} \lambda_{\text{pose}} L_{\text{pose}} + L_{\text{latent}}, & t \in \mathcal{S}_1 \\ \lambda_{\text{detail}} L_{\text{detail}} + L_{\text{latent}}, & t \in \mathcal{S}_2 \\ \lambda_{\text{low}} L_{\text{low}} + L_{\text{latent}}, & t \in \mathcal{S}_3 \end{cases} $$

In the implementation, the early phase is the first 10% of timesteps, the middle phase is the following 60%, and the remaining timesteps form the final phase. The weights $\lambda_{\text{pose}}$, $\lambda_{\text{detail}}$, and $\lambda_{\text{low}}$ are all set to $0.1$.

The three auxiliary losses are:

• Pose-dominant loss: one-step sample the latent, decode it, extract pose maps from the predicted image and target image, and minimize MSE between them.
• Detail-dominant loss: apply a Canny-based edge extractor to predicted and target images, and minimize MSE on the extracted high-frequency detail maps.
• Low-level loss: use LPIPS between the predicted and target RGB images to improve perceptual quality.

The supplementary material further clarifies that the pose maps are implemented with Gaussian maps to preserve differentiability, and the Canny-based detail operator is adapted into a fixed convolutional kernel for the same reason. The paper also notes that the phase-specific training is intended to avoid the need for a more expensive multi-loss setup with extra pose encoders and VAE decoders operating simultaneously.

Training Setup and Data

EchoMimicV2 is trained in two stages. In the first stage, the model learns to generate target frames from reference images and complete pose conditions, with audio attention masked. This stage uses 8 A100 GPUs for 10,000 iterations, batch size 4 per GPU, and resolution $768 \times 768$.

The APDH schedule is then applied in a series of additional training segments: 10,000 iterations with iterative pose sampling and pose dropout increasing from 0% to 20%; 10,000 iterations using $\mathcal{P}^{-\text{lips}}$ with Lips Partial Attention; 10,000 iterations using $\mathcal{P}^{-\text{head}}$ with Head Partial Attention on headshot data from EchoMimic; another 10,000 iterations on half-body data; and finally 10,000 iterations using $\mathcal{P}^{\text{hands}}$ with global audio cross-attention.

In the second stage, only the temporal-attention modules are optimized, while all other modules are frozen. This stage runs for 30,000 steps with batch size 4. For both stages, the learning rate is $10^{-5}$ and the classifier-free guidance scale for reference images and audio is set to 2.5.

Training Datasets

The training data is explicitly divided into three parts:

• Fully pose-driven dataset: HumanVid, with about 20,000 high-quality human-centric videos and 2D pose annotations.
• Half-body dataset: curated from internet videos, focused on half-body speaking scenarios, spanning 160 hours and including more than 10,000 identities.
• Headshot dataset: derived from the training data used in EchoMimic, containing 540 hours of talking-head videos.

This combination supports the paper’s staged training narrative: first learn general pose-driven motion, then reduce the pose dependency and inject progressively stronger audio control, while reusing headshot data to compensate for limited half-body facial data.

Novel Half-Body Evaluation Benchmark

The paper introduces EMTD (EchoMimicV2 Testing Dataset) as a new benchmark for audio-driven half-body human animation. It contains 65 high-definition TED videos from YouTube at 1080P, with 110 annotated, clear, front-facing half-body speech segments. The authors describe EMTD as a benchmark intended to fill a gap in evaluation coverage, since existing public datasets do not directly target this setting.

Evaluation Metrics

The reported evaluation spans both image/video quality and controllability. The metrics are:

• FID and FVD for visual quality.
• SSIM and PSNR for low-level reconstruction quality.
• E-FID for expression authenticity.
• Sync-C and Sync-D from SyncNet for audio-lip synchronization.
• CSIM for identity consistency.
• HKC and HKV for hand quality and hand motion richness.

This metric mix is important because EchoMimicV2 is not only evaluated as a generic video generator; it is also assessed on whether it preserves identity, synchronizes speech, and improves hand motion in a half-body setting.

Qualitative Results

The paper presents multiple qualitative comparisons showing that EchoMimicV2 can generate diverse half-body animations from different reference images, hand pose sequences, and audio clips. The authors claim the model generalizes across characters and gesture styles, with visually coherent motion and improved synchronization of facial expression and gesture.

In comparisons against pose-driven baselines such as AnimateAnyone and MimicMotion, the paper reports stronger structural integrity and identity consistency, especially in the face and hands. The authors emphasize that this is notable because half-body generation requires both accurate body structure and detailed local realism.

For audio-driven half-body baselines, the paper compares against Vlogger and CyberHost using results obtained from their project materials. EchoMimicV2 is reported to yield better image quality and more natural motion under the same reference image setting.

A particularly important qualitative result is hand synthesis. The authors note that hands are generally hard to generate, even for strong text-to-image systems, and the issue is worse in audio-driven settings. EchoMimicV2 is reported to repair or synthesize hands even when the reference image has missing or deformed hands. This supports the paper’s claim that the combination of hand pose and audio can recover detailed upper-body motion and local anatomy.

Quantitative Results

The main quantitative comparison table reports that EchoMimicV2 outperforms AnimateAnyone and MimicMotion across all listed metrics on EMTD. The strongest gains are seen in perceptual quality, identity consistency, hand motion quality, and audio synchronization.

Relative to MimicMotion, the full model improves FID from 53.47 to 49.33, FVD from 622.62 to 598.45, SSIM from 0.702 to 0.738, PSNR from 19.278 to 21.986, E-FID from 2.628 to 2.218, Sync-D from 7.958 to 7.021, Sync-C from 1.495 to 7.219, HKC from 0.907 to 0.923, HKV from 24.82 to 25.28, and CSIM from 0.526 to 0.558. The especially large improvement in Sync-C suggests that the APDH schedule and audio diffusion significantly help audio-conditioned synchronization under the paper’s metric definition.

Ablation Study

The paper includes a broad ablation table to isolate the effects of pose sampling, audio diffusion, headshot augmentation, phase-specific loss, and the overall APDH training strategy. Below is the ablation table as reported.

What the Ablations Show

The ablation results support the paper’s narrative in several ways. First, the initial pose stage and iterative pose sampling both matter, because they provide a strong motion prior and stabilize training before pose information is reduced. Second, the spatial pose sampling design is important not because it boosts all metrics by itself, but because it enables the model to remain functional with much less pose information; the authors note that the full pose condition has limited impact once spatial sampling is removed, which means the condition is redundant for the final half-body setup.

Third, headshot augmentation has a large effect on synchronization metrics, especially Sync-C and Sync-D, indicating that facial data transfer is useful when combined with head partial attention. Fourth, the three phase-specific losses each contribute to different aspects of quality: $L_{\text{pose}}$ helps the overall structure and synchronization under incomplete pose, $L_{\text{detail}}$ improves local realism and expression-related metrics, and $L_{\text{low}}$ improves SSIM and PSNR.

The paper also includes a straightforward baseline that directly trains on half-body audio and hand pose conditions without APDH or PhD Loss. That baseline is consistently worse than the full model, supporting the claim that simple condition concatenation is not enough and that staged harmonization is required for stable training with simplified conditions.

The supplementary analysis additionally shows that if hand pose is removed entirely, the model can still learn hand rhythm and local hand quality from audio, but the range of motion becomes restricted. This supports the authors’ claim that hand pose is useful for gesture diversity even when audio already drives general motion.

Interpretation of the Main Claims

The strongest conceptual contribution of EchoMimicV2 is not merely that it uses audio and pose, but that it re-allocates responsibility between those conditions over the course of training. Pose is used to establish body structure, then gradually reduced; audio is introduced first locally at the lips, then expanded to the face, and finally to broader body motion. This creates a training curriculum in which the model can learn half-body animation without requiring every condition at every stage.

The second major contribution is the use of partial attention as a data and control mechanism. Head partial attention is especially notable because it lets headshot data contribute to facial expressiveness without changing the inference-time interface. This is a pragmatic design for systems teams because it reduces the number of controls that must be exposed to end users while still benefiting from more abundant head-region data during training.

The third contribution is the phase-specific loss. Rather than adding more auxiliary modules, the paper moves supervision into different denoising phases. That makes the training objective closer to what the model is expected to produce at each stage of generation: coarse pose first, fine detail second, and perceptual quality last. The supplementary visualizations reinforce this intuition by showing that the model’s intermediate outputs improve in precisely that order.

Limitations

The paper is explicit about two limitations. First, the method still requires predefined hand pose sequences, which means human input is needed for high-quality animation. This limits end-to-end usability and means the system is not yet fully audio-to-hand-pose generation in a single step. The authors state that future work will explore generating hand pose sequences directly from audio.

Second, performance declines on non-cropped reference images, especially full-body images. EchoMimicV2 is reported to work well on cropped half-body references, but the method is less robust when the reference image contains the full body and more background context. This suggests the current model is specialized for cropped semi-body scenarios rather than general unconstrained portrait-to-body animation.

Conclusion

EchoMimicV2 proposes a simplified half-body animation pipeline that combines a ReferenceNet-based diffusion backbone with APDH training, head partial attention, temporal refinement, and phase-specific denoising supervision. The paper’s main empirical claim is that this combination yields better qualitative realism and better quantitative performance than prior pose-driven and audio-driven baselines, while reducing unnecessary control complexity. Its benchmark EMTD and the ablation table are both central to the paper’s contribution, because they support the claim that the method is specifically tuned to half-body animation rather than generic talking-head generation.

From a systems perspective, the work is notable because it reframes control signals as resources to be scheduled rather than always-on inputs. That makes EchoMimicV2 a useful reference point for teams building conversational or talking-head systems that need more expressive upper-body motion without introducing a brittle multi-condition inference stack.