EchoMimicV3

EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation

EchoMimicV3 is a unified 1.3B-parameter model for multi-task, multi-modal human animation that supports text, image, and audio-driven tasks efficiently. It uses novel fusion and training methods to deliver fast, high-quality talking-head animation without separate models for each task.

• talking-head
• lip-sync
• audio-driven
• multimodal
• face-animation
• gesture

Authors: Rang Meng, Yan Wang, Weipeng Wu, Ruobing Zheng, Yuming Li, Chenguang Ma

Categories: cs.CV

Published 2025-07-05 · Updated 2026-03-02

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub963 stars12 watchers113 forksApache-2.0
• Hugging Face0 downloads41 likes

Impact

• Semantic Scholar28 citations1 influential citation

Abstract

Recent work on human animation usually incorporates large-scale video models, thereby achieving more vivid performance. However, the practical use of such methods is hindered by the slow inference speed and high computational demands. Moreover, traditional work typically employs separate models for each animation task, increasing costs in multi-task scenarios and worsening the dilemma. To address these limitations, we introduce EchoMimicV3, an efficient framework that unifies multi-task and multi-modal human animation. At the core of EchoMimicV3 lies a threefold design: a Soup-of-Tasks paradigm, a Soup-of-Modals paradigm, and a novel training and inference strategy. The Soup-of-Tasks leverages multi-task mask inputs and a counter-intuitive task allocation strategy to achieve multi-task gains without multi-model pains. Meanwhile, the Soup-of-Modals introduces a Coupled-Decoupled Multi-Modal Cross Attention module to inject multi-modal conditions, complemented by a Multi-Modal Timestep Phase-aware Dynamical Allocation mechanism to modulate multi-modal mixtures. Besides, we propose Negative Direct Preference Optimization, Phase-aware Negative Classifier-Free Guidance (CFG), and Long Video CFG, which ensure stable training and inference. Extensive experiments and analyses demonstrate that EchoMimicV3, with a minimal model size of 1.3 billion parameters, achieves competitive performance in both quantitative and qualitative evaluations.

---

Introduction and Problem Setting

EchoMimicV3 addresses a practical bottleneck in human animation: recent systems improve quality by adopting large-scale video diffusion models, but these models are slow, expensive, and often specialized to a single task. The paper frames the challenge as simultaneously achieving fast inference, high quality, stronger generalization, and unified support for multiple human-animation tasks inside one compact model.

The authors argue that two factors drive the current inefficiency: parameter inflation in large video models and model fragmentation across tasks such as lip sync, text-to-video, image-to-video, and first-last-frame-to-video. Their response is a single 1.3B-parameter framework built on a compact video diffusion backbone rather than a very large LVDM. The design goal is not only to be smaller, but to make the smaller model behave like a multi-task, multi-modal expert system.

The core thesis is that the model can unify tasks and modalities if all of them are reformulated as masked reconstruction problems and if training/inference are structured to exploit the different roles of text, image, and audio across diffusion timesteps. The paper’s main contributions are organized around three components: Soup-of-Tasks, Soup-of-Modals, and a new training and inference strategy.

Overall Framework

EchoMimicV3 is initialized from Wan2.1-FUN-inp-480p-1.3B and targets talking-human generation under combinations of reference image, audio, and text prompt, without requiring explicit 2D or 3D pose conditioning. The full pipeline is illustrated in the paper’s training figure, which emphasizes that task unification, multi-modal fusion, and negative-sample rejection are integrated into a single end-to-end framework.

The backbone operates in latent diffusion space. The model reuses the video model’s masked-sequence input convention and attaches task-specific masks and modality-specific cross-attention branches. In this sense, EchoMimicV3 is not a new video architecture from scratch; rather, it is a task-and-modality reprogramming of a compact diffusion transformer.

Soup-of-Tasks: Unifying Multiple Animation Tasks

Task reformulation as spatial-temporal masked reconstruction

The Soup-of-Tasks view treats diverse animation tasks as variants of masked reconstruction. The transformer backbone already accepts a binary mask sequence concatenated with the video latent, and the paper leverages this interface to encode task identity through the mask pattern alone. The authors define distinct mask inputs for text-to-video, image-to-video, first-last-frame-to-video, and lip synchronization:

$$ \mathsf{M}_{\text{T2V}},\; \mathsf{M}_{\text{I2V}},\; \mathsf{M}_{\text{FLF2V}},\; \mathsf{M}_{\text{lip}} $$

Conceptually, lip synchronization becomes reconstruction of the mouth region, while the other tasks become reconstruction of intermediate temporal frames. This lets step-wise diffusion and patch-wise masked modeling coexist without changing the backbone architecture.

Counter-intuitive task schedule

The training schedule deliberately goes against standard curriculum learning. Instead of easy-to-hard progression, the model first trains on the most difficult tasks with the highest masking ratios, because those tasks are said to align better with the pretrained model’s native behavior. Only later are simpler tasks introduced. The paper describes this as a hard-to-easy schedule in the introduction and as a counter-intuitive schedule in the method section.

The intuition is that starting with high-mask, high-difficulty tasks preserves the pretrained model’s ability to reconstruct broad video structure. Once the anchor task is established, simpler tasks are merged in to refine the task soup without destabilizing the pretrained representation.

Implicit task mixture with EMA

To avoid catastrophic forgetting while combining tasks, the authors use an inter-task Exponential Moving Average strategy. A pretrained-like anchor task is trained first, then additional tasks are integrated through EMA updates, implicitly mixing multiple task experts inside one model. The paper positions this as a way to achieve multi-task gains without maintaining separate task-specific models.

In ablations, removing EMA hurts performance, especially when tasks are jointly trained, and is associated with motion degradation.

Soup-of-Modals: Multi-Modal Fusion Across Timestep Phases

Coupled-Decoupled Multi-Modal Cross Attention

To inject text, audio, and image conditioning into the compact backbone, EchoMimicV3 uses a Coupled-Decoupled Multi-Modal Cross Attention module (CDCA). Text is encoded by umT5, audio by an audio encoder, and image by CLIP. The three modalities are then fused with shared-query, modality-specific key/value attention:

$$ z_o = \sum_{c \in \{t,i,a\}} W(c,\tau)\, \operatorname{CA}_c\bigl(Q_{\text{shared}}, K(c), V(c)\bigr) $$

Here, $c$ indexes text, image, and audio; $\tau$ denotes the diffusion timestep; and $W(c,\tau)$ is a timestep-dependent weight for each modality expert. The module is described as similar in spirit to IP-Adapter-style conditioning, but extended to multiple modalities and fused with phase-aware weights.

Multi-Modal PhDA: phase-aware dynamical allocation

The second part of the Soup-of-Modals is Multi-Modal timestep Phase-aware Dynamical Allocation (PhDA), which decides how much each modality should contribute at each diffusion phase. The paper argues that different modalities matter most at different phases: text remains broadly important throughout; image is most influential in early and middle phases; audio is especially relevant in the initial phase.

The weight schedule is piecewise-defined. If $\tau$ is the current timestep and $\mathcal{B}_c^1, \mathcal{B}_c^2$ are the left and right critical phase boundaries for modality $c$, then:

$$ W(c,\tau) = \begin{cases} 0.5 & \text{if } \tau < \mathcal{B}_c^1, \\ m\tau + b & \text{if } \mathcal{B}_c^1 \le \tau < \mathcal{B}_c^2, \\ 1 & \text{if } \tau \ge \mathcal{B}_c^2. \end{cases} $$

The authors describe the schedule as being sampled from a meta-distribution grounded in a phase-specific modal importance law. In ablation, misaligning the modality-phase correspondence harms the target behavior: omitting audio in early phases weakens lip sync, removing text in early/mid phases can cause motion collapse, and excluding image in early phases hurts identity preservation.

Audio injection details

Audio is handled with explicit temporal alignment to latent frames. Because the latent diffusion transformer downsamples time by a factor $r$, one latent frame corresponds to $r$ audio tokens. The audio stream is segmented into chunks of $r \times \alpha$ features, where $\alpha$ is the number of audio embeddings corresponding to one video frame. The segment center is selected, then expanded by $r + e$ features in both directions to form frame-aligned audio segments.

The aligned segments are injected through an audio cross-attention expert. The audio branch output is further modulated by a binary facial-region mask $\mathbb{M}_{\text{face}} \in \{0,1\}$ so that audio conditioning focuses on the face area and better supports natural lip synchronization and facial motion.

Training Strategy: Negative DPO inside an SFT Cycle

Motivation

Traditional Direct Preference Optimization is described as expensive and sensitive to preference-data quality. EchoMimicV3 replaces paired preference data with pairing-free negative samples derived from intermediate SFT checkpoints. The idea is to use suboptimal generations as negative evidence and then iteratively suppress those failure modes during training.

Negative Direct Preference Optimization

During SFT, the model periodically saves intermediate checkpoints $\{\mathcal{M}_{s_i}\}$. For each stage, a checkpoint generates videos, the authors identify negative samples from those outputs, and user feedback is used to annotate what is undesirable. Importantly, the method does not require explicit $(p, y^-, y^+)$ pairing. The negative preference data are denoted as $(p^-, y^-)$.

The paper states the stage-$i$ objective in simplified form as:

$$ \mathcal{L}^{i}_{\text{NDPO}}(\theta) = \mathbb{E}_{(p^-, y^-)}\bigl[-\log(\pi_\theta(y^- \mid p^-) + 1)\bigr]. $$

Operationally, this objective penalizes the model’s tendency to generate undesirable distributions. The training pipeline alternates between Negative DPO and SFT, forming a DPO-SFT cycle: the negative stage removes bad behaviors, and the SFT stage recovers and strengthens positive generation ability.

The paper’s ablation indicates that SFT alone is weaker than the NDPO-SFT cycle, and that a conventional paired DPO setup also underperforms. The authors attribute this to the lack of broad paired preference coverage and argue that their pairing-free negative data are more efficient.

Inference Strategy: Negative CFG and Long-Video Generation

Phase-aware Negative-enhanced CFG (PNG)

Because the model learns to reject bad samples during training, the authors extend that behavior into inference with Phase-aware Negative classifier-free Guidance (PNG). PNG strengthens negative prompts differently across diffusion phases. Motion-related negative prompts are emphasized early, while detail-related negative prompts are emphasized later. The intended effect is to suppress artifacts such as unnatural gestures, identity drift, and color inconsistencies.

In ablation, removing PNG leaves the model’s output qualitatively worse, especially in gesture quality. The paper specifically notes degradation when the initial timesteps of PNG are disabled.

Long Video CFG

For long clips, the paper improves the common sliding-window approach by carefully smoothing classifier-free guidance in overlapping regions. The authors identify overlap handling as the main source of unnatural transitions, color discrepancies, and identity inconsistency.

For overlap frame $f$, the proposed guided noise prediction is:

$$ \hat{\epsilon}_{\theta}^{w_o}(f) = \epsilon_{\theta}^{w}(f) + s\left(\sum_{i \in \{w,w+1\}} \alpha^i \epsilon_{\theta}^{i}(f) - \epsilon_{\theta}^{w}(\varnothing)\right), $$

with the overlap interpolation defined as

$$ \sum_{i \in \{w,w+1\}} \alpha^i \epsilon_{\theta}^{i}(f) = \left(1 - \frac{f}{N}\right)\epsilon_{\theta}^{w}(f) + \frac{f}{N}\epsilon_{\theta}^{w+1}(f). $$

Here, $s$ is the guidance scale, $N$ is the overlap length, and the two windows are blended linearly across the overlap. The paper reports that this reduces overexposure and color mismatch across windows and allows longer videos to be produced with smoother temporal continuity.

Experimental Setup

Implementation details

The model is trained on 64 GPUs with 96 GB memory each, using a learning rate of $10^{-4}$. The input video length is set to 113. The classifier-free guidance scale is 3 for text and 9 for audio. For efficiency, the authors pre-extract VAE latents and caption embeddings for the entire training set.

Training data

EchoMimicV3 is trained on a mix of the EchoMimicV2 dataset, the HDTF dataset, and additional self-collected data. The total training corpus is approximately 1,500 hours of video. The preprocessing pipeline includes audio synchronization and subtitle removal to improve data quality.

Evaluation metrics

The paper evaluates image quality with Fréchet Inception Distance ($\text{FID}$), video quality with Fréchet Video Distance ($\text{FVD}$), perceptual quality with IQA, aesthetic quality with ASE, lip-sync quality with Sync-C and Sync-D, and consistency with Vbench2.0 metrics: identity consistency ($\text{ID}$), human anatomy ($\text{HA}$), clothing consistency ($\text{HC}$), and background consistency ($\text{BC}$). For quantitative evaluation, 300 generated videos are randomly selected.

Main Results

The paper compares EchoMimicV3 against EchoMimicV2, HunyuanAvatar, OmniAvatar, Hallo3, MultiTalk, FantasyTalk, and related settings. The main message is that a 1.3B model can remain competitive with much larger systems, including 14B-parameter baselines, while also supporting multiple tasks in one framework.

In the main quantitative table, EchoMimicV3 achieves especially strong results on identity preservation, clothing consistency, background consistency, aesthetics, and balanced video quality. On some individual metrics, larger models still score better in isolation; for example, HunyuanAvatar has the highest Sync-C in the table. The paper’s claim is not that EchoMimicV3 wins every metric outright, but that it delivers a strong overall tradeoff with substantially lower parameter count and much better task unification.

Below is the full comparison table reported in the paper.

The paper also reports that, with TeaCache, the method can generate a 5-second talking-human video in about 4 minutes using 25 inference steps, and that talking-head generation is achieved within 5 inference steps. The headline efficiency claim is an approximately $18\times$ speedup compared with 14B-parameter models in the cited comparison.

Multi-Task Capability

Beyond talking-head generation, the paper demonstrates that the same model can handle multiple animation tasks, including lip synchronization (LC), image-audio-to-video (IA2V), and first-and-last-frame-audio-to-video (FLFA2V). The qualitative results suggest that the shared masking-and-modality design is flexible enough to support different generation modes without separate task-specific models.

Ablation Studies

The paper includes a detailed ablation table covering task scheduling, EMA, modality allocation, Negative DPO, PNG, and Long Video CFG. These ablations are important because they show that the paper’s gains are not just from backbone scale; they depend on the specific training and inference recipe.

Task scheduling and EMA

Task Schedule A is an easy-to-hard sequence that starts with lip sync and progresses toward image-to-video, while Task Schedule B is uniform random sampling. The paper reports that the counter-intuitive schedule used by EchoMimicV3 performs better than both alternatives, especially for lip synchronization, body motion accuracy, and identity preservation. This supports the claim that task allocation matters as much as architecture in compact multi-task diffusion.

Removing EMA hurts performance across tasks, and the authors specifically note motion issues when jointly training without the moving-average mechanism.

Modality-phase allocation

The three modality ablations show that the phase schedule is nontrivial. The paper’s interpretation is that audio should be emphasized early for lip sync, text should not be dropped too early because it helps maintain motion coherence, and image conditioning must remain present in the earlier phases to preserve identity.

Negative DPO

The NDPO-SFT cycle outperforms SFT alone and a conventional SFT + paired DPO setup. The authors say that paired DPO tends to suffer from identity consistency problems when preference pairs are drawn only from the original training set, whereas Negative DPO can generate additional paired preference signals more efficiently because it is driven by negative examples extracted from intermediate checkpoints.

PNG and Long Video CFG

PNG improves gesture quality by rejecting negative behaviors more strongly at the appropriate timesteps. Long Video CFG improves temporal consistency over sliding windows, reducing overexposure and color discontinuity in long clips. These two inference-time mechanisms are important because they extend the training-time negative-sample idea into generation-time control.

Qualitative Comparison

The paper includes side-by-side qualitative comparisons with prior methods for talking-human animation. The qualitative figure is used to illustrate the visual benefits of the compact unified model, especially in identity consistency, motion realism, and overall aesthetics. This is the primary visual evidence supporting the claim that EchoMimicV3 can be competitive with larger systems.

What Is New Relative to Prior Human-Animation Systems

Relative to prior work, the paper’s novelty is less about a new video backbone and more about the combination of: (1) a task-unifying masked reconstruction perspective, (2) phase-aware multi-modal attention allocation, (3) pairing-free negative preference optimization embedded into SFT, and (4) phase-aware guidance at inference. The result is a single model that is intended to replace multiple specialized experts while remaining small enough to be computationally practical.

The related-work discussion places the method in the context of diffusion-based video generation, talking-head synthesis, and multi-modal avatar systems. EchoMimicV3 is positioned as a response to the fact that DiT/LVDM-based human animation methods have improved quality but often struggle with lip sync, long-duration generation, and operational efficiency.

Stated Contributions

• A lightweight human-animation framework that supports both multi-task unification and multi-modal conditioning in a single 1.3B model.
• Soup-of-Tasks, which uses shared masked sequence inputs and a counter-intuitive inter-task training schedule to unify tasks without separate experts.
• Soup-of-Modals, which combines a Coupled-Decoupled Multi-Modal Cross Attention module with a phase-aware modality allocation mechanism.
• Negative DPO embedded into SFT to dynamically reject undesirable outputs using pairing-free negative samples.
• Phase-aware Negative CFG and Long Video CFG for more vivid outputs and more stable long-form generation.
• Empirical competitiveness against much larger models, with the paper emphasizing strong overall quality and efficiency rather than raw scale.

Limitations and Practical Considerations

The provided paper text does not contain a dedicated limitations section, so the summary below is limited to what is explicitly stated or directly implied by the method and experiments. First, the model still depends on a fairly large training corpus of about 1,500 hours and on substantial compute for training on 64 GPUs. Second, the training and inference recipes are relatively elaborate: the gains depend on task scheduling, EMA, Negative DPO, PNG, and Long Video CFG, which means the system is simple in parameter count but not trivial in optimization. Third, while the method is compact relative to 14B baselines, the paper still acknowledges that some larger models can outperform it on isolated metrics, so the contribution is best understood as an efficiency-quality-task-unification tradeoff rather than a universal win on every score.

Conclusion

EchoMimicV3 is a compact 1.3B-parameter human-animation system that unifies multiple tasks and multiple conditioning modalities through a carefully engineered masking, attention, training, and guidance strategy. Its main technical idea is to turn task diversity into mask diversity, turn modality importance into phase-aware attention weights, and turn bad generations into training signal through pairing-free negative optimization. The empirical story is that these ingredients let a relatively small model remain competitive with much larger baselines while supporting more than one animation regime inside a single framework.