EchoMimic

EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions

EchoMimic is a diffusion-based system that generates lifelike talking-head videos by combining audio and facial landmarks. It supports audio-only, landmark-only, or combined conditioning to produce natural and stable portrait animations, addressing limitations of previous single-signal methods.

• audio-driven
• talking-head
• portrait-animation
• face-animation
• face-reenactment
• lip-sync

Demos

The selected demos showcase EchoMimic's capability to generate lifelike portrait animations driven by audio inputs in multiple languages and styles, including speech in Chinese and English as well as singing. These videos highlight the method's stability and naturalness in lip-syncing and facial movements, addressing common issues of instability from audio-only and unnaturalness from key-point-only approaches. When evaluating, focus on the smoothness, expressiveness, and synchronization of the facial animations with the audio cues.

Authors: Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma

Categories: cs.CV

Published 2024-07-11 · Updated 2024-07-12

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads156 likes

Impact

• Semantic Scholar216 citations45 influential citations

Abstract

The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.

---

Introduction

EchoMimic addresses audio-driven portrait animation, specifically the problem of generating lifelike talking-head videos from a reference portrait together with audio, facial landmarks, or both. The paper frames the challenge as a tradeoff between two common conditioning signals: audio is semantically rich but can be unstable because it is relatively weak as a control signal, while facial landmarks provide stronger geometric guidance but can overconstrain motion and lead to unnatural results. The stated goal of EchoMimic is to combine these signals in a single diffusion-based system that can operate in three modes: audio only, landmark only, and audio plus selected landmarks.

The paper positions its contribution in the context of diffusion-model-based portrait animation, where recent methods have improved realism but still struggle with temporal coherence, facial structure, and the balance between flexibility and control. EchoMimic claims that concurrent training on audio and landmarks, together with a novel training strategy, enables better stability than audio-only methods and more natural outputs than landmark-only methods. The method is evaluated on public datasets and on a collected dataset, with quantitative metrics and qualitative comparisons reported against several open-source baselines.

Stated contributions in the paper are:

• a diffusion-based portrait animation framework that jointly uses audio and facial landmarks;
• a training strategy that supports three inference modes: audio only, landmark only, and audio plus selected landmarks;
• a landmark handling strategy for robust landmark-based control and a part-aware motion synchronization method for inference alignment;
• a timestep-aware spatial loss to improve facial detail learning in pixel space;
• extensive comparisons on HDTF, CelebV-HQ, and a collected dataset.

Background and Problem Setting

The method is built on the Stable Diffusion family of latent diffusion models. The paper reviews the standard pipeline in which an image $x_0$ is encoded by a VAE into a latent $z_0 = E(x_0)$, Gaussian noise is added to produce $z_t$ at diffusion step $t$, and a UNet denoiser predicts the noise $epsilon_\theta(z_t,t,c)$ conditioned on an input context $c$. The training objective in this setup is the standard denoising loss:

$$ \mathcal{L} = \mathbb{E}_{t,c,z_t,\epsilon}\left[\left\lVert \epsilon - \epsilon_\theta(z_t,t,c) \right\rVert^2\right]. $$

EchoMimic extends this latent-diffusion backbone from image generation to portrait video generation by injecting multiple conditioning streams: a reference image for identity preservation, audio for speech-driven motion, and landmarks for explicit geometric control. The paper’s motivation is that a single conditioning source is insufficient for robust portrait animation across styles and identities.

Method

Overall Architecture

The core of EchoMimic is a denoising UNet that operates over multi-frame latents, with three specialized attention mechanisms inside each transformer block:

• Reference-Attention to connect the current video frame with the reference portrait;
• Audio-Attention to incorporate speech cues into the denoising process;
• Temporal-Attention to model dependencies across frames and promote temporal coherence.

The framework also includes a Reference UNet, an Audio Encoder, and a Landmark Encoder. The design is illustrated in the pipeline figure below.

Denoising UNet

The denoising UNet is adapted from SDv1.5 and modified for multi-frame generation. The paper emphasizes that each transformer block contains the three attention layers listed above. The Reference-Attention layer receives keys and values from the reference-image pathway, the Audio-Attention layer introduces speech-conditioning at the spatial level, and the Temporal-Attention layer performs self-attention over the temporal axis to capture motion patterns between frames.

For the temporal module, the hidden state is described as $h \in \mathbb{R}^{b \times f \times d \times h \times w}$, where $b$ is batch size, $f$ is the number of frames, $d$ is feature dimension, and $h,w$ are spatial dimensions. It is reshaped to $\mathbb{R}^{(b \times h \times w) \times f \times d}$ so that self-attention can be applied along the frame dimension. The paper’s intent is to let each spatial location attend across time, improving frame-to-frame consistency.

Reference UNet

The reference image pathway is used to preserve identity and background consistency. The Reference UNet mirrors the SDv1.5 architecture and runs in parallel with the denoising UNet. It encodes the reference portrait without adding noise, and its self-attention features are reused as the key/value inputs for the Reference-Attention layers in the denoising network. The paper notes that an empty text placeholder is supplied to the ReferenceNet cross-attention layer so that no extraneous textual content contaminates the reference features.

Audio Encoder

Speech-driven motion is encoded using features extracted from a pretrained Wav2Vec model. The paper states that the representation for each generated frame is formed by concatenating features from adjacent audio frames, so that the model can exploit both past and future speech context. These audio features are then injected through Audio-Attention in the denoising UNet, where cross-attention is performed between the latent code and the audio-conditioned features following the Reference-Attention stage.

Landmark Encoder

The Landmark Encoder is described as a lightweight convolutional module that maps a facial-landmark image into a latent-aligned feature representation. Those landmark features are added element-wise to the multi-frame latents before the denoising UNet processes them. This direct fusion is intended to provide precise geometric guidance for facial structure and articulation while keeping the implementation simple.

Spatial Loss

Because latent-space resolution is relatively coarse compared with pixel-level facial detail, the paper introduces a timestep-aware spatial loss to supervise image reconstruction directly in pixel space. The predicted latent is first mapped back to $z_0$ through the sampler, then decoded by the VAE to obtain an image prediction $I_p$. The spatial supervision uses both an $L_2$ reconstruction term and LPIPS:

$$ L_{\text{spatial}} = w(t)\left[L_2(I_p, I_{GT}) + LPIPS(I_p, I_{GT})\right]. $$

The timestep weighting is defined as

$$ w(t) = \cos\left(\frac{t\pi}{2T}\right), $$

where $T$ is the total diffusion horizon. The overall objective is then written as

$$ Obj = L_{\text{latent}} + \lambda L_{\text{spatial}}. $$

The paper’s rationale is that large diffusion timesteps are harder to optimize, so the cosine weight down-weights their contribution and helps the model learn facial structure more effectively.

Training Strategy

EchoMimic uses a two-stage training procedure:

• Stage 1: train the Reference UNet and denoising UNet on single-frame data to learn image-audio and image-pose relationships. The temporal-attention layer is not inserted at this stage.
• Stage 2: insert the temporal-attention layer into the denoising UNet and train the overall pipeline on 12-frame videos. The paper says only the temporal model is trained in this stage while other parts are frozen.

The authors also add two implementation-level techniques intended to improve robustness and quality:

• Random Landmark Selection (RLS): the face is split into parts, including eyebrows, eyes, pupils, nose, and mouth; one or more parts are randomly dropped during training to make landmark conditioning more robust.
• Audio augmentation: noise and perturbations are added to audio to make the audio conditioning more resilient, analogous to image augmentation.

The paper further reports that the training routine randomly omits the reference image, guiding audio, and motion frames with probability $5\%$ to improve robustness. The motion module is initialized with pretrained weights from AnimateDiff.

Inference and Motion Alignment

At inference time, the model supports three operating modes: audio-driven, landmark-driven, and audio-plus-selected-landmark-driven generation. The paper notes that audio-only inference is straightforward, but pose- or landmark-driven inference requires alignment between the driving pose and the reference portrait.

To address limitations of full-face affine warping, EchoMimic proposes part-aware motion synchronization. The face is split into parts, a global transformation is estimated on the full face, and then residual transformations are computed for each part and added to the global matrix to obtain the final transform. The paper argues that this better matches the shape of the reference face and allows landmark motion to be mapped more faithfully across unusual portrait styles.

The paper’s qualitative discussion claims that this part-aware mapping can preserve identity while adapting landmarks to portraits with substantially different face shapes. In the example described in the text, the method can project a small mouth from the reference image onto a much larger mouth in the target portrait.

Experimental Setup

Implementation Details

Experiments were conducted on 8 NVIDIA A100 GPUs. Training is described as two segments of 30,000 steps each, with batch size 4 and a resolution of $512 \times 512$. The learning rate is $10^{-5}$ throughout both stages. In the second phase, the paper states that 14 video frames are generated per iteration, with latent variables from the motion module integrated with the first 2 actual video frames to preserve continuity. For inference, the system merges noisy latents with features from the most recent 2 motion frames from the previous step to maintain sequential coherence.

Datasets

The paper trains on approximately 540 hours of web-collected talking-head video, corresponding to about 130,000 clips of roughly 15 seconds each, and supplements this corpus with HDTF and CelebV-HQ. The data-cleaning pipeline keeps videos of a single speaking person with strong lip-audio correspondence and removes clips with scene cuts, strong camera motion, excessive facial motion, or fully profile viewpoints. Facial landmarks are extracted using MediaPipe.

Evaluation Metrics

The paper evaluates using FID, FVD, SSIM, and E-FID. Lower FID and FVD indicate more realistic images and videos, SSIM measures structural similarity, and E-FID is computed from expression parameters extracted via a face-reconstruction method to measure expression fidelity.

Baselines

EchoMimic is compared with SadTalker, AniPortrait, V-Express, and Hallo on HDTF, CelebV-HQ, and the collected dataset. The paper reports both quantitative and qualitative comparisons.

Quantitative Results

On HDTF, EchoMimic achieves the best score on all reported metrics: FID $29.136$, FVD $492.784$, SSIM $0.812$, and E-FID $1.112$. The paper interprets this as strong evidence of improved visual fidelity, temporal coherence, and lip synchronization.

On CelebV-HQ, the paper reports that all methods perform worse than on HDTF because the video quality is lower. EchoMimic obtains the best FID and E-FID, while Hallo achieves the best FVD and SSIM. The exact numbers are:

On the collected dataset, EchoMimic again reports the lowest FID, FVD, and E-FID, while its SSIM is very close to the best result. The paper interprets this as evidence that the method generalizes across more challenging and diverse scenarios.

Qualitative Results

The qualitative section emphasizes three driving modes. In audio-driven generation, the method is said to handle diverse speech inputs and produce visually consistent animations across portrait styles.

In landmark-driven generation, the paper highlights the benefit of part-aware motion synchronization for transferring expression while preserving portrait identity. The method is also presented as capable of handling large pose changes and reproducing nuanced expressions more faithfully than prior landmark-mapping approaches.

In audio plus selected landmark generation, the paper argues that users can maintain lip synchronization while explicitly controlling additional facial actions such as blinking or eye closure. This mode is presented as the most controllable setting and is used to demonstrate that selected landmarks can constrain specific facial regions without fully overriding the audio signal.

Ablation Study

Effect of Motion Synchronization

The paper uses landmark mapping examples to validate part-aware motion synchronization. The qualitative claim is that the new alignment procedure better adapts driving landmarks to the reference face shape than previous full-face warping.

Effect of Driving Mode

The paper also reports a driving-mode ablation on HDTF. It interprets the three settings as a spectrum of control strength: audio-only has the weakest geometric constraint and therefore the most freedom, landmark-only has the strongest constraint and gives the closest resemblance to the original video, and audio-plus-landmarks lands in between.

Here, $A$ denotes audio-only driving, $L$ denotes landmark-only driving, and $A+L$ denotes audio plus landmarks with the mouth region excluded. The reported numbers show that strong landmark constraints reduce reconstruction distance substantially on HDTF, while the combined mode remains close to landmark-only performance. The paper uses this ablation to support the claim that the system can trade off control and naturalness through conditioning design.

Limitations and Future Work

The paper explicitly lists two main limitations. First, the framework is still an extension of Stable Diffusion-style image processing to the video domain, rather than a purpose-built video generation architecture. The authors suggest that future work could re-formulate the approach around more native video frameworks such as 3DVAE or DiT. Second, inference speed remains a challenge, and the paper points to SD acceleration methods as a route toward real-time generation. The authors connect this to potential applications in digital-human interaction and real-time conversation systems.

Conclusion

EchoMimic is presented as a diffusion-based portrait animation system that jointly models audio and facial landmarks to improve the stability, realism, and controllability of talking-head generation. The paper’s central technical idea is to combine reference-image preservation, audio conditioning, landmark conditioning, and temporal attention within a unified latent-diffusion pipeline. Across HDTF, CelebV-HQ, and the collected dataset, the paper reports strong quantitative performance and qualitatively plausible animations in audio-driven, landmark-driven, and audio-plus-landmark settings. The authors conclude that the approach is a promising step toward more expressive and editable portrait animation.