EMO

EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

EMO generates expressive and realistic talking head videos from a single image and audio, without relying on 3D models or landmarks. It uses weak controls to enable natural facial expressions and head motions for talking and singing, preserving identity and ensuring smooth long-duration video synthesis.

• talking-head
• audio-driven
• portrait-animation
• face-animation
• head-pose

Demos

The demos showcase EMO's ability to generate expressive portrait videos from a single reference image and vocal audio, supporting both singing and talking inputs. Evaluate the naturalness and accuracy of facial expressions, head pose variations, and lip-sync synchronization with different languages and rapid rhythms. The videos highlight the system's capacity to animate diverse characters, including AI-generated avatars and historical portraits, maintaining their identity over time.

Authors: Linrui Tian, Qi Wang, Bang Zhang, Liefeng Bo

Categories: cs.CV

Published 2024-02-27 · Updated 2024-08-08

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub7,616 stars328 watchers931 forks

Impact

• Semantic Scholar266 citations30 influential citations

Abstract

In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

---

1. Problem Setting and Core Idea

EMO addresses audio-driven talking head generation with a focus on expressiveness, identity preservation, and long-duration continuity. The paper argues that many prior approaches rely on strong intermediate controls such as 3D morphable models, facial landmarks, blendshapes, or predefined pose sequences. Those controls can improve lip synchronization or motion stability, but they also restrict the range of facial nuance and head movement, especially for spontaneous expressions and singing. EMO instead learns a direct audio-to-video synthesis pathway that generates portrait videos from a single reference image and audio, without using explicit 3D or landmark intermediates.

The design goal is not to strictly constrain the face, but to guide it with weak conditions that stabilize generation while preserving generative freedom. In the paper’s framing, this lets the model better capture the one-to-many nature of speech-to-motion correspondence: the same audio can support different plausible head motions and expressions, and those motions can vary across speaking styles and singing.

2. Method Overview

EMO is built on top of Stable Diffusion as the backbone diffusion framework and adapts it from image generation to video generation. The model denoises a sequence of latent video frames rather than a single image latent, and it adds modules for temporal coherence, identity conditioning, audio conditioning, and weak spatial/motion control.

The base diffusion objective follows the standard denoising loss:

$$\mathcal{L} = \mathbb{E}_{t,c,z_t,\epsilon}\left[\|\epsilon - \epsilon_\theta(z_t,t,c)\|^2\right]$$

where the conditioning signal $c$ is text in ordinary Stable Diffusion, but in EMO the paper replaces text conditioning with reference features and audio features. The architecture has four key pieces: Backbone Network, ReferenceNet, Audio Layers, and Temporal Modules. Two weak conditioning mechanisms, the Face Locator and Speed Layers, further guide motion without imposing a rigid pose or expression template.

2.1 Backbone Network and ReferenceNet

The Backbone Network has a UNet structure similar to SD 1.5. Unlike standard SD, the paper does not use prompt embeddings, so the original cross-attention layers are adapted into reference-attention layers that attend to image features instead of text features. This is the mechanism that transfers identity information from the reference image into the generated video.

The ReferenceNet is a UNet-like network with the same architecture as the backbone and initialized from the same SD weights. It processes the reference portrait image and extracts feature maps from its self-attention layers. During denoising, the backbone uses those feature maps through reference-attention to preserve the person’s appearance across frames.

The paper also reuses the extracted reference features across the denoising process, which avoids repeatedly recomputing the same image features and keeps inference overhead manageable.

2.2 Audio Layers

Audio drives the speaking or singing motion. EMO uses a pretrained Wav2Vec encoder to extract audio features, then concatenates features from neighboring frames to form the per-frame audio representation. For the $f$-th generated frame, the paper defines an audio embedding by concatenating the features from nearby audio segments:

$$\mathbf{A}^{(f)} = \bigoplus\{A^{(f-m)},\ldots,A^{(f)},\ldots,A^{(f+m)}\}$$

where $m$ is the number of additional features taken from each side. This is meant to account for the fact that facial motion is not driven only by the current instant of audio; nearby context can matter for anticipation effects such as mouth opening before speech or inhalation-like pauses. The audio embedding is injected through audio-attention layers that follow the reference-attention layers in the backbone.

2.3 Temporal Modules and Motion Frames

To produce temporally coherent video, EMO inserts self-attention temporal layers at every resolution level of the backbone. The feature tensor $x \in \mathbb{R}^{b \times c \times f \times h \times w}$ is reshaped so attention operates across the frame dimension $f$, enabling the model to learn correlations among adjacent generated frames.

A notable design choice is the use of motion frames for clip-to-clip continuity. For a new clip, EMO takes the last $n$ frames from the previous generated clip and feeds them to the ReferenceNet as motion context. Those features are then merged with the temporal backbone during denoising. For the first clip, the motion frames are initialized as zero maps. The paper emphasizes that this feature reuse supports long videos with seamless transitions, and that the motion frames are only encoded once, so they do not substantially increase inference time.

2.4 Weak Conditions: Face Locator and Speed Layers

Instead of strong geometric controls, EMO uses two weak control signals. The first is the Face Locator, which receives a face-region mask and encodes it with lightweight convolutional layers. The mask is defined as the union of face regions across the clip, $\mathbf{M} = \bigcup_{i=1}^{f} M^i$, and is added to the noisy latent before denoising. This tells the generator where the face should appear without forcing a fixed pose or rigid location.

The second is the Speed Layer, which models head rotation speed in a weakly controlled way. The paper discretizes speed into buckets and maps the measured head velocity $w^f$ into a speed vector. For bucket $i$ with center $c_i$ and radius $r_i$, the value is

$$s_i = \tanh\left(\frac{w^f - c_i}{r_i} \cdot 3\right)$$

and the per-frame speed embedding is formed by concatenating neighboring frame speeds:

$$\mathbf{S}^f = \bigoplus\{\mathbf{s}^{(f-m)},\ldots,\mathbf{s}^{(f)},\ldots,\mathbf{s}^{(f+m)}\}$$

This speed embedding is passed through an MLP and then used in temporal cross-attention. The paper’s intent is to stabilize rotation frequency across clips while still allowing the network freedom to generate natural motion. The authors explicitly frame both the face locator and speed layers as weak conditions, not strict controls.

3. Training Strategy

EMO is trained in three stages.

• Stage 1: image pretraining. The backbone, ReferenceNet, and Face Locator are trained on single frames. The backbone receives one frame, while ReferenceNet receives a different randomly selected frame from the same clip. Both networks initialize from Stable Diffusion.
• Stage 2: video training. Temporal modules and audio layers are introduced. The model samples $n + f$ contiguous frames from a clip, where the first $n$ are motion frames. The temporal modules initialize from AnimateDiff.
• Stage 3: speed-layer training. The speed layers are added, and the paper trains only the temporal modules and speed layers. Audio layers are deliberately omitted at this stage so that the model does not learn a shortcut in which motion is driven by the speed signal instead of audio.

The rationale for this staged schedule is that audio, expression, mouth movement, and head motion are correlated, and joint optimization of audio and speed layers can weaken the model’s audio-driven behavior. The final stage therefore isolates speed modeling to reduce interference.

4. Datasets, Preprocessing, and Implementation Details

The training data combines three sources: about 250 hours of internet-collected talking-head videos, HDTF, and VFHQ. The paper reports that HDTF contains about 15.8 hours and roughly 362 unique identities, while VFHQ contains 16k high-resolution talking-video clips without audio. VFHQ is only used in the first training stage because it lacks audio. The collected internet dataset has over 10,000 unique identities and includes both English and Chinese content.

The authors deliberately avoid VoxCeleb because its frequent centering on facial centroids can create unstable camera motion. They also state that the collected data keeps the subject position, camera angle, and background relatively fixed, and that each frame contains a single individual.

Preprocessing uses PySceneDetect to segment videos into temporally coherent clips of roughly 3 to 12 seconds, mitigating discontinuities from scene switches and jump cuts. Clips are cropped using expanded facial bounding boxes, converted to 30 FPS, and resized/cropped to 512 × 512. The labels and auxiliary signals are produced with MediaPipe for face boxes and facial landmarks, Wav2Vec for audio embeddings, and 6-DoF head-pose estimation for frame-by-frame rotation velocities.

The implementation settings reported in the paper are: batch size 48 for stage 1, batch size 4 for stages 2 and 3, video length $f = 12$, motion frames $n = 4$, additional audio features $m = 2$, and learning rate $10^{-5}$ for all stages. During inference, EMO uses DDIM sampling for 40 steps. The main paper reports roughly 15 seconds per 12-frame batch, while the supplement reports about 18 seconds per 12 frames on an A100 GPU under 40 denoising steps.

5. Experimental Setup and Metrics

For comparisons, the authors split HDTF into 90% training and 10% test sets, with no overlap in character identity between the split subsets. To probe harder and more diverse conditions, they also extract 1k clips from the internet dataset, each around 4 seconds, with many clips showing expressive portrait motions and singing. The paper compares against Wav2Lip, SadTalker, DreamTalk, and MakeItTalk; Diffused Heads is only compared qualitatively because the released model is trained on CREMA with a green background and suffers from low resolution and error accumulation.

The evaluation metrics are:

• FID for per-frame image quality;
• F-SIM for identity preservation, treated as a population-reference metric where values closer to the ground-truth reference are better;
• FVD for video-level quality;
• SyncNet score for lip synchronization;
• E-FID for expression realism, computed by extracting expression parameters from face reconstruction and then applying FID to those expression features.

The paper also reports a user study in the supplement with 20 participants evenly split between 10 males and 10 females, with ages ranging from 20 to 60. Participants rated lip sync and vividness on a 1–5 scale.

5.1 Quantitative Results

The authors report that EMO performs especially well on FID, FVD, and E-FID, while Wav2Lip retains the best SyncNet scores because it is trained with SyncNet as a discriminator. EMO’s SyncNet score is not the best, but the paper emphasizes that the method produces much more vivid and expressive motion than SyncNet-optimized baselines.

The paper’s main quantitative claims are straightforward: EMO achieves the best overall frame quality and video coherence on both HDTF and the internet dataset, and it is strongest on expression realism by a wide margin. The ablation without the 250-hour internet dataset is still competitive, which the authors interpret as evidence that the public datasets alone provide a workable base and that the extra data mainly improves dynamics and expression variety.

5.2 User Study

In the user study, EMO is rated best on both lip synchronization and especially vividness, reinforcing the paper’s central claim that direct audio-to-video diffusion can produce more lively portraits than methods that overconstrain motion.

6. Qualitative Findings

The qualitative comparisons emphasize three recurring behaviors. First, compared with Wav2Lip, SadTalker, DreamTalk, and MakeItTalk, EMO generates more dynamic head motion and a wider range of facial expression. Wav2Lip is described as producing blurry mouth regions and mostly static head pose and eye motion when driven from a single reference image. DreamTalk’s style clips can distort identity and limit dynamics. SadTalker and DreamTalk are less expressive in head movement than EMO.

Second, the model generalizes across portrait styles. Even though training is described on realistic video data, the paper shows examples using reference images from different T2I-generated styles, including realistic, anime, and 3D. With the same vocal input, the resulting clips maintain roughly consistent lip synchronization across styles.

Third, EMO produces richer movement for audio with strong tonal variation, especially singing. The paper highlights that high-pitched segments can elicit more intense expressions, and that motion frames allow the video to continue for long durations as long as the input audio is long enough.

The supplementary comparison with Diffused Heads shows another practical advantage: EMO’s results are presented as higher resolution and less affected by error accumulation across successive frames, whereas Diffused Heads is reported to degrade over time.

7. Ablations and Control Analysis

The ablation study focuses on two weak controls: speed layers and the face locator. The paper’s interpretation is that these controls stabilize generation without collapsing diversity.

7.1 Speed Layer Ablation

The speed layer is evaluated on HDTF by assigning different speed values during inference. The paper measures Mean Velocity, Velocity Variance, and Variance of Mean Velocities (VMV). The results show that speed conditioning reduces both within-clip velocity variance and cross-clip variance, making head motion more stable.

The authors also report a practical heuristic: speech-driven cases generally use speed values from 0.1 to 1.0, while singing scenarios can use higher settings around 1.0 to 1.3. Speeds above about 1.5 can become unnaturally rapid and jittery.

The paper concludes that the speed layer successfully influences actual motion speed while also improving consistency. Importantly, it is still only a weak prior: it nudges motion frequency rather than forcing exact rotation trajectories.

7.2 Face Locator Ablation

The face locator controls where the face can appear and how much room is allowed for movement. The paper shows that enlarging the face region gives the character more room to swing the head, increasing motion range; increasing height can promote nodding; and using a white mask yields the least specific guidance, allowing facial generation in arbitrary positions. This supports the broader thesis that EMO uses weak spatial conditions rather than hard pose constraints.

7.3 Inference Steps and Temporal Stability

The supplement reports that inference step count matters strongly. Fewer than 20 denoising steps tends to produce temporal inconsistencies and many artifacts; the range from 20 to 35 improves quality but can still show jitter; and more than 35 steps gives noticeably more stable and coherent results. This aligns with the main paper’s use of 40 DDIM steps for generation.

The supplement also clarifies that clip transitions can still be sensitive to jump cuts, because the model does not use strong motion control signals such as pose sequences. Motion frames help with continuity, but the authors acknowledge that they are not a perfect substitute for explicit cross-clip motion supervision.

8. Limitations

The paper is explicit about several limitations. First, because EMO does not use strong control signals, it may accidentally generate body parts such as hands when the audio carries strong emotion. The authors link this to the training data: expressive clips often co-occur with more body movement, but the dataset is heavily head-centric and contains only about 3% frames with hands. This mismatch can create incorrect limb renderings or other artifacts.

Second, the model can occasionally produce subtitle-like patterns, attributed to internet training clips containing embedded subtitles. The paper notes that this is a known artifact category in image/video generation and is not unique to EMO.

Third, the reliance on audio as the principal control signal means the output expression is learned from correlations in the training data rather than explicitly user-defined emotion states. As a result, the generated emotion can be less controllable than a system with dedicated emotion inputs.

Fourth, the method is more time-consuming than diffusion-free talking-head approaches. The supplement reports about 12 frames per 18 seconds on an A100 GPU under 40 denoising steps. Finally, because the system is weakly conditioned, it can still be sensitive to scene discontinuities and jump cuts in the source data.

9. Takeaway

EMO’s main contribution is a fully generative audio-driven portrait-video framework that avoids explicit 3D or landmark intermediates, yet still preserves identity and generates expressive motion. The architectural novelty lies in combining a diffusion backbone with reference-attention, audio-attention, temporal modules, motion frames, and two weak controls for face location and motion speed. The experimental results support the paper’s claim that this design improves expressiveness and realism while remaining compatible with long-form portrait video generation, including singing and style-diverse portraits.