Hallo

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

Hallo presents a hierarchical audio-driven synthesis for portrait animation, decomposing lip, expression, and pose motions for precise lip-sync and diverse motion from speech audio. It generates high-quality, temporally consistent animations in an end-to-end diffusion framework without intermediate facial models.

• talking-head
• audio-driven
• lip-sync
• head-pose
• face-animation
• portrait-animation

Authors: Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, Siyu Zhu

Categories: cs.CV

Comment: 20 pages

Published 2024-06-13 · Updated 2024-06-16

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads162 likes

Impact

• Semantic Scholar216 citations53 influential citations

Abstract

The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion. Our proposed network architecture seamlessly integrates diffusion-based generative models, a UNet-based denoiser, temporal alignment techniques, and a reference network. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities. Through a comprehensive evaluation that incorporates both qualitative and quantitative analyses, our approach demonstrates obvious enhancements in image and video quality, lip synchronization precision, and motion diversity. Further visualization and access to the source code can be found at: https://fudan-generative-vision.github.io/hallo.

---

Introduction

Hallo addresses portrait image animation from a single reference portrait and a driving speech audio clip. The paper positions this task as the joint problem of (i) keeping lip motion, facial expression, and head pose synchronized with audio and (ii) generating outputs that are both visually high-fidelity and temporally stable across frames. The central design choice is to stay fully within an end-to-end diffusion formulation rather than relying on intermediate parametric facial representations such as landmarks, 3DMM coefficients, or other hand-crafted motion proxies.

The main contribution is a hierarchical audio-driven visual synthesis module that decomposes audio-conditioned motion guidance into lip, expression, and pose regions, then fuses them adaptively. In the authors’ framing, this allows the model to control motion diversity more flexibly while improving audio-visual alignment. The system combines a Stable Diffusion-style backbone, a UNet denoiser, a ReferenceNet for identity and texture preservation, and temporal alignment for sequence coherence.

Problem Setting and High-Level Approach

The task is: given a single face image and an audio clip, generate a video with $S$ frames that preserves the source identity while producing mouth, facial-expression, and head-pose motion consistent with speech. The paper emphasizes that prior pipelines often depend on intermediate facial representations, which can limit expressiveness and introduce errors. Hallo instead follows the latent diffusion paradigm used in recent end-to-end portrait animation work and adds finer-grained audio-to-motion alignment inside the denoising network.

Conceptually, the method has three major parts:

• Identity preservation through a pre-trained face encoder and a ReferenceNet conditioned on the first reference frame.
• Audio-driven motion generation through wav2vec-based audio features and hierarchical cross-attention to lip, expression, and pose regions.
• Temporal consistency through a video diffusion architecture with temporal alignment and motion-module initialization from AnimateDiff.

Methodology

Latent Diffusion Background

The paper builds on latent diffusion. Given an image $I$, an encoder maps it to a latent representation $z_0 = \mathcal{E}(I)$. During training, noise is added over timesteps to obtain $z_t$, and the denoising objective is the standard noise-prediction loss

$$ L = \mathbb{E}_{\mathcal{E}(I),\, c_{\text{embed}},\, \epsilon \sim \mathcal{N}(0,1),\, t} \left[\lVert \epsilon - \epsilon_\theta(z_t, t, c_{\text{embed}}) \rVert_2^2\right]. $$

Here, $\epsilon_\theta$ is the denoising UNet with residual and attention blocks, and the paper notes that sampling at inference can be performed with DDIM-style deterministic denoising. Hallo follows this general formulation but replaces the usual text conditioning with audio and reference-image conditioning.

Cross-Attention as Motion Guidance

Hallo uses cross-attention to inject motion guidance into the latent denoising process. In the paper’s notation, queries are derived from the noisy latent and keys/values from the conditioning signal. A generic form is

$$ Q = W_Q z_t, \qquad K = W_K c_{\text{embed}}, \qquad V = W_V c_{\text{embed}}, $$

with the cross-attention output computed by the attention operator. This mechanism is the basis for both identity conditioning and the proposed audio-to-region motion control.

Hierarchical Audio-Driven Visual Synthesis

The key novelty is the hierarchical decomposition of audio-conditioned motion into lip, expression, and pose components. For each frame $s$, the model maintains a latent $z_t^{(s)}$ at diffusion timestep $t$. The face image is first processed to create masks for the three motion regions. The paper uses MediaPipe landmarks to define bounding boxes for the lip and expression regions, then derives masks as follows:

$$ M_{\text{lip}} = Y_{\text{lip}}, \qquad M_{\text{exp}} = (1 - M_{\text{lip}}) \odot Y_{\text{exp}}, \qquad M_{\text{pose}} = 1 - M_{\text{exp}}. $$

The audio embedding for each frame is first produced by wav2vec. The paper concatenates the final 12 layers of wav2vec features and passes them through three linear layers to obtain per-frame audio embeddings $c_{\text{audio}}^{(s)}$.

Audio is then injected into the video latent via cross-attention:

$$ o_t^{(s)} = \operatorname{CrossAttn}(z_t^{(s)}, c_{\text{audio}}^{(s)}). $$

The resulting feature map is split by the three masks into pose, expression, and lip regions:

$$ b_t^{(s)} = o_t^{(s)} \odot M_{\text{pose}}, \qquad f_t^{(s)} = o_t^{(s)} \odot M_{\text{exp}}, \qquad l_t^{(s)} = o_t^{(s)} \odot M_{\text{lip}}. $$

These region-specific outputs are then fused by an adaptive convolutional module. The paper also studies different fusion mechanisms and finds that the chosen zero convolution formulation gives the best overall balance on synchronization and video-quality metrics.

Network Architecture

Hallo is built on a Stable Diffusion 1.5 latent diffusion backbone with three major subsystems:

• UNet denoiser for latent noise prediction.
• ReferenceNet to inject identity and texture information from the reference image into the generative process.
• Temporal alignment / motion module to preserve coherence across frames and support incremental video generation.

The paper states that text conditioning is not used; instead, audio serves as the main motion driver. ReferenceNet is implemented as a UNet-like network with the same layer count as the denoiser. The first frame of the video clip serves as the reference image during training.

Temporal alignment is implemented by concatenating motion frames from the previous inference step. In the reported implementation, the model uses the last $2$ frames from the prior clip as the initial $k$ frames for the next clip. During second-stage training, the motion module is initialized from AnimateDiff weights.

Training and Inference

Two-Stage Training

The training pipeline is explicitly two-stage:

1. Stage 1: single-frame generation. The model is trained on reference-image / target-frame pairs. The VAE encoder-decoder and the face encoder are frozen, while the spatial cross-attention modules in ReferenceNet and the denoising UNet are optimized. The goal is to improve identity-faithful single-frame synthesis.
2. Stage 2: video-sequence generation. The model is trained on reference images, input audio, and target video sequences. The spatial modules are frozen, and the new focus is on learning hierarchical audio-visual cross-attention plus the motion module for temporally coherent generation.

Training uses 8 NVIDIA A100 GPUs. Both stages run for $30{,}000$ steps with batch size $4$ and resolution $512 \times 512$. The learning rate is $1 \times 10^{-5}$ in both stages. Each second-stage training sample contains $14$ video frames, and the motion-module latents are concatenated with the first $2$ ground-truth frames for continuity. To regularize conditioning, the reference image, guidance audio, and motion frames are each dropped with probability $0.05$ during training.

Inference Procedure

At inference time, the model receives a single reference image and a driving audio clip, then synthesizes a video sequence. Long sequences are generated incrementally: the last $2$ frames from the previous clip are used to condition the next clip. This is intended to reduce discontinuities when generating longer videos.

Datasets and Evaluation Protocol

Training Data

The paper reports a mixture of HDTF and internet-sourced videos. Before filtering, the dataset statistics are:

After cleaning, the final training set includes HDTF with $188$ clips ($6.47$ hours) and internet-sourced data with $1635$ clips ($102.39$ hours). The paper says the cleaning process keeps single-person speaking videos with strong audio-lip consistency and removes clips with scene changes, large camera motion, excessive facial motion, or fully side-facing views. MediaPipe is used to estimate expression and lip activity ranges for mask construction.

Evaluation Metrics

The paper uses five metrics:

• FID and FVD: lower is better; measure image/video realism relative to real data.
• Sync-C: higher is better; evaluates content-level lip synchronization.
• Sync-D: lower is better; evaluates dynamic synchronization.
• E-FID: lower is better; an image-fidelity metric based on Inception features.

The reported evaluation split uses $90\%$ of identity data for training.

Quantitative Results

HDTF

On HDTF, Hallo achieves the best overall image and video quality metrics, with the lowest FID, FVD, and E-FID among the compared methods. Its synchronization scores are competitive and close to real video, though not always the single best among the baselines on every sync metric.

CelebV

On CelebV, Hallo again leads the comparison in overall visual quality and lip synchronization according to the paper’s metrics. The method achieves the best FID, FVD, Sync-C, and E-FID, while remaining competitive on Sync-D.

Proposed Wild Dataset

The paper also reports results on its proposed “wild” dataset, which is intended to stress test robustness under more diverse and challenging conditions. Hallo obtains the best FID, FVD, Sync-C, and E-FID, with competitive Sync-D.

Qualitative Robustness

The paper qualitatively emphasizes that the method can handle different portrait styles and audio styles, and that it produces more diverse head and expression motion than prior methods. The qualitative figures also suggest the model can better preserve identity while varying motion in a controllable way.

Ablation Studies

Hierarchical Audio-Visual Cross Attention

The ablation study isolates lip, expression, and pose attention channels. The main trend is that adding the full hierarchy improves overall performance most consistently, especially when the three region-specific branches are combined. The paper presents this as evidence that fine-grained regional audio-visual alignment is better than a single undifferentiated cross-attention path.

Attention Weighting Mechanism

The paper compares direct addition, self-attention, and zero convolution as fusion strategies. Direct addition gives the lowest FID, but the authors select zero convolution because it gives the best overall trade-off across video quality, synchronization, and perceptual fidelity.

Motion Control and CFG Scales

The paper studies controllability by adjusting hierarchical weights for lip, expression, and pose. Increasing lip weight is reported to improve lip synchronization while slightly hurting image/video quality; increasing expression weight can improve image/video quality slightly but may reduce synchronization. The default setting is chosen by the adaptive weighting process rather than by maximizing a single metric.

The selected CFG setting in the paper is $\lambda_a = 3.5$ and $\lambda_i = 3.5$, which is described as a balance between visual fidelity and motion diversity.

Identity Personalization

The paper also reports identity-specific fine-tuning. The qualitative result is that personalizing on data from a target identity improves how well the synthesized animation captures that identity’s characteristic expressions and pose patterns.

Efficiency Analysis

The efficiency study suggests that the hierarchical audio-driven visual synthesis module has negligible runtime and memory overhead in the reported configuration, while resolution is the main driver of computational cost.

Limitations and Social Risks

The paper explicitly lists four limitations: stronger audio-visual synchronization, more robust temporal coherence for rapid or complex motion, better computational efficiency, and improved diversity control that preserves identity integrity. These are framed as open directions for future work rather than solved problems.

The authors also discuss social risks. Because the method can generate highly realistic talking portraits from audio and images, it could be misused for deceptive content such as deepfakes. The paper recommends ethical guidelines, responsible use practices, transparent data policies, informed consent, and privacy safeguards.

Conclusion

In summary, Hallo is an end-to-end diffusion-based portrait animation system that replaces coarse intermediate facial representations with a hierarchical audio-driven visual synthesis module. By separately modeling lip, expression, and pose motion and then fusing those regions adaptively, the method improves visual realism, temporal consistency, and control over motion diversity. Across HDTF, CelebV, and the authors’ proposed wild dataset, the reported results show clear gains in FID, FVD, and E-FID, with strong lip-synchronization performance and practical controllability through hierarchical weighting.