Hallo2

Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

Hallo2 enables long-duration, 4K resolution audio-driven portrait animation with optional text prompts for expression control. It reduces appearance drift and temporal artifacts using motion-frame augmentation, achieving high-fidelity, controllable hour-long talking-head videos beyond prior short-clip methods.

• talking-head
• portrait-animation
• audio-driven
• voice-cloning
• emotion
• head-pose

Authors: Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, Jingdong Wang

Categories: cs.CV

Published 2024-10-10 · Updated 2024-10-14

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads137 likes

Impact

• Semantic Scholar108 citations24 influential citations

Abstract

Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities. First, we extend the method to produce long-duration videos. To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration. Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension. By integrating a high-quality decoder, we realize visual synthesis at 4K resolution. Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes. Project page https://fudan-generative-vision.github.io/hallo2

---

Problem Setting and High-Level Contribution

Hallo2 addresses a central limitation in portrait image animation: existing diffusion-based talking-head systems perform well on short clips, but struggle when the generation horizon extends to minutes or longer, and they typically operate at relatively low output resolution. The paper targets a combined setting of long-duration, audio-driven, and high-resolution portrait animation from a single reference image, with optional text prompts for expression control. The authors position Hallo2 as an extension of Hallo that adds three capabilities at once: robust long-horizon generation, 4K synthesis, and textual controllability for expression and motion.

The central design choice is to keep the subject identity anchored in a single reference image while using previous generated frames only as motion carriers. To make that incremental setting work over long sequences, the paper introduces corruption-based augmentation on the motion-conditioned frames so that appearance information from prior frames does not dominate or drift away from the reference identity. For high resolution, the method adds a temporal alignment module over discrete latent code prediction. For controllability, it injects text embeddings through adaptive normalization in the denoising network.

Why Long-Duration Portrait Animation Is Hard

The introduction frames the core challenge as a tradeoff between two imperfect strategies for long video generation. In parallel generation, multiple clips are synthesized independently and then stitched together, but continuity constraints between adjacent chunks can blur motion and distort expressions or poses. In incremental generation, each new segment conditions on previously generated frames, which better supports continuity but makes the system vulnerable to error accumulation: any contamination in appearance, background, or facial motion can propagate forward and worsen with time. Hallo2 follows the incremental paradigm, but attempts to reduce this propagation by weakening appearance leakage from past frames and by training the model to recover from corrupted motion conditions.

The paper also motivates the need for text control. Audio is sufficient for lip motion and coarse speaking-style dynamics, but it is not always enough to specify affect or fine expression changes. Hallo2 therefore adds textual prompts as an extra condition so that users can steer expressions and head motion beyond what the audio alone implies.

Preliminaries: Latent Diffusion and Conditioning

The paper uses the standard latent diffusion formulation. An image is encoded into a latent $\mathbf{z}_0 = \mathcal{E}(\mathbf{I})$ by a pretrained VAE encoder, noised to $\mathbf{z}_t$ according to the forward process, and reconstructed by a denoising network $\boldsymbol{\epsilon}_{\theta}(\mathbf{z}_t, t, \mathbf{c})$. The training objective is the usual weighted noise-prediction loss

$$\mathcal{L} = \mathbb{E}_{\mathbf{z}_0,\mathbf{c},\boldsymbol{\epsilon},t}\left[\omega(t)\lVert \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\mathbf{z}_t,t,\mathbf{c})\rVert_2^2\right].$$

Conditioning is implemented with cross-attention. In the paper’s notation, queries come from the noisy latent, while keys and values come from conditioning inputs. Hallo2 uses multiple modalities in $\mathbf{c}$: a reference-image embedding, audio features, and text embeddings. This is the backbone that lets the model preserve identity from one input image while synchronizing motion to the audio stream and optionally following text prompts.

Method

Hallo2 is organized into three major extensions over the earlier Hallo-style portrait animation pipeline: long-duration animation via motion-frame corruption, high-resolution enhancement via discrete code prediction with temporal alignment, and textual prompt control via adaptive normalization. The resulting pipeline keeps the appearance anchored in the reference image, uses past frames only as motion guides, and then upsamples the generated sequence to 4K while preserving temporal consistency.

1) Long-Duration Animation by Patch-Drop and Gaussian-Noise Augmentation

The key insight is that incremental generation should not allow previous frames to dominate identity formation. Hallo2 therefore corrupts the appearance content of motion-condition frames while preserving coarse structure and motion cues. The conditioning sequence is composed of prior generated frames $\{\mathbf{I}_{t-1},\dots,\mathbf{I}_{t-N}\}$, but before they are fed to the denoiser, the paper applies patch drop and then injects Gaussian noise.

Patch drop partitions a frame into non-overlapping patches of size $p \times p$ and samples a binary mask for each patch with retention probability $1-r$, where $r$ is the drop rate. In image space, this removes fine appearance details such as texture, color shifts, and local artifacts while leaving enough spatial scaffolding for the model to infer motion continuity. In the implementation described in the main method section, the same idea is applied in latent space after encoding the prior frames, which is consistent with the diffusion pipeline used for conditioning.

After patch dropping, the paper adds latent Gaussian noise,

$$\hat{\mathbf{z}}_{t-i} = \tilde{\mathbf{z}}_{t-i} + \boldsymbol{\eta}_{t-i}, \qquad \boldsymbol{\eta}_{t-i} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I}),$$

so that the denoiser learns to recover motion-related cues even when the conditioning frames are partially corrupted. The resulting corrupted motion latents $\{\hat{\mathbf{z}}_{t-i}\}$ are concatenated with the reference-image latent and audio/text conditions inside the denoising U-Net through cross-attention. The paper’s stated goal is to force the model to extract appearance primarily from the reference image and use motion frames only for temporal dynamics such as lip movement, expression change, and head pose.

2) High-Resolution Enhancement with Temporal Alignment

Hallo2 adds a separate enhancement stage to produce 4K output. The paper builds on a discrete codebook prediction approach inspired by CodeFormer/VQ-style latent representations and extends it into the temporal domain. The generated low-resolution video is encoded into latent tokens, which are processed by a Transformer block that includes both spatial self-attention and temporal alignment.

In the Transformer, the latent sequence is first flattened across spatial dimensions and processed with self-attention within each frame. The hidden states are then reshaped so that temporal attention can be applied across frames at each spatial location. This design explicitly separates intra-frame modeling from inter-frame coherence, which is important for preserving fine facial details when upscaling to 4K. In other words, the module is not just a super-resolution decoder; it is a spatiotemporal consistency mechanism for high-resolution facial synthesis.

The paper describes two ways to extract the input latent features for this stage. The simpler option uses latents directly from the diffusion model, but it requires end-to-end training of the full super-resolution module. The preferred option, given the scarcity of super-resolution training data, passes features through the diffusion decoder and then a low-quality decoder, so that only a lightweight temporal alignment module needs to be trained. The authors report that this second design performs better under limited training data.

3) Textual Prompt Control via Adaptive Layer Normalization

To make the system more controllable, Hallo2 adds text prompts as an optional condition. A text embedding is extracted with CLIP and passed through a zero-initialized MLP to produce scale and shift vectors $\gamma$ and $\beta$. These are applied after layer normalization in the denoising U-Net, between the cross-attention and audio-attention parts of the network:

$$\mathbf{X}_{\text{adapted}} = \gamma \odot \operatorname{LayerNorm}(\mathbf{X}_{\text{cross}}) + \beta + \mathbf{X}_{\text{cross}}.$$

The paper’s qualitative examples indicate that the textual prompts are used to modulate facial expressions and emotional tone, rather than to fully replace the audio-driven motion. This means the prompt acts as a controllable expression prior layered on top of the speaking signal.

4) Network Composition

The network has three main components. First, a ReferenceNet embeds the reference portrait and its background. Second, a denoising U-Net synthesizes frames from a noisy latent $\mathbf{z}_t$ conditioned on the reference image, audio, text, and augmented motion frames. The audio embedding is extracted using a 12-layer wav2vec encoder, and the text embedding comes from CLIP. Third, a high-resolution enhancement module uses vector quantization and temporal alignment to produce the final 4K frames.

The paper emphasizes that the reference image provides appearance, the motion frames provide temporal cues, and the audio/text conditions provide controllable articulation and expression. This separation is important because it explains why the augmentation strategy focuses on motion frames: if those frames leak too much appearance, the model can drift away from the original identity over long sequences.

Training and Inference

Two-Stage Training

The training recipe is explicitly two-stage. In stage one, the model is trained for ordinary audio-driven portrait generation at $512 \times 512$ resolution. A random frame from a video clip serves as the reference image, adjacent frames serve as targets, and the reference-image encoder, VAE encoder/decoder, and facial image encoder are frozen. The optimization focuses on the spatial cross-attention modules in both ReferenceNet and the denoising U-Net, together with motion modules intended to improve smoothness and temporal coherence.

Stage two turns the model into a long-duration generator. Here, the model is trained with patch drop plus Gaussian noise on the motion frames, so it learns to tolerate corrupted conditioning and still produce stable identity and motion over long sequences. Text prompts are also introduced in this stage for expression control. For the high-resolution branch, the paper trains the VAE encoder/codebook prediction weights and the temporal alignment module, using VFHQ as super-resolution training data.

The reported optimization details are: training on 8 NVIDIA A100 GPUs, stage one for 30,000 steps with step size 4, stage two for 28,000 steps with batch size 4 and learning rate $10^{-5}$, and super-resolution temporal-alignment training for 550,000 steps with learning rate $10^{-4}$. The motion module is initialized from Animatediff weights, and the super-resolution part from CodeFormer weights. The paper states that around 160 hours of video data are used across the two stages.

Inference Pipeline

At inference time, the model receives a single reference image, driving audio, an optional textual prompt, and motion frames augmented by patch drop and Gaussian noise. It then generates a sequence of portrait frames that follow the audio while remaining visually tied to the reference image. The high-resolution module then upsamples these frames to as much as $4096 \times 4096$ resolution. The paper reports that evaluation is performed on generated videos of about 4 minutes duration, using looped audio for the benchmark comparisons, and that the final system can extend to tens of minutes in qualitative demonstrations.

Experimental Setup

The paper evaluates on three datasets: HDTF, CelebV, and a new dataset introduced by the authors called Wild. Wild contains 2,019 clips totaling approximately 155.9 hours, with diverse lip motion, facial expressions, and head poses. The authors use this dataset to test long-duration behavior and generalization in more unconstrained settings.

The reported metrics are standard for the field: $\operatorname{FID}$ for image realism, $\operatorname{FVD}$ for video quality, $\operatorname{Sync\text{-}C}$ for lip synchronization consistency, $\operatorname{Sync\text{-}D}$ for temporal consistency of lip motion, and $\operatorname{E\text{-}FID}$ for expression fidelity. Lower is better for $\operatorname{FID}$, $\operatorname{FVD}$, $\operatorname{Sync\text{-}D}$, and $\operatorname{E\text{-}FID}$, while higher is better for $\operatorname{Sync\text{-}C}$.

For fair comparison against earlier methods, the paper disables the high-resolution enhancement module in the quantitative benchmarks and evaluates all methods at $512 \times 512$ resolution. This is important because the compared baselines do not include a 4K enhancement stage.

Quantitative Results

CelebV

On CelebV, Hallo2 achieves the best values on $\operatorname{FID}$, $\operatorname{FVD}$, and $\operatorname{E\text{-}FID}$ among the compared methods, while its $\operatorname{Sync\text{-}C}$ is slightly below the best baseline and the real-video reference. The paper uses this result to argue that the long-duration setting is handled well without collapsing visual quality.

The main take-away on CelebV is that incremental long-duration generation becomes much harder for prior methods as inference length grows: the paper specifically notes severe degradation for Hallo and for parallel-generation baselines such as EchoMimic. Hallo2 reduces this deterioration and maintains lower image/video distances over the full long-horizon setting.

HDTF

On HDTF, Hallo2 again reports the best overall fidelity and expression metrics. The paper highlights especially strong $\operatorname{FID}$ and $\operatorname{E\text{-}FID}$, while the synchronization metrics remain competitive with the best baselines and close to the real-video reference on this benchmark.

The paper uses the FID-versus-time curve to underscore a specific failure mode of long-duration generation: as inference time increases, the image quality of existing diffusion-based methods degrades substantially. Hallo2 is presented as more stable in that regime, which supports the paper’s claim that the augmentation strategy mitigates appearance drift and accumulated artifacts.

Wild Dataset

On the proposed Wild dataset, Hallo2 obtains the best reported $\operatorname{FID}$ and $\operatorname{E\text{-}FID}$, and its synchronization metrics are competitive with the other methods. The table in the paper reports $\operatorname{FVD}=360.192$ for Hallo2. The authors describe this as evidence of coherent video structure under a challenging, more unconstrained setting, though the numerical comparison should be read directly from the table because the metric is defined as lower-is-better.

Ablation Studies

The ablations concentrate on the long-duration augmentation strategy and the high-resolution module. The reported conclusions are consistent: patch drop is the dominant component for long-horizon stability, Gaussian noise adds a smaller but still measurable benefit, and temporal alignment improves super-resolution quality.

Patch Drop Size

The paper evaluates different patch sizes for patch drop on CelebV. A patch size of $0$ means no patch drop. The best values are achieved when the patch size is $1$, which is the setting used in the final implementation.

Patch Drop Rate

The authors also vary the patch drop rate. The best reported setting is a drop rate of $0.25$, which again matches the final implementation.

Gaussian Noise and Patch Drop Combination

The combination study shows that Gaussian noise alone helps slightly, patch drop provides the major improvement, and the combination of patch drop plus Gaussian noise is the best-performing configuration. This is the strongest evidence in the paper that the long-duration robustness comes from training the model to ignore appearance contamination in the motion frames while still preserving motion continuity.

High-Resolution Enhancement and Temporal Alignment

The paper provides qualitative evidence that high-resolution enhancement improves sharpness and facial detail, and that temporal alignment reduces inter-frame artifacts compared with simpler image-based enhancement methods. In the paper’s framing, temporal alignment is not a cosmetic add-on; it is what allows the high-resolution decoder to preserve coherence while details are being restored.

Text Prompt Control

The qualitative prompt ablation demonstrates that text can meaningfully alter expression and emotional tone. The paper’s examples suggest the prompt is especially useful when users want a speaking face to look happier, more serious, or otherwise more expressive than the audio alone would indicate.

Attention Visualization

The attention maps show the intended functional split between reference image and motion frames. According to the paper, the reference image dominates overall appearance and background, while the motion frames focus on facial regions related to movement. This supports the claim that patch-drop augmentation shifts appearance responsibility back to the reference image and leaves the motion frames to encode dynamics.

Qualitative Observations

Beyond the main benchmark tables, the paper reports that Hallo2 can animate a variety of portrait styles, including oil paintings, anime-style portraits, and portraits generated by other models. This is not framed as a separate benchmark, but as evidence that the approach is robust to different input domains and not limited to photographic faces.

Limitations and Future Work

The paper is explicit about three limitations. First, the system relies on a single reference image, which restricts the diversity of expressions and poses that can be generated. The authors suggest that multiple references or stronger conditioning mechanisms would be needed for broader variation. Second, patch-drop augmentation, while effective, can introduce artifacts; future work should explore alternative corruption strategies or adaptive corruption policies that are more content-aware. Third, 4K generation is computationally expensive, so real-time deployment would require significant optimization and hardware acceleration.

These limitations are important because they point to the main tradeoff of the approach: the paper succeeds by aggressively stabilizing identity and temporal coherence, but that same design choice can limit variation and increases computational burden at high output resolution.

Conclusion

Hallo2 is a substantial engineering extension of portrait image animation built on latent diffusion. Its main contributions are: a corruption-based strategy for long-duration incremental generation, a temporal-alignment super-resolution module for 4K synthesis, and a text-conditioning pathway for controllable expressions. The empirical results support the paper’s claim that these additions improve robustness over long horizons and preserve more stable identity, motion, and fine facial detail than prior methods. Within the scope of the paper’s experiments, the method is especially strong on long-duration fidelity and expression quality, while remaining extensible to prompt-based control and very high output resolution.