Hallo3

Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer

Hallo3 uses a pretrained transformer-based video model for audio-driven portrait animation, enabling dynamic, realistic videos from complex, non-frontal images. It preserves identity over time and adds immersive backgrounds and foreground motions, surpassing prior U-Net methods.

• talking-head
• portrait-animation
• audio-driven
• lip-sync

Authors: Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, Siyu Zhu

Categories: cs.CV, cs.GR, cs.LG

Published 2024-12-01 · Updated 2025-03-13

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads0 likes

Impact

• Semantic Scholar124 citations24 influential citations

Abstract

Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/.

---

1. Problem setting, motivation, and main idea

Hallo3 tackles audio-driven portrait image animation in a more challenging regime than prior talking-head systems: the reference portrait is not assumed to be frontal or tightly centered, and the generated video is expected to contain dynamic foreground objects, dynamic backgrounds, and realistic motion while preserving identity over extended durations. The paper argues that existing portrait animation methods still struggle with three recurring failure modes: non-frontal viewpoints, motion of objects around the portrait, and immersive background generation. These issues are especially visible when the source portrait shows a side profile, when the subject is holding or interacting with objects, or when the scene itself is active rather than static.

The key design choice is to move away from the U-Net-based diffusion backbones used by many prior talking-head systems and instead build on a pretrained transformer-based video generative model (CogVideoX-5B-I2V). The paper presents this as the first application of a pretrained DiT-style video generator to audio-driven portrait animation. That change is not a cosmetic backbone swap: it makes earlier identity-maintenance, audio-conditioning, and long-video extrapolation tricks designed for U-Net architectures no longer directly applicable, so the paper redesigns the conditioning pathway around the new backbone.

The proposed system combines three ingredients:

• Identity preservation through an identity reference network built from a causal 3D VAE and stacked transformer layers.
• Speech audio conditioning through cross-attention over Wav2Vec features, with comparisons to adaptive normalization and self-attention alternatives.
• Video extrapolation with motion frames from the previously generated segment, enabling long-duration generation beyond the native clip length of the backbone.

Across the paper, the authors emphasize that the goal is not just lip-synced facial motion, but a more general form of realistic portrait video synthesis that can handle head pose diversity, accessories, foreground motion, and background dynamics.

2. Backbone architecture and conditioning design

The backbone is initialized from CogVideoX-5B-I2V and uses the same general transformer-video formulation: a 3D causal VAE compresses the video into latents, the T5 text encoder provides semantic conditioning, and the transformer denoiser operates in latent space with temporal positional encoding. The paper notes that 3D rotational positional encoding helps capture inter-frame relationships over time and strengthen long-range dependencies.

Let $\mathbf{z}_t$ denote the latent video tokens at diffusion step $t$ and $\mathbf{c}_{\text{text}}$ denote the text embedding. The baseline transformer denoiser processes the concatenated latent sequence and text sequence through expert transformer blocks with adaptive layer normalization. The method then extends this baseline with two additional conditions: a speech audio condition $\mathbf{c}_{\text{audio}}$ and an identity appearance condition $\mathbf{c}_{\text{id}}$.

2.1 Audio conditioning

For audio, the paper uses Wav2Vec features. The audio representation is built by concatenating embeddings from the final twelve Wav2Vec layers and then passing them through three successive linear layers to obtain frame-level audio features:

$$\mathbf{c}_{\text{audio}}^{(f)} = \mathcal{L}_3\big(\mathcal{L}_2(\mathcal{L}_1(\mathbf{c}_{\text{audio}}))\big).$$

These frame-specific audio features are injected into the denoising transformer using cross-attention. The paper explicitly compares three fusion strategies for speech audio conditioning: self-attention, adaptive normalization, and cross-attention. Cross-attention is the best performer in both quantitative and qualitative ablations, especially on the lip-synchronization metrics.

Conceptually, the denoising latents attend to the audio features as queries-to-keys/values interaction, so the generated motion can be aligned with phonetic timing and prosody. The text states that this is more suitable than AdaLN-style injection for richer sequential audio inputs.

The paper summarizes this cross-attention update in the denoiser as:

$$\mathbf{z}_t = \text{CrossAttention}(\mathbf{z}_t, \mathbf{c}_{\text{audio}}^{(f)}).$$

2.2 Identity reference network

Identity preservation is handled by a dedicated identity reference network. The core motivation is that, as the generated clip gets longer, facial identity in DiT-based generation can drift even if speech alignment remains plausible. The paper therefore treats the reference portrait as a single frame, encodes it with a causal 3D VAE, and feeds the resulting latent tokens through a stack of transformer layers. The implementation uses 42 transformer layers, matching the denoising network so that the reference pathway and generation pathway stay semantically and scale-consistent.

If $\mathbf{I}_{\text{ref}}$ is the reference image, the identity encoder produces:

$$\mathbf{z}_{\text{id}} = \mathcal{E}_{3D}(\mathbf{I}_{\text{ref}}).$$

The reference features are then injected into the denoising stream using self-attention-like fusion:

$$\mathbf{z}_{t,\text{enhanced}} = \text{SelfAttention}(\mathbf{z}_t, \mathbf{z}_{\text{id}}).$$

The paper compares four identity-conditioning strategies: no identity condition, face attention, face adaptive norm, identity reference network, and a hybrid that combines face attention with the identity reference network. The reference-network design is important because it retains global appearance cues from the portrait while avoiding the distortion that can arise when a face embedding is used too narrowly. The authors explicitly note that using the same causal 3D VAE weights and the same number of transformer layers in both the reference network and the denoising network helps maintain feature compatibility.

2.3 Temporal motion frames for long videos

To extend generation beyond the native clip length, the paper introduces motion frames: the last $n$ frames from the previously generated segment are re-encoded and used as extra conditioning for the next segment. These are passed through the 3D VAE to obtain latent codes. The remaining positions are zero-padded, then concatenated with Gaussian noise and patchified into vision tokens. This gives the model a short-term temporal anchor when generating the next clip.

The source describes inference as repeatedly feeding the last generated frames back into the next generation stage. In the main text, the default inference description uses the last two frames; the ablation table then studies $n \in \{1,2,4,8\}$ and shows that $n=2$ gives the best overall trade-off, while $n=1$ gives the strongest lip-sync metrics.

This mechanism is the paper’s answer to a core limitation of video diffusion transformers: they can generate only a limited number of frames in one pass, so long-form portrait animation requires an explicit autoregressive-like stitching strategy at the latent level.

3. Training, data curation, and evaluation protocol

The training pipeline is built in two phases.

• Identity consistency phase: the model learns to preserve the portrait appearance. The 3D VAE and face image encoder are frozen, while the 3D full-attention layers in both the reference network and the denoising network, plus the face-attention layers in the denoising network, are updated. Inputs include a randomly sampled reference image from the training video, a textual prompt, and a face embedding extracted by InsightFace.
• Audio-driven video generation phase: audio-attention modules are inserted into each transformer block of the denoising network, and only these audio modules are updated. Inputs are a reference image, an audio embedding, and a textual prompt.

The model is trained with a v-prediction diffusion loss. The implementation details reported in the paper are: initialization from CogVideoX-5B-I2V, 64 NVIDIA A100/H100 GPUs depending on the source section, 20,000 steps per phase, batch size $1$ per GPU, learning rate $10^{-5}$, and training resolution $480\times720$. The source also states that conditioning inputs are dropped with probability $0.05$, and motion frames are randomly masked with probability $0.25$ to improve robustness.

Text prompts are generated with MiniCPM in training and are described as covering human appearance, actions, and environmental background. The paper also notes the use of InsightFace for face embeddings and Wav2Vec2/Wav2Vec for audio embeddings. For data curation, the authors combine three sources: the HDTF dataset, a large YouTube collection, and a large movie collection. After cleaning, they report a final training set of about 134 hours, composed of 6 hours from HDTF, 72 hours from YouTube, and 56 hours from movies.

The data filtering pipeline is explicitly multi-stage: single-speaker extraction, motion filtering, and post-processing. The paper mentions filters for head motion, head pose, camera motion, and lip-synchronization quality; it then crops clips to a 3:2 aspect ratio and extracts face and audio embeddings. The goal is to reduce noisy internet video while keeping the diversity needed for non-frontal views and complex scenes.

3.1 Evaluation metrics

The experiments use standard image/video realism and sync metrics on benchmark sets, and motion-consistency metrics on the wild set:

• FID and FVD for image/video distribution similarity.
• Sync-C and Sync-D for audio-visual lip synchronization.
• E-FID for quality on Celeb-V / EMO-style evaluation.
• V-bench-style measures for the wild set: dynamic degree via RAFT and subject consistency via DINO feature similarity.

4. Quantitative results

The paper reports results on HDTF, Celeb-V, and a newly proposed wild dataset with 34 representative cases. The wild set includes a broad range of head proportions, head poses, static and dynamic scenes, and complex headwear and clothing. It is intended to stress the exact failure modes that motivated Hallo3: side-face portraits, dynamic foreground objects, and active backgrounds.

4.1 Benchmark results on HDTF and Celeb-V

On HDTF, Hallo3 achieves the best FID and FVD, indicating improved realism and temporal quality, but it does not beat Hallo on synchronization. The authors attribute this to the more dynamic and less constrained training data used for their method: compared with methods tuned for mostly static, frontal talking-face clips, Hallo3 is trained to handle larger head/body motion and dynamic scenes, which better matches the intended setting but slightly lowers lip-sync metrics.

On Celeb-V, Hallo3 again leads on FID, FVD, and E-FID, while Hallo retains the best Sync-C. The takeaway is consistent across both benchmarks: the DiT-based system improves realism and generalization, especially for video quality, while synchronization remains more sensitive to the diversity and complexity of the training regime.

4.2 Wild dataset results

The wild set is where Hallo3 is most clearly differentiated from prior methods. It achieves the best synchronization, the strongest subject motion, the strongest background motion, and the best subject/background FVD. Qualitatively, the paper reports that prior methods often freeze on side views, distort faces, or animate only the mouth region, whereas Hallo3 can animate the subject while also handling surrounding objects and moving environments.

5. Ablation studies and design decisions

The supplementary material contains ablations that explain why the final system is structured the way it is. The recurring theme is that each conditioning route is useful for a different aspect of the final video: audio governs lip motion, the reference pathway governs identity, text governs scene and motion diversity, and motion frames stabilize long-range continuity.

5.1 Audio conditioning ablation

Cross-attention is the clear winner. The paper interprets this as evidence that audio is best used as a rich conditional sequence that the visual latent tokens can query, rather than being injected only by normalization shifts or by a generic self-attention path. The gain is especially large for lip synchronization: Sync-C jumps from roughly $1.3$--$1.4$ to $4.601$, and Sync-D drops from about $13.5$ to $10.416$.

5.2 Identity conditioning ablation

This ablation shows two important points. First, simply adding face embeddings is not sufficient: face attention and face adaptive norm both degrade the generation quality relative to the identity reference network. Second, the identity reference network alone already improves identity preservation and temporal quality, while combining it with face attention gives the best overall result. The paper’s qualitative discussion says that the reference pathway preserves global appearance cues, while the face encoder adds higher-level semantic facial information.

5.3 Motion frame ablation

The motion-frame study reveals a practical trade-off. A single motion frame gives the best synchronization, but two motion frames are best for the overall visual metrics. Larger values eventually hurt both image/video quality and sync stability. This supports the paper’s default design choice of using a very short history window for long-form generation.

5.4 CFG scale study

The CFG study clarifies how the three condition streams interact. Increasing the audio CFG scale helps synchronization; increasing the text CFG scale pushes the scene toward stronger dynamics; increasing the image CFG scale improves subject consistency. The paper chooses $\lambda_a = 3.5$, $\lambda_t = 3.5$, and $\lambda_i = 1.0$ as a balanced operating point.

6. Limitations, safety, and what the paper leaves open

The paper does not claim to solve all portrait-animation failure cases. Its stated limitations are that, although identity preservation and lip synchronization improve, the model still has difficulty rendering intricate facial expressions in dynamic environments, especially under varying illumination. The authors also say that robustness to a wider range of perspectives, interactions, backgrounds, and facial accessories remains an open problem, and they suggest that real-time feedback mechanisms could improve interactivity for live media and augmented reality applications.

On the safety side, the authors explicitly call out the usual risks associated with high-fidelity portrait animation: deepfake misuse, privacy concerns, and consent. The paper frames responsible dataset usage, transparent policies, and regulatory compliance as necessary parts of future deployment, rather than optional add-ons.

7. Overall technical takeaway

Hallo3 is best understood as a DiT-first portrait animation system rather than a U-Net system retrofitted with a new encoder. That distinction matters because the entire conditioning stack is redesigned to match a transformer video backbone. The identity reference network preserves appearance through a causal 3D VAE plus transformer stack; audio is injected through cross-attention over Wav2Vec features; motion frames allow the model to continue long videos with temporal continuity; and the training data is deliberately expanded beyond frontal talking-head clips to include non-frontal views and dynamic scenes. The result is a model that trades some lip-sync dominance on easy benchmark settings for substantially better realism, dynamics, and robustness in harder real-world portrait animation scenarios.