HunyuanVideo-Avatar

HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters

HunyuanVideo-Avatar is a multimodal diffusion transformer that generates high-fidelity, dynamic audio-driven human animations for multiple characters with precise emotion alignment. It uniquely balances natural motion with character consistency and supports independent audio control for each character.

• audio-driven
• talking-head
• avatar
• emotion
• full-body
• face-animation

Demos

These demos showcase HunyuanVideo-Avatar's ability to generate high-fidelity audio-driven human animations for multiple characters with dynamic and emotion-controllable motions. Key highlights include precise alignment of lip-sync and facial emotion with the audio, strong character consistency across varied styles, and robust multi-character interaction driven by independent audio streams. When evaluating, watch for the fluidity of dynamic motions, emotion accuracy, and clarity of multi-character dialogues.

Authors: Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, Qinglin Lu

Categories: cs.CV

Published 2025-05-26 · Updated 2025-06-03

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub2,128 stars25 watchers343 forksTencent Hunyuan Community License Agreement
• Hugging Face0 downloads328 likes

Impact

• Semantic Scholar56 citations7 influential citations

Abstract

Recent years have witnessed significant progress in audio-driven human animation. However, critical challenges remain in (i) generating highly dynamic videos while preserving character consistency, (ii) achieving precise emotion alignment between characters and audio, and (iii) enabling multi-character audio-driven animation. To address these challenges, we propose HunyuanVideo-Avatar, a multimodal diffusion transformer (MM-DiT)-based model capable of simultaneously generating dynamic, emotion-controllable, and multi-character dialogue videos. Concretely, HunyuanVideo-Avatar introduces three key innovations: (i) A character image injection module is designed to replace the conventional addition-based character conditioning scheme, eliminating the inherent condition mismatch between training and inference. This ensures the dynamic motion and strong character consistency; (ii) An Audio Emotion Module (AEM) is introduced to extract and transfer the emotional cues from an emotion reference image to the target generated video, enabling fine-grained and accurate emotion style control; (iii) A Face-Aware Audio Adapter (FAA) is proposed to isolate the audio-driven character with latent-level face mask, enabling independent audio injection via cross-attention for multi-character scenarios. These innovations empower HunyuanVideo-Avatar to surpass state-of-the-art methods on benchmark datasets and a newly proposed wild dataset, generating realistic avatars in dynamic, immersive scenarios.

---

1. Problem Setting, Motivation, and Core Idea

HunyuanVideo-Avatar addresses three persistent limitations in audio-driven human animation: motion quality (videos need to be dynamic without losing identity), emotion alignment (facial expression should follow the affective content of the audio), and multi-character control (different people in the same scene should be independently driven by different audio streams). The paper argues that existing portrait animation methods are too spatially narrow, while full-body methods still struggle with realistic motion, audio-emotion mismatch, and multi-person dialogue generation.

The proposed solution is a multimodal diffusion transformer (MM-DiT) system built on HunyuanVideo. It introduces three specialized modules: a Character Image Injection Module for consistency without sacrificing motion diversity, an Audio Emotion Module for transferring emotion cues from a reference image into the generated video, and a Face-Aware Audio Adapter for localized audio injection in multi-character scenes.

At a high level, the model takes a reference character image, driving audio, and a face mask as input, then generates single- or multi-character talking videos. The main design principle throughout the paper is to move from brittle, inference-time conditioning tricks toward conditioning pathways that better match training behavior, are spatially localized, and can support multiple independent control signals.

2. Backbone and Overall Architecture

The backbone is HunyuanVideo, a video generation model based on the MM-DiT architecture. The paper states that the method uses HunyuanVideo for the core denoising backbone and, in the experimental setup, uses HunyuanVideo-I2V as the base model for HunyuanVideo-Avatar. The architecture is extended with image, audio, and emotion conditioning pathways rather than replaced wholesale.

2.1 Preliminaries and Latent Construction

The method first resizes the reference image to the video-frame resolution and encodes both the reference image and the noisy video through a pretrained 3D VAE. The image latent is denoted as $v_R \in \mathbb{R}^{w \times h \times c}$, and the video latent as $v_{\text{noise}} \in \mathbb{R}^{f \times w \times h \times c}$, where $f$ is the number of frames. Tokenizers map these latents into token sequences, and the image latent is repeated across time and concatenated with the video representation so the model can propagate identity information along the temporal axis.

The paper describes the final conditioning input as a token concatenation of the repeated reference-image branch and the image-token branch. In words, the model fuses a temporally repeated character image embedding with the noisy video latent, then feeds the combined tokens into the MM-DiT backbone.

To better preserve identity without simply copying the reference image into the output, the image latent is assigned a temporal position before the first frame, and its spatial position is shifted. The paper writes this as a 3D-RoPE offset of the form $\operatorname{RoPE}_{z_I}(l,i,j)=\operatorname{RoPE}(-1,i+w,j+h)$, where $(-1)$ places the image before the first frame and $(w,h)$ adds the spatial shift.

2.2 Flow-Matching Training Objective

Training follows a flow-matching formulation. A clean latent $z_1$ is paired with Gaussian noise $z_0 \sim \mathcal{N}(0,I)$, and a time $t \in [0,1]$ is sampled from a logit-normal distribution. The intermediate sample $z_t$ is obtained by linear interpolation. The model predicts the velocity field $u_t = \frac{d z_t}{d t}$ with output $\lambda_t$, and the optimization objective is the mean squared error between predicted and true velocities:

$$ \mathcal{L}_{\text{generation}} = \mathbb{E}_{t, z_0, z_1}\left\lVert \lambda_t - u_t \right\rVert^2. $$

This objective is the only explicit generation loss described in the main method section, and it is used to learn the conditional video distribution under the combined image, audio, and emotion inputs.

2.3 Character Image Injection Module

This module is the paper’s main answer to the tension between consistency and dynamics. The authors note that conventional reference-image padding schemes improve identity preservation but often make motion less natural and create a mismatch between training and inference. Removing padding frames improves motion but weakens character consistency. The proposed module is designed to avoid this trade-off.

The paper compares three conditioning strategies:

• Token: the reference image and video are processed with the same tokenizer and concatenated at the token level.
• Token + Channel: the character image is repeated over time, concatenated with the video in the channel dimension, and then fused with a separate image tokenizer.
• Token + Add: the reference image is repeated over time, encoded through a separate tokenizer, and added to the video latent through a projection module built from fully connected layers.

The paper reports that Token + Add performs best overall among the three variants and is the chosen design. It is described as better at balancing motion quality and character integrity than token-only or channel-based fusion. The paper also states that copying the backbone tokenizer weights to the extra image tokenizer speeds up convergence.

2.4 Face-Aware Audio Adapter

The Face-Aware Audio Adapter is designed for multi-character scenes. Audio features are extracted with Whisper, and face regions are localized with InsightFace. Because the video latent is temporally compressed by the 3D VAE, the audio sequence is padded and aggregated so that each compressed video frame aligns with a block of audio tokens. The paper states that the audio features have shape $n' \times 10 \times d$ before alignment and are rearranged to match the compressed video structure as $[b,(n+1),40,d]$ after grouping every four audio frames.

A face mask is aligned to the same compressed temporal grid, with the initial frame mask set to 1. This mask localizes where the audio should act, which is crucial for scenes with multiple people. The adapter then performs spatial cross-attention independently at each time step, so each audio segment only interacts with the spatial tokens of its corresponding frame. In effect, the audio signal is prevented from leaking into unrelated characters or unrelated temporal positions.

The update can be summarized as a masked, per-frame cross-attention injection:

$$ y''_{t,A} = y'_{t,A} + \alpha_A \cdot \operatorname{CrossAttn}(g_A, y'_t) \cdot g_M, $$

where $\alpha_A$ controls the strength of audio conditioning and $g_M$ is the aligned face mask. The paper’s qualitative ablation shows that without the mask, audio affects all characters; with the mask, only the intended face region is animated, and the target can be switched by moving the mask.

2.5 Audio Emotion Module

The Audio Emotion Module addresses the mismatch between the affective tone of speech and the facial expression of the generated character. The paper does not attempt to infer emotion directly from the audio alone; instead, it uses an emotion reference image whose features are extracted by the pretrained 3D VAE. These features are injected into the backbone’s Double Block through an FC layer and spatial cross-attention, with the reference image acting as key/value and the video latent as query.

The paper emphasizes that this module works better in a Double Block than in a Single Block. The authors report that Single Block insertion did not allow the model to effectively learn or express emotional cues, whereas Double Block insertion enabled more reliable emotion-to-expression transfer. The update can be written as:

$$ y''_{t,A,E} = y'_{t,A} + \gamma_E \cdot \operatorname{CrossAttn}(\operatorname{FC}(E_{\text{ref}}), y'_{t,A}), $$

where $\gamma_E$ is a learnable scaling factor. This module is the paper’s main mechanism for emotion style control, but it requires a manually chosen reference image for the desired affect.

2.6 Long Video Generation

The authors note that HunyuanVideo-13B is limited to generating 129 frames, which is often shorter than the audio duration. To handle longer clips, they adopt Time-aware Position Shift Fusion from Sonic. The idea is to denoise the video in overlapping or shifted audio-video segments, then stitch successive segments using a starting offset. The paper states that the offset $\alpha$ is set between 3 and 7 frames and that the method introduces no extra training or inference cost.

This strategy is used to reduce jitter and abrupt temporal transitions when generating longer dialogue videos.

3. Training Setup, Data, and Evaluation Protocol

The implementation uses a two-stage training recipe. In the first stage, the model is trained only on audio data to establish basic audio-visual alignment. In the second stage, the authors mix audio and image data in a 1:1.5 ratio to improve motion stability. Training resolution ranges from $704 \times 704$ to $704 \times 1216$. The paper states that LLaVA and the 3D VAE are frozen, while all other learnable parameters are updated.

Training is run on 160 GPUs with 96 GB memory each, using a global batch size of 40 and learning rate of 10^{-5}.

The training set is built by filtering data with LatentSync to remove audio-visual asynchrony and by using tools such as Koala-36M to remove low-brightness and low-aesthetic samples. The final training corpus contains 500,000 samples with character audio, totaling about 1,250 hours.

Evaluation is performed on two portrait datasets and one new full-body benchmark:

• CelebV-HQ: used for portrait evaluation in diverse scenes.
• HDTF: used for portrait evaluation with high-resolution videos and many subjects.
• Wild full-body test set: a newly constructed benchmark containing 250 videos and 200 identities, spanning different races, ages, genders, styles, and initial actions.

The paper uses Q-align to score video quality (IQA) and aesthetics (ASE), FID and FVD to compare generated and real distributions, VBench smoothness for motion stability, and Sync-C for audio-video synchronization. In addition to objective metrics, the authors run a user study with 30 participants who score lip synchronization (LS), identity preservation (IP), full-body naturalness (FBN), and facial naturalness (FCN) on a 1–5 scale.

4. Quantitative and Qualitative Results

The paper compares against state-of-the-art portrait and full-body baselines. For portrait animation, the baselines are Sonic, EchoMimic, EchoMimic-V2, and Hallo-3. For full-body animation, the baselines are Hallo3, FantasyTalking, and OmniHuman-1.

4.1 Qualitative Findings

On portrait benchmarks, the authors report that the method generates more natural and vivid facial expressions and more aesthetically pleasing videos than the baselines. On the full-body wild dataset, they emphasize more realistic foreground and background motion, better lip synchronization, and stronger identity consistency, especially in scenarios involving character interactions and body movement. The appendix further shows results for multi-character dialogue, realistic scenes, diverse stylized characters, and emotion control.

4.2 Quantitative Results: Portrait and Full-Body

From the reported numbers, the method is strongest on the portrait benchmarks in IQA, ASE, and FID, and it achieves the best FVD on HDTF among the compared methods. On the full-body set, it is best on all five reported metrics. Notably, the portrait Sync-C values are not the best; Sonic is higher on Sync-C for both CelebV-HQ and HDTF, while the proposed method still wins on visual quality and distribution-based metrics.

4.3 User Study

The user study suggests a more nuanced picture. HunyuanVideo-Avatar leads on identity preservation and lip synchronization, which aligns with the design of the character image injection module and the face-aware audio adapter. However, OmniHuman-1 receives stronger scores for facial naturalness and full-body naturalness, which the paper attributes partly to its super-resolution advantage and the fact that HunyuanVideo-Avatar inherits some limitations from the HunyuanVideo base model.

5. Ablations and Module-Level Analysis

5.1 Character Image Injection Module Ablation

The paper evaluates the three injection strategies with subjective scores on video quality (VQ), motion diversity (MD), identity preservation (IP), and lip synchronization (LS). The authors conclude that the Token + Add design gives the best overall balance, especially for dynamics and identity consistency, and is the strongest choice among the tested variants.

5.2 Audio Emotion Module Ablation

The qualitative ablation indicates that text-only emotion guidance is insufficient for reliable facial affect control. When the emotion reference image is injected through the Audio Emotion Module, the model better transfers emotional cues to the generated character face. This supports the paper’s claim that emotion style is most effectively controlled by conditioning on an explicit emotional reference rather than relying on the audio stream alone.

5.3 Face-Aware Audio Adapter Ablation

The ablation shows that, without the face mask, the audio influences all visible characters in the scene. With the face mask enabled, only the targeted character is driven by the audio, and the control can be moved to another person by moving the mask. This is the key mechanism that makes multi-character dialogue generation feasible.

6. Appendix: Additional Visualizations

The appendix provides extra qualitative examples that reinforce the main claims: robustness in multi-character scenarios, realism in ordinary scenes, generalization to diverse stylized characters, and fine-grained emotion control.

7. Limitations and Societal Implications

The paper is explicit about several limitations. First, emotion control still depends on emotion reference images rather than direct emotion inference from audio, which increases user burden and prevents the model from naturally handling changing emotions within a single clip. Because each reference image corresponds to a single emotion, audio that contains multiple emotional states may be difficult to model correctly.

Second, generation is still slow. The authors report that a 10-second video at $720 \times 1216$ resolution with 50 inference steps takes about 60 minutes, making the current system unsuitable for real-time applications. They identify faster generation and interactive feedback as future goals.

In the societal-impacts discussion, the paper positions the work as supporting real-time digital humans by improving expressiveness and naturalness, but it also notes that practical deployment remains limited by the cost and latency of large generative models.

8. Conclusion

HunyuanVideo-Avatar is presented as a high-fidelity audio-driven human animation system that extends an MM-DiT video backbone with three targeted mechanisms: character-image injection for consistency and motion, face-aware audio injection for multi-character control, and emotion-reference conditioning for affect alignment. The reported results show clear gains over prior methods on portrait and full-body benchmarks, especially in identity preservation, aesthetic quality, and multi-character audio-driven generation. The main unresolved issues are dependence on emotion reference images and non-real-time inference speed.