OmniHuman-1

OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models

OmniHuman-1 scales human animation by mixing image, text, audio, and pose inputs during training to produce realistic, flexible videos of diverse portrait styles and motions. This approach surpasses prior audio-driven methods in realism and supports varied driving signals and full-body generation.

• talking-head
• talking-face
• audio-driven
• full-body
• portrait-animation
• gesture
• multimodal

Demos

The selected demos showcase OmniHuman-1's ability to generate highly realistic human videos from a single input image and diverse motion signals, including audio-only, video-only, and combined driving inputs. Watch for the model's handling of various input aspect ratios (portrait, half-body, full-body), the naturalness of motion and gesture synchronization with speech, and the diverse styles including cartoons and animals. The demos also highlight OmniHuman's superior realism in lighting, texture, and motion compared to prior methods.

Authors: Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang

Categories: cs.CV

Comment: ICCV 2025, Homepage: https://omnihuman-lab.github.io/

Published 2025-02-03 · Updated 2025-06-30

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar128 citations11 influential citations

Abstract

End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (https://omnihuman-lab.github.io)

---

Introduction

OmniHuman-1 addresses a central scaling problem in end-to-end human animation: existing audio-driven and video-driven systems can produce convincing talking-head results, but they are typically trained on heavily filtered, narrow datasets and do not scale like modern general video generation models. The paper argues that this data bottleneck is the main reason human animation has not yet exhibited the same scaling behavior seen in text-to-video and image-to-video diffusion transformers. Rather than treating audio as the sole driving signal, OmniHuman introduces a mixed-conditioning training paradigm that leverages text, audio, and pose together, so that data discarded for a strong condition can still be used under weaker conditions.

The core claim is that human animation can scale if the training recipe is changed: weaker conditions can absorb more diverse data, while stronger conditions are used more sparingly and later in training. This lets the model learn from substantially larger and more varied motion data without being over-constrained by audio-only filtering criteria such as lip-sync quality or pose stability. The result is a unified human animation model that supports face close-ups, portraits, half-body and full-body subjects, talking and singing, human-object interaction, challenging body poses, and diverse image styles.

The paper’s main contributions, as presented in the body of the paper, are:

• a Diffusion Transformer-based human animation framework that can be conditioned simultaneously on image, text, audio, and pose;
• an omni-conditions training strategy with two principles for scaling data and balancing condition strength;
• an efficient conditioning design that avoids duplicating an entire reference network by reusing the main denoising backbone for appearance conditioning;
• strong empirical results on portrait animation, body animation, and pose-driven body animation.

Method

Overall design

OmniHuman is built on top of an MMDiT-style video diffusion backbone, initialized from a general text-to-video model pretrained on large-scale text-video data. The model operates in a compact latent space obtained via a causal 3D VAE, so the diffusion model works on compressed video latents rather than raw frames. The paper’s design goal is to keep modality-specific parameters minimal and to let the shared backbone learn most of the multimodal interaction.

The system is organized into two parts: (1) the OmniHuman model, which supports simultaneous conditioning on text, image, audio, and pose; and (2) the omni-conditions training strategy, which progressively introduces conditions from weaker to stronger motion relevance and adjusts their sampling ratios.

Condition injection design

The paper divides modalities into two roles: driving conditions and appearance conditions. The implementation is intentionally lightweight so that most of the model capacity remains inside the shared diffusion backbone.

For audio, the model uses wav2vec features. These multi-scale acoustic features are compressed with an MLP to match the hidden size of the MMDiT backbone and aligned to the video frame rate of $25$ fps. Features from adjacent timestamps are concatenated to form frame-wise audio tokens, which are injected into every MMDiT block through frame-wise cross-attention.

For pose, the authors use a pose guider to encode a sequence of skeleton maps. The resulting pixel-aligned pose features are concatenated across adjacent frames into frame-wise pose tokens. These pose tokens are then stacked with the noisy latent representations along the channel dimension, which helps with visual alignment and motion modeling.

For text, OmniHuman reuses the original MMDiT text branch, following standard text-to-image and text-to-video practice.

For appearance conditioning, the paper does not duplicate the denoising backbone with a separate reference network. Instead, the reference image and noisy video latents are flattened into token sequences, packed together, and jointly processed by the same DiT backbone. This allows reference tokens and video tokens to interact through self-attention across the network. To distinguish them, the 3D RoPE temporal component is zeroed for reference tokens while remaining unchanged for video tokens. The stated benefit is a much smaller parameter overhead than a full reference branch while still enabling strong identity preservation.

For long video continuation, the model additionally uses motion frames: the latents of the last generated frames are concatenated with noise latents for the next segment. The paper’s inference discussion later specifies that the last five frames of the previous segment are used as motion frames for the following segment.

Omni-conditions training strategy

OmniHuman is trained through a three-stage curriculum that progressively turns a generic text-to-video model into a mixed-conditioned human animation model. The three training task types are:

• $I + T \rightarrow V$ : image and text to video;
• $I + T + A \rightarrow V$ : image, text, and audio to video;
• $I + T + A + P \rightarrow V$ : image, text, audio, and pose to video.

The two guiding principles are:

• Principle 1: stronger-conditioned tasks can reuse weaker-conditioned data and thus scale training data;
• Principle 2: the stronger the condition, the lower its training ratio should be.

Principle 1 means that data that might be filtered out for an audio-driven task can still be used for text-conditioned training. In Stage 1, the model excludes audio and pose entirely and is trained only on image-text-to-video data, maximizing data usage. Principle 2 addresses the fact that strong conditions, especially pose, can dominate learning if introduced too aggressively. The paper therefore introduces weaker conditions earlier or at higher ratios, and stronger conditions later or at lower ratios, so that all conditions contribute meaningfully without causing the model to overfit to a single control signal.

In the main text, the authors describe the final transition as follows: Stage 2 drops only the pose condition, and Stage 3 incorporates all conditions. They also report that text, audio, and pose sampling ratios are progressively reduced when present, and that better-balanced ratios improve generalization and motion quality.

Inference strategy

At inference time, OmniHuman activates conditions according to the task. For audio-driven animation, all conditions except pose are active. For mixed audio-plus-pose driving, all conditions are active. For pose-only driving, audio is disabled. The paper states a general rule that when a condition is activated, all conditions with weaker motion influence are also activated unless unnecessary.

The authors use an image captioning model to generate the text input for all animation tasks. They also apply classifier-free guidance only to audio and text, not to pose, and use a guidance scale of $6.5$. The motivation is that weaker conditions benefit from guidance, while pose is already a strong motion control signal. For long videos, temporal coherence is maintained by reusing the last five frames of the previous segment as motion frames for the next segment.

Experimental Setup

Data

Training uses $18.7$K hours of in-house human-related video data, filtered by aesthetics, image quality, motion amplitude, and similar criteria common in video generation. Of this data, $13\%$ is selected using lip-sync and pose visibility criteria so that audio and pose modalities can be used. The paper emphasizes that the training composition is adjusted to fit the omni-condition strategy rather than to isolate a single modality.

For evaluation, the paper follows the portrait animation setup of Loopy and the half-body setup of CyberHost. The portrait test set is built by randomly sampling $100$ videos from public datasets CelebV-HQ and RAVDESS. The body animation test set uses CyberHost’s test split of $269$ body videos with $119$ identities spanning multiple races, ages, genders, and initial poses.

Baselines

The paper compares against representative portrait animation baselines SadTalker, Hallo, VExpress, EchoMimic, Loopy, and Hallo-3, as well as body animation baselines DiffTED, DiffGest + MimicMotion, and CyberHost. A separate comparison is also reported against pose-driven body animation methods DisCo, AnimateAnyone, MimicMotion, and CyberHost.

Metrics

The reported metrics cover both visual quality and motion fidelity. The paper uses FID and FVD to compare generated samples against reference images/videos, IQA and ASE from q-align to assess no-reference image quality and aesthetics, Sync-C to measure audio-visual synchronization, and HKV and HKC to characterize hand motion richness and hand quality. For pose-driven evaluation, the paper additionally reports AKD.

Training is done with AdamW at a learning rate of $5 \times 10^{-5}$, batch size $256$, weight decay $0.01$, and gradient clipping at $1.0$. The work is trained on $400$ A100 GPUs, with each phase taking about $10$ days. The authors report ablations at $480$P resolution and their broader comparisons at $720$P resolution.

Qualitative Results and Generalization

The paper emphasizes that OmniHuman is not only a better audio synchronizer, but also a broader human motion generator. The qualitative examples show improved mouth motion, gesturing, and object interaction. The model is reported to support stylized humanoid characters, $2$D cartoon characters, and even non-human images animated in an anthropomorphic way. It also preserves the visual style of the input image, including anime-like mouth movement patterns.

A notable application discussed by the authors is speaker control via classifier-free guidance: by adjusting the regions influenced by guidance, the user can specify which person or region should speak in the generated video. This is presented as an example of the flexibility enabled by the mixed-conditioning design.

Quantitative Comparisons

Audio-conditioned portrait animation

On CelebV-HQ and RAVDESS, OmniHuman is compared with specialized audio-driven portrait animation systems. The model ranks best across nearly all metrics on both datasets. The gains are especially clear on synchronization and motion quality, while maintaining strong image and video fidelity.

The reported pattern is that OmniHuman improves synchronization, fidelity, and visual realism while remaining competitive or best on perceptual metrics. Compared with Loopy, which is one of the strongest portrait baselines, OmniHuman improves CelebV-HQ Sync-C from $4.849$ to $5.199$ and reduces FID from $33.204$ to $31.435$. On RAVDESS, it improves Sync-C from $4.814$ to $5.255$ and FVD from $16.134$ to $15.906$. These improvements support the paper’s claim that mixed-condition training yields a more capable unified model rather than a narrower audio-sync model.

Audio-conditioned body animation

In body animation, OmniHuman is compared to DiffTED, DiffGest + MimicMotion, and CyberHost. The comparison is important because full-body generation is where hands, object interaction, and pose diversity become most difficult. OmniHuman again shows the best overall performance.

The body-animation results are especially important because the paper’s main promise is broader motion modeling, not only lip-sync. OmniHuman improves both hand quality and motion richness: HKV rises sharply to $47.561$, compared with $24.733$ for CyberHost, while HKC improves from $0.884$ to $0.898$. It also achieves the best reported FID and FVD in the body setting.

Pose-driven body animation

The paper also reports a pose-driven comparison to demonstrate that OmniHuman can function as a unified model beyond audio-only animation. Even though the baselines are specialized pose-driven methods, OmniHuman remains competitive or better on all reported metrics.

This result supports the paper’s argument that the mixed-conditioned training recipe is not just a hack for audio-driven talking heads; it also transfers to pose-conditioned body animation and yields strong driving accuracy and visual quality.

Ablation Studies

Principle 1: scaling with weaker conditions

The first ablation examines how much text-conditioned data should be introduced when audio- and pose-conditioned data are limited. The paper reports that adding more text-conditioned data improves core metrics such as FVD, FID, lip-sync accuracy, and gesture quality, because data that would otherwise be discarded can be used under the weaker text condition. At the same time, the authors observe that the image quality score may drop as the model begins to follow the distribution of the input images rather than the narrower high-quality training subset; they interpret this as improved generalization rather than a failure.

The reported trend is that more text-conditioned data improves the model’s core motion modeling, especially on more diverse datasets. The best-performing entries in this block occur when the model can exploit more mixed data rather than relying only on a small, tightly filtered audio subset.

Principle 2: training order and condition ratio

The second ablation studies the order in which audio and pose are introduced. The paper’s conclusion is that introducing pose too early harms quality, because the stronger signal can dominate learning and reduce the model’s ability to learn robust motion from weaker audio cues. When pose is introduced later, the final model retains quality and gains the flexibility to support both audio-driven and mixed audio-plus-pose driving.

The most important finding here is that the order matters. The paper reports that IPA underperforms, while IAP preserves quality and enables mixed audio-pose driving. The qualitative analysis in the paper shows that IAP also decouples hand motions from the audio signal, reducing overly exaggerated gestures and making the motion more natural. Between the two ratio variants, the setting with audio stronger than pose, $A>P$, is preferred over the opposite setting.

Comparison with a pretrained image-to-video model

The paper also compares OmniHuman against a pretrained image-to-video baseline to illustrate the effect of omni-condition training. The visual comparison shows that OmniHuman improves local human features such as gestures and yields more natural overall motion. This supports the claim that mixed-condition training can enhance a general video model even when the inference task is human animation.

Key Takeaways from the Paper’s Experiments

• Data scaling matters: using more mixed-condition training data, especially text-conditioned data, improves motion and synchronization instead of simply overfitting to a narrow audio-only subset.
• Condition order matters: pose should not be introduced too early; later introduction preserves quality and enables better mixed driving.
• One model covers multiple regimes: the same system performs well on portrait, half-body, and pose-driven tasks rather than needing separate specialized pipelines.
• Broader generalization: the model supports diverse aspect ratios, image styles, cartoon-like inputs, and anthropomorphic animation of non-human images.

Limitations

The paper is explicit that mixed-conditioning does not fully eliminate failure modes. Because audio correlates only weakly with motion, the model can still produce uncoordinated or overly expressive motion. Object interactions can also look unrealistic at times. The authors attribute these issues partly to insufficient training coverage: when the input image is far from the training distribution, the model tends to generate unnatural outputs. To stabilize synthesis, the system uses a relatively high CFG scale, but that in turn can increase overfitting. The proposed direction for future work is to add richer motion conditions, such as style, intensity, and intention, to improve motion naturalness.

Conclusion

OmniHuman proposes a unified, one-stage human animation framework that treats scaling as a conditioning problem rather than only a data-cleaning problem. By mixing text, audio, and pose during training, and by carefully controlling the order and sampling ratios of those conditions, the model can exploit much more of the available human video data. The paper reports strong results on audio-driven portrait animation, audio-driven body animation, and pose-driven body animation, while also showing broader flexibility in input types and visual styles than prior specialized systems.