OmniHuman-1.5

OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation

OmniHuman-1.5 creates avatars that combine reactive lip-sync with high-level planning driven by multimodal inputs, enabling expressive and context-aware motions. It extends avatar animation beyond rhythmic mimicry to semantically coherent actions in complex scenes with multiple characters and non-humans.

• avatar
• talking-head
• multimodal
• llm
• audio-driven
• emotion
• gesture

Authors: Jianwen Jiang, Weihong Zeng, Zerong Zheng, Jiaqi Yang, Chao Liang, Wang Liao, Han Liang, Yuan Zhang, Mingyuan Gao

Categories: cs.CV

Comment: Homepage: https://omnihuman-lab.github.io/v1_5/

Published 2025-08-26 · Updated 2025-08-26

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar28 citations9 influential citations

Abstract

Existing video avatar models can produce fluid human animations, yet they struggle to move beyond mere physical likeness to capture a character's authentic essence. Their motions typically synchronize with low-level cues like audio rhythm, lacking a deeper semantic understanding of emotion, intent, or context. To bridge this gap, \textbf{we propose a framework designed to generate character animations that are not only physically plausible but also semantically coherent and expressive.} Our model, \textbf{OmniHuman-1.5}, is built upon two key technical contributions. First, we leverage Multimodal Large Language Models to synthesize a structured textual representation of conditions that provides high-level semantic guidance. This guidance steers our motion generator beyond simplistic rhythmic synchronization, enabling the production of actions that are contextually and emotionally resonant. Second, to ensure the effective fusion of these multimodal inputs and mitigate inter-modality conflicts, we introduce a specialized Multimodal DiT architecture with a novel Pseudo Last Frame design. The synergy of these components allows our model to accurately interpret the joint semantics of audio, images, and text, thereby generating motions that are deeply coherent with the character, scene, and linguistic content. Extensive experiments demonstrate that our model achieves leading performance across a comprehensive set of metrics, including lip-sync accuracy, video quality, motion naturalness and semantic consistency with textual prompts. Furthermore, our approach shows remarkable extensibility to complex scenarios, such as those involving multi-person and non-human subjects. Homepage: \href{https://omnihuman-lab.github.io/v1_5/}

---

1. Problem Setting and High-Level Idea

OmniHuman-1.5 addresses a limitation that the authors argue is shared by many recent video-avatar systems: they can generate visually fluent human motion, but their behavior is often driven by low-level correlations such as audio rhythm rather than by higher-level intent, emotion, or scene context. In the paper's framing, prior systems behave like a mostly reactive System 1, while human-like avatars should also include a deliberative System 2 that plans motion in a semantically coherent way.

The core goal is therefore not just lip-synchronized animation, but generation of videos that are physically plausible, semantically coherent, and expressive across audio, text, and reference image conditioning. The model is presented as a framework for audiovisual avatar generation that can also generalize to more difficult settings, including multi-person scenes and non-human subjects.

2. Main Contributions

• Agentic semantic planning. The model uses multimodal large language models (MLLMs) to analyze the input context and produce a structured textual plan that captures persona, emotion, speech content, intent, and environment.
• Multimodal DiT rendering with conflict-aware fusion. The video synthesis backbone is a Multimodal Diffusion Transformer (MMDiT) with dedicated branches for video, text, and audio, fused through shared self-attention rather than standard audio cross-attention.
• Pseudo Last Frame identity strategy. Instead of conditioning on a training-time reference image in the usual way, the method trains on native first/last frames and uses the user's reference image as a pseudo last frame at inference to preserve identity without forcing the image to appear literally in the generated sequence.
• Generalization beyond canonical talking heads. The paper reports qualitative and quantitative results on non-human subjects, dialogue turn-taking, and multi-person interaction scenarios.

3. Method Overview

The full pipeline is organized as a dual-system framework. A deliberative planning module first produces a high-level schedule, and a reactive diffusion renderer turns that schedule into frames. The backbone is a pre-trained DiT video generator operating in latent space, trained with a flow-matching objective over a compact 3D VAE representation. The authors emphasize that standard components such as autoregressive long-video generation and latent diffusion are conventional; the paper's novelty lies primarily in how planning and multimodal conditioning are structured to reduce conflicts and improve semantic coherence.

4. Agentic Reasoning: Deliberative Control via MLLMs

The agentic module is the paper's explicit attempt to model the deliberative side of avatar behavior. It takes as input the reference image, audio, and an optional text prompt. From these, it generates two kinds of semantic guidance:

• Reasoning text, which is the explicit chain-of-thought-like textual output used directly as a condition for generation.
• Reasoning latents, which are intermediate MLLM features extracted from the analyzer and optionally fused into the generator.

In the default configuration, the paper primarily relies on reasoning text rather than latent conditioning.

4.1 Two-Stage Reasoning Pipeline

The reasoning process is split across two MLLMs. The first, called the Analyzer, receives the reference image, an auxiliary caption of that image, the audio clip, and the optional prompt. It performs iterative reasoning to infer the character's persona, speaking style, speech content, emotion, intent, and environmental context, and then compiles this into a structured representation, typically a JSON-like object. The second MLLM, the Planner, consumes the Analyzer output plus the original image and turns this semantic understanding into a shot-level action plan. Each shot corresponds to one generation pass of the diffusion model, so the plan is effectively a motion schedule for the video.

The resulting schedule is meant to preserve a coherent character persona over time, rather than simply reacting to local audio cues. This is the paper's main mechanism for injecting high-level intent into motion generation.

4.2 Reflective Re-planning

For longer videos, the authors add an optional reflection loop. During autoregressive synthesis, the Planner can re-examine the most recently generated frames together with the original reference image and revise the next part of the plan. The motivation is to correct semantic drift and reduce accumulated inconsistency in long-form generation. The paper explicitly notes, however, that this reflection introduces extra inference overhead and was therefore disabled for quantitative comparisons.

4.3 Latent Feature Conditioning

The authors also experimented with using MLLM internal features as an additional conditioning signal. Concretely, they extracted audio token representations from the final transformer layer of the Analyzer after multimodal cross-attention had enriched them with higher-level semantics, then concatenated these "reasoning-infused" audio latents with raw audio features before feeding them to the DiT. This variant was explored but not chosen as the final default.

5. Reactive Rendering: Multimodal Diffusion with Conflict Mitigation

The rendering stage turns the semantic schedule into frames. The authors argue that conditioning design is critical because audio, text, and identity cues all want to influence motion, but naïvely combining them can create conflicts. Their solution is an MMDiT with three symmetric branches and two additional training/inference strategies: the pseudo last frame and branch warm-up.

5.1 Pseudo Last Frame Identity Strategy

The paper devotes significant attention to the role of the reference image. In many avatar systems, a reference image is sampled from the training video and used for identity preservation. The authors argue that this can create a spurious correlation: the model learns that the reference image should literally recur in the generated clip, which restricts motion and conflicts with audio/text-driven action. Sampling reference images from outside the clip can solve that artifact but introduces another problem, because the model may then learn that the target output should deviate strongly from the reference.

Their alternative is to remove the reference image from training entirely and instead train with native signals: the ground-truth first and last frames of a clip. At inference, the user's reference image is placed in the last-frame slot, with positional encoding shifted so that the model treats it as temporally distant from the generated content. The authors call this a pseudo last frame. It acts as a soft identity guide without forcing literal copying.

5.2 Symmetric Multimodal Fusion

Instead of injecting audio via extra cross-attention layers, OmniHuman-1.5 gives audio its own branch, architecturally symmetric to the video and text branches. All three modalities are then fused at each layer through shared multi-head self-attention. The authors argue that this enables iterative refinement and true joint modeling, since all tokens can attend to each other within a common semantic space.

The paper also reports that this architecture is not easy to train directly. If all branches are optimized jointly from the start, the model may over-rely on the dense audio stream and ignore text guidance. If the pre-trained branches are frozen, the audio branch can overfit and mistakenly absorb non-audio factors such as lighting or camera motion. To solve this, the authors introduce a two-stage warm-up: first train the full three-branch model jointly to establish a division of labor, then construct the final model by reusing the warmed-up audio branch while reinitializing the text and video branches from their pre-trained weights, followed by fine-tuning.

6. Training Setup and Data

The model is built on a pre-trained MMDiT backbone and trained in the latent space of a pre-trained 3D VAE using a flow-matching objective. The implementation details reported in the paper are fairly concrete:

• Clip format: most experiments use 120-frame clips at 24 fps and 480p resolution on the short side.
• Upsampling: a separate super-resolution model with the same architecture is used to upscale outputs to 720p or 1080p.
• Long video generation: videos can be generated autoregressively by using the end of one clip as the start of the next.
• Optimizer: AdamW.
• Learning rate: $5 \times 10^{-5}$.
• Global batch size: 256.
• Gradient clipping: norm clipped to 1.0.
• Compute: training was run on 256 compute nodes.
• Stages: a 3-day audio branch warm-up, a 7-day main training phase, and a 1-day fine-tuning phase on high-quality data.

The training set consists of 15,000 hours of filtered video data. The paper states that a lip-sync model was used to identify and discard audio from videos with poor audio-lip correlation; these discarded or filtered samples account for 70% of the data and were used with audio dropout during training. For the final fine-tuning stage, the authors ranked data by quality metrics and selected the top 100 hours.

7. Evaluation Protocols and Datasets

The evaluation design is intentionally broad. The authors note that standard DiT-based avatar methods already do well on conventional speaking-head scenarios, so they built two more difficult test sets to stress generalization:

• Single-subject benchmark: 150 cases spanning real human portraits, AIGC figures, anime characters, and animals. Experts manually paired each image with audio, including speech, singing, and theatrical performances.
• Multi-subject benchmark: 57 cases with the same diversity, but in multi-character interaction settings.
• Text-conditioning evaluation: experts wrote descriptive prompts for all 150 single-subject cases so that adherence to textual guidance could be measured.
• Prior benchmark compatibility: the paper also follows established settings on 100 videos from CelebV-HQ for talking-head evaluation and the CyberHost test set with 269 videos and 119 identities for full-body generation.

The paper evaluates with a mixture of objective and subjective metrics:

• Image/video quality: FID, FVD, no-reference IQA, and aesthetics score (ASE).
• Audio-visual synchronization: Sync-C for single-subject cases and Sync-D for multi-person evaluation.
• Hand/motion dynamics: Hand Keypoint Confidence (HKC) and Hand Keypoint Variance (HKV).
• User studies: pairwise GSB comparisons, a defect-focused study of lip-sync inconsistency (LSI), motion unnaturalness (MU), and image distortion (ID), plus a best-choice Top-1 selection task.

The subjective evaluation uses 40 participants. In pairwise comparison, participants view two videos in random order and choose the better one; the paper reports a Good/Same/Bad score computed as $(\text{Wins} - \text{Loses})/(\text{Wins} + \text{Loses} + \text{Ties})$. In the best-choice task, participants select the single best video among all methods, yielding a Top-1 rate.

8. Main Quantitative Results

8.1 Portrait Animation on CelebV-HQ

On the portrait benchmark, OmniHuman-1.5 performs competitively with the strongest prior methods, including OmniHuman-1, and is best or near-best on the reported metrics.

The model is not dramatically ahead on every portrait metric, but it is consistently competitive, and the authors interpret this as evidence that portrait generation is already a strong regime for the underlying family of methods. The more important gains appear in harder settings where semantic planning matters more.

8.2 Full-Body Animation on CyberHost

On full-body synthesis, the paper reports stronger advantages in motion dynamics while preserving quality and lip-sync. Compared with recent baselines, OmniHuman-1.5 obtains the best IQA, ASE, FID, and HKV, while staying competitive on Sync-C and HKC.

The strongest motion-related result is HKV, which the authors interpret as evidence that the model produces more dynamic large-scale body movement. The paper also notes that the small decrease in HKC relative to OmniHuman-1 is outweighed by the large gain in motion variance and the overall subjective preference results.

8.3 User Preference Studies

The subjective studies are an important part of the paper's claims because the authors argue that objective metrics under-capture contextual coherence, intent alignment, and perceptual plausibility.

The paper also reports pairwise GSB results against proprietary systems, but the systems are anonymized as CS-A, CS-B, CS-C, and CS-D to comply with EULAs. The qualitative takeaway is that OmniHuman-1.5 is preferred for contextual coherence, which the authors emphasize as a dimension missed by traditional audio-sync metrics.

9. Ablation Studies

The ablation studies are central to the paper's argument because they separate the contribution of the reasoning module from the contribution of the multimodal architecture.

9.1 Agentic Reasoning Ablation

The paper first removes parts of the reasoning pipeline on the single-subject test set. Quantitatively, low-level metrics such as IQA and Sync-C change only modestly across ablations, which the authors interpret as evidence that these metrics do not capture the semantic role of reasoning. By contrast, HKV and the subjective motion-naturality results degrade more substantially as reasoning is reduced.

In the pairwise user study, the full model reduces motion unnaturalness from 0.58 to 0.37 and improves the overall GSB score from $-0.29$ to $+0.29$, while lip-sync inconsistency stays unchanged at 0.12. This supports the paper's claim that reasoning improves semantic motion quality rather than just audio synchronization.

9.2 Conditioning Architecture Ablation

The second ablation study varies the diffusion conditioning design while keeping the reasoning module fixed. The tested variants include replacing the proposed multimodal attention with ordinary cross-attention, removing the multimodal warm-up, conditioning on a reference image, and removing both the reference image and pseudo frame.

In the objective ablation table, the proposed full model achieves the best HKV, which is the clearest motion-related signal. The authors also report that standard cross-attention performs worse than their symmetric multimodal fusion, and that omitting the pseudo last frame or multimodal warm-up harms either motion dynamics or synchronization.

9.3 Text-Only and Base-Model Comparisons

The paper includes a user study where the audio component is omitted to isolate visual fidelity and motion, comparing the model against its base pretrained generator under text-only conditioning. The reported GSB scores show that the proposed method preserves text-following ability while improving motion naturalness and visual quality: text alignment changes only slightly ($-0.02$), while motion and visual quality improve by $+0.18$ and $+0.14$, respectively.

10. Generalization and Multi-Person Extension

A key claim of the paper is that the framework extends beyond standard single-person talking heads. The authors report qualitative generalization to anthropomorphic and animal subjects, conversational turn-taking, and multi-person scenes. For dialogue, they show that the model can switch between speaking and idle states according to the conversational audio, suggesting that the reasoning module provides context beyond raw lip synchronization.

10.1 Multi-Person Adaptation

To extend the method to multi-person animation, the authors make two modifications. First, audio injection is restricted to a speaker-specific mask so that the model can condition only the masked regions during multimodal attention. The mask is produced by a lightweight plug-and-play predictor that tracks speakers through motion and occlusion. Second, the Planner is augmented to take the mask as input so that it can identify the active speaker and reason about who should speak or remain idle. The rest of the reasoning pipeline remains unchanged.

On the multi-person test set, the full model improves both objective motion statistics and pairwise subjective judgments relative to the ablation without reasoning and relative to InterActHuman.

11. Qualitative Analyses

The qualitative section is used to support the paper's central thesis that semantic planning matters. One figure demonstrates the reflection process: without reflection, the plan can become logically inconsistent, for example by generating an action like "Rubs the surface" after "Takes out letter," which causes the letter to vanish and breaks semantic continuity. With reflection, the plan is revised to remain coherent with the emerging scene.

Another qualitative comparison shows OmniHuman-1.5 generating actions with stronger semantic alignment than OmniHuman-1. In the examples described in the paper, the model turns the character's head when the speech says "Mary," applies eyeliner when makeup is mentioned, and animates a glowing crystal ball in response to a wizard's incantation. The authors present these as examples of contextual reasoning that standard objective metrics do not adequately reward.

12. Stated Limitations and Practical Trade-Offs

The paper does not present a dedicated limitations section in the main body, but it does state several practical trade-offs and constraints that function as limitations. The clearest one is that the reflection mechanism improves logical consistency but adds inference overhead, so it is excluded from the paper's quantitative comparisons. The authors also note that injecting reasoning latents can increase nuance in facial expression but may suppress large dynamic actions, which they describe as a trade-off rather than a clear win.

A second implicit limitation is that many of the reported gains are demonstrated on custom benchmarks and curated test sets designed to stress generalization. The paper is explicit that conventional talking-head benchmarks are already largely saturated for strong DiT-based methods, so the proposed advantage is most visible in more complex settings where semantic planning is important. This means the headline benefit is not simply better lip-sync, but better semantic control under harder conditions.

In the broader-impacts discussion, the authors acknowledge the misuse risk of realistic avatar generation and recommend visible watermarks, filtering of malicious or inappropriate prompts, and invisible traceable watermarks for accountability. These are presented as deployment-time safeguards rather than changes to the core method.

13. Bottom-Line Assessment

OmniHuman-1.5 proposes a clear conceptual shift for avatar generation: instead of mapping audio directly to motion, it first uses MLLM-based reasoning to infer a structured semantic plan and then renders that plan with a multimodal diffusion transformer designed to reduce conflicts among audio, text, and identity cues. The experimental story is strongest where semantic coherence and motion richness matter most: user studies, full-body generation, and multi-person interaction. The paper's technical contribution is therefore less about a new basic rendering primitive than about organizing planning and conditioning so that an avatar can appear to have an "active mind" rather than only a reflexive response to sound.