Kling-Avatar

Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis

Kling-Avatar grounds multimodal instructions into a coherent storyline to guide photorealistic, long-duration avatar animation. It combines multimodal language modeling for semantic planning with parallel video synthesis to produce vivid, expressive, and semantically aligned talking-head videos.

• multimodal
• talking-head
• avatar
• lip-sync
• audio-driven
• emotion
• portrait-animation

Demos

These demos showcase Kling-Avatar's ability to generate high-quality, long-duration avatar animations guided by multimodal instructions. Watch for precise lip-audio alignment, natural dynamic body motions, rich emotional expressions, and fluent transitions in extended video sequences. The generalization demos highlight the framework's flexibility across diverse avatars and scenarios, confirming its robustness for varied open-world usage.

Authors: Yikang Ding, Jiwen Liu, Wenyuan Zhang, Zekun Wang, Wentao Hu, Liyuan Cui, Mingming Lao, Yingchao Shao, Hui Liu, Xiaohan Li, Ming Chen, Xiaoqiang Liu, Yu-Shen Liu, Pengfei Wan

Categories: cs.CV

Comment: Technical Report. Project Page: https://klingavatar.github.io/

Published 2025-09-11 · Updated 2025-09-17

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar22 citations3 influential citations

Abstract

Recent advances in audio-driven avatar video generation have significantly enhanced audio-visual realism. However, existing methods treat instruction conditioning merely as low-level tracking driven by acoustic or visual cues, without modeling the communicative purpose conveyed by the instructions. This limitation compromises their narrative coherence and character expressiveness. To bridge this gap, we introduce Kling-Avatar, a novel cascaded framework that unifies multimodal instruction understanding with photorealistic portrait generation. Our approach adopts a two-stage pipeline. In the first stage, we design a multimodal large language model (MLLM) director that produces a blueprint video conditioned on diverse instruction signals, thereby governing high-level semantics such as character motion and emotions. In the second stage, guided by blueprint keyframes, we generate multiple sub-clips in parallel using a first-last frame strategy. This global-to-local framework preserves fine-grained details while faithfully encoding the high-level intent behind multimodal instructions. Our parallel architecture also enables fast and stable generation of long-duration videos, making it suitable for real-world applications such as digital human livestreaming and vlogging. To comprehensively evaluate our method, we construct a benchmark of 375 curated samples covering diverse instructions and challenging scenarios. Extensive experiments demonstrate that Kling-Avatar is capable of generating vivid, fluent, long-duration videos at up to 1080p and 48 fps, achieving superior performance in lip synchronization accuracy, emotion and dynamic expressiveness, instruction controllability, identity preservation, and cross-domain generalization. These results establish Kling-Avatar as a new benchmark for semantically grounded, high-fidelity audio-driven avatar synthesis.

---

Introduction

Kling-Avatar tackles a specific but important gap in audio-driven avatar generation: most existing systems can synchronize lips and mimic low-level motion, but they do not explicitly model the communicative intent behind multimodal instructions. The paper argues that this causes semantic conflicts across modalities and weak narrative coherence, for example when the audio suggests a sad or angry tone but the generated avatar remains emotionally mismatched. The proposed solution is a cascaded framework that combines multimodal instruction understanding with photorealistic portrait generation, aiming to produce long-duration videos that are not only visually realistic, but also semantically grounded and controllable.

The core idea is to move from shallow cue matching to a two-level planning-and-rendering pipeline. First, an MLLM Director transforms audio, image, and text cues into a structured storyline that captures high-level semantics such as character motion, emotional trajectory, camera behavior, background layout, and visual style. Second, the model uses blueprint keyframes from that storyline to generate multiple sub-clips in parallel, refining local details while preserving global intent. The paper positions this design as especially useful for long-duration use cases such as digital human livestreaming, podcasting, public speaking, education, and vlogging.

The authors summarize their main contributions as follows:

• Unified multimodal instruction grounding. The paper introduces an MLLM Director that converts multimodal evidence into a unified high-level plan rather than treating each condition independently.
• Cascaded long-duration synthesis. A two-stage generation pipeline first establishes global semantics and then refines local dynamics in parallel sub-clips.
• Curated training and benchmark construction. The work includes a multi-stage data filtering pipeline and a 375-sample benchmark covering diverse, challenging scenarios.
• High-fidelity and generalizable outputs. The system reports strong performance on lip synchronization, visual quality, controllability, identity preservation, and cross-domain generalization.

Method

Problem formulation

The system takes a conditioning image, an audio signal, and a text prompt, and outputs a fluent portrait video with accurate lip synchronization, expressive motion, and long-horizon temporal stability. The method is built on top of a Video Diffusion Transformer backbone that was pretrained on large-scale data, and then augmented for audio-driven video synthesis with additional cross-modal conditioning paths.

MLLM Director for multimodal instruction grounding

The first stage is a semantic planning component called the MLLM Director. Instead of directly pushing audio and image features into the generator and hoping the model infers intent, Kling-Avatar converts the multimodal inputs into an explicit storyline. The paper describes the following pipeline:

• Audio understanding: Qwen2.5-Omni is used to extract transcription and emotion from the audio, producing an audio caption.
• Image understanding: Qwen2.5-VL is used to caption the reference image.
• Prompt fusion: the audio caption, image caption, and user prompt are combined and fed into the MLLM Director.
• Storyline generation: the director outputs a coherent storyline using a three-shot in-context template.

The storyline is explicitly prioritized by user knowledge, then audio, then image references. It encodes scene-level and character-level factors such as character features, background layout, actions, camera planning, visual style, and emotional shifts. This storyline is then injected into the video model through text cross-attention to produce a blueprint video.

Cascaded generation for long-duration synthesis

The second stage turns the blueprint into a long video by splitting the sequence into sub-clips and generating them in parallel. The method first segments the timeline evenly according to the desired number of clips. Around each segmentation point, it chooses a high-quality anchor frame that preserves identity, contains meaningful motion, avoids occlusion, and shows expressive facial detail. These frames serve as the first and last frame conditions for adjacent sub-clips.

During sub-clip generation, the MLLM Director further decomposes the global storyline into temporally localized semantic plans. These localized plans, together with time-aligned audio conditioning, guide the denoising process so that each clip remains locally expressive while still matching the global narrative. To reduce mismatches between anchor frames and speech timing, the system also uses an audio-conditioned interpolation strategy to synthesize transition frames and improve temporal coherence.

The paper emphasizes that the pipeline is naturally parallelizable: each clip can be generated independently, so increasing the number of anchors allows arbitrarily long videos without a proportional runtime increase. This is one of the paper's main arguments for why the approach is suitable for real-world long-duration avatar applications.

Training-time alignment strategies

The paper does not introduce a new loss formulation, but it does describe several practical strategies that improve synchronization and robustness during training:

• Sliding-window audio injection: audio features are injected into the audio cross-attention layer so that each video token attends only to temporally aligned audio tokens with a small padding window.
• Mouth-region loss reweighting: DWPose is used to locate the mouth region, and the denoising loss in that area is weighted more heavily to strengthen lip-audio correspondence.
• Random empty-padding augmentation: empty pixels are randomly added around frames during training to reduce the face's relative size, improving robustness to small-face and long-shot cases.
• Frozen text cross-attention: the text cross-attention layer is frozen to preserve the controllability of the base model and prevent overfitting to talking-head specifics.

Inference-time stabilization

At inference, Kling-Avatar adds two extra techniques to improve identity consistency and lip control:

• Negative-frame classifier-free guidance (CFG): the authors analyze identity-drift artifacts such as texture distortion, blur, exaggerated contrast and saturation, and color shifts. They manually corrupt the reference image to mimic these failure modes and use the degraded image as a negative CFG signal, pushing the denoising process away from drift-prone directions.
• Audio-attention strengthening: because ground-truth mouth masks are unavailable at inference, the model increases audio cross-attention values to reinforce lip-audio alignment.

Data and benchmark construction

Training data

The training corpus consists of thousands of hours of audio-visual content collected from public datasets and self-collected videos, including film clips, speeches, monologues, interviews, and singing performances. The paper makes an explicit methodological point that data quality matters more than raw scale for this task: a smaller amount of clean talking segments is more useful than indiscriminately increasing dataset size with long-tail noisy samples.

To enforce quality, the authors build a filtering pipeline with several expert models:

• Lip-clarity filtering: a discriminator is trained on synthetic positive/negative pairs created by perturbing the mouth region in high-quality talking-head videos, and is used to reject visually ambiguous or motion-blurred lip regions.
• Temporal-continuity detection: manually assembled discontinuous clips are used as negatives for a temporal coherence discriminator, combined with PySceneDetect to remove temporally broken segments.
• Audio-visual synchronization scoring: SyncNet is used to score frame-level A/V alignment, and clips below a calibrated threshold are discarded.
• Aesthetic quality assessment: video aesthetic scoring methods are used to retain only visually appealing videos above a calibrated threshold.

After these automatic filters, the retained samples are manually curated, yielding hundreds of hours of high-quality human portrait videos for supervision.

Benchmark

To evaluate instruction-grounded avatar synthesis, the paper constructs a benchmark of 375 image-audio-prompt samples. The benchmark is intentionally diverse and difficult:

• Images: reference images are split equally between real-video frames and AI-generated content. The set includes 340 human portraits across different races, in both full-body and half-body formats, plus 35 non-human examples such as cartoon, anime, and animal characters. Resolutions range from 480p to 1080p and include vertical, horizontal, and square aspect ratios.
• Audio: the audio portion contains both speech and singing, with 150 Chinese, 150 English, 35 Korean, and 40 Japanese clips. Clip duration ranges from 8 seconds to 2 minutes, and the samples cover multiple speaking rates and expressive styles.
• Prompts: prompts are manually annotated with explicit instructions for emotion, action, camera movement, and background layout. The emotion set includes calm, excitement, confusion, sadness, surprise, and anger, each with multiple intensity levels.

This benchmark is designed to test not only lip synchronization, but also control response, identity preservation, and the ability to maintain coherence under challenging multimodal instructions.

Training and inference details

Implementation

The model is built on a pretrained Video Diffusion Transformer and extended with an audio cross-attention layer for audio-to-video generation. Audio features are extracted with a pretrained Whisper encoder, and text conditioning uses a T5 encoder. Optimization uses AdamW with a learning rate of $10^{-5}$. The framework supports training at arbitrary video resolutions from 480p to 1080p, and inference can produce videos up to 1080p at 48 fps.

Evaluation protocol

The main evaluation is a human preference protocol using Good / Same / Bad judgments. For each benchmark sample, three participants compare Kling-Avatar with a baseline, and the final label is determined by majority vote. The paper reports the score as $\frac{G+S}{B+S}$, i.e., the proportion of cases where the system is judged better than or at least not worse than the baseline.

In addition to the overall score, the authors evaluate four sub-dimensions:

• Lip synchronization: naturalness of mouth motion and audio-visual alignment.
• Visual quality: aesthetic appeal, structural coherence, and clarity.
• Control response: whether emotions, actions, and camera motions follow the prompt; for OmniHuman-1, which does not accept prompt input, this dimension is evaluated in terms of how audio controls body motion.
• Identity consistency: preservation of the reference identity and characteristic dynamics.

Baselines

The paper compares Kling-Avatar mainly against OmniHuman-1 and HeyGen, chosen because they represent competitive current systems. The authors note that HeyGen is a commercial digital-human product and that OmniHuman-1 is the strongest public benchmark-style baseline in the study.

Experiments and results

The experimental section focuses on human-preference comparisons rather than objective metric suites. The authors argue that subjective GSB judgments better capture the semantic, expressive, and aesthetic properties that matter for avatar generation. The main findings are that Kling-Avatar outperforms both baselines on the overall benchmark and on most sub-dimensions, with especially strong gains in lip synchronization and instruction-following expressiveness.

Quantitative GSB results

The reported numbers show that Kling-Avatar is consistently preferred over OmniHuman-1 across all categories and dimensions. Against HeyGen, it is particularly strong in lip synchronization and visual quality, while some control-response and identity-consistency values are closer to parity. The paper interprets this as evidence that Kling-Avatar retains vividness and diversity better than looping-style generation, while remaining competitive on stability and identity preservation.

Qualitative observations

The paper presents qualitative evidence that the model can follow rich instructions about emotion, camera behavior, and body motion while maintaining accurate lip synchronization. The generated videos are described as vivid, coherent, and responsive, with stronger generalization to open scenarios such as multi-person scenes, cartoons, anime, and other stylized or non-human characters. The long-duration examples show stable identity and continuous motion over extended time spans, including background changes, head movement, and hand gestures.

Ablations

The provided LaTeX source does not include a dedicated ablation table or ablation section with numerical component removals. Instead, the paper explains the purpose of each design choice procedurally: the MLLM Director is introduced to lift generation from low-level tracking to intent understanding, the cascaded parallel strategy is used to scale to long durations, and the training/inference heuristics are presented as practical stabilizers for synchronization and identity preservation. Because no ablation numbers are reported in the supplied text, any quantitative decomposition of these contributions would be speculative.

Stated limitations and future work

The paper does not include a formal limitations section, but it does state two clear future directions. First, the authors plan to incorporate additional objective metrics to complement the human-preference GSB protocol. Second, they plan to extend comparisons to other commercial systems such as Higgsfield and Hedra. The evaluation setup is therefore intentionally centered on subjective human judgments, and the paper frames this as a practical choice for assessing the semantic and aesthetic properties of avatar generation.

Conclusion

Kling-Avatar proposes a semantically grounded approach to talking-head and portrait animation generation. Its main technical contribution is the combination of multimodal understanding, explicit storyline planning, and cascaded parallel video synthesis. The MLLM Director turns audio, image, and prompt information into a blueprint-level plan; the second stage uses anchor keyframes and first-last-frame conditioning to generate long videos in parallel; and the training/inference heuristics improve lip synchronization and identity stability. On the 375-sample benchmark, the system is reported to outperform strong baselines on overall human preference and to generalize well across speech, singing, and stylized portrait scenarios. Within the scope of the provided paper, Kling-Avatar is presented as a strong step toward controllable, high-fidelity, long-duration avatar animation.