KlingAvatar 2.0
KlingAvatar 2.0 Technical Report
KlingAvatar 2.0 generates long, high-resolution avatar videos with strong lip-sync, identity preservation, and multimodal instruction following. It uses a spatio-temporal cascade and LLM-based director for coherent, expressive, and multi-person talking-head video synthesis.
Links
Paper & demos
Abstract
Avatar video generation models have achieved remarkable progress in recent years. However, prior work exhibits limited efficiency in generating long-duration high-resolution videos, suffering from temporal drifting, quality degradation, and weak prompt following as video length increases. To address these challenges, we propose KlingAvatar 2.0, a spatio-temporal cascade framework that performs upscaling in both spatial resolution and temporal dimension. The framework first generates low-resolution blueprint video keyframes that capture global semantics and motion, and then refines them into high-resolution, temporally coherent sub-clips using a first-last frame strategy, while retaining smooth temporal transitions in long-form videos. To enhance cross-modal instruction fusion and alignment in extended videos, we introduce a Co-Reasoning Director composed of three modality-specific large language model (LLM) experts. These experts reason about modality priorities and infer underlying user intent, converting inputs into detailed storylines through multi-turn dialogue. A Negative Director further refines negative prompts to improve instruction alignment. Building on these components, we extend the framework to support ID-specific multi-character control. Extensive experiments demonstrate that our model effectively addresses the challenges of efficient, multimodally aligned long-form high-resolution video generation, delivering enhanced visual clarity, realistic lip-teeth rendering with accurate lip synchronization, strong identity preservation, and coherent multimodal instruction following.
Introduction
KlingAvatar 2.0 is a technical report on audio-driven avatar video synthesis for long-duration, high-resolution human video generation with strong lip synchronization, identity preservation, and multimodal instruction following. The paper positions the problem as a limitation of prior avatar generators: as videos get longer and higher resolution, models tend to suffer from temporal drifting, quality degradation, and weaker adherence to the user’s prompt. The proposed system is built as a unified framework that combines a spatio-temporal cascade generator, a multimodal planning module called the Co-Reasoning Director, a negative-prompt refinement module, and a multi-character control mechanism.
The report is framed around the practical needs of conversational AI and talking-head systems: coherent speech-driven facial motion, believable upper-body and full-body animation, camera motion that follows instructions, and support for multiple people with identity-specific audio control. The paper argues that existing methods are good at short clips or simpler talking portraits, but are less efficient and less stable for extended, cinematic, open-domain avatar videos.
Problem Setting and High-Level Design
The system takes as input a reference image, one or more audio streams, and textual instructions. From these modalities, it generates long-form avatar videos that should satisfy several simultaneous goals: visual fidelity, speech synchronization, semantic adherence to the text, coherent camera and body motion, emotional expressiveness, and robustness over long time horizons. The authors describe their design as an extension of Kling-Avatar, but with a stronger emphasis on long-video efficiency and multimodal planning.
The core idea is to separate generation into stages that are easier to control. Instead of directly generating a long high-resolution video in one pass, KlingAvatar 2.0 first creates a low-resolution blueprint video that captures global semantics and motion. Those keyframes are then refined into higher-resolution, temporally coherent segments. This cascade is meant to reduce the failure modes common in long video generation: it keeps global structure stable while allowing local refinement of appearance, lip motion, and motion details.
A second key idea is to use an LLM-based planning layer to convert sparse or ambiguous multimodal inputs into shot-level storylines. The paper argues that multimodal conflicts are common in avatar generation—for example, audio emotion may not match the text, or a prompt may underspecify a camera action. The planning layer is designed to reason over these conflicts before diffusion generation begins.
Method
Spatial-Temporal Cascade Modeling
The generation pipeline is organized as a nested spatial-temporal cascade built on audio-driven DiT video diffusion models and pretrained video diffusion backbones. The paper describes the following stages:
- Blueprint generation: a low-resolution diffusion model produces a blueprint video that captures the global content, layout, and motion of the scene.
- High-resolution keyframe refinement: representative low-resolution keyframes are upscaled by a high-resolution DiT, adding detail while preserving identity and scene composition.
- Sub-clip generation: a low-resolution video diffusion model expands the high-resolution anchor keyframes into audio-synchronized sub-clips using a first-last-frame conditioned generation strategy.
- Transition synthesis: an audio-aware interpolation strategy generates transition frames to improve temporal continuity and synchronization.
- Final super-resolution: a high-resolution video diffusion model performs super-resolution on the low-resolution sub-clips to produce temporally coherent, high-fidelity segments.
The paper does not provide a formal objective for this cascade in the excerpted body, but the description makes clear that the architecture is meant to decompose a difficult long-video generation problem into manageable stages. The low-resolution blueprint handles global planning; the later stages focus on localized spatio-temporal refinement and visual fidelity. The authors emphasize that this design helps mitigate temporal drifting and improves computational efficiency for long-form generation.
Co-Reasoning Director
The Co-Reasoning Director is a multimodal planning module that reasons jointly over audio, image, and text using a multi-turn dialogue. It is instantiated with three modality-specialized experts:
- Audio-centric expert: performs transcription and paralinguistic analysis, including emotion, prosody, and speaking intent.
- Visual expert: summarizes appearance, layout, and scene context from the reference image.
- Textual expert: interprets the user instruction, integrates conversational history from the other experts, and produces a coherent storyline plan.
The experts reason in several rounds with chain-of-thought style dialogue. According to the paper, this helps resolve modality conflicts and fill in underspecified details such as implied actions or camera movements. The output is a structured storyline that decomposes the video into a sequence of shots, which then guides the diffusion backbone. For long videos, the director further refines the global storyline into segment-level plans aligned with the audio timeline.
A related component is the negative director. Rather than relying on a small, generic list of negative prompts for the entire video, the paper uses detailed, shot-specific negative prompts that suppress implausible poses, artifacts, and mismatched emotional or motion styles. The authors explicitly mention examples such as reducing the likelihood of a sad expression when the intended emotion is happy, or suppressing motion that is too fast or too slow relative to the target style.
Multi-Character Control
KlingAvatar 2.0 extends single-speaker avatar generation to multi-character scenes with identity-specific audio control. The paper’s key observation is that deep features in DiT blocks become semantically coherent and spatially organized, which makes them suitable for character-aware masking. Based on this, the system attaches a mask-prediction head to selected deep DiT blocks.
The mechanism works as follows: for a specified character in the first frame, the system encodes identity crops using the same patchification scheme as the video tokens, but without adding noise to the reference tokens. It then computes cross-attention between deep video latent tokens and the identity tokens for each person, and uses MLP modules to regress per-frame character masks. Ground-truth masks are downsampled to match the resolution of the intermediate latent features. During training, the main DiT backbone is frozen and only the mask-prediction modules are optimized. During denoising, the predicted masks gate identity-specific audio injection to the relevant spatial regions.
The paper also describes an automated annotation pipeline for producing per-character masks at scale. The pipeline uses YOLO for person detection, DWPose for keypoint estimation, and SAM2 for segmentation and temporal tracking. Characters are detected in the first frame, keypoints are estimated for each detection, and the resulting boxes and keypoints seed SAM2 to segment and track individuals over time. Generated masks are validated using frame-level YOLO and DWPose outputs, and misaligned or low-overlap segments are filtered out.
Accelerated Video Generation
The paper also discusses acceleration via distillation. It compares trajectory-preserving distillation methods such as PCM and DCM with distribution-matching distillation such as DMD, and states that trajectory-preserving distillation was selected after considering training stability, inference flexibility, cost, and final generation quality. The authors further mention customized time schedulers that analyze the base model’s behavior at different timesteps to balance speedup and performance. They also introduce a multi-task distillation paradigm, which is described as producing synergistic gains across tasks.
The paper does not spell out a full mathematical derivation of the distillation loss in the provided body text, so the safe takeaway is architectural and empirical: acceleration is treated as a system-level capability, not as the core novelty of the report.
Training Data and Curation
The paper states that the training dataset expands upon the earlier Kling-Avatar system and includes a substantially larger collection of high-quality, cinematic-level video data. The dataset covers multilingual and multi-character conversational scenarios. The authors emphasize extensive filtering to preserve visual fidelity and consistent audio-lip synchronization, but they do not disclose exact dataset sizes, sample counts, or a full data composition breakdown in the main text.
For multi-character data specifically, the report highlights the need for large-scale mask annotation and describes the automated pipeline above as a practical solution for producing per-character video masks. This is important because the multi-character capability depends on reliable region-level supervision for identity-specific audio gating.
Evaluation Protocol
The paper follows the human preference-based subjective evaluation protocol from Kling-Avatar. The evaluation set contains 300 test cases, each with a paired image, audio, and text prompt. These are split evenly across 100 Chinese speech, 100 English speech, and 100 singing samples. Human annotators perform Good/Same/Bad pairwise comparisons between KlingAvatar 2.0 and baseline systems.
The main metric is reported as $(G+S)/(B+S)$, where higher values indicate stronger human preference. The authors also break the evaluation into finer-grained criteria:
- Face–lip synchronization: alignment between speech and lip/facial motion, including continuity and naturalness.
- Visual quality: sharpness, detail fidelity, temporal consistency, and artifact resistance.
- Motion quality: plausibility and smoothness of body, head, and camera motion.
- Motion expressiveness: diversity and emotional richness of lip, facial, and full-body motion.
- Text relevance: consistency between generated video dynamics and the text instruction.
Experimental Results
The paper compares KlingAvatar 2.0 against HeyGen, Kling-Avatar, and OmniHuman-1.5. The reported pairwise GSB results show that the new system is preferred across all compared dimensions, with especially strong gains in motion expressiveness and text relevance. The authors emphasize that the system also improves visual clarity, camera motion, and lip-sync fidelity, though the table itself is reported as pairwise preference ratios rather than absolute scores.
| GSB comparison | Overall | Face-Lip Sync. | Visual Qual. | Motion Qual. | Motion Expr. | Text Rel. |
|---|---|---|---|---|---|---|
| Ours vs. HeyGen | 1.26 | 0.86 | 1.76 | 0.88 | 1.53 | 1.39 |
| Ours vs. Kling-Avatar | 1.73 | 0.80 | 0.89 | 1.13 | 2.47 | 3.73 |
| Ours vs. OmniHuman-1.5 | 1.94 | 1.02 | 1.99 | 1.06 | 1.13 | 1.08 |
The authors’ qualitative analysis focuses on three recurring strengths. First, the system produces more natural hair dynamics and more vivid facial expressions than the baselines. Second, it adheres better to specified camera motion, including a bottom-to-top camera trajectory. Third, it follows fine-grained textual instructions more faithfully, such as a prompt describing a character turning to the front and folding her hands in front of her chest.
The representative results figure shows that the spatio-temporal cascade and co-reasoning planner can generate emotionally expressive characters with realistic motion dynamics and accurate lip synchronization, while also supporting multi-character generation with identity-specific audio control.
Ablations and Component Analysis
The paper includes a focused ablation on the negative director. The corresponding qualitative comparison shows that adding shot-specific negative prompts improves facial expressiveness, strengthens temporal stability and emotional controllability, and reduces lighting and exposure artifacts. This supports the claim that negative prompting is not merely cosmetic, but helps align the generated content with the intended storyline and emotional direction.
Beyond the negative director, the body text also gives qualitative evidence for the value of the cascade architecture and the co-reasoning planner, but it does not present a full numeric ablation table in the provided LaTeX. The main argument is structural: the cascade helps with long-form efficiency and detail preservation, while the planning layer improves multimodal alignment and story coherence.
Stated Contributions and Novelty
- Spatial-temporal cascade framework: a hierarchical generation design that progressively upsamples across time and space to support long, high-resolution avatar videos.
- Co-Reasoning Director: a multi-expert, multi-turn multimodal planner that converts audio, image, and text into shot-level storylines.
- Negative director: detailed negative prompting at the shot level to suppress artifacts and mismatched emotional or motion styles.
- Multi-character multi-audio control: identity-aware mask prediction and gated audio injection for multi-person conversational scenes.
- Large-scale data curation: an automated annotation pipeline using YOLO, DWPose, and SAM2 to support multi-character training data.
Limitations and Scope
The paper does not include a dedicated limitations section. Based on the main text alone, the most visible constraints are practical rather than theoretical: the system depends on a multi-stage cascade, the planning layer introduces additional complexity, and the multi-character mode relies on automated mask annotation and identity-specific audio control. The report also indicates that several training and distillation details are implementation-dependent and not fully specified in the main body. No explicit failure analysis, latency numbers, or hard failure cases are provided in the paper text supplied here.
Another scope consideration is that the evaluation is human-preference driven and centered on 300 curated test cases. That is appropriate for avatar quality assessment, but it means the paper’s reported evidence is strongest for perceptual quality, alignment, and controllability rather than for exhaustive coverage of all possible deployment conditions.
Conclusion
KlingAvatar 2.0 presents a system-level approach to audio-driven digital human generation that combines long-horizon planning, staged spatio-temporal refinement, and multi-character control. The paper’s central claim is that long-form, high-resolution avatar synthesis becomes more reliable when global semantics are planned first, local motion is refined in stages, and modality conflicts are resolved before diffusion generation. In the reported experiments, this design yields strong human-preference scores, especially for motion expressiveness and text relevance, while also improving lip synchronization, visual quality, and multi-person controllability.
For a talking-head or conversational-AI team, the main practical takeaway is that the report treats avatar generation not as a single diffusion pass but as a coordinated multimodal system: planner, generator, negative prompt refiner, and character-specific control all work together to stabilize long-form output.