LongCat-Video-Avatar 1.5

LongCat-Video-Avatar 1.5 Technical Report

An open-source audio-driven avatar video system prioritizing production readiness. It blends large-scale curated data, advanced audio encoding, and reinforcement learning to deliver stable, identity-consistent, long-duration videos with accurate lip-sync and full-body motion in diverse real and stylized scenarios.

• talking-head
• avatar
• audio-driven
• lip-sync
• full-body
• emotion

Authors: Meituan LongCat Team, Xunliang Cai, Meng Cheng, Feng Gao, Zhe Kong, Jiamu Li, Le Li, Weiheng Li, Hongyu Liu, Shuai Tan, Xiaoming Wei, Tianyu Yang, Yong Zhang

Categories: cs.CV

Comment: Homepage: https://meigen-ai.github.io/LongCat-Video-Avatar-1.5-Page/ Github: https://github.com/meituan-longcat/LongCat-Video

Published 2026-05-26 · Updated 2026-05-26

Links

Paper & demos

• arXiv

Code & resources

• GitHub4,562 stars45 watchers730 forksMIT
• Hugging Face2,175 downloads600 likes

Impact

• Semantic Scholar0 citations

Abstract

Despite advances in audio-driven video generation, achieving commercial-grade stability remains challenging. We present LongCat-Video-Avatar 1.5, an upgraded open-source framework prioritizing systematic engineering and production-readiness over architectural novelty. By upgrading the audio encoder to Whisper Large and meticulously scaling our training recipes, v1.5 achieves accurate lip-synchronization, full-body temporal stability, and robust long-video generation with strict identity consistency. Through rigorous data curation and RLHF Training, the model readily generalizes to stylized domains such as anime and animals, and natively handles complex real-world conditions, such as multi-person interactions and object handling. Furthermore, addressing the practical demands of industrial deployment, we employ advanced step distillation to accelerate inference to an optimal 8 NFE, achieving a favorable trade-off between serving efficiency and visual fidelity. The superiority of our approach is validated through extensive quantitative metrics and a rigorous human evaluation conducted on a comprehensive benchmark of over 500 diverse test cases. Results show that v1.5 achieves competitive or superior performance compared to leading closed-source systems (e.g., HeyGen, OmniHuman 1.5, Kling Avatar 2.0) across human-likeness ratings and expert-level quality assessments on our benchmark. With its open-source release, LongCat-Video-Avatar 1.5 narrows the gap between academic research prototypes and commercial-grade deployment.

---

1. Problem Setting and Core Claim

LongCat-Video-Avatar 1.5 is presented as an open-source system for audio-driven avatar video generation that is optimized for production readiness rather than architectural novelty. The paper frames the central problem as the gap between research-quality demos and commercial-grade systems: real deployment requires not only plausible facial motion, but also stable identity over long horizons, full-body temporal consistency, robust lip synchronization, support for difficult real-world scenes, and efficient inference cost.

The authors argue that these requirements cannot be met by model design alone. Instead, v1.5 is built around a system-level recipe that combines (i) large-scale data curation and filtering, (ii) a stronger audio encoder, (iii) a unified DiT-based diffusion architecture, (iv) per-frame reinforcement learning from human feedback via GRPO, and (v) few-step distillation to reach an efficient 8 NFE inference regime. The paper’s claimed improvements are broad: accurate lip motion, temporal stability, strict identity preservation, generalization to stylized domains such as anime and animals, and robust behavior in multi-person and object-interaction scenarios.

The paper’s stated contributions are consistent with this systems-first philosophy:

• a commercial-grade open-source audio-driven video framework with stronger lip synchronization, temporal stability, identity consistency, and broad open-domain generalization;
• a step-distilled inference pipeline that runs at 8 denoising steps while preserving favorable visual quality;
• a multi-stage training pipeline that includes supervised pretraining, GRPO-based preference optimization, and DMD-based acceleration;
• specialized data pipelines for multi-person conversations, silent motion, and emotion-aware generation.

2. Data Curation and Dataset Construction

A major part of the paper is devoted to data engineering. The authors emphasize that the capability gap in avatar generation is often caused by noisy, heterogeneous training data rather than by a lack of model capacity. Their solution is a multi-stage data pipeline that converts diverse raw videos into structured, filterable, and reusable training samples.

2.1 General-purpose pipeline

The general pipeline is designed to support the full range of capabilities needed for avatar generation: identity preservation, audio-driven mouth motion, facial expression modeling, upper-body and full-body motion, hand-object interaction, camera-motion control, and style generalization. To achieve this, the paper organizes raw videos by their functional role rather than by source. The listed source types include close-up face videos, interview videos, acted performance videos, interaction videos, music videos, and stylized animation content.

The paper is explicit that simply mixing these sources would be harmful because they differ in face scale, body composition, camera motion, audio quality, language distribution, and caption granularity. The goal is to transform heterogeneous raw material into a unified data representation with consistent annotations.

Offline annotation

The offline stage constructs reusable metadata over full videos or pre-cut clips. The annotation schema includes human presence, face geometry, body composition, visual quality, audio availability, lip synchronization, speech and language, camera motion, motion speed, and semantic captions. The purpose is to make later sample selection explicit and controllable.

Important offline annotations include:

• Human-centric structure: face location, landmarks, detection confidence, person count, visible body region, and body composition.
• Audio and lip-sync: raw audio extraction, vocal-track separation, and synchronization estimation for talking videos.
• Visual quality: perceptual quality, text coverage, borders, black borders, brightness anomalies, compression artifacts, and corrupted frames.
• Camera and motion: camera type, camera motion, and motion speed.
• Semantic captions: multilingual captions at different granularities, including detailed, summary, and temporal-span captions.

The temporal-span captioning is important because long videos or acted scenes may contain local windows that do not match a single global description. The paper therefore explicitly pairs sampled clips with local text descriptions to reduce text-video mismatch.

Online clip-level validation and condition construction

The online stage validates the actual sampled training clip. Even if a full video is globally acceptable, a local window may still contain transitions, black frames, white flashes, sudden frame jumps, under/over-exposure, residual borders, or abnormal motion. The paper therefore performs clip-level filtering on duration, frame rate, resolution, brightness, black/white pixel ratio, border artifacts, frame jumps, and motion intensity.

The same structured annotations are also converted into text conditions. In particular, camera motion, shot size, lens type, and visual style can be appended to the original caption, making controllable factors explicit in the conditioning signal.

2.2 Specialized pipelines for under-covered scenarios

The paper identifies three settings that remain difficult for existing avatar systems: multi-person interaction, silent motion, and emotional expression. It therefore adds specialized curation pipelines on top of the general framework.

Multi-person data

Multi-person curation uses ByteTrack-based person tracking to extract person-level trajectories and separate dynamic people from static human-like artifacts. For videos with multiple people, the system applies an active speaker detection pipeline based on TalkNet and UniTalk, with YOLOv6 used as the detection backbone in the implementation described by the authors. The result is a set of track-level speaker activity labels that associate face trajectories with speaking intervals.

These labels are used to discard intervals with concurrent speaking and to retain non-overlapping single-speaker segments, thereby reducing speaker ambiguity during training.

Silent data

The silent pipeline collects non-speaking human clips so that the model can learn to preserve natural stillness when there is no speech. This is important because an audio-driven model should not hallucinate unwanted mouth motion in silent intervals; it also needs to model gaze, head pose, posture, gestures, and object interactions without speech supervision.

The silent pipeline decomposes long videos into short clips and evaluates each clip independently. A conservative two-stage multimodal verification strategy is used: Qwen3-Omni provides an initial silent/non-silent decision, and Qwen3-VL independently re-checks the same clip. A clip is retained only if both models agree that the subject is not speaking. After clip-level decisions are aggregated, the full video is kept only if all sampled clips are consistently silent.

Emotion data

The emotion pipeline is designed to capture temporal emotional dynamics rather than static facial expressions. The paper defines a six-class taxonomy:

1. high-arousal emotional expression,
2. context- or plot-driven reaction,
3. spontaneous expressive speech with low arousal,
4. non-verbal dominant emotion,
5. emotion regulation or suppression, and
6. temporal emotion dynamics.

Qwen3-Omni is used for initial labeling, but the authors impose hard exclusion rules: synthetic content, more than two subjects, identity switches, or subjects that occupy a very small part of the frame are assigned a null label. The valid clips are then refined using EmotiEffLib frame-level recognition. For each emotion class, the system averages the confidence of the top-$N$ frames, where $N = 10$ or $20$, and retains a clip only if the dominant emotion confidence exceeds $0.7$. Neutral predictions are corrected by also recording the second-best emotion class, which is used to recover weaker expressive signals.

The retained emotion clips are re-captioned using a context-aware annotation scheme that describes spatial environment, interpersonal relationships, and plot progression. The captions are intentionally objective and focus on observable signals such as forehead wrinkles, eyebrow position, gaze, blinking, head displacement, head tilt, posture shifts, shoulder shrugging, and hand gestures.

3. Method

3.1 Unified DiT-based architecture

LongCat-Video-Avatar 1.5 inherits the unified DiT-based video diffusion architecture from v1.0. The backbone is built on a 3D VAE and a Diffusion Transformer with 3D self-attention, text cross-attention, and a feed-forward network in each block. Text embeddings are encoded with a UMT5 encoder, while 3D Rotary Position Embeddings are used to represent spatiotemporal position in the visual tokens.

The architecture supports multiple task configurations through different latent inputs:

• Text-to-video: only noise latents are provided.
• Text-image-to-video: a reference latent is temporally concatenated with the noise latents.
• Video continuation: context latents are concatenated with the noise latents as additional conditioning.

To add audio-driven generation, the authors insert an audio cross-attention layer after text cross-attention in each DiT block. They keep adaptive layer normalization before the audio cross-attention as a gating mechanism, which helps avoid destabilizing the pre-trained visual priors while introducing audio control more gradually.

3.2 Audio feature extraction with Whisper Large

A central change in v1.5 is the replacement of the Wav2Vec2 audio encoder used in v1.0 with Whisper Large. The paper states that Whisper Large has about 1.5 billion parameters, is pre-trained on 680,000 hours of multilingual speech data, and works on Mel spectrograms rather than raw waveforms. The motivation is empirical: the authors report smoother and more natural lip dynamics than with Wav2Vec2, together with stronger multilingual robustness and richer acoustic representations.

Because Whisper has a 30-second context limit, the system uses a sliding-window strategy for long audio streams. The encoder yields 33 hidden states at an internal frame rate of 50 Hz. These are grouped into four blocks of eight layers plus one singleton layer, and each group is mean-pooled to form a 5-channel representation. The 50 Hz features are then linearly interpolated to the target video rate of 25 FPS, producing an audio embedding of shape $(T, 5, 1280)$, where $T$ is the number of video frames.

Since the VAE compresses videos with a temporal factor of $4\times$, the audio side is also temporally compressed with an audio projector that aggregates nearby context and aligns the audio sequence length with the latent video sequence. The authors present this alignment as a key ingredient for stable audio-to-mouth synchronization.

3.3 Multi-reward GRPO at the per-frame level

The reinforcement-learning stage extends the LongCat-Video GRPO formulation from video-level rewards to per-frame rewards. If $r_{k,j}^i$ is the reward of temporal partition $j$ of sample $i$ under reward model $R_k$, then the normalized relative advantage is defined as

$$ \hat A_{k,j}^i = \frac{r_{k,j}^i - \mu_{k,j}}{\sigma_{k,j}^{\max}}, $$

where $\mu_{k,j}$ is the group mean and $\sigma_{k,j}^{\max}$ is the maximum group standard deviation for reward $R_k$ at the same temporal partition. The final advantage is a weighted sum over reward models:

$$ \hat A_{\text{total},j}^i = \sum_k w_k \hat A_{k,j}^i. $$

Compared with video-level reward assignment, this per-frame formulation provides finer credit assignment and is intended to better penalize localized errors such as short-lived motion collapse, hand deformation, or local inconsistency.

The paper also adds two task-aware refinements: a first-frame hand-presence check for image-to-video and video-continuation tasks, and a multi-clip rollout strategy for long-horizon continuation. In the multi-clip setting, earlier clips are used as temporal context, but only the final clip participates in GRPO optimization.

3.4 Few-step generation via distillation

To reduce inference cost, the authors adapt Distribution Matching Distillation 2. Their goal is to compress the multi-step teacher into an efficient generator that preserves the teacher’s distribution. The implementation is engineered to avoid the VRAM burden of maintaining three separate models at once. Instead, they use a single base DiT backbone with multiple LoRA adapters and dynamically mount either a Generator LoRA or a Fake Score LoRA, while the base DiT serves as the real-score model.

The distillation target is an 8-step sampler. The authors keep the scheduler from earlier stages in both the real and fake score functions so that the noise scale remains consistent. They also reduce the classifier-free guidance scales for both text and audio to $4.0$ to mitigate oversaturation during distillation. The paper’s main trade-off is explicit: the accelerated model is faster and often more stable, but the base model retains stronger expressive richness and slightly better human-likeness on some metrics.

3.5 Multi-person conversation

For two-person conversational generation, the paper follows the MultiTalk strategy and uses L-RoPE to associate each speaker region with the correct audio condition. Reference attention maps are used to establish region-level correspondence between characters and speech. A practical issue arises when extra people appear in the reference image: visually similar background characters can be incorrectly absorbed into the target regions and driven by the target speech signal.

To reduce this attribution ambiguity, the authors add bounding-box annotations and treat non-target characters as separate categories during attention-map estimation. When additional person boxes are available, they also assign an explicit silent audio track to non-target regions. This prevents background characters from inheriting the mouth motion of the target speakers and is illustrated by the silent-background condition in the figure above.

4. Training Recipe

The training pipeline has three progressive stages: base model training, RLHF training, and acceleration training. The overall goal is to start with a stable generative foundation, then align outputs with human preferences, and finally distill the model for efficient serving.

4.1 Base model training

The generative process uses flow matching. Given a clean video latent $x_0$, Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$, and a timestep $t \in [0,1]$, the noisy latent is defined by linear interpolation:

$$ x_t = (1 - t) \cdot x_0 + t \cdot \epsilon. $$

The network predicts the velocity $v_{\text{pred}}(x_t, c, t; \theta)$ conditioned on task inputs $c$ such as text, audio, and conditional image/video latents. The objective is the mean squared error against the ground-truth velocity $v_t = x_0 - \epsilon$:

$$ \mathcal{L} = \mathbb{E}_{\epsilon, x_0, c, t} \left\| v_{\text{pred}}(x_t, c, t; \theta) - v_t \right\|^2. $$

The base training is progressive. It begins with low-resolution pretraining to learn coarse audio-to-face correspondence, then moves to higher-resolution training for detailed synthesis, then introduces a reference-image module for identity preservation, and finally extends to multi-person dialogue data.

The training stages reflect a clear curriculum: first learn speech-driven generation, then improve spatial fidelity, then add identity conditioning, then mix resolutions, and finally specialize for multi-person conversations.

4.2 RLHF training

After supervised training, the model is further optimized with GRPO using multiple quality-related reward signals. The paper reports a maximum rollout length of 5 clips for the multi-clip extension, with the actual rollout length randomly sampled during training. Only the final clip contributes to GRPO optimization; earlier clips are used as context. For image-to-video and video-continuation tasks, MediaPipe hand detection is used to keep hand-relevant examples in the preference-learning mix.

4.3 Acceleration training

The distillation stage compresses the model to 8 denoising steps. The authors report that extending this stage too long degrades visual realism, so they stop after 400 training steps, which they describe as the best point before quality decline. The generator learning rate is set to $2 \times 10^{-5}$, the fake-score learning rate to $4 \times 10^{-6}$, and the update ratio between the generator and fake scorer is $1:5$.

5. Evaluation and Reported Results

The benchmark is built on EvalTalker and contains more than 400 samples; the authors add over 50 stylized images such as cartoons and animals, resulting in a total of 508 image-audio pairs. The evaluation set covers multiple application scenarios, including news broadcasting, education, entertainment, and commercial use, in both Chinese and English, and in both realistic and animated visual styles.

Difficulty is varied along audio dimensions such as speaking speed, fluency, emotion, and paralinguistics, and along visual dimensions such as person count, pose, background complexity, and occlusion. The paper evaluates outputs using four expert-level perceptual dimensions:

• Rationality: whether movement and interactions obey physical and biomechanical plausibility.
• Harmony: audio-visual synchronization and visual naturalness of face and body motion.
• Stability: temporal consistency, including jump cuts, blur, tone drift, and resolution shifts.
• Consistency: identity preservation across time.

The paper evaluates against LC-Video-Avatar 1.0, InfiniteTalk, OmniHuman-1.5, HeyGen, Hedra, Kling Avatar 2.0, and OmniAvatar. In the reported evaluation protocol, LC-Video-Avatar 1.5 refers to the accelerated 8 NFE model.

The study combines three tracks: 770 crowdsourced evaluators produced 13,240 anthropomorphism judgments, 10 domain experts performed structured four-way analysis, and a pairwise A/B test compared LC-Video-Avatar 1.5 against three commercial systems.

The human-likeness ranking is summarized qualitatively as follows: in the single-person setting, LC-Video-Avatar 1.5, LC-Video- Avatar 1.0, and InfiniteTalk form the leading tier, followed closely by HeyGen and OmniHuman-1.5. In the multi-person setting, the two LC variants remain the strongest among methods that support multi-person synthesis, clearly ahead of InfiniteTalk. Overall, the authors argue that the remaining perceptual gap is driven mainly by physical rationality failures and imperfect audio-visual synchronization.

5.1 Expert-level quality analysis

In the stability analysis, the authors highlight that OmniHuman 1.5 suffers from substantial tone-error accumulation, while the reference skip-attention mechanism inherited from v1.0 remains effective in suppressing error propagation. The accelerated v1.5 model has slightly worse tone-error issue rate than v1.0, which the paper explicitly frames as an intentional trade-off for much faster inference. In contrast, the accelerated model achieves the lowest frame-jump issue rate among compared methods, and the temporal profile visualizations are presented as evidence of improved camera stability and reduced frame skipping relative to v1.0.

In the harmony analysis, the main focus is lip synchronization, expression-motion alignment, and face-body synchronization. The paper reports that the switch from Wav2Vec2 to Whisper Large reduces issue rates for these metrics relative to v1.0. Qualitative examples in talking-head, music, anime, performance, and emotional-expression scenarios are used to support the claim that v1.5 better aligns mouth motion with audio and handles expressive content more naturally.

For rationality, the paper attributes the improvement largely to GRPO, which penalizes unnatural or physically implausible outputs, and also notes that DMD distillation contributes by reducing hand distortion and exaggerated facial expressions. For consistency, the authors report that v1.5 performs best among the compared systems in identity preservation.

The pairwise A/B test against commercial systems is reported as favoring LC-Video-Avatar 1.5 over HeyGen, OmniHuman-1.5, and Kling Avatar 2.0, with the strongest preference margin over Kling Avatar 2.0.

5.2 Base versus accelerated variant

The paper is unusually explicit about the trade-off introduced by acceleration. The base model uses a much longer inference path, while the accelerated version is the one evaluated as LC-Video-Avatar 1.5 in the main benchmark. The base model retains a slight advantage in single-person human-likeness and harmony, while the accelerated model is better on multi-person human-likeness and on most stability-related metrics.

Interpreting this table in the paper’s own framing: the accelerated model is substantially more stable and more rational, with a modest decline in harmony, while also keeping consistency slightly better. The base model is described as more expressive, with richer motion diversity and more nuanced facial behavior.

6. Limitations and Future Work

The paper is candid that audio-visual harmony remains an open problem across the field, even when models achieve strong stability and identity preservation. It also notes that current virtual human systems still leave room for improvement in physical plausibility and fine-grained synchronization between speech and motion.

A second limitation concerns long-horizon identity maintenance. The authors observe that many existing approaches rely too heavily on fixed reference frames, which can cause repetitive motion patterns and unnatural camera transitions constrained by the reference view. Their stated future direction is an unbounded, infinite-length generation framework that preserves identity without rigid dependence on static reference frames.

In short, the paper’s main technical message is that a commercially relevant avatar generator can be built through disciplined data curation, a stronger audio encoder, per-frame preference optimization, and aggressive distillation. The model’s strongest reported gains are in stability, rationality, and identity consistency, while the remaining headroom is concentrated in audio-visual harmony and long-horizon realism.