StableAvatar

StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation

StableAvatar is an end-to-end video diffusion transformer enabling infinite-length audio-driven avatar videos. It tackles long-video issues by aligning audio conditioning with diffusion dynamics, ensuring identity consistency and natural lip-sync across extended videos without post-processing.

• talking-head
• avatar
• audio-driven
• lip-sync
• face-animation

Authors: Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Chong Luo, Zuxuan Wu, Yu-Gang Jiang

Categories: cs.CV

Published 2025-08-11 · Updated 2025-08-11

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face65 downloads80 likes

Impact

• Semantic Scholar37 citations5 influential citations

Abstract

Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency. This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing. Conditioned on a reference image and audio, StableAvatar integrates tailored training and inference modules to enable infinite-length video generation. We observe that the main reason preventing existing models from generating long videos lies in their audio modeling. They typically rely on third-party off-the-shelf extractors to obtain audio embeddings, which are then directly injected into the diffusion model via cross-attention. Since current diffusion backbones lack any audio-related priors, this approach causes severe latent distribution error accumulation across video clips, leading the latent distribution of subsequent segments to drift away from the optimal distribution gradually. To address this, StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents error accumulation via time-step-aware modulation. During inference, we propose a novel Audio Native Guidance Mechanism to further enhance the audio synchronization by leveraging the diffusion's own evolving joint audio-latent prediction as a dynamic guidance signal. To enhance the smoothness of the infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy that fuses latent over time. Experiments on benchmarks show the effectiveness of StableAvatar both qualitatively and quantitatively.

---

1. Problem Setting and Core Idea

StableAvatar addresses a specific but important failure mode in audio-driven avatar video generation: existing diffusion-based systems can synthesize convincing short clips, but they tend to degrade sharply when asked to produce long-form or effectively unbounded videos. The paper argues that the dominant failure source is not only temporal length itself, but audio modeling. Prior systems typically extract audio features with an off-the-shelf encoder such as Wav2Vec and inject those features directly into a diffusion backbone through cross-attention. Because the backbone has no strong audio prior, this creates latent distribution mismatch that accumulates across clips, causing the latent trajectory to drift away from the target distribution over time. The resulting symptoms are body distortions, face deformation, color drift, and poor lip synchronization once videos exceed roughly 15 seconds.

The proposed system is an end-to-end video diffusion transformer built on Wan2.1 that aims to generate infinite-length audio-driven avatar videos without post-processing. The key design principle is to make audio conditioning time-step aware and to use the diffusion model’s own evolving joint audio-latent prediction as a form of inference-time guidance. In addition, the system introduces a long-video denoising strategy that fuses overlapping latent windows with smooth, logarithmic weights.

2. Overall Architecture

StableAvatar follows a DiT-style conditional diffusion pipeline. The model takes a reference image, audio, and a text prompt as inputs. The backbone is a pre-trained Wan2.1-I2V 1.3B model; the audio encoder is trained from scratch. The reference image is incorporated through two pathways:

• It is concatenated with zero-filled frames along the temporal axis and encoded by a frozen 3D VAE. The resulting latent code is then concatenated with compressed video frames and binary masks, where the first frame is marked 1 and subsequent frames are 0.
• It is also encoded by a CLIP image encoder, and the resulting image embeddings are injected into each image-audio cross-attention block of the denoising transformer to preserve appearance and identity.

During inference, the original input video frames are replaced with random noise while the remaining conditions stay unchanged. This setup lets the model generate long videos autoregressively in latent space while keeping the reference identity and the conditioning signals fixed.

3. Why Long-Form Avatar Generation Fails in Prior Work

The paper’s central diagnosis is that latent distribution error accumulation is caused by the way audio is handled. If audio embeddings are simply injected into a diffusion transformer that has no audio-specific prior, each clip introduces a small mismatch between the audio-conditioned latent and the backbone’s natural latent manifold. When clips are chained for longer generation, these mismatches accumulate. The result is a gradual departure from the optimal latent distribution and a measurable increase in distortion and synchronization error.

This problem is distinct from the usual “long video” issue in text-to-video generation. In audio-driven avatar animation, the model must simultaneously maintain identity, facial motion, body consistency, and audio synchronization, and the audio signal has to remain semantically aligned with the latent state across many denoising steps and many video clips. The authors therefore argue that prior long-video tricks such as motion frames or conventional sliding windows are helpful for smoothing but do not solve the underlying distribution drift.

4. Timestep-aware Audio Adapter

The main training-time contribution is the Timestep-aware Audio Adapter. The adapter refines Wav2Vec audio embeddings so that they interact with the current diffusion timestep and with the current latent state before being injected into the backbone. The paper’s key intuition is that timestep embeddings and latent distributions are tightly coupled in diffusion pretraining: each timestep corresponds to a different latent regime. By conditioning audio on timestep information, the adapter encourages the network to model a joint audio-latent distribution instead of treating audio as an external feature stream.

In the notation of the paper, raw audio features $ \mathbf{a} $ are first contextualized by concatenating neighboring audio embeddings:

$$ \mathbf{e}_{\text{aud}}^{i} = \operatorname{Concat}(\mathbf{a}^{i-k}, \ldots, \mathbf{a}^{i}, \ldots, \mathbf{a}^{i+k}) $$

where $2k+1$ is the context length. The adapter then performs a sequence of normalization, MLP projection, affine modulation with timestep embeddings, and cross-attention with the current latent $ \mathbf{z}_t $. The refined audio embedding is then injected into the DiT via cross-attention together with the image embedding.

Conceptually, the adapter has three roles:

• Contextualization: nearby audio frames are aggregated so that the current frame can use preceding and following phonetic cues.
• Timestep-aware modulation: audio embeddings are modulated using the diffusion timestep embedding, aligning audio conditioning with the latent scale and distribution at that denoising step.
• Latent-aware refinement: cross-attention with the current latent explicitly exposes the audio branch to the state of the generative process.

The adapter is the paper’s primary mechanism for reducing clip-to-clip drift. Unlike simply improving audio features, it tries to make audio conditioning compatible with diffusion dynamics.

5. Audio Native Guidance

For inference, StableAvatar replaces conventional classifier-free guidance with Audio Native Guidance. The guiding idea is that the refined audio embedding used by the adapter is itself dependent on the latent state. Therefore, the guidance signal should not treat audio as an independent external condition; instead, it should guide the sampler toward the joint distribution of latent and audio representations.

The paper writes the modified sampling objective as a combination of three denoising evaluations, with guidance scales $\alpha$ and $\beta$:

$$ (1+\alpha+\beta)\,D([\mathbf{z}_t, \bar{\mathbf{a}}_t], \mathbf{y}, \mathbf{I}, \mathbf{A}; \theta) - \alpha\,D([\mathbf{z}_t, \bar{\mathbf{a}}_t], \mathbf{y}, \mathbf{I}, \emptyset; \theta) - \beta\,D([\mathbf{z}_t, \emptyset], \mathbf{y}, \mathbf{I}, \mathbf{A}; \theta) $$

In prose, this means the sampler uses the model’s own joint audio-latent prediction as a dynamic guidance signal, instead of only extrapolating from unconditional and conditionally guided predictions as in CFG. The paper also notes that including the reference image and text prompt inside the guidance computation substantially increases GPU cost without materially improving visual quality, so they are not used as guidance factors.

The authors present this as a way to keep the denoising trajectory anchored to the audio-latent manifold throughout sampling, which is especially important in long or infinite-length generation where small errors can snowball.

6. Dynamic Weighted Sliding-window Strategy

To improve temporal continuity across arbitrarily long videos, StableAvatar adds a Dynamic Weighted Sliding-window Strategy during inference. The latent sequence is denoised in windows of length $l$ with overlap length $m$. Adjacent windows share overlapping latent regions, and the overlap is fused with a logarithmic weighting schedule based on relative frame position.

The implementation described in the paper proceeds over denoising steps $t = T, \ldots, 1$. For each window, the model denoises the current segment, then blends the overlap with the previous segment using weights derived from a logarithmically transformed linear ramp. This means early overlap positions change more strongly than later ones, producing a progressive smoothing effect that reduces visible seams between clips.

The paper contrasts this with conventional sliding-window denoising. The main advantage of the new scheme is not simply overlap, but the specific dynamic weighting of the overlap. The fused latents are injected back into the neighboring windows, so both edges of the central region are already blended when they are revisited.

7. Training Objective and Data

StableAvatar is trained with a reconstruction objective on diffusion latents, and only a subset of the full model is trainable: the attention modules of the DiT and the Audio Adapter. The training loss is piecewise and uses face and lip masks extracted by Mediapipe to emphasize the most important regions for avatar animation:

$$ \mathcal{L} = \begin{cases} \mathbb{E}\big[\| (\mathbf{z}_{gt} - \mathbf{z}_{\theta}) \odot \mathbf{M}_{face} \|^2\big], & 0.4 \le q < 0.5 \\ \mathbb{E}\big[\| (\mathbf{z}_{gt} - \mathbf{z}_{\theta}) \odot \mathbf{M}_{lip} \|^2\big], & q \ge 0.5 \\ \mathbb{E}\big[\| (\mathbf{z}_{gt} - \mathbf{z}_{\theta}) \odot (1 + \mathbf{M}_{face} + \mathbf{M}_{lip}) \|^2\big], & \text{otherwise} \end{cases} $$

Here, $q$ is uniformly sampled from $[0,1]$. This piecewise loss explicitly separates supervision for facial region quality and lip synchronization, rather than applying a uniform reconstruction target over all pixels or latents.

The training dataset consists of three parts: Hallo3, Celebv-HQ, and videos collected from the internet, including BiliBili, YouTube, and TikTok. The authors filter the collected data using SyncNet and Q-Align for lip-sync and video fidelity, and use InsightFace to discard videos with facial confidence below $0.9$. The final training corpus contains approximately 1200 hours of video.

Optimization details reported in the paper are: 20 epochs, 64 NVIDIA A100 80G GPUs, batch size 1 per GPU, learning rate $10^{-5}$, guidance scales $\alpha = 4.5$ and $\beta = 3.0$. The Audio Encoder is trained from scratch, while the DiT backbone starts from pre-trained Wan2.1-I2V 1.3B weights.

8. Evaluation Protocol and Benchmarks

The main evaluation uses HDTF and AVSpeech, following prior work. Since open-source test splits are unavailable in earlier papers, the authors randomly select 100 videos from each dataset, with lengths between 5 and 20 seconds, and report the average duration as about 10 seconds. To test long-form behavior, they also construct Long100, a new benchmark of 100 unseen internet videos that are 2 to 5 minutes long. Long100 is presented as significantly more challenging than HDTF or AVSpeech because it contains more complex rhythms, varied human appearances, full-body and half-body shots, and more object interactions.

The paper evaluates with the following metrics:

• FID and FVD for image and video quality.
• CSIM for cosine similarity of facial embeddings and identity consistency.
• Sync-C and Sync-D for lip synchronization.
• IQA and ASE from Q-Align for quality and aesthetics.
• CIEDE in the error-accumulation study to measure color drift.

9. Main Quantitative Results

On the main benchmark table, StableAvatar is compared against GAN-based and diffusion-based baselines including SadTalker, AniPortrait, Sonic, EchoMimic, Hallo3, FantasyTalking, HunyuanAvatar, MultiTalk, and OmniAvatar. The strongest result is on Long100, where StableAvatar substantially improves identity and synchronization metrics while reducing distortion and drift. The authors emphasize that all competitors are re-trained on the same training dataset before Long100 evaluation to make the comparison fair.

The three values in each row correspond to HDTF / AVSpeech / Long100. The most striking result is on Long100: StableAvatar reports FID $57.18$, FVD $504$, CSIM $0.849$, Sync-C $8.24$, Sync-D $6.79$, IQA $3.84$, and ASE $2.39$. Compared with the previous strongest Wan2.1-14B-based OmniAvatar, the paper highlights an especially large improvement in long-video identity and synchronization, and states that StableAvatar outperforms the competitor by 80.3% in CSIM and 85.2% in Sync-C on Long100.

The qualitative discussion matches the table: earlier methods can handle the first few seconds reasonably well, but beyond about 15 seconds they begin to show lip-sync failures, body/face distortion, and color drift. StableAvatar is reported to preserve identity and produce coherent motion even after 3500+ frames in a single pass.

The paper also includes a quality-drift plot showing that StableAvatar’s degradation remains negligible as frame count grows:

The authors state that, in theory, the system can synthesize hours of video without significant quality degradation, although this is presented as an extrapolation from the observed drift curve rather than a measured hours-long benchmark.

10. Ablation Studies and Mechanistic Evidence

The core ablation table shows that removing any of the three main components sharply hurts long-video performance. In particular, removing the Audio Adapter causes the largest degradation, with FVD jumping to 1802 and CSIM dropping to 0.457. Removing guidance or DWSW is less catastrophic than removing the adapter, but both still reduce fidelity and synchronization.

The paper then drills down into audio modeling.

This ablation supports the paper’s specific design claims: timestep-aware modulation matters, random modulation is not enough, and cross-attention between audio embeddings and latents is also necessary. The authors interpret these results as evidence that the adapter is not just a stronger audio encoder, but a mechanism for aligning audio with the diffusion latent trajectory.

The error-accumulation study compares early frames with late frames. The paper defines $A$ as frames 1--200 and $B$ as frames 3500--3700, and uses CIEDE to quantify color drift.

The error study is one of the strongest pieces of evidence in the paper. The baseline performs much worse in the later segment $B$, which is consistent with the claim that audio injection produces clip-level latent drift. Adding the Audio Adapter stabilizes the late frames, and adding guidance on top of that further improves quality and synchronization. The reduction in CIEDE from 2.318 to 0.523 between Baseline (B) and Adapter + G (B) is presented as evidence that the proposed audio-related components materially suppress color drift over long generation horizons.

Audio guidance is also studied against CFG:

The paper reports that CFG helps only modestly, while Audio Native Guidance gives a much larger improvement. This supports the claim that using the model’s internal joint audio-latent prediction is more effective than treating audio as an external condition independent of the latent state.

The authors conclude that prior long-video strategies mainly improve smoothness but do not remove clip-boundary jitter as effectively as the proposed logarithmically weighted fusion. In the supplementary ablation on weight assignment, the logarithmic schedule outperforms fixed weights and uniform arithmetic gaps:

11. Efficiency, Generalization, and Additional Applications

The supplementary results make an important practical point: despite using a much smaller backbone than the 14B-model family, StableAvatar is reported to be substantially more efficient. For generating 81 frames at $480 \times 832$ resolution, the paper reports 18.4G GPU memory and 2.32 minutes latency for StableAvatar, compared with 40.83G and 20.31 minutes for OmniAvatar.

The paper uses this table to argue that StableAvatar is not only stronger in quality, but also more practical in resource-constrained settings. It also notes that the model can animate full-body and half-body characters, multiple protagonists, and even cartoon avatars, suggesting that the method is not limited to close-up talking heads.

In the user study, participants were shown the reference image and audio, then asked to compare paired outputs on lip alignment, appearance alignment, background alignment, and identity alignment. StableAvatar is preferred by large margins over all compared methods. The reported preference rates are especially high on identity and appearance alignment.

12. Limitations and Ethical Considerations

The paper reports one clear failure case: when the reference subject is a non-human or fantastical creature, the model struggles to locate and animate the lips because the anatomy differs too much from common human patterns. The proposed future direction is to add an extra reference network that explicitly captures semantic details from the input image.

The ethical note is straightforward: because the system can animate a given reference image from audio, it has legitimate applications in virtual reality and digital-human creation, but it also raises the risk of misleading or deceptive content generation. The authors recommend sensitive-content detection as one mitigation strategy.

13. Takeaway

StableAvatar’s main contribution is to show that long-form, identity-preserving audio-driven avatar generation is not just a matter of better post-processing or smoother window stitching. Instead, the paper argues that the central obstacle is how audio enters the diffusion model. By making audio conditioning timestep-aware, guiding sampling with the model’s own joint audio-latent prediction, and blending overlapping latent windows with dynamic logarithmic weights, StableAvatar achieves the paper’s headline goal: end-to-end generation of infinite-length avatar videos with substantially improved identity consistency, synchronization, and visual stability.