Loopy

Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency

Loopy is an audio-only conditioned diffusion model for generating natural talking-head videos by leveraging long-term motion dependency. It removes spatial constraints and learns motion patterns from extended temporal context, improving natural facial expressions and head movements directly from audio.

• talking-head
• avatar
• audio-driven
• audio2face
• portrait-animation
• head-pose

Demos

These demos showcase Loopy's ability to generate vivid, natural portrait animations from audio alone, capturing lifelike long-term motion dependencies such as head movements, emotion-driven eye and eyebrow gestures, and non-speech motions like sighing. Watch for the model's high motion diversity in adapting the same reference image to different audio styles, including rapid speech and singing, and its superior quality compared to recent methods, all without requiring spatial motion templates during inference.

Authors: Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, Yanbo Zheng

Categories: cs.CV

Comment: ICLR 2025 (Oral), Homepage: https://loopyavatar.github.io/

Published 2024-09-04 · Updated 2025-04-04

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar110 citations11 influential citations

Abstract

With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

---

Introduction

Loopy addresses a recurring problem in end-to-end audio-driven portrait video generation: audio alone is a weak control signal for full head-and-face motion, so many recent diffusion-based systems add auxiliary spatial constraints such as face locators, speed layers, or movement templates to stabilize output. According to the paper, these extra controls reduce motion freedom and often bias the generator toward static or template-like behavior instead of learning natural motion directly from data.

The central observation behind Loopy is that two kinds of information are being underused in prior work. First, short motion histories typically used by existing systems are too brief to convey motion style; with only a few preceding frames, the model can easily extract appearance but not longer-term temporal regularities such as when blinks or subtle head turns should happen. Second, audio and portrait motion are only weakly correlated at the pixel level, so the generator can spend capacity modeling irrelevant background changes rather than the motion of the face itself.

Loopy is designed to remove the need for manually specified spatial motion templates at inference time. It is an audio-only conditioned portrait diffusion model that tries to learn natural motion patterns directly from longer temporal context, while also strengthening the audio-to-motion pathway through an intermediate motion-latent representation.

The paper positions Loopy against recent diffusion-based talking-head systems such as Hallo, EchoMimic, and VExpress, and argues that the key differentiator is not just improved conditioning, but the explicit modeling of long-term motion dependency.

Method

Loopy is built on top of Stable Diffusion in latent space. The paper keeps the standard latent-diffusion objective and updates the denoising network so it can operate on video sequences rather than a single image. Let $z_0$ be the latent encoding of a video frame and $z_t$ the noisy latent at diffusion step $t$. The training objective is the standard noise-prediction loss:

$$ \mathcal{L} = \mathbb{E}_{z_t, c, \epsilon \sim \mathcal{N}(0,1), t}\left[\lVert \epsilon - \epsilon_\theta(z_t, t, c) \rVert_2^2\right]. $$

In Loopy, the condition set $c$ is not text as in vanilla Stable Diffusion. Instead, it includes a reference image latent $c_{\mathrm{ref}}$, audio embeddings $c_{\mathrm{audio}}$, motion-frame latents $c_{\mathrm{mf}}$ from preceding clips, and during training additional motion-related supervision in the form of head-movement variance and expression variance.

The overall denoising system uses a dual U-Net design. One U-Net branch acts as a reference network and processes the reference image latent, while the denoising U-Net processes the noisy video latents. The reference network also processes motion-frame latents so their extracted features can be reused later in temporal attention. During spatial attention, the denoising U-Net incorporates reference features from the reference network, allowing appearance and identity cues to be transferred into the video generation pipeline.

Framework

The input to the denoising network is a sequence of noisy latents, not a single image latent. The model also consumes the reference latent, audio features, and motion-frame latents. The paper emphasizes that Loopy’s temporal design is intended to prevent the model from over-relying on appearance cues from the reference image or from short motion history alone.

In the spatial attention blocks, features from corresponding locations in the reference network are concatenated with the denoising features along the spatial dimension before attention is applied. This is the mechanism by which identity and appearance are injected into the video generator.

Inter- and Intra-Clip Temporal Modeling

The main temporal contribution of Loopy is the separation of temporal modeling into two distinct layers:

• Inter-clip temporal layer: handles relationships between the current clip and motion frames from preceding clips.
• Intra-clip temporal layer: models temporal dependencies inside the current clip itself.

This differs from prior end-to-end talking-head diffusion models, which typically feed motion-frame and noisy features into a single temporal layer. Loopy’s decomposition gives the model a clearer division of labor: one layer captures longer-range motion context, while the other focuses on short-range temporal coherence.

For the inter-clip path, motion-frame latent features extracted by the reference network are concatenated with noisy-latent features along the temporal dimension. Learnable temporal embeddings are added to distinguish token types, and self-attention is then computed over the concatenated temporal tokens. The intra-clip path excludes motion-frame features and only processes the current clip’s noisy latents.

To extend the receptive field further, Loopy introduces the temporal segment module (TSM). The paper’s motivation is that simply making the motion history longer can help capture motion style, but only if the model can process long histories efficiently and at multiple temporal scales. The TSM therefore partitions the preceding motion frames into segments, samples representative frames from each segment, and recombines them into a compact motion-frame latent sequence.

If the stride is $s$ and the expansion ratio is $r$, then the number of frames in the $i$-th segment is $s \cdot r^{i-1}$. With the default setting reported in the paper, $s=4$ and $r=2$, and the total number of segments is 5. This expands the temporal coverage to more than 100 frames, corresponding to about 5 seconds at 25 fps, which is roughly 30 times the original short motion window used by prior methods.

The paper gives the output of the temporal segment module as:

$$ c^{o}_{\mathrm{mf}, i} = c_{\mathrm{mf}, \left[ \sum_{j=0}^{k-1} r^j s + r^k (i \bmod s) \right]}, \qquad k = \left\lfloor \frac{i}{s} \right\rfloor. $$

Intuitively, nearby motion frames are preserved at higher detail, while farther history is compressed more aggressively. The authors report that this multi-scale temporal abstraction helps the model learn long-term motion style while keeping computation manageable.

Audio Condition and Audio-to-Latents

Audio features are extracted with wav2vec. Following EMO, the hidden states from all wav2vec layers are concatenated to obtain multi-scale audio features. For each video frame, Loopy concatenates the audio features from the two preceding and two succeeding frames, producing a 5-frame audio embedding for the current frame.

The first audio-conditioning path is standard cross-attention: noisy latents serve as queries, and audio embeddings provide keys and values. The resulting attended audio feature is added to the noisy latent features after the inter-clip temporal layer.

The second path is the paper’s more novel audio-to-latents module. Its purpose is to move beyond direct audio injection and instead project audio into a shared latent space with strongly motion-correlated supervision. During training, this module can be driven by one of three inputs with equal probability: audio embeddings, absolute head-movement variance, or expression variance. At test time, it receives only audio.

The motion-related supervision is derived from facial keypoints detected by DWPose. The paper defines head movement using the variance of the nose-tip position across the current 12 frames, and expression movement using the variance of the upper-face keypoints relative to the nose tip. These two signals are treated as strongly motion-correlated conditions, while audio is the weakly correlated signal that benefits from alignment to them.

Implementation-wise, the module maintains a set of learnable embeddings of size 128. Condition features are mapped through fully connected layers into query vectors, while the learnable embeddings act as key and value tokens. Attention is computed over the token dimension, with features unified to 256 channels for QKV computation. The resulting motion latents are then projected to 1280 channels and added to the timestep embedding. The paper’s core claim is that this helps weak audio signals exploit a latent representation learned from stronger motion supervision, which improves subtle facial expressions and reduces attention drift toward irrelevant regions.

Training and Inference Strategy

Loopy uses several condition dropout and masking strategies to prevent the model from collapsing onto the easiest cue. During training, the audio condition and motion latents are masked to zero with 10% probability. The reference image has a 15% dropout probability, and when the reference is dropped, the motion frames are also dropped from the spatial and temporal attention paths. Motion frames additionally have an independent 40% probability of being masked to all-zero features.

Training is performed in two stages, following prior work such as AnimateAnyone and EMO. In stage 1, the model is trained without temporal layers and without the audio-condition module, effectively learning an image-level pose-variation task from a single target frame and a reference image. In stage 2, the stage-1 weights initialize the full system, and the inter/intra-clip temporal modules plus the audio-conditioning module are added for end-to-end training.

Inference uses classifier-free guidance with three forward passes. The paper defines the final noise estimate as:

$$ e_{\mathrm{final}} = \text{audio\_ratio} \cdot (e_{\mathrm{audio}} - e_{\mathrm{ref}}) + \text{ref\_ratio} \cdot (e_{\mathrm{ref}} - e_{\mathrm{base}}) + e_{\mathrm{base}}, $$

where $e_{\mathrm{audio}}$ uses all conditions, $e_{\mathrm{ref}}$ drops the audio condition, and $e_{\mathrm{base}}$ further drops the reference image. The reported guidance weights are 5 for audio and 3 for the reference. The sampler is DDIM with 25 denoising steps. Ungenerated parts of the motion-frame sequence are masked to zero until they are produced.

Experimental Setup

The training set is assembled from multiple sources, including face-related datasets, general-purpose datasets, and online platforms. The authors filter out videos with low lip-sync scores, excessive head movement, and large rotations. This yields 174 hours of training data.

The test sets include 100 sampled videos each from CelebV-HQ, RAVDESS, and HDTF. To test open-set generalization, the authors also build an additional set of 20 portrait images and 20 audio clips covering real people, anime, side faces, humanoid crafts, speech, singing, rap, and emotionally rich speech.

Training details are specific and fairly heavy-weight: 24 Nvidia A100 GPUs, batch size 24, AdamW optimizer, learning rate $10^{-5}$, and two training stages of 4 days each. Videos are processed at 25 fps and cropped to $512 \times 512$. The generated clip length is 12 frames. The motion history is set to 124 preceding frames, which the temporal segment module compresses to 20 motion-frame latents. For training the audio-to-latents module, the paper uses DWPose-derived head-motion and expression statistics from the current 12 frames.

Evaluation uses a mixture of standard and task-specific metrics:

• IQA for image quality.
• VBench smooth for motion stability.
• Sync-C and Sync-D for audio-visual synchronization.
• FVD, FID, and E-FID for realism and distributional similarity.
• Glo, Exp, DGlo, and DExp for global motion and dynamic facial expression quality beyond mouth motion.

For the open-set set, the paper uses subjective voting by 10 users on six dimensions: identity consistency, video quality, audio-emotion matching, motion diversity, motion naturalness, and lip-sync accuracy.

Baselines include SadTalker, Hallo, Hallo* (Hallo trained on the authors’ data), VExpress, and EchoMimic. The paper also reports a Loopy model trained without collected data for reference in one HDTF comparison.

Quantitative Results

Across the reported datasets, Loopy consistently improves synthesis quality and temporal behavior relative to recent audio-driven portrait diffusion baselines. The strongest patterns in the results are: better overall video quality, better or competitive lip-sync, stronger motion realism, and notably improved stability under diverse input conditions.

CelebV-HQ

CelebV-HQ is the more realistic and challenging setting because it contains celebrities speaking in both indoor and outdoor scenes with diverse poses. The paper reports that Loopy performs best on the main quality and synchronization metrics and also improves the motion-specific measures tied to facial dynamics.

On CelebV-HQ, Loopy achieves the best reported IQA, Sync-C, Sync-D, FVD-R, FVD-I, and FID, and also the best dynamic-expression deviation $\mathrm{DExp}$. The paper notes that the global-motion deviation $\mathrm{DGlo}$ is competitive but not the very lowest among all methods, which reflects the fact that Loopy is trying to increase motion richness rather than remain as static as possible.

RAVDESS

RAVDESS is the emotional-expression benchmark. The paper uses it to test whether Loopy can generate more expressive talking portraits instead of merely lip-synced but rigid videos. The results show clear gains in video realism and the emotion-sensitive metric E-FID.

On RAVDESS, Loopy matches or exceeds the strongest baselines on almost every main metric, including the best FVD-R, FVD-I, FID, and E-FID. The authors also highlight that some methods can appear strong on SyncNet-style scores while still producing overly static motion; Loopy is intended to avoid that failure mode by directly modeling motion diversity.

Open-Set Generalization and Human Preference Study

The open-set set is intended to probe generalization rather than benchmark-style similarity to a familiar training distribution. It includes image styles such as real people, anime, side faces, and humanoid crafts, paired with speech, singing, rap, and emotionally rich audio. In the user study, Loopy is reported to consistently and significantly outperform the compared methods across these diverse conditions.

On the open-set and HDTF evaluations, Loopy again leads the table in the key quality and synchronization metrics. The HDTF row labeled "Loopy (HDTF)" shows the same model evaluated under a training regime without the collected data used for the full model’s reference setting; the full Loopy model still achieves the strongest results among the compared methods.

Ablations and Analysis

The ablation study isolates the two central design choices: the inter/intra-clip temporal structure and the audio-to-latents module. Removing the inter-clip temporal layer, removing the temporal segment module, or removing the audio-to-latents module all degrade the results. The paper states that the biggest drop comes from removing the dual temporal design, which supports the claim that long-term motion dependency is the primary source of the improved motion realism.

The table above also explores the long-horizon motion setting. Simply increasing the motion-frame length to 20 while keeping a single temporal layer improves dynamics but causes a substantial stability drop, especially visible in the open-set smoothness score. In contrast, when the model is allowed to use 20 motion frames together with the inter/intra-clip temporal split, stability is improved, indicating that long histories are useful only if the temporal architecture can separate short-range and long-range roles.

The authors further vary the temporal-segment hyperparameters. With $r=2$, smaller segment stride values tend to underperform, which the paper interprets as a sign that too few motion frames leave the model with too little temporal information and encourage appearance extraction instead of motion-style learning. Among the sampling strategies, uniform sampling is reported as the best default. Mean sampling and random sampling are both weaker, suggesting that a more structured abstraction of long-range motion is better for the inter-clip temporal layer than pooling-like or stochastic selection.

The ablations make two broader points. First, the dual temporal decomposition is not just a cosmetic architectural change; it materially improves the balance between motion richness and temporal stability. Second, the temporal-segment design is sensitive to how long motion histories are abstracted. The reported default behavior favors a structured, multi-scale sampling pattern over naive alternatives.

Qualitative Behavior

The qualitative analysis in the paper stresses three recurring strengths: identity preservation, motion amplitude, and image quality. Loopy is shown on diverse input styles, and the authors claim that it can produce more natural head motion and more detailed facial expressions than prior methods, while still maintaining stable portrait appearance.

Conclusion and Scope

The paper’s conclusion is straightforward: by combining inter/intra-clip temporal modeling with an audio-to-latents bridge, Loopy can generate talking-head portraits with more lifelike motion and stronger audio-motion alignment, without needing explicit motion templates or spatial constraints. The empirical evidence is strongest on metrics tied to realism, dynamics, and synchronization, and the open-set study suggests the gains transfer across a wider variety of portrait styles and audio types.

The paper does not present a dedicated limitations section. From the reported experimental setup, its scope is clearly centered on portrait/talking-head generation, with 12-frame outputs at 25 fps, $512 \times 512$ crops, and facial-motion supervision derived from keypoints during training. The paper does not report broader non-portrait generation, runtime analysis, or explicit failure cases in the main text.