InfiniteTalk

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

InfiniteTalk enables sparse-frame video dubbing by preserving keyframes for identity and gestures while allowing audio-driven full-body and facial motion editing. This balances appearance stability with flexible motion for natural, synchronized long-video dubbing beyond traditional mouth-only approaches.

• dubbing
• talking-head
• audio-driven
• full-body
• face-animation
• streaming

Authors: Shaoshu Yang, Zhe Kong, Feng Gao, Meng Cheng, Xiangyu Liu, Yong Zhang, Zhuoliang Kang, Wenhan Luo, Xunliang Cai, Ran He, Xiaoming Wei

Categories: cs.CV

Comment: 11 pages, 7 figures

Published 2025-08-19 · Updated 2025-08-19

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub7,229 stars61 watchers1,274 forksApache-2.0
• Hugging Face17,265 downloads237 likes

Impact

• Semantic Scholar32 citations4 influential citations

Abstract

Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.

---

Introduction and problem setting

InfiniteTalk addresses audio-driven video dubbing, but with a broader editing target than conventional talking-head systems. The paper argues that mainstream dubbing methods are too narrowly focused on mouth-region synthesis: they can improve lip synchronization, yet they often leave head pose, facial expression, torso motion, and camera behavior essentially unchanged. That produces a mismatch between speech prosody and the rest of the performer, which is especially noticeable in emotionally expressive or full-body footage.

The central idea is a new task formulation called sparse-frame video dubbing. Instead of conditioning on a single image or densely constraining every frame, the method preserves only a small set of reference keyframes from the source video. Those keyframes are intended to preserve identity, iconic gestures, emotional cadence, and camera trajectory, while still allowing the generator to alter lips, facial expressions, head turns, and body dynamics so they can follow the dubbed audio naturally. The paper emphasizes that this is not just a stronger talking-head model; it is a video-to-video dubbing paradigm for long-form, full-body audiovisual alignment.

The teaser figure summarizes the motivation: compared with mouth-only dubbing, sparse-frame dubbing keeps only sparse anchor frames and gives the generator freedom to produce synchronized mouth, facial, and body motion. This framing is important because the paper’s claimed improvement is not limited to lip-sync quality; it is about holistic audiovisual plausibility over long sequences.

Why naive image-to-video dubbing fails

The paper spends a substantial part of the method section on analyzing why a straightforward reuse of audio-conditioned image-to-video models does not solve sparse-frame dubbing. Two baseline regimes are examined: initial-frame conditioning (an I2V-style setup) and initial-plus-terminal-frame conditioning (an FL2V-style setup). The failure modes are different and reveal the core design constraints of the task.

In the I2V case, the system is initialized from a single source frame and then rolled forward autoregressively across chunks. The paper reports that this causes error accumulation: identity drifts, background colors shift, and visual details degrade as the generated sequence gets farther away from the first frame. In other words, the model can move, but it cannot reliably remember the source appearance over long horizons.

In the FL2V case, each chunk is conditioned on both its start and end frames. This reduces drift, but the conditioning becomes too rigid. When the boundary frames are similar, the generator collapses into minimal-motion interpolation: facial expression, head motion, and body gestures become stiff and under-responsive to the audio. The paper highlights that this is a particularly bad fit for dubbing, where speech energy and emotional tone often require motion that deviates from the reference pose.

Both setups also create inter-chunk discontinuities. Because they only inject static image conditions, they do not explicitly carry motion momentum from the previous segment into the next. The result is abrupt temporal jumps at chunk boundaries even when each chunk individually looks plausible.

This analysis motivates the rest of the paper: a useful dubbing model must simultaneously support long-range identity preservation, soft control over motion magnitude, and streaming continuation across chunks.

InfiniteTalk method overview

InfiniteTalk is built as a streaming audio-driven generator on top of a conditional flow-matching video model. The architecture takes three conditioning sources: an audio embedding, a reference frame embedding, and context frames from the preceding chunk. The context frames provide motion momentum for seamless continuation, while the reference frame anchors identity, background, and camera motion.

The implementation uses an audio embedder, a video VAE, and a diffusion transformer. The paper states that the model is based on MeiGen-MultiTalk, a 14B-parameter DiT that supports audio-driven image-to-video generation at multiple resolutions. The audio encoder is wav2vec2, and the reference image encoder is CLIP/H. The key architectural modification is that each transformer block includes both audio cross-attention and reference cross-attention, which allows the denoiser to adapt jointly to speech and reference appearance.

Conditional flow matching formulation

The generator is trained with conditional flow matching. Let the latent video be $\mathbf{x}$, and let the conditioning set be $\mathbf{c} = \{y, a, \mathbf{x}_{\text{ref}}, \mathbf{x}_{\text{tran}}\}$, where $y$ is the text prompt embedding, $a$ is the audio embedding, $\mathbf{x}_{\text{ref}}$ is the reference-frame latent, and $\mathbf{x}_{\text{tran}}$ denotes transition/context frames. The paper defines the interpolated path

$$q_t(\mathbf{x} \mid y, a) = (1-t)\,p(\mathbf{x} \mid y, a) + t\,q(\mathbf{x} \mid y, a),$$

and samples a noisy latent as

$$\mathbf{x}_t = (1-t)\,\mathbf{x}_1 + t\,\mathbf{x}_0, \qquad \mathbf{x}_1 \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).$$

The DiT learns a velocity field $\mathbf{v}_\theta(\mathbf{x}_t \mid \mathbf{c})$ that approximates $\mathbf{x}_1 - \mathbf{x}_0$ under the conditional flow-matching objective:

$$\mathcal{L}_{\mathrm{fm}} = \mathbb{E}_{t,\mathbf{x}_0,\mathbf{x}_1,\mathbf{c}}\,\left\|\mathbf{v}_\theta(\mathbf{x}_t \mid \mathbf{c}) - (\mathbf{x}_1 - \mathbf{x}_0)\right\|_2^2.$$

At inference time, an ODE solver is used to sample from the learned flow. The paper’s formulation is important because it makes the generator naturally compatible with streaming chunk-by-chunk synthesis.

Streaming chunk construction

For training, the paper does not require paired dubbed-video supervision. It is sufficient to use videos with audio tracks. A source video is encoded into latents, a reference frame is sampled uniformly at random from the video, and the context frames are set to the first $4(t_c-1)+1$ frames of the source sequence. The paper’s implementation uses $9$ context images, corresponding to $t_c = 3$ context-frame latents. During autoregressive generation, the first chunk uses the first source frame as the reference frame and no context frames. For later chunks, the model uses the last $4(t_c-1)+1$ frames from the previous generated chunk as context, plus the first image of the current input chunk as the reference frame.

The latent tensor is assembled by concatenating the noisy video, the clean context frames, a padded reference frame, and a binary reference mask. In symbolic form, the paper describes:

$$\mathbf{z}_1 = \operatorname{concat}((\mathbf{x}_{\text{context}}, \mathbf{x}_t), 2), \quad \mathbf{z}_2 = \operatorname{concat}((\mathbf{x}_{\text{ref}}, \mathbf{0}), 2), \quad \mathbf{m} = \operatorname{concat}((\mathbf{1}, \mathbf{0}), 2), \quad \mathbf{z} = \operatorname{concat}((\mathbf{z}_1, \mathbf{z}_2, \mathbf{m}), 1).$$

Conceptually, this lets the transformer see both the current denoising state and the temporally adjacent clean frames, while also marking the spatial-temporal location of the reference image.

Soft conditioning and reference-frame sampling

A major insight of the paper is that the position of the reference frame during training controls the strength of conditioning. The model should not always be forced to reproduce the reference frame at the exact timestamp where it appears; instead, the reference should sometimes act as a soft stabilizer that preserves identity and camera continuity while leaving enough freedom for audio-driven motion.

The paper studies four training-time sampling strategies:

• M0: reference frames sampled uniformly at random from the current chunk. This produces excessive control strength and can cause the model to replicate reference content at the wrong time.
• M1: references restricted to the first or last frame of the chunk. This behaves like a rigid boundary-conditioned model and tends to suppress motion expressiveness.
• M2: references taken from temporally distant chunks, for example segments separated by more than 5 seconds. This weakens control too much and leads to accumulation errors in color and background consistency.
• M3: references taken from adjacent chunks, for example within about 1 second of the input chunk. This gives moderate control strength, preserving identity and camera motion without forcing exact reproduction.

The paper’s conclusion is that chunk-level distance is the dominant factor. Near-chunk references strike the best balance between stability and freedom: they anchor appearance enough to prevent drift, but they are not so tightly coupled that the generator collapses into motion copying. This is the core mechanism behind the paper’s claim of soft conditioning.

Camera control

The source reference frames provide a global constraint on camera trajectory, but the paper notes that subtle intra-chunk camera motion still needs extra help. To address this, the authors experiment with two external camera-control plug-ins: SDEdit and Uni3C. SDEdit injects camera information by adding the source video to the initialization noise at a scale $t_0$, so denoising begins from $t = t_0$ rather than from pure noise. Uni3C injects camera motion with a ControlNet-like architecture. The paper reports that Uni3C can better enforce camera motion but may weaken background preservation.

Training and implementation details

The implementation section provides several practical details that matter for reproducibility. Training uses roughly 2,000 hours of video containing talking people, and the model is trained on a cluster of 64 NVIDIA H100 80G GPUs. The context window contains 9 images, the video chunk length is 81 frames, and the model autoregressively generates 72 frames per chunk during long-video synthesis. The output resolution used for testing is 480×480.

The paper’s setting is important because it clarifies the scale of the system: InfiniteTalk is not a small post-processing network. It is a large streaming generative model trained on a substantial corpus, with a transformer backbone that is expected to preserve long-range structure while reacting to audio at each denoising step.

Evaluation protocol

The authors evaluate on three benchmarks: HDTF and CelebV-HQ for facial dynamics, and EMTD for full-body movement. To build a dubbing-style test set, they randomly sample 40 videos per dataset and replace the original audio with mismatched tracks, yielding a total of 120 videos. This allows the model to be tested under realistic dubbing conditions without requiring manually curated dubbed ground truth.

The automatic metrics are:

• FID for per-frame visual quality.
• FVD for temporal coherence.
• Sync-C and Sync-D from SyncNet for lip synchronization.
• CSIM for identity preservation.

The authors also run a human study because they argue that no existing automatic metric reliably captures full-body motion-to-audio alignment. The human evaluation collects 340 responses from 17 participants on 40 EMTD dubbing results, and asks participants to rank outputs on gesture synchronization, head motion alignment, lip synchronization, identity consistency, and overall naturalness.

Quantitative results

The paper compares InfiniteTalk against two categories of baselines: traditional dubbing methods such as MuseTalk and LatentSync, and audio-driven image-to-video generators such as FantacyTalking, Hallo3, OmniAvatar, and MultiTalk. The authors repeatedly stress that these categories are not directly comparable on all metrics: mouth-only dubbing systems often look excellent on FID/FVD because most of the frame is copied from the input, while image-to-video systems are asked to generate much more motion and therefore face a different quality-synchronization trade-off.

Comparison with traditional video dubbing methods

Against traditional dubbing methods, InfiniteTalk is strongest where the new task definition matters most: lip and full-body synchronization. On all three datasets, the model achieves higher Sync-C and lower Sync-D than the mouth-only baselines. This is consistent with the paper’s thesis that editing only the oral region is not enough for believable dubbing. The trade-off is that FID, FVD, and CSIM are often less favorable than methods that preserve almost the entire original frame, which the paper explicitly notes is not a fair proxy for dubbing quality.

Comparison with audio-driven image-to-video models

These results show a more nuanced picture. On the I2V-style comparison, InfiniteTalk is consistently among the strongest methods on synchronization metrics, and on EMTD it is best across all reported metrics. On HDTF and CelebV-HQ, the method often matches or slightly trails the best visual-quality metrics from strong baseline generators, but it maintains better audiovisual alignment. The paper’s interpretation is that sparse-frame dubbing intentionally shifts the optimization target away from static frame preservation and toward full-body audiovisual coherence.

Human evaluation for lip and body synchronization

The human study strongly favors InfiniteTalk on both lip and body synchronization. The paper highlights this as important evidence that current automatic metrics do not fully capture the benefits of full-body dubbing, especially when the head and torso motion are supposed to reinforce speech rather than remain frozen.

Ablation study

The ablation study focuses on the reference-frame positioning strategies introduced earlier. The reported values on EMTD are:

The trends are consistent with the paper’s hypothesis about conditioning strength. M1 produces the best FID and FVD among the four variants, but it also harms synchronization because the boundary conditioning becomes too strict. M2 is clearly the worst overall, confirming that excessively distant references fail to stabilize appearance and background. M3 gives the best balance: it achieves the strongest synchronization metrics while retaining competitive visual quality. M0 sits in between, but its random same-chunk sampling can over-constrain the model at the wrong timestamps.

The qualitative comparison supports the tables: traditional dubbing methods can look visually clean but remain body-static, whereas InfiniteTalk produces more plausible face, head, and body motion that follows the audio cadence. The paper also includes camera-control visualizations showing that InfiniteTalk alone preserves only coarse camera behavior, while SDEdit or Uni3C can add finer camera motion, each with its own trade-off.

Key takeaways and practical limitations

• Main conceptual contribution: sparse-frame video dubbing, which preserves only sparse keyframes instead of every frame or only the mouth region.
• Main modeling contribution: a streaming conditional flow-matching generator with audio cross-attention, reference cross-attention, and context-frame continuation.
• Main algorithmic insight: reference-frame placement during training determines control strength; near-chunk references are the best compromise between identity preservation and motion freedom.
• Main empirical claim: the method improves lip, head, and body synchronization, and human raters prefer it for both lip and body alignment.

The paper does not include a separate formal limitations section, but several practical constraints are visible from the text. First, the method is computationally heavy: the implementation uses a 14B DiT, a large-scale training corpus, and 64 H100 GPUs. Second, full-body motion evaluation still depends partly on human judgment because the authors state that there is no reliable automatic metric for full-body motion-audio alignment. Third, camera control is not fully solved inside the core generator; the paper relies on external plug-ins such as SDEdit or Uni3C for finer trajectory control, and those options introduce their own trade-offs, especially around background preservation. Finally, the reported metric trade-offs show that if one optimizes for static similarity alone, the result can look better under conventional frame-based measures while being less convincing as a dubbing system.

Overall, InfiniteTalk is best understood as a long-form, streaming, audio-driven video dubbing system that explicitly balances three competing goals: preserving source identity, enabling natural audio-driven motion, and maintaining continuity across chunks. Its novelty lies less in introducing a new backbone than in rethinking how conditioning should be structured for dubbing when the target is a whole human performer rather than only the mouth region.