SyncCache

SyncCache: Exploiting Asymmetric Dynamics for Fast Audio-Driven Portrait Animation

Training-free caching for audio-driven portrait animation that exploits asymmetric dynamics in diffusion transformers. By caching stable visual blocks while refreshing audio blocks, it achieves up to 4.12× speedup while preserving precise lip sync—addressing memory and fidelity failures of prior caching methods.

• talking-head
• audio-driven
• lip-sync
• portrait-animation
• low-latency
• realtime

Authors: Juncheng Ma, Yuxuan Du, Yanan Sun, Zhening Xing, Changlin Li, Zhenyu Tang, Bo Li, Peng-Tao Jiang, Li Yuan, Daquan Zhou, Yonghong Tian

Categories: cs.CV, cs.SD, eess.AS

Comment: ECCV 2026

Published 2026-06-29 · Updated 2026-06-29

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Diffusion Transformers (DiTs) have significantly advanced audio-driven portrait animation, but their high computational cost leads to substantial inference latency. Although training-free diffusion caching accelerates inference significant, existing methods are primarily developed for text-conditioned generation and overlook the spatial and modality imbalances inherent in audio-driven portrait animation. In this paper, we propose SyncCache, a training-free caching acceleration method tailored for DiT-based portrait animation that explicitly exploits asymmetric dynamics. Specifically, high-frequency dynamics driven by audio conditions and concentrated in human regions are more challenging and critical to cache and reuse than the low-frequency visual background in portrait animation. First, we introduce Spatially-Asymmetric Probing to prioritize error sensitivity in dynamic human region. Second, through Modality-Decoupled Caching, we bypass heavy DiT block by reusing stable inter-block residuals, while continuously recomputing lightweight audio blocks to preserve precise lip synchronization. Furthermore, we introduce a cache ratio to control cache capacity and formulate memory-adaptive cache selection as an offline dynamic programming problem without online overhead. Extensive experiments demonstrate that SyncCache achieves superior speed-quality trade-offs, delivering up to 4.12x acceleration on HunyuanVideo-Avatar and 3.75x on Wan-S2V with near-lossless visual fidelity and precise audio alignment.

---

Introduction and Motivation

Diffusion Transformers (DiTs) have become the dominant backbone for video generation and, more recently, for audio-driven portrait animation — the task of synthesising a realistic talking-face video from a reference portrait image and a driving audio clip. State-of-the-art open-source systems such as HunyuanVideo-Avatar and Wan-S2V deliver compelling visual quality, but their inference cost is prohibitive: generating a 15-second clip on HunyuanVideo-Avatar requires approximately 10 minutes on 8× A800 GPUs, severely hampering creative iteration in production workflows.

Training-free diffusion caching — exploiting the temporal redundancy of adjacent denoising steps to skip or approximate expensive computations — has emerged as a practical remedy. Methods such as TeaCache, MagCache, and TaylorSeer have demonstrated meaningful speedups on text-to-video models. However, none of these methods were designed for the unique dynamics of audio-driven portrait animation, and they fail in characteristic ways when naively applied:

• Timestep-level caching (TeaCache, MagCache, EasyCache): caches and skips entire denoising steps under the assumption of synchronous modality dynamics. This preserves slow-changing visual content but interrupts the continuous injection of rapidly varying audio signals, leading to blurry lip movements and degraded synchronisation at high speedup ratios.
• Module-level caching (TaylorSeer, SpeCa, ClusCa): independently caches feature maps of every layer, scaling memory footprint to $O(L)$ where $L$ is the number of layers. At the resolutions and durations required by production portrait animation, this causes out-of-memory (OOM) failures on standard 80 GB A800 GPUs.

The authors identify the root cause of these failures as a failure to account for two fundamental asymmetries in portrait animation:

1. Spatial asymmetry: Critical high-frequency motion (lips, facial muscles, hand gestures) is concentrated on the human subject, while the background is a static, low-frequency anchor consistent with the reference image.
2. Modality asymmetry: The visual self-attention and MLP blocks dominate computation, while the audio cross-attention blocks are lightweight (contributing less than 1% of end-to-end latency) yet are the sole carriers of the high-frequency lip-sync signal.

To address both asymmetries simultaneously within a single, training-free framework, this paper introduces SyncCache.

Background: Flow Matching and DiT-based Portrait Animation

The models targeted by SyncCache are built on Flow Matching, which formulates generative modelling as a deterministic transport governed by an ODE. Given data sample $x_0 \sim p_{\text{data}}$ and noise $x_T \sim p_{\text{noise}}$, the linear interpolation at timestep $t \in [0, T]$ is:

$$x_t = \left(1 - \frac{t}{T}\right)x_0 + \frac{t}{T}x_T$$

A timestep-conditioned velocity field $v_\theta(x_t, t, c)$ is learned to approximate the denoising direction $(x_0 - x_T)$, where $c$ denotes optional conditioning signals (text, reference image, audio). Sampling integrates this velocity field along the straight-line trajectory from noise to data.

In DiT-based portrait animation pipelines, the architecture interleaves heavyweight DiT backbone blocks (comprising self-attention and MLP layers) with lightweight audio cross-attention blocks. The audio blocks act as high-frequency local controllers, injecting precise lip-sync and human motion signals into the massive backbone at each block boundary.

Method: SyncCache

SyncCache is composed of three mutually complementary components: Spatially-Asymmetric Probing (SAP), Modality-Decoupled Caching (MDC), and Memory-Adaptive Optimal Selection (MAOS).

Spatially-Asymmetric Probing (SAP)

The first challenge in any caching scheme is deciding when to skip computation and reuse the cache, and when to trigger a full forward pass. Prior work (e.g., TeaCache) employs the output of the first DiT block as a lightweight proxy, reasoning that its temporal variation strongly correlates with the overall change in the network output across denoising steps. The paper confirms this empirical observation on both HunyuanVideo-Avatar and Wan-S2V:

Let $p^t$ denote the output of the first block at denoising step $t$. A naive global relative L1 error would be:

$$e_t = \frac{\|p^t - p^{t+1}\|_1}{\|p^{t+1}\|_1}$$

The key insight of SAP is that this global metric is systematically biased: the static background inflates the denominator and dilutes the signal from highly dynamic human regions. To correct for this, SyncCache modulates the probing error using a spatial human mask $M$. For models that provide a mask as a standard auxiliary input (e.g., HunyuanVideo-Avatar, Hallo3), $M$ is obtained for free. For mask-free architectures (e.g., Wan-S2V), a lightweight human detector with only 46 million parameters is run once per video, adding only 0.07 seconds of latency. The spatially-asymmetric error at timestep $t$ is:

$$\hat{e}_t = \frac{\|(p^t - p^{t+1}) \odot (1 + \omega M)\|_1}{\|p^{t+1} \odot (1 + \omega M)\|_1}$$

where $\omega$ is a human emphasis weight that amplifies sensitivity in the masked region ($\omega = 2$ is used throughout, empirically robust for $\omega \in [2, 4]$), and $\odot$ denotes element-wise multiplication. Errors from cached steps are accumulated; once $\sum \hat{e}_t > \delta$ for a user-specified tolerance $\delta$, a full forward pass is triggered. This ensures timely recomputation precisely when human-region dynamics demand it, rather than when the low-frequency background changes.

An important robustness property of SAP is that it uses a soft multiplicative mask rather than a hard crop. If the human detector returns a low-confidence mask or fails entirely, the formulation naturally degrades gracefully towards the uniform probing baseline — it does not impose incorrect spatial asymmetries.

Modality-Decoupled Caching (MDC)

SAP determines when to cache, but MDC determines what to cache. The central observation is that the inter-block residuals connecting successive audio blocks — i.e., the feature tensor after one audio block and before the next heavyweight DiT block — exhibit high temporal stability across consecutive denoising steps. These residuals capture the slow evolution of the global visual representation across DiT blocks.

Formally, during a full-computation timestep $t_a$, SyncCache caches the inter-block residual $r_{i,j}^{t_a}$ between the $i$-th and $j$-th audio block boundaries. During subsequent cache-reuse timesteps, the expensive DiT backbone blocks between these boundaries are completely bypassed by directly reusing $r_{i,j}^{t_a}$. Crucially, the lightweight audio blocks themselves are always recomputed at every timestep, regardless of whether the backbone is cached. This decoupling has two consequences:

• High-frequency audio signals — lip positions, muscle activations, emotional cues — are injected freshly at every denoising step, preserving synchronisation.
• The expensive self-attention and MLP computations (which together account for more than 99% of latency) are skipped for the cached steps.

Memory-Adaptive Optimal Selection (MAOS)

Caching residuals at every audio block boundary incurs a memory footprint of $O(L_a)$, where $L_a$ is the total number of audio blocks. For long video generation or VRAM-constrained deployments, this can still exceed available memory. The paper introduces a continuous cache ratio $\sigma \in (0, 1]$ to limit the cache to a subset of $n = \lfloor L_a \sigma \rfloor$ residual boundaries. The question becomes: which $n$ boundaries to select?

The paper shows empirically that temporal stability varies substantially across network depths, making uniform or random boundary selection suboptimal. To address this, MAOS formulates the selection as a dynamic programming (DP) optimisation problem. The temporal instability score for the residual segment spanning from the $i$-th to the $j$-th audio block over the full denoising trajectory of length $T$ is:

$$\gamma_{i,j} = \sum_{t=0}^{T-1} \frac{\|r_{i,j}^t - r_{i,j}^{t+1}\|_1}{\|r_{i,j}^{t+1}\|_1}$$

The DP then selects the $n$ boundaries that minimise the total instability across all cached segments. The complete procedure is encapsulated in Algorithm 1:

1. Calibration: Run a single full denoising pass on any sample, computing $\gamma_{i,j}$ for all block pairs $(i, j)$.
2. DP search: Solve the DP to obtain the optimal boundary set $\mathcal{B} = \{b_1, \ldots, b_n\}$ for the requested $\sigma$.
3. Deployment: Use the precomputed $\mathcal{B}$ for all subsequent generation calls — no online overhead.

A critical finding is that the optimal boundary set $\mathcal{B}$ is remarkably robust across diverse inputs: calibrating on a single sample, an average over all samples, or even a silent sample with minimal lip motion produces near-identical results. This means a one-time offline calibration pass fully determines the optimal cache plan for any memory budget $\sigma$.

Experimental Setup

Models and Inference Protocols

SyncCache is evaluated on two strong open-source DiT-based portrait animation models:

• HunyuanVideo-Avatar: Built on HunyuanVideo, which injects audio conditioning by inserting audio blocks after dual-stream DiT blocks. Standard inference uses 50 denoising steps.
• Wan-S2V: Built on Wan-14B, injecting audio via eleven audio blocks after backbone blocks, using a 40-step UniPC sampler.

Evaluation Metrics

Three axes of evaluation are reported:

• Visual quality: LPIPS (lower is better), PSNR (higher is better), SSIM (higher is better), FID (lower is better), FVD (lower is better) — computed by comparing accelerated outputs against original 50/40-step outputs.
• Audio-visual synchronisation: Sync-C (higher is better) and Sync-D (lower is better) from SyncNet — widely used in the portrait animation community.
• Inference efficiency: Speedup ratio and absolute latency in seconds.

All experiments are conducted on NVIDIA A800 80 GB GPUs with PyTorch and FlashAttention. Latency benchmarks use 8× A800 GPUs with FSDP for HunyuanVideo-Avatar and FSDP + Ulysses sequence parallelism for Wan-S2V. The evaluation set is the EMTD dataset (from EchoMimicV2), comprising solo-talking and semi-body human videos.

Baselines

Comparisons include: $\Delta$-DiT, TeaCache, MagCache, DiCache, CGCache, and the TaylorSeer-series (TaylorSeer, SpeCa, ClusCa).

Quantitative Results

HunyuanVideo-Avatar

SyncCache-slow reduces end-to-end latency from approximately 10 minutes to 157 seconds while achieving the best visual fidelity across LPIPS, SSIM, FID, and FVD among all compared methods. Most critically, it achieves near-lossless lip synchronisation with Sync-C of 6.944 versus 6.963 for the original model. At the aggressive 4.12× speedup of SyncCache-fast, both perceptual quality and lip-sync remain superior to all baselines, even those operating at much lower speedup ratios. The TaylorSeer-series methods fail entirely with OOM errors at production-scale video settings.

Wan-S2V

Wan-S2V is a particularly challenging testbed due to its compressed 40-step sampler and complex dynamic patterns. SyncCache dominates across all quality metrics and achieves 3.75× speedup — outperforming all baselines in both speedup ratio and all quality dimensions. The step-wide skipping of TeaCache and MagCache interrupts the continuous audio feed, causing lip-sync misalignment that is avoided by MDC's continuous audio block refresh.

Qualitative Comparisons

Visualisations confirm the quantitative findings. Traditional caching methods, operating under synchronous modality dynamics, lose high-frequency details — manifested as blurry hand gestures and lip-sync artifacts. SyncCache at 4.12× achieves notably sharper lip-audio alignment than all baselines, even those operating at far lower speedup ratios.

Multi-Person Scenarios

Portrait animation is increasingly applied to multi-speaker scenarios using dynamic spatial masks to control the range of audio cross-attention. SyncCache integrates seamlessly with this masking strategy. MDC's continuous refreshing of audio conditions — combined with the spatial mask information — ensures per-speaker lip synchronisation remains accurate at every denoising step, whereas TeaCache suffers synchronisation failures due to weak or interrupted audio control.

Ablation Studies

Component Contributions

Effect of Spatially-Asymmetric Probing: Removing SAP (reverting to a naive uniform probing metric) causes a clear degradation in spatial visual fidelity across all metrics. Without SAP, the global probing metric is diluted by the static background, causing premature cache reuse that fails to preserve high-frequency human details.

Effect of Modality-Decoupled Caching: Disabling MDC — equating to a rigid synchronous caching strategy where audio blocks are also skipped — causes significant temporal alignment degradation, with Sync-C dropping from 6.944 to 6.822. Restoring continuous audio block recomputation recovers near-lossless lip-sync with only a 1-second latency penalty (156 s → 157 s), demonstrating that keeping the lightweight audio stream refreshed is practically "free" in terms of latency while being essential for audio-visual alignment.

Memory-Adaptive Optimal Selection Robustness

These ablations (conducted at $\sigma = 0.4$ on HunyuanVideo-Avatar) confirm two key claims:

1. DP selection is critical: Replacing DP-optimised boundaries with arbitrary selection degrades LPIPS from 0.1016 to 0.1369 and reduces Sync-C from 6.944 to 6.847, highlighting that temporal stability varies meaningfully across layers and must be explicitly optimised.
2. Calibration is robust: The optimal boundary set determined by a single sample, the full dataset average, or even a silent sample (with minimal lip motion) produces nearly identical results. This definitively demonstrates that optimal cache boundaries depend on the model's inter-layer temporal dynamics rather than input-specific content, validating the offline calibration strategy.

System-Level Memory Efficiency

A key systems contribution of SyncCache is its predictable, bounded memory footprint. TaylorSeer and FoRA, which cache feature maps for every layer, exhibit memory requirements that grow rapidly with video resolution and duration, quickly exceeding 640 GB total VRAM on the 8-GPU setup. SyncCache's inter-block residual caching — limited to audio block boundaries and further constrained by $\sigma$ — maintains a stable and predictable memory footprint across all tested resolutions and durations, enabling deployment on hardware that would OOM for existing module-level methods.

Key Contributions and Novelty

The authors identify the following novel aspects of SyncCache:

• First multimodal-aware caching framework for portrait animation: SyncCache is the first training-free caching method explicitly designed for the dual-asymmetry (spatial and modality) of audio-driven DiT-based portrait animation. Prior work applies text-to-video caching heuristics without accounting for these structure-specific properties.
• Spatially-Asymmetric Probing: The human-mask-weighted probing error formulation formally encodes the spatial prior that critical dynamics are human-centric, enabling more responsive and precise cache triggering than global metrics.
• Modality-Decoupled Caching: The structural separation of the heavyweight visual backbone from the lightweight audio stream — caching stable inter-block residuals while always recomputing audio blocks — is a novel architectural insight that preserves lip-sync at extreme speedups with negligible latency cost for the audio blocks.
• Memory-adaptive offline DP: Formulating cache boundary selection as a dynamic programming problem over inter-block temporal instability scores, solved once offline and applicable to any memory budget $\sigma$, is a principled and practical solution to the memory scalability problem that afflicts prior module-level caching approaches.

Limitations and Future Directions

The paper does not enumerate explicit limitations in a dedicated section. However, several implicit constraints can be noted from the experimental setup and method design:

• SyncCache is evaluated specifically on two DiT-based portrait animation models. While the principles of spatial and modality asymmetry are general, the method's effectiveness on other architectures with different audio injection mechanisms (e.g., U-Net-based systems) is not demonstrated.
• The human emphasis weight $\omega$ and tolerance threshold $\delta$ are hyperparameters that require some tuning, though $\omega$ is reported to be empirically robust in the range $[2, 4]$.
• The lightweight human detector (46M parameters) adds a minor dependency for mask-free models, though its 0.07-second latency is negligible in practice.
• Evaluation is conducted on the EMTD dataset focusing on solo-talking and semi-body scenarios; performance on highly unconstrained in-the-wild videos with complex camera motion, multiple subjects with heavy occlusion, or non-frontal poses is not systematically studied.

Conclusion

SyncCache is a training-free, memory-adaptive caching strategy that achieves state-of-the-art inference acceleration for DiT-based audio-driven portrait animation by explicitly exploiting the asymmetric dynamics inherent to the task. Three tightly integrated components work in concert: Spatially-Asymmetric Probing amplifies sensitivity to high-frequency human-region dynamics during cache decision-making; Modality-Decoupled Caching bypasses computationally expensive visual blocks via stable inter-block residuals while always recomputing lightweight audio blocks to preserve lip synchronisation; and Memory-Adaptive Optimal Selection uses offline dynamic programming to analytically determine the optimal cache boundaries for any given VRAM budget. The result is up to 4.12× acceleration on HunyuanVideo-Avatar and 3.75× on Wan-S2V, with near-lossless visual fidelity and audio alignment — substantially outperforming all compared baselines across every evaluated metric, while maintaining a stable memory footprint that scales gracefully with video resolution and duration.