Sonic

Sonic: Shifting Focus to Global Audio Perception in Portrait Animation

Sonic is an audio-driven portrait animation method focusing on global audio perception to produce natural, temporally consistent talking-face videos. It disentangles audio into intra- and inter-clip perception to guide lip sync, expression, and head motion without relying on visual cues.

• audio-driven
• talking-face
• portrait-animation
• lip-sync
• head-pose
• face-animation

Authors: Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, Qinglin Lu, Chengjie Wang

Categories: cs.MM, cs.CV, cs.GR, cs.SD, eess.AS

Comment: refer to our main-page \url{https://jixiaozhong.github.io/Sonic/}

Published 2024-11-25 · Updated 2025-06-05

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3,258 stars56 watchers291 forksCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International
• Hugging Face0 downloads28 likes

Abstract

The study of talking face generation mainly explores the intricacies of synchronizing facial movements and crafting visually appealing, temporally-coherent animations. However, due to the limited exploration of global audio perception, current approaches predominantly employ auxiliary visual and spatial knowledge to stabilize the movements, which often results in the deterioration of the naturalness and temporal inconsistencies.Considering the essence of audio-driven animation, the audio signal serves as the ideal and unique priors to adjust facial expressions and lip movements, without resorting to interference of any visual signals. Based on this motivation, we propose a novel paradigm, dubbed as Sonic, to {s}hift f{o}cus on the exploration of global audio per{c}ept{i}o{n}.To effectively leverage global audio knowledge, we disentangle it into intra- and inter-clip audio perception and collaborate with both aspects to enhance overall perception.For the intra-clip audio perception, 1). \textbf{Context-enhanced audio learning}, in which long-range intra-clip temporal audio knowledge is extracted to provide facial expression and lip motion priors implicitly expressed as the tone and speed of speech. 2). \textbf{Motion-decoupled controller}, in which the motion of the head and expression movement are disentangled and independently controlled by intra-audio clips. Most importantly, for inter-clip audio perception, as a bridge to connect the intra-clips to achieve the global perception, \textbf{Time-aware position shift fusion}, in which the global inter-clip audio information is considered and fused for long-audio inference via through consecutively time-aware shifted windows. Extensive experiments demonstrate that the novel audio-driven paradigm outperform existing SOTA methodologies in terms of video quality, temporally consistency, lip synchronization precision, and motion diversity.

---

Introduction

Sonic is a one-stage audio-driven portrait animation method that aims to move beyond the common practice of stabilizing talking-face generation with auxiliary visual motion cues such as motion frames, overlapping clips, or other spatial priors. The paper argues that, for audio-driven animation, the audio signal itself is the most appropriate and unique prior for driving lip movement, facial expression, and head motion. Its central thesis is that existing methods underuse global audio perception, and that this underuse leads to reduced naturalness, temporal jitter, and inconsistent long-video generation.

The method is designed for the standard setting of a single portrait reference image plus an input audio track. Sonic is positioned as a unified paradigm that separates audio understanding into intra-clip and inter-clip perception. Intra-clip perception focuses on extracting temporal cues inside each audio segment to guide local facial dynamics, while inter-clip perception bridges adjacent segments so long-audio inference can remain coherent across clip boundaries.

The paper’s framing is especially important for talking-head / conversational-AI applications where both synchronization and conversational expressiveness matter. Instead of using external visual motion hints to regularize generation, Sonic attempts to let audio alone determine the motion trajectory, with explicit mechanisms for long-range audio context, disentangled motion control, and seamless long-audio denoising.

Method Overview

Sonic consists of three core components: context-enhanced audio learning, motion-decoupled controller, and time-aware position shift fusion. The overall system is initialized from stable-video-diffusion-xt-1-1 for both the spatial and temporal modules, and it is trained in a single stage. In inference, the model uses multi-condition classifier-free guidance (CFG) and supports controllable motion strength through a dynamic scale parameter.

Context-Enhanced Audio Learning

The first module is intended to extract long-range temporal cues from audio and inject them directly into the diffusion U-Net, without relying on motion frames or other visual motion priors. The paper uses Whisper-Tiny as the audio encoder, emphasizing that it is lighter than the Wav2Vec-based encoders used by many prior works. For each audio frame, the features from the last layers of five stages are concatenated to build a multi-scale representation. The authors state that each video frame is conditioned on about 0.2 s of audio context.

After feature extraction, three linear layers project the audio tokens to the cross-attention dimension. If the transformed audio embedding is written as $c_a \in \mathbb{R}^{b \times f \times d \times c}$, then the spatial latent features $z_s$ are updated by audio-guided cross-attention inside the face region mask $M$:

$$z_s' = z_s + \operatorname{CrossAttn}(Q(z_s), K(c_a), V(c_a)) \cdot M.$$

Here, $M$ is determined from the joint face bounding boxes so that the audio guidance is focused on the talking-head area. This design is meant to improve both appearance fidelity and local motion alignment by letting audio directly condition the spatial denoising process.

Sonic also introduces a temporal audio cross-attention path. The latent after temporal self-attention is reshaped to $z_t \in \mathbb{R}^{(b \times h \times w) \times f \times c}$. To keep computation manageable, the audio representation is average-pooled along the temporal axis to produce temporal embeddings $c_{ta}$, which are then repeated across spatial locations and injected into the temporal denoising blocks:

$$z_t' = z_t + \operatorname{CrossAttn}(Q(z_t), K(c_{ta}), V(c_{ta})).$$

The paper’s interpretation is that audio provides a long-range signal containing tone, rhythm, and speech speed, which implicitly encode priors for expression intensity and head movement. Temporal audio attention is therefore responsible for motions that are strongly synchronized with speech, such as lip movement and content-related facial expression changes.

Motion-Decoupled Controller

The second module handles motion factors that are only weakly correlated with audio, especially habitual head motion and perspective variation. The goal is to disentangle these motion patterns into separate, user-controllable components so that animation can remain expressive without collapsing into over-smoothing or over-constraint.

During training, two integer motion buckets in the range $[0, 128]$ are derived from the clip: the translation bucket $m_t$ is computed from the variance of bounding boxes, and the expression bucket $m_e$ is computed from the variance of relative landmarks. These are encoded with position embeddings and a linear projection:

$$emb = W[\operatorname{Pe}(m_t), \operatorname{Pe}(m_e)].$$

In inference, Sonic can either use user-adjusted motion settings or predict them automatically from the audio and the reference image. The predicted buckets are generated by an audio-to-bucket network $E_b(c_a, R_{img})$ and then scaled by a dynamic factor $\beta$:

$$m_t', m_e' = \beta \times E_b(c_a, R_{img}).$$

The paper reports three scale settings: $\beta = 0.5$ for mild dynamics, $\beta = 1.0$ for moderate dynamics, and $\beta = 2.0$ for intense dynamics. This mechanism is intended to support both automatic motion prediction and direct controllability for lip, expression, and head motion, making the system more suitable for interactive use cases.

Time-Aware Position Shift Fusion

The third module is the method’s long-video strategy. Existing approaches often rely on motion frames or overlap-based segment fusion to maintain temporal consistency, but these methods either restrict receptive field or add computational overhead. Sonic instead proposes time-aware position shift fusion, which expands intra-clip audio perception into a global inter-clip mechanism by progressively shifting the denoising window across time.

The mechanism is organized as two nested loops. The outer loop follows the reverse diffusion process over timesteps, while the inner loop slides over the long sequence with non-overlapping clips. At each timestep, the start point is shifted by an accumulated offset $\alpha_\Sigma$, and the model denoises the current clip conditioned on the correspondingly shifted audio segment. When the shifted index exceeds the sequence length, circular padding is used so that the beginning of the clip can be wrapped to the end.

In algorithmic form, the model denoises clips $z_t^{[s,e]}$ from audio segments $c_a^{[s,e]}$ and updates the accumulated offset as $\alpha_\Sigma \leftarrow \alpha_\Sigma + \alpha$ after each diffusion step. The paper emphasizes that this strategy preserves the audio-video correspondence while enlarging temporal context, and that it does so without extra training cost or extra inference time from overlap recomputation.

The paper also provides a simple compute comparison. If a single forward pass has FLOPs $\Omega$, the video length is split into $n$ clips of length $f$, and denoising uses $T$ steps, then Sonic’s shift-fusion cost is written as $\Omega \times T \times n$. By contrast, overlap-based fusion reprocesses overlapped latent regions, and motion-frame based approaches add additional reference-network and motion-module overhead. The stated point is that Sonic improves long-range coherence while remaining efficient.

Experiments

Experimental Setup

Training is performed in a single stage. The model is initialized from stable-video-diffusion-xt-1-1. To make the network robust to missing conditions, the training data is manipulated so that 5% of samples drop audio, another 5% drop the image condition, and another 5% drop both conditions. At inference time, the paper uses multi-condition CFG with the reference-image CFG set to 2.0 and the audio CFG set to 7.5. The full model uses $\beta = 1.0$ and shift offset $\alpha = 7$.

The training set is formed by aggregating open-source data from VFHQ, CelebV-Text, and VoxCeleb2. Evaluation is conducted on two commonly used test sets: HDTF and CelebV-HQ. The paper states that it randomly selects 100 four-second clips from HDTF and 50 four-second clips from CelebV-HQ.

The reported metrics include FID and FVD for visual fidelity and video realism; E-FID and F-SIM for expression quality and facial similarity; Sync-C and Sync-D from SyncNet for lip synchronization; Smooth from VBench for temporal smoothness; and Dynamic for motion amplitude in ablations.

Main Quantitative Results

Sonic is compared against SadTalker, AniPortrait, V-Express, Hallo, Hallo2, and EchoMimic. The paper reports that Sonic achieves the best or near-best performance on most metrics, with particularly strong gains in image/video quality, synchronization, and smoothness.

On HDTF, Sonic attains the best FID, FVD, Sync-C, Sync-D, and Smooth. On CelebV-HQ, Sonic again obtains the best FID, Sync-C, Sync-D, E-FID, F-SIM, and Smooth, while its FVD is second best behind Hallo2. The runtime reported in the table is 17.04, which is faster than motion-frame or overlap-based long-video strategies discussed in the paper’s inference comparison, but slower than some lightweight baselines that do not provide the same range of motion and quality.

Qualitative Behavior

The qualitative comparisons show that prior methods tend to depend on motion frames or other stabilization signals, which can limit motion richness and make facial movement feel constrained. Sonic instead leverages audio-centric long-range perception, and the paper claims this produces broader expression variation and more natural head movement. The authors also show results for multiple portrait styles, including real human, animation, and 3D portraits, and for audio durations ranging from around 20 seconds up to 10 minutes.

User Study

The paper reports a subjective evaluation with 40 participants who scored results from five methods on a 1-to-5 scale along four dimensions: lip sync, motion diversity, identity consistency, and video smoothness. Sonic is ranked highest in all four categories.

Ablation Studies

The ablations isolate the contribution of temporal audio attention, motion control, and shift fusion on CelebV-HQ. Removing any of these components hurts synchronization, expression fidelity, or motion richness. The motion controller is especially important on CelebV-HQ, which contains more movement that is only weakly correlated with audio.

The paper’s key ablation observations are:

• Removing temporal audio attention lowers lip-sync and worsens E-FID, showing that local speech context matters for fine expression control.
• Removing the motion controller causes the largest drop in dynamic range, indicating that explicit disentanglement is important for non-speech motion such as head movement.
• Removing shift fusion degrades both synchronization and smoothness in long-video inference, confirming the importance of inter-clip audio perception.
• Increasing $\beta$ amplifies motion diversity and sync, but overly large values can slightly reduce stability and expression similarity.

Long-Video Inference Comparison

For extended generation, Sonic is compared with motion-frame-based and overlap-based strategies using 8 frames. The shift-fusion method achieves the best synchronization and smoothness while also being the fastest of the three reported long-video strategies.

The paper also notes that the smoothest result in the ablation over the offset parameter occurs at $\alpha = 7$, matching the value used in the full model.

Discussion, Scope, and Limitations

The paper does not include a separate limitations section, so the following remarks are restricted to what is explicitly stated in the body. Sonic is designed for the specific problem of audio-driven portrait animation from a single reference image and audio track. It is trained and evaluated with pretrained components, notably Whisper-Tiny for audio encoding and Stable Video Diffusion initialization for the denoising backbone. The method also relies on face bounding boxes and relative landmarks to form motion buckets, which means its controllable motion design is tied to the availability of these preprocessing signals during training.

Within that scope, the authors claim the main advantage is a better balance of video quality, temporal consistency, lip synchronization, and motion diversity without resorting to motion-frame guidance. The paper’s experiments support this claim on HDTF and CelebV-HQ, and the qualitative examples suggest robustness across different portrait styles and long audio durations.

Conclusion

Sonic proposes a global-audio-first paradigm for portrait animation. Its technical contribution is the combination of context-enhanced audio learning for intra-clip control, a motion-decoupled controller for explicit head/expression disentanglement, and time-aware position shift fusion for efficient long-video inference. Across the reported benchmarks and user study, the method improves synchronization, motion diversity, and temporal smoothness relative to the compared baselines, while also providing a controllable dynamic range through the scale parameter $\beta$.