Wan-S2V

Wan-S2V: Audio-Driven Cinematic Video Generation

Wan-S2V advances audio-driven video generation from simple talking heads to cinematic scenes by using text for scene planning and audio for detailed facial and body motion. It achieves superior identity and motion consistency and supports long-form video and precise lip-sync editing.

• audio-driven
• full-body
• face-animation
• lip-sync
• gesture
• multimodal

Demos

These demos showcase Wan-S2V's capability to generate high-quality cinematic videos from static images and audio input, featuring natural facial expressions, synchronized lip movements, realistic body gestures, and professional camera work. Highlights include diverse scenarios such as a stormy sea scene with atmospheric effects, dynamic movement on a moving train, and engaging dialogue scenes with subtle environmental interactions. Watch for synchronized audio-visual expressions, smooth motion, and cinematic camera dynamics.

Authors: Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, Lian Zhuo

Categories: cs.CV

Published 2025-08-26 · Updated 2025-08-26

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar48 citations9 influential citations

Abstract

Current state-of-the-art (SOTA) methods for audio-driven character animation demonstrate promising performance for scenarios primarily involving speech and singing. However, they often fall short in more complex film and television productions, which demand sophisticated elements such as nuanced character interactions, realistic body movements, and dynamic camera work. To address this long-standing challenge of achieving film-level character animation, we propose an audio-driven model, which we refere to as Wan-S2V, built upon Wan. Our model achieves significantly enhanced expressiveness and fidelity in cinematic contexts compared to existing approaches. We conducted extensive experiments, benchmarking our method against cutting-edge models such as Hunyuan-Avatar and Omnihuman. The experimental results consistently demonstrate that our approach significantly outperforms these existing solutions. Additionally, we explore the versatility of our method through its applications in long-form video generation and precise video lip-sync editing.

---

Introduction and Problem Setting

Wan-S2V is an audio-driven video generation system aimed at extending audio-conditioned human animation beyond the usual talking-head setting into cinematic and film/television-style scenes. The paper argues that prior audio-driven methods perform well for speech and singing, but struggle when the scene requires richer body motion, character interactions, and dynamic camera-related content. The central design idea is to combine text and audio as complementary controls: text is used for global scene and motion planning, while audio is used for fine-grained facial expression, head motion, and local character action control.

The system is built on the Wan video foundation model, with the main reported model referred to as Wan-S2V-14B. The paper also states that a smaller 1.3B variant is trained with the same general approach, though the provided experimental section focuses on the 14B model and does not report a separate quantitative table for the smaller model.

The paper’s stated contributions are:

• Extending audio-driven generation from short talking-head clips to more complex cinematic scenarios.
• Introducing a long-video stabilization strategy based on more efficient use of motion-frame history.
• Building a large, filtered, human-centric training corpus combining open datasets and internally collected data.

Data Processing Pipeline

The paper places substantial emphasis on data construction, arguing that cinematic audio-driven generation requires not only speech clips but also videos with intentional human activity, visible faces, and sufficiently rich motion. The training corpus is assembled from two main sources: (1) automated screening of large-scale open video datasets such as OpenHumanViD and koala36m, and (2) manual curation of higher-quality public videos featuring speaking, singing, and dancing. The result is described as a large human-centric pool that forms the basis for training.

A pose-tracking stage uses VitPose to detect 2D pose and then converts the tracks to DWPose. The tracked poses are used in two ways: as an optional multi-modal control signal, and as a filtering tool to remove videos in which the human subject is too small temporally or spatially. The pipeline also keeps only videos with consistent, visible faces across the sequence, so that the model can learn audio-conditioned facial dynamics from reliable examples.

Additional filtering uses pretrained video quality models and other heuristics:

• Clarity via the Dover metric.
• Motion stability via optical flow estimated with UniMatch.
• Face/hand sharpness using a Laplacian operator on facial and hand regions.
• Aesthetic quality with an improved aesthetic predictor.
• Subtitle occlusion detection with OCR to remove clips where subtitles cover faces or hands.
• Audio-visual alignment using Light-ASD to discard clips where the audio is not synchronized with the active speaker, or where no active speaker exists.

The paper also generates dense video captions with Qwen2.5-VL 72B. The captioning prompt is designed to explicitly describe camera angle, subject appearance, subject actions, and background/environmental details. The captioning instructions intentionally avoid subjective judgments or emotional interpretations so that captions remain faithful to generation-relevant content rather than opinionated description.

Model Architecture

The model is formulated as a diffusion-style video generator trained with flow matching. Given a reference image, an audio signal, and a text prompt, the model generates a video that is synchronized with the audio while preserving the reference image content. The paper states that generation is not simply image-to-video starting from the input image; instead, the model denoises multi-frame noise latents into consecutive video frames conditioned on the reference frame, motion frames, audio, and prompt.

Let the target RGB video frames be $X \in \mathbb{R}^{F \times H \times W \times 3}$, and let the 3D VAE encode them to a latent tensor $x_0$. During training, noise $\epsilon$ is added using a time variable $t \in [0,1]$ according to:

$$x_t = t\epsilon + (1-t)x_0$$

The model predicts the velocity field, written in the paper as $\frac{dx}{dt} = \epsilon - x_0$. During inference, the noisy latent is recovered into the target latent under the conditions of the reference image, motion frames, audio, and prompt.

Visual tokenization. The reference image, target frames, and motion frames are encoded with a 3D VAE. Their latent tensors are then patchified, flattened, and concatenated into a sequence of visual tokens. Motion frames are optional but important for temporal continuity across clips. To keep long-range history while limiting compute, the paper applies a FramePack-style compression strategy in which earlier motion frames are compressed more aggressively than recent ones.

Audio encoding and injection. Raw audio is first encoded with Wav2Vec. To better capture diverse acoustic cues, the system uses a weighted average over layers, following the paper’s citation to the EMO-style audio representation. This allows shallow rhythmic/emotional information and deeper lexical information to be combined. The resulting audio features are then temporally compressed with multiple causal 1D convolutions, producing frame-aligned audio tokens $a_i$.

Instead of applying full 3D attention between all visual tokens and all audio tokens, the model performs attention between each audio slice $a_i$ and the corresponding visual segment $x_{t i}$. This segment-wise design reduces compute while preserving time alignment between audio and video tokens.

Conceptually, the architecture is designed to keep global motion control and local audiovisual synchronization separate enough to remain manageable, while still letting them interact in a single generative framework.

Training and Implementation

The implementation section describes a three-stage training process built on a pretrained Wan model:

1. Audio encoder training.
2. Training on speech videos.
3. Training on film-and-television plus speech videos, followed by a high-quality supervised fine-tuning stage.

Training uses a hybrid parallel strategy combining FSDP with context parallelism. FSDP shards model parameters across 8 GPUs in a single node and is used to fit the large Wan-S2V-14B model into 80 GB per GPU. For the context-parallel component, the paper combines RingAttention and Ulysses, similar to the cited USP-style approach. This is reported to reduce a single training iteration from roughly 100 seconds to about 12 seconds on 8 GPUs.

The authors state that this setup supports models larger than 16B parameters, including the audio encoder and cross-attention components, and enables training with up to 48 frames at 1024×768 resolution on 8 GPUs.

To support variable video sizes, the paper uses a variable-length resolution training strategy. The token count after patchification is compared against a maximum limit $M$; clips exceeding $M$ are resized or cropped until their token count is at or below the limit, while shorter clips are used as-is.

Experimental Setup

The training set is built from filtered OpenHumanViD data plus an internal talking-head dataset. The evaluation section focuses on the EMTD dataset proposed by EchoMimicV2, which mainly contains solo-talking videos. The paper compares against several open-source and recent audio-driven methods, including EchoMimicV2, MimicMotion, EMO2, FantasyTalking, and Hunyuan-Avatar. Qualitative comparisons also include OmniHuman (spelled “Ominihuman” in one caption in the provided text).

The reported metrics are:

• FID, SSIM, and PSNR for frame quality.
• FVD for video coherence.
• CSIM for identity consistency between the reference image and generated frames.
• Sync-C for audio-lip synchronization.
• HKC and HKV for hand quality and hand motion diversity.
• EFID for expression divergence relative to ground truth.

Quantitative Results

The table below reproduces the reported numbers from the paper. Lower is better for FID, FVD, EFID; higher is better for the remaining metrics.

The paper’s narrative claims broad superiority across quality, realism, and hand motion. However, the reported table shows a more nuanced picture: Wan-S2V achieves the best FID, SSIM, PSNR, and CSIM, and its FVD is essentially tied with EMO2. On the other hand, EMO2 has the strongest reported HKC and HKV, while HY-Avatar obtains the highest Sync-C. This makes the overall result set strong but not uniformly dominant across all metrics.

Qualitative Findings

In the visual comparisons, the paper reports that Hunyuan-Avatar often exhibits facial distortion and identity drift during large motions, while OmniHuman tends to produce very limited motion amplitudes, often remaining close to the pose of the reference image. Wan-S2V is said to better preserve identity under motion while generating a wider range of body and camera-related dynamics.

For long-form generation, the paper argues that film-style outputs require temporal consistency across multiple clips, including consistent motion direction, speed, camera movement, and identity. The method leverages FramePack to encode more motion frames, which helps preserve long-range motion trends. In the example described in the paper, a moving train keeps a coherent direction and speed across clips, whereas OmniHuman fails to preserve the motion trend as reliably.

The paper further claims that when generating a new clip after previous clips, the earlier clips can be treated as motion frames. With FramePack, not only is the overall motion trend preserved, but object identity across clips is also better maintained. The example in the paper shows a character picking up a piece of paper that remains visually consistent across clips, whereas without FramePack the object appearance can drift.

Limitations and Scope

The paper is explicit that truly complex film and television problems remain unresolved. In particular, the authors identify difficult cases such as nuanced multi-person interactions and precise camera control driven solely by audio. The provided text does not include a dedicated ablation section or a detailed failure-case analysis, so the empirical support is primarily a combination of pipeline descriptions, qualitative comparisons, and a single quantitative benchmark table.

Another practical limitation is that the strongest reported model is large and requires sophisticated parallel training infrastructure. The paper addresses this through FSDP and context parallelism, but the method still inherits the inference and deployment challenges typical of large diffusion/video foundation models.

Conclusion

Wan-S2V presents a full pipeline for audio-driven cinematic video generation: aggressive human-centric data filtering, dense captioning for text control, a flow-matching video model with audio injection, and a training system capable of scaling large Wan-based models. The paper’s strongest empirical signals are improved frame quality, identity consistency, and long-range temporal coherence, along with qualitative evidence that the model can better handle motion-heavy scenes than prior audio-driven baselines. The remaining gap is the broader cinematic problem of fully reliable multi-person, camera-aware generation conditioned only by audio.

The authors position Wan-S2V as the first release in their Vida research series and indicate future work on more advanced character control and dancing generation.