VASA-1

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

VASA-1 generates lifelike, real-time talking faces from a single image and speech audio. It uniquely models holistic facial dynamics and head motion together, producing natural expressions, gaze, blinking, and head pose with interactive control and low latency, surpassing prior lip-sync focused methods.

• talking-face
• audio-driven
• face-animation
• lip-sync
• head-pose
• gaze
• emotion
• realtime

Authors: Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo

Categories: cs.CV

Comment: NeurIPS 2024 (Oral) Camera ready. Project webpage: https://www.microsoft.com/en-us/research/project/vasa-1/

Published 2024-04-16 · Updated 2024-10-31

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar228 citations21 influential citations

Abstract

We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only generating lip movements that are exquisitely synchronized with the audio, but also producing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos. Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method not only delivers high video quality with realistic facial and head dynamics but also supports the online generation of 512x512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors.

---

Introduction

VASA-1 is a framework for generating lifelike, audio-driven talking faces from a single static portrait image and a speech audio clip. The paper targets a gap that remains after years of progress in lip-sync synthesis: many systems can align the mouth to speech, but still fail to produce the broader set of behaviors that make a face feel alive, including subtle expressions, gaze shifts, blinking, and natural head motion. The authors frame these broader behaviors as visual affective skills and argue that they are essential for convincing conversational avatars.

The central design choice is to generate holistic facial dynamics and head movement in a learned latent space, rather than predicting individual motion factors with separate modules. In the paper's formulation, the motion latent represents lip motion, non-lip facial expression, eye gaze, blinking, and head pose in a unified way. This contrasts with prior pipelines that often decompose the task into multiple specialized subproblems, such as separate regressors for lip-related motion, pose, and blink behavior. VASA-1 instead aims to model the joint distribution of these elements and condition it on audio plus optional controls.

The result is a system that the paper reports can generate 512$\times$512 videos in real time, with up to 40 FPS in online streaming mode and negligible starting latency, while also improving realism, expressiveness, and controllability over prior audio-driven talking-face methods.

Key contributions

• Holistic motion generation: the model treats facial dynamics and head pose as a single generative problem and learns a diffusion model over the joint motion latent sequence.
• Expressive, disentangled face latent space: the paper extends a 3D-aided face reenactment representation with additional losses to better separate identity, head pose, and facial dynamics while preserving rich appearance detail.
• Optional controllability: the generator can be conditioned not only on audio, but also on main gaze direction, head-to-camera distance, and an emotion offset.
• Real-time synthesis: the method is designed for online generation, and the reported runtime makes it suitable for interactive conversational settings.
• New evaluation tools: the paper introduces a data-driven audio-pose metric called CAPP and also reports a pose variation intensity score to better characterize head motion vividness.

Method

The method has two major stages. First, it learns an expressive and disentangled face latent space from video. Second, it trains a diffusion transformer to generate the latent motion sequence conditioned on audio and optional control signals. At inference time, the system extracts the source identity and appearance from a single image, generates motion latents from the speech audio, and decodes the result back into video frames.

Overall pipeline

The system operates in a latent space built around a 3D-aided representation. A source portrait is decomposed into a canonical 3D appearance volume $\mathbf{V}^{app}$, an identity code $\mathbf{z}^{id}$, a head pose code $\mathbf{z}^{pose}$, and a facial dynamics code $\mathbf{z}^{dyn}$. A decoder then reconstructs the face from these components, with inverse warping applied to map the canonical appearance back to the posed view.

Expressive and disentangled face latent space construction

The authors start from the 3D-aided face reenactment framework of prior work, because 3D feature volumes can represent facial appearance more faithfully than 2D feature maps and can naturally support rigid and non-rigid motion through warping. Their goal is not just reconstruction quality, but a latent space with two properties at once: expressiveness and disentanglement. Expressiveness is needed to preserve appearance details and subtle dynamics; disentanglement is needed so that the motion generator can manipulate pose and expression without entangling them with identity.

The paper notes that the original losses from the base 3D-aided framework were not sufficient: facial dynamics and head pose remained poorly separated, and identity could also leak into motion. To fix this, the authors add two losses.

• Pairwise consistency loss $l_{consist}$: for two frames $\mathbf{I}_i$ and $\mathbf{I}_j$ sampled from the same video, they swap pose and dynamics across frames and require the resulting transfers to agree. Concretely, $$ \hat{\mathbf{I}}_{j,\mathbf{z}^{pose}_i} = \mathcal{D}(\mathbf{V}^{app}_j, \mathbf{z}^{id}_j, \mathbf{z}^{pose}_i, \mathbf{z}^{dyn}_j) $$ and $$ \hat{\mathbf{I}}_{i,\mathbf{z}^{dyn}_j} = \mathcal{D}(\mathbf{V}^{app}_i, \mathbf{z}^{id}_i, \mathbf{z}^{pose}_i, \mathbf{z}^{dyn}_j), $$ then minimize the discrepancy between the two transferred results. This encourages facial dynamics and head pose to become more separable.
• Cross-identity similarity loss $l_{cross\_id}$: for source and driving videos from different subjects, the method transfers pose and dynamics from one subject to another while preserving the source identity. A cosine similarity loss on deep face identity features, extracted with ArcFace-style embeddings, penalizes identity drift in cross-reenactment.

The paper emphasizes that these losses are critical for high-quality talking-face generation, especially for subtle but important behaviors such as side glances, asymmetrical lip shapes, and nontrivial coupling between pose and expression.

Holistic facial dynamics generation with a diffusion transformer

After the latent space is learned, the system extracts motion sequences from real talking-face videos and trains a diffusion model on them. The key modeling choice is to represent the motion sequence as a single latent stream that contains both head pose and facial dynamics:

$$ \mathbf{X} = \{[\mathbf{z}^{pose}_i, \mathbf{z}^{dyn}_i]\}_{i=1}^{W}. $$

The corresponding audio is converted to synchronized features with a pretrained Wav2Vec2 encoder, producing a sequence $\mathbf{A} = \{\mathbf{f}^{audio}_i\}$. The paper's generator is a transformer-based denoiser $\mathcal{H}$ trained with a simplified diffusion objective that predicts the clean latent directly:

$$ \mathbb{E}_{t \sim \mathcal{U}[1,T],\, \mathbf{X}^0,\, \mathbf{C} \sim q(\mathbf{X}^0, \mathcal{C})} \left[\|\mathbf{X}^0 - \mathcal{H}(\mathbf{X}^t, t, \mathbf{C})\|^2\right]. $$

The forward diffusion process is the standard Gaussian noising chain: $$ q(\mathbf{X}^t \mid \mathbf{X}^{t-1}) = \mathcal{N}(\mathbf{X}^t; \sqrt{1-\beta_t}\,\mathbf{X}^{t-1}, \beta_t \mathbf{I}). $$ Instead of modeling only audio-to-lip motion, the transformer learns the joint distribution of lip motion, non-lip facial expression, eye gaze, blinking, and head pose.

Conditioning signals and controllability

The primary conditioning signal is audio, but the method also supports three optional control variables: main gaze direction $\mathbf{g} = (\theta, \phi)$, head-to-camera distance $d$, and emotion offset $\mathbf{e}$. The gaze control specifies a desired looking direction, the distance control changes the scale of the face in the frame, and the emotion offset acts as a global adjustment rather than a full emotion override. The authors explicitly note that emotion is often inferable from speech, so $\mathbf{e}$ is meant to gently modulate the output rather than force implausible emotion-audio mismatches.

The full input condition used by the diffusion model is written as $\mathbf{C} = [\mathbf{X}^{pre}, \mathbf{A}^{pre}; \mathbf{A}, \mathbf{g}, d, \mathbf{e}]$, where the prefix terms provide the last few frames of the previous window to support smooth sliding-window generation. The conditions are concatenated with noise along the temporal dimension.

The paper uses classifier-free guidance during inference. If $\lambda_{\mathbf{c}}$ is the guidance scale for condition $\mathbf{c}$, then the predicted clean motion is $$ \hat{\mathbf{X}}^0 = \left(1 + \sum_{\mathbf{c} \in \mathbf{C}} \lambda_{\mathbf{c}}\right)\mathcal{H}(\mathbf{X}^t, t, \mathbf{C}) - \sum_{\mathbf{c} \in \mathbf{C}} \lambda_{\mathbf{c}}\,\mathcal{H}(\mathbf{X}^t, t, \mathbf{C}|_{\mathbf{c}=\emptyset}). $$

During training, each condition is randomly dropped to enable guidance at test time. The paper uses a drop probability of $0.1$ for each condition, except for the prefix conditions $\mathbf{X}^{pre}$ and $\mathbf{A}^{pre}$, which are dropped with probability $0.5$ so that the model can handle the first window when no history is available. The last few audio frames may also be dropped randomly to make the model robust when the input audio is shorter than the generation window.

Inference procedure

At test time, the system extracts $\mathbf{V}^{app}$ and $\mathbf{z}^{id}$ from the source portrait, computes audio features for the whole speech clip, splits the audio into segments, and generates motion latents window by window in a sliding manner. The final video is then synthesized by the decoder using the generated pose and dynamics latents together with the source appearance and identity.

Training setup and datasets

For face latent learning, the paper uses VoxCeleb2, which contains talking-face videos from about 6K subjects. The authors reprocess the dataset and discard clips with multiple people or low visual quality. For motion generation, they train on VoxCeleb2 plus an additional high-resolution talking-video dataset collected by the authors, which contains about 3.5K subjects. The total training set is about 500K clips, each 2 to 10 seconds long.

The face latent model is reported to have about 200M parameters and takes about 7 days to train on 4 NVIDIA RTX A6000 GPUs. The diffusion transformer has about 29M parameters and takes about 3 days to train. The diffusion network itself is an 8-layer transformer encoder with embedding dimension 512 and 8 attention heads.

The default inference setup uses a forward-facing main gaze condition, the average head distance over training videos, and an empty emotion offset. The paper sets the CFG weights to $\lambda_{\mathbf{A}} = 0.5$ and $\lambda_{\mathbf{g}} = 1.0$, and uses 50 sampling steps in the default configuration.

The authors also train the CAPP evaluation network on 2K hours of real-world audio-pose data, using 3-second windows. The audio encoder is initialized from a pretrained Wav2Vec2 model, and the pose encoder is a randomly initialized 6-layer transformer.

Evaluation protocol

The paper evaluates on two benchmarks. The first is a VoxCeleb2 subset: 46 subjects from the test split, with 10 clips per subject, for a total of 460 clips. These clips are 5 to 15 seconds long, with most under 10 seconds, and largely consist of interviews and news reports. The second benchmark, called OneMin-32, contains 32 one-minute clips of 17 individuals, sourced mostly from online coaching sessions and educational lectures. This second benchmark is intended to test longer-generation stability and more diverse speaking styles.

For quantitative evaluation, the paper reports:

• Audio-lip synchronization: SyncNet confidence score $S_C$ and feature distance $S_D$.
• Audio-pose alignment: CAPP, a contrastive audio-pose pretraining score inspired by CLIP-style paired representation learning.
• Pose variation intensity: $\Delta P$, the mean angle difference between adjacent head poses, as a measure of motion vividness.
• Video quality: Fréchet Video Distance over 25-frame clips, denoted $\mathrm{FVD}_{25}$.

The compared methods are MakeItTalk, Audio2Head, and SadTalker. For deterministic methods, the authors generate one video per audio input; for stochastic methods, they sample three videos per audio and average the metrics. Because the compared methods use different pose parameterizations, the paper re-extracts head poses from the generated frames before computing pose-based metrics.

Main quantitative results

Across both benchmarks, VASA-1 is reported to achieve the best results on all evaluated metrics. The gains are particularly strong for lip synchronization, pose alignment, and video quality on the longer OneMin-32 benchmark. The paper also reports that the model trained with only 10% of the data remains competitive, showing that the approach is not overly dependent on massive training data, although additional data improves motion diversity and video quality.

The paper highlights several patterns in these numbers. First, VASA-1 has substantially better lip synchronization than prior methods. Second, its pose alignment is also stronger, especially on the longer OneMin-32 benchmark. Third, the motion intensity $\Delta P$ is higher than competing methods, indicating more vivid head motion, though still below the intensity seen in real video. Finally, the FVD result on OneMin-32 is much lower than the baselines, showing a large gain in overall visual realism and temporal quality.

Interestingly, the reported lip-sync scores on generated outputs can exceed those measured on real videos. The authors attribute this to the effect of audio classifier-free guidance, which biases the model toward stronger alignment when enabled.

Qualitative behavior and controllability

Visually, the method is reported to produce not just mouth movements but also spontaneous-looking conversational cues such as gaze changes, blinking rhythms, and small facial expression shifts. The paper emphasizes that the model is able to retain the identity of the source portrait while transferring motion, and that the same motion sequence can be applied to different source images without collapsing identity.

The control variables work as intended: gaze direction changes the looking direction, head distance changes the apparent scale and camera proximity, and emotion offset nudges the output toward different emotional expressions. The paper notes that these controls are global and easy to interpret, which is valuable for interactive systems that need predictable user-facing knobs.

Identity and motion disentanglement

The paper presents examples showing that the same latent motion can be transferred across multiple identities while preserving each person's distinct appearance. This suggests that the latent construction does indeed separate the identity information from the motion variables sufficiently well for reenactment-like transfer.

Head pose and facial dynamics disentanglement

Another qualitative test isolates pose from facial dynamics. By holding one factor constant and varying the other, the model preserves the intended motion source without accidentally changing the other factor too much. This is the key property the added $l_{consist}$ loss is designed to enforce.

Out-of-distribution robustness

The paper also shows examples with non-photorealistic images, singing audio, and non-English speech. These are outside the training distribution, yet the system still produces convincing talking-face videos that remain synchronized with the input audio. The authors present this as evidence that the model has some robustness beyond the exact training domain, though they still frame the main use case as photorealistic conversational avatars.

Ablation analysis

Why CAPP is meaningful

The CAPP metric is intended to measure audio-pose synchronization in a data-driven way, rather than relying on ambiguous notions of speech "beats". The paper evaluates CAPP by perturbing ground-truth audio-pose pairs and checking whether the score decreases as alignment worsens.

These tests show that CAPP is sensitive to temporal misalignment and also reacts to changes in motion intensity, which supports its use as a pose-audio synchronization metric.

Classifier-free guidance trade-offs

The paper studies the audio CFG scale $\lambda_{\mathbf{A}}$, the gaze CFG scale $\lambda_{\mathbf{g}}$, and the number of sampling steps. Increasing $\lambda_{\mathbf{g}}$ improves gaze control accuracy, while increasing $\lambda_{\mathbf{A}}$ tends to improve lip synchronization and overall video quality, at the cost of slightly less stable or less natural pose behavior when pushed too far.

The default choice $\lambda_{\mathbf{A}}=0.5$ and $\lambda_{\mathbf{g}}=1.0$ reflects a balance between synchronization, controllability, and sample fidelity. The authors also report that reducing the diffusion sampling steps from 50 to 10 speeds up the motion-generation module by about 5$\times$, but with some loss in motion intensity and overall quality.

Effect of $l_{consist}$ and $l_{cross\_id}$

The qualitative ablation on $l_{consist}$ shows that without this term, the model has trouble separating subtle facial dynamics from pose. In particular, it may miss side glances or mouth asymmetries that are entangled with head orientation. For cross-identity reenactment, the paper reports that adding $l_{cross\_id}$ improves the identity-preservation score from 0.72 to 0.80 on a 5,400-clip evaluation built from all 108 VoxCeleb2 test subjects.

The paper also studies training-data scale. A model trained on only 10% of the motion data (about 50K clips) still performs strongly on synchronization and alignment, though FVD and motion intensity degrade relative to the full-data model. This suggests the method is effective even at smaller scale, while benefiting from more data for motion diversity.

Runtime and efficiency

One of the headline claims of the paper is that the method is not only high quality but also efficient enough for interactive use. The reported offline batch generation speed is 45 FPS at 512$\times$512 resolution, and the online streaming mode reaches up to 40 FPS with only about 170 ms of initial latency on a desktop PC with a single NVIDIA RTX 4090 GPU. The authors present this as a major advantage over prior diffusion-based talking-face systems, which generally offer strong quality but are much slower.

Limitations and responsible use

The authors explicitly list several limitations. First, the model currently handles human regions only up to the torso, so extending it to full upper-body generation would require additional modeling. Second, although the method uses 3D latent representations, it does not rely on a more explicit 3D face model, which can lead to artifacts such as texture sticking caused by neural rendering. Third, the current system does not explicitly model non-rigid elements such as hair and clothing, which could benefit from a stronger video prior. Finally, the authors note that more diverse talking styles and emotions would improve expressiveness and controllability in future versions.

The appendix also discusses responsible AI concerns. The paper states that the method is intended for positive avatar applications and not for deception, but it acknowledges misuse risks such as impersonation. To help with misuse detection, the authors trained a neural-network detector that distinguishes real videos from generated videos with 97.8% accuracy. They also mention ongoing work on using generated talking-face data to improve forgery detection models.

Conclusion

VASA-1 combines an expressive disentangled face latent space with a diffusion transformer for audio-conditioned holistic motion generation. According to the paper, this design substantially improves lip synchronization, head motion naturalness, and overall realism while also enabling real-time synthesis. The main conceptual advance is the shift from separately regressing different motion components to modeling the whole talking-face behavior jointly in a latent space that can still be controlled by audio and simple interpretable signals.

For talking-head and conversational-AI systems, the paper is significant because it moves toward avatars that are not just intelligible, but socially and visually convincing: they maintain identity, speak in sync, and exhibit the kinds of micro-behaviors that people associate with genuine interaction.