AniPortrait

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

AniPortrait generates photorealistic talking-head videos from audio and a reference portrait through a two-stage process: audio to 3D facial landmarks, then diffusion-based video synthesis. It offers high-quality, temporally coherent animations with controllable facial motion and supports editing and face reenactment.

• audio-driven
• speech-driven
• talking-head
• portrait-animation
• face-animation
• head-pose

Demos

AniPortrait demonstrates high-quality photorealistic portrait animations driven by audio input and reference images or videos, exemplifying both self-driven animations and face reenactment. Key evaluations include natural lip synchronization, realistic head pose control, and fidelity to the source portrait's identity and expression. The pipeline diagram illustrates the progressive stages enabling these results, highlighting the model's diverse driving modalities and pose retargeting capability.

Authors: Huawei Wei, Zejun Yang, Zhisheng Wang

Categories: cs.CV, cs.GR, eess.IV

Published 2024-03-26 · Updated 2024-03-26

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub5,020 stars60 watchers617 forksApache-2.0
• Hugging Face0 downloads124 likes

Abstract

In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait

---

Introduction

AniPortrait addresses audio-driven portrait animation: given an audio clip and a single reference portrait image, the goal is to synthesize a video in which lip motion, facial expression, and head pose remain synchronized with the speech while preserving the identity and appearance of the reference subject. The paper positions this problem as difficult because realistic portrait animation requires both high visual fidelity and temporal consistency, and because prior generators based on GANs, NeRFs, or motion decoders often struggle with generalization and stability.

The core idea is to split the problem into two stages. First, audio is converted into an intermediate facial representation: a sequence of $3$D facial meshes and head poses, which are then projected into $2$D facial landmarks. Second, these landmarks are used to condition a diffusion-based video generator that produces a photorealistic, temporally coherent portrait animation. The authors emphasize that using an explicit intermediate representation improves controllability and enables downstream editing or reenactment.

The paper’s main design choice is to combine audio-to-landmark prediction with a landmark-conditioned diffusion video synthesis stage. This is intended to preserve the semantic structure of speech-driven motion in an interpretable form while leveraging the image quality and generative robustness of diffusion models.

Overview of the Two-Stage Framework

The overall system is composed of two modules: Audio2Lmk and Lmk2Video. Audio2Lmk maps a speech sequence $\mathcal{A}^{1:T} = (\mathbf{a}^1, \ldots, \mathbf{a}^T)$ to a sequence of $3$D facial meshes $\mathcal{M}^{1:T} = (\mathbf{m}^1, \ldots, \mathbf{m}^T)$ and head poses $\mathcal{P}^{1:T} = (\mathbf{p}^1, \ldots, \mathbf{p}^T)$. Lmk2Video then consumes the resulting landmark sequence $\mathcal{L}^{1:T} = (\mathbf{l}^1, \ldots, \mathbf{l}^T)$ together with a reference portrait image $I_{ref}$ and synthesizes a video sequence $\mathcal{I}^{1:T} = (\mathbf{I}^1, \ldots, \mathbf{I}^T)$.

A notable property of the design is that the facial motion is first expressed in a compact geometric form and only later rendered into pixels. This makes the motion easier to control than a direct end-to-end audio-to-video approach. The paper also notes that the two stages are trained within the same overall framework, although their optimization procedures differ.

Method

Audio2Lmk: audio to $3$D facial geometry and pose

In Audio2Lmk, the input is a sequence of speech snippets $\mathcal{A}^{1:T}$. The target is to predict both a facial mesh sequence and a pose sequence. Each mesh $\mathbf{m}^t$ lies in $\mathbb{R}^{N \times 3}$, meaning that the model predicts $N$ mesh vertices in $3$D at each time step. Each pose vector $\mathbf{p}^t$ is $6$-dimensional and encodes rotation and translation.

The paper uses a pre-trained wav2vec backbone to extract speech features. The rationale is that wav2vec captures pronunciation and intonation in a way that generalizes well across audio inputs. For the audio-to-mesh path, these features are passed through a very simple predictor consisting of two fully connected layers. The authors explicitly note that this lightweight design is sufficient and efficient for inference.

The audio-to-pose path uses the same wav2vec family of features, but the weights are not shared with the audio-to-mesh module. The paper argues that pose depends more on rhythm and tone than on the fine-grained phonetic content needed for mouth shape. To model temporal dependence, a transformer decoder is used, and the audio features are injected through cross-attention.

Both audio-to-mesh and audio-to-pose are trained with a simple $L_1$ loss. After predicting the $3$D mesh and pose, the method applies perspective projection to obtain $2$D facial landmarks. These landmarks become the condition for the second stage.

In effect, this stage turns raw speech into an interpretable geometric control signal. Because the intermediate representation is explicit, it can later be edited or reused for related tasks such as reenactment.

Lmk2Video: landmark-conditioned photorealistic video synthesis

The second stage takes a reference portrait image $I_{ref}$ and landmark sequence $\mathcal{L}^{1:T}$ and produces the final portrait video. The architecture is inspired by AnimateAnyone and uses Stable Diffusion 1.5 as the image-generation backbone, augmented with a temporal motion module to transform multi-frame noise into coherent video frames. A ReferenceNet, mirroring the backbone structure, extracts appearance information from the reference image so that identity and visual style remain consistent across frames.

A key contribution of the paper is the redesign of the pose-guidance pathway. The authors observe that a simple PoseGuider with only a few convolution layers and a late fusion of landmark features is not expressive enough to capture subtle lip dynamics. To address this, they adopt a ControlNet-like multi-scale strategy: landmark features at corresponding scales are injected into different blocks of the diffusion backbone. This retains a lightweight design while improving the network’s ability to model fine facial motion.

The paper further adds the reference image’s landmark as an additional input. The PoseGuider uses cross-attention between the reference landmarks and the per-frame target landmarks, providing extra cues about how facial geometry relates to appearance. The stated purpose is to help the model generate more accurate motion, especially around the mouth region.

Conceptually, the second stage can be viewed as learning a conditional video distribution of the form $$p(\mathcal{I}^{1:T} \mid I_{ref}, \mathcal{L}^{1:T}),$$ where the landmarks constrain motion and the reference image constrains identity. The motion module is responsible for temporal coherence, while the diffusion backbone and ReferenceNet support high visual fidelity.

Training and Implementation Details

Audio2Lmk training

For Audio2Lmk, the authors use wav2vec$2.0$ as the backbone. They annotate the training data using MediaPipe to extract $3$D meshes and $6$D poses. The audio-to-mesh model is trained on an internal dataset consisting of nearly one hour of high-quality speech data from a single speaker. To make MediaPipe’s $3$D mesh extraction stable, the speaker is instructed to keep a steady head position and face the camera throughout recording.

Audio2Pose is trained on HDTF. Optimization uses Adam with a learning rate of $10^{-5}$. Training is performed on a single A100 GPU.

Lmk2Video training

For the video synthesis stage, the paper uses a two-step training procedure. In the first step, the model trains the $2$D component of the backbone, the ReferenceNet, and the PoseGuider, while the temporal motion module is excluded. In the second step, all other components are frozen and the motion module alone is trained. The authors state that this staged procedure is used to stabilize learning.

Training data for Lmk2Video comes from two large-scale facial video datasets: VFHQ and CelebV-HQ. The videos are processed with MediaPipe to extract $2$D landmarks. To make the model more sensitive to articulation around the mouth, the upper and lower lips are rendered in different colors in the pose image derived from landmarks. All images are resized to $512 \times 512$.

The video model is trained on $4$ A100 GPUs, with two days spent on each of the two training steps. The optimizer is AdamW with a learning rate of $10^{-5}$.

Experiments and Qualitative Results

The paper’s experimental section is qualitative rather than benchmark-heavy in the provided text. The main reported result is that AniPortrait produces portrait animations that are visually realistic, temporally stable, and expressive. The authors claim improvements in facial naturalness, pose diversity, and overall visual quality compared with prior approaches.

A central advantage of the method is that it operates through an intermediate $3$D representation. The paper highlights that this allows motion editing: one can extract landmarks from a source and alter the identity to obtain a face reenactment effect. This makes the representation useful beyond direct speech-to-video synthesis.

The paper’s results figure is used to support the claim that the generated animations are striking in both quality and realism. Although the provided text does not report numerical metrics, an explicit comparison table, or detailed ablation scores, the narrative emphasizes that the diffusion-based rendering stage yields convincing appearance quality while the landmark conditioning provides motion controllability.

Design Rationale and What Is Novel

The main novelty of AniPortrait is not a single architectural trick, but a careful combination of representation choices and generation stages:

• Intermediate geometric control: speech is first transformed into $3$D facial meshes and head pose, then projected to $2$D landmarks.
• Separate modeling of facial content and pose: mesh prediction and pose prediction use different heads, with pose explicitly modeled by a transformer decoder.
• Diffusion-based rendering: landmark sequences are rendered to video with Stable Diffusion 1.5 plus a temporal motion module, improving realism and temporal consistency.
• Multi-scale pose guidance: landmark information is injected at multiple scales rather than only at the input, addressing fine mouth motion.
• Reference landmark interaction: the model uses the reference portrait’s landmark as additional input and cross-attends to it, giving more structure to motion generation.

These choices align the system with a broader trend in controllable generation: use explicit, editable motion representations to steer a powerful generative backbone rather than asking the backbone to discover all motion semantics from scratch.

Limitations and Future Work

The paper is unusually direct about limitations. First, the method depends on intermediate $3$D representations, and obtaining large-scale, high-quality $3$D data is expensive. Second, despite the improved realism, the authors acknowledge that the generated facial expressions and head poses still cannot fully avoid an uncanny-valley effect. This is important for conversational-AI or talking-head systems, because small artifacts in the mouth or eyes can be highly noticeable.

The stated future direction is to move away from the intermediate $3$D representation and predict portrait videos directly from audio, following the direction of EMO. The motivation is to achieve even more impressive generation quality by avoiding the bottleneck of explicit 3D supervision.

Takeaways for Talking-Head / Conversational AI Systems

For teams building talking-head systems, AniPortrait is most relevant as a demonstration that a two-stage, geometry-then-rendering pipeline can combine controllability with strong visual quality. The first stage provides interpretable motion signals that can be edited, synchronized, or reused; the second stage uses a modern diffusion video generator to improve photorealism.

The paper also highlights a practical engineering pattern: if a simple conditioning mechanism underperforms on lip motion, multi-scale conditioning and reference-aware attention can strengthen local articulatory fidelity without requiring a large increase in parameter count. In short, AniPortrait is designed to be both expressive and controllable, with the main trade-off being the need for intermediate 3D supervision and the remaining gap to perfectly natural facial motion.

Conclusion

AniPortrait proposes a diffusion-based framework for audio-driven portrait animation that first predicts $3$D facial mesh and pose from speech, converts them into $2$D landmarks, and then renders a temporally stable portrait video conditioned on those landmarks and a reference image. The paper’s contribution is a controllable, interpretable pipeline that leverages diffusion models for realism while preserving explicit motion structure for editing and reenactment. Its limitations center on the cost of $3$D supervision and the fact that generated expressions still show some uncanny-valley effects, motivating future work toward direct audio-to-video generation.