SkyReels-A1

SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers

SkyReels-A1 animates static portraits by transferring facial expressions and body motion with strong identity preservation and temporal coherence. It uses 3D expression-aware landmarks and a novel identity alignment to overcome challenges like distortion and unrealistic dynamics in portrait animation.

• portrait-animation
• face-animation
• face-reenactment
• talking-face
• head-pose
• full-body
• avatar

Authors: Di Qiu, Zhengcong Fei, Rui Wang, Jialin Bai, Changqian Yu, Mingyuan Fan, Guibin Chen, Xiang Wen

Categories: cs.CV

Published 2025-02-15 · Updated 2025-02-15

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub584 stars11 watchers61 forksOther (Skywork Community License)
• Hugging Face28 downloads67 likes

Impact

• Semantic Scholar39 citations4 influential citations

Abstract

We present SkyReels-A1, a simple yet effective framework built upon video diffusion Transformer to facilitate portrait image animation. Existing methodologies still encounter issues, including identity distortion, background instability, and unrealistic facial dynamics, particularly in head-only animation scenarios. Besides, extending to accommodate diverse body proportions usually leads to visual inconsistencies or unnatural articulations. To address these challenges, SkyReels-A1 capitalizes on the strong generative capabilities of video DiT, enhancing facial motion transfer precision, identity retention, and temporal coherence. The system incorporates an expression-aware conditioning module that enables seamless video synthesis driven by expression-guided landmark inputs. Integrating the facial image-text alignment module strengthens the fusion of facial attributes with motion trajectories, reinforcing identity preservation. Additionally, SkyReels-A1 incorporates a multi-stage training paradigm to incrementally refine the correlation between expressions and motion while ensuring stable identity reproduction. Extensive empirical evaluations highlight the model's ability to produce visually coherent and compositionally diverse results, making it highly applicable to domains such as virtual avatars, remote communication, and digital media generation.

---

1. Problem Setting and Core Idea

SkyReels-A1 addresses portrait image animation: generating a temporally coherent video of a person from a single static reference portrait while transferring motion from a driving video. The paper is motivated by recurring failure modes in prior portrait animation systems, especially when the task moves beyond clean head-only reenactment to more diverse body proportions. The authors emphasize three major challenges: identity distortion, background instability, and unrealistic facial dynamics. They argue that these issues become more pronounced when the animated subject occupies a half-body or full-body composition rather than a tightly cropped face.

The proposed solution is a video diffusion Transformer (DiT) framework that uses expression-aware motion conditioning and a lightweight facial identity alignment path. The central claim is that a strong video DiT prior can better integrate facial details, body motion, and background coherence than prior warping-centric or UNet-based approaches, provided that the motion representation and identity conditioning are designed carefully.

2. Claimed Contributions

• A portrait animation framework built on a video diffusion Transformer, aimed at improving motion transfer precision, identity retention, and temporal coherence.
• An expression-aware conditioning pipeline that uses facial expression-aware landmarks as motion descriptors, intended to better capture subtle facial motions such as eyebrow raising, lip curvature, eye gaze, and other micro-expressions.
• A facial image-text alignment module that maps facial image embeddings into the text feature space, allowing the base model to reuse its pretrained text-conditioning pathway for identity preservation with relatively few additional parameters.
• A three-stage training strategy that progressively learns motion control, identity preservation, and joint refinement.

3. Architecture Overview

SkyReels-A1 is built on top of a pretrained video diffusion Transformer, specifically described as being based on CogVideoX-5B. The model operates in latent space using a 3D variational autoencoder (VAE). The core DiT block contains 3D spatial-temporal self-attention and a feed-forward network, and the text embedding pathway uses a T5 encoder in the general formulation of the underlying framework.

The system takes two inputs: a driving video clip and a reference portrait image $I_{\mathrm{ref}}$. The driving video is used to extract facial expression-aware landmarks, which serve as motion descriptors. These landmarks are encoded and injected into the latent diffusion model as conditioning signals. In parallel, the reference portrait is used to extract facial identity information that is aligned to the model’s text-conditioning space.

A simplified view of the model is:

• Motion path: extract expression-aware landmarks from the driving video; encode them with a VAE-based pose guider and a 3D causal encoder; inject the resulting latent motion signal into the DiT input stream.
• Identity path: crop the face region from the reference portrait with a face extractor $\mathcal{E}_{\mathrm{face}}$, encode it using a vision encoder $\mathcal{E}_{\mathrm{vision}}$ (SigLIP), and project it into the text feature space via a learnable MLP $\mathcal{P}$.
• Generation path: the DiT denoises the latent video conditioned on motion and identity information.

3.1 Diffusion and Conditioning Formulation

The paper reviews standard diffusion training in latent space. Given latent variables $\mathbf{x}_t$ at diffusion step $t$, the reverse process is modeled as

$$p_{\theta}(\mathbf{x}_t \mid \mathbf{x}_{t+1}) = \mathcal{N}(\mathbf{x}_t; \widetilde{\mu}_t, \widetilde{\beta}_t \mathbf{I}).$$

Training follows the common noise-prediction objective:

$$\mathcal{L}_{\mathrm{noise}} = w \cdot \mathbb{E}_{t, c_{\mathrm{txt}}, \epsilon \sim \mathcal{N}(0,1)}\left[\left\|\epsilon - \epsilon_{\theta}(\mathbf{x}_t, t, c_{\mathrm{txt}})\right\|^2\right].$$

In the controllable extension used by SkyReels-A1, the denoiser becomes conditional on both text and image-derived features: $\epsilon_{\theta}(\mathbf{x}_t, t, c_{\mathrm{txt}}, c_{\mathrm{img}})$, where $c_{\mathrm{img}} = \tau_{\mathrm{img}}(r)$ is the visual condition extracted from a reference image $r$.

3.2 Expression-Aware Landmarks

A key design choice is the use of 3D facial expression-aware landmarks rather than plain 2D facial landmarks. The authors argue that 2D landmarks often suffer from expression-reference misalignment at inference time, especially when the target expression differs strongly from the source or when the driving face is not frontal. They also note that generic 3D keypoints obtained from third-party tools may be too coarse to capture fine-grained facial dynamics.

To address this, they adopt a 3D facial expression framework with a neural rendering module, replacing a conventional differentiable rendering approach. The stated goal is to recover higher-precision 3D keypoints and better encode nuanced motion, so that subtle expression changes remain consistent under strong pose changes and in more complex body compositions.

3.3 Landmark Guider

The landmark guider is described as a spatio-temporal alignment module built around a 3D causal encoder. Its job is to project the motion signal and the video latent into a shared latent representation, improving synchronization between the driving motion and the generated frames. The encoder is carefully fine-tuned so it captures motion representation more effectively while preserving temporal coherence.

Conceptually, this is a direct latent-space alignment mechanism rather than explicit warping. The paper positions it as one of the reasons the model can better preserve motion realism without the artifacts sometimes caused by landmark-to-image warping pipelines.

3.4 Facial Image-Text Alignment Module

For identity conditioning, the paper avoids heavy cross-attention adapters and instead leverages the pretrained text pathway. The reference portrait is cropped to the face region:

$$I_{\mathrm{face}} = \mathcal{E}_{\mathrm{face}}(I_{\mathrm{ref}}).$$

A SigLIP vision encoder then produces a facial identity embedding:

$$F_{\mathrm{id}} = \mathcal{E}_{\mathrm{vision}}(I_{\mathrm{face}}).$$

This embedding is mapped into the text feature space with a lightweight MLP:

$$F'_{\mathrm{id}} = \mathcal{P}(F_{\mathrm{id}}).$$

The stated motivation is twofold: first, the face encoder captures detailed identity cues; second, projecting into the text embedding space allows the method to reuse the pretrained DiT backbone without adding a large number of extra parameters. The paper explicitly notes that this approach is easier to train than cross-attention-based identity adapters.

4. Training Strategy

SkyReels-A1 uses a three-stage progressive training pipeline designed to decouple motion learning from identity learning before combining them in a final joint refinement stage.

4.1 Stage 1: Motion-Driven Training

In the first stage, the goal is to inject motion control into the video generation process. The 3D landmark guider is initialized from a pretrained 3D causal encoder but remains frozen. The landmark latent is concatenated with the noise input of the image-to-video backbone. To adapt the model to landmark-conditioned generation while keeping the base image-to-video capabilities intact, only the convolutional layers inside the PatchEmbedding module are trained.

4.2 Stage 2: Identity-Preserving Training

The second stage focuses on identity consistency. The text branch is retained even though portrait animation does not inherently require text, because the authors want to preserve and reuse pretrained multimodal capabilities. Facial features are extracted with the vision encoder and projected into the text feature space using a trainable linear mapping. During this stage, only the projection layer is optimized; the rest of the model remains fixed.

The manuscript also mentions mask-based supervision to encourage identity consistency across expressions and motion, but the only explicit loss formula provided is the face-aware reweighting loss described below.

4.3 Stage 3: Multi-Module Joint Fine-Tuning

In the final stage, the 3D landmark guider, the DiT block, and the linear projection layer are jointly optimized. This stage is intended to tighten the coupling among motion, facial identity, and the generative prior, and to improve generalization to diverse portraits and motion styles.

4.4 Face-Aware Loss

The paper introduces a face-aware loss that increases the contribution of high-motion regions such as the face. Optical flow is estimated using RAFT. The flow magnitude is averaged to obtain a threshold, and pixels with flow above this threshold are treated as foreground motion regions. A normalized motion mask is then used to reweight the per-pixel noise-prediction error.

The loss is written as:

$$L_{\mathrm{face}} = \frac{1}{N H' W'} \sum_{i=1}^{N} \sum_{x=1}^{H'} \sum_{y=1}^{W'} \mathcal{M}_{i,\mathrm{norm}} \cdot \left[\epsilon_i(x,y) - \hat{\epsilon}_i(x,y)\right]^2,$$

where $\epsilon_i(x,y)$ and $\hat{\epsilon}_i(x,y)$ are the target and predicted noise at location $(x,y)$ for frame $i$, and $H'$ and $W'$ are the latent spatial dimensions. The intent is to emphasize reconstruction quality in visually important, highly dynamic regions.

5. Data, Filtering, and Implementation Details

The model is trained on a combination of public datasets and a collected internet dataset. The paper lists the following sources: NeRSemble, HDTF, DFEW, RAVDESS, Panda70M, and approximately 1W character video clips collected from the internet.

The preprocessing pipeline consists of three steps: single-character extraction, motion filtering, and post-processing. The authors first clean the videos to isolate single-character clips and reduce camera artifacts and background noise. They then detect facial keypoints with MediaPipe to estimate head angles and mouth variation, keeping only clips with significant head motion and expressive facial movement. Finally, videos are cropped or padded to a fixed resolution of $480 \times 720$. A random frame is selected from each clip, and the face is encoded into an embedding with the CLIP/SigLIP-style encoder used by the identity branch.

Implementation details reported by the paper are as follows:

• Base model: CogVideoX-5B.
• Training schedule: 2K steps in stage 1, 2K steps in stage 2, and 1K steps in stage 3.
• Batch size: 512.
• Learning rate: $10^{-5}$ for the first two stages and $10^{-6}$ for the final stage.
• Optimizer: AdamW.
• Hardware: 32 NVIDIA A800 GPUs.
• Inference: DDIM sampler with classifier-free guidance scale set to 3.
• Reference portraits: generated by Flux and sourced from Pexels, as stated in the paper.

6. Evaluation Protocol

The experiments are set up as a cross-identity motion transfer task. A reference portrait is randomly chosen from 100 in-the-wild images, and the driving sequence comes from the test dataset. The authors compare against three baselines: Follow-Your-Emoji, LivePortrait, and the commercial system Act-One.

The reported metrics are:

• Identity preservation: FaceSim-Arc and FaceSim-Cur, both based on cosine similarity between source and generated images.
• Image quality: FID computed using a pretrained evaluation network.
• Motion accuracy: L1 differences between extracted facial blendshapes and head poses of the driving and generated frames, using FaceAnalysis and OpenPose.

7. Quantitative Results

The paper reports the following comparison on portrait animation quality and motion fidelity. Lower is better for image quality, expression distance, and pose distance; higher is better for identity similarity.

The clearest quantitative advantage reported for SkyReels-A1 is image quality, where it achieves the best FID-like score among the listed methods. On identity similarity, the model is competitive with Act-One but does not surpass it in the reported numbers. Likewise, its expression and pose distances are strong but slightly behind Act-One in the table. The paper nevertheless frames the overall system as a substantial advance because it produces coherent animations with strong visual fidelity and better generalization to varied body proportions.

8. Qualitative Findings

The qualitative results emphasize three recurring strengths. First, the model can transfer mouth motion and eye gaze with relatively fine detail. Second, it preserves the appearance of the source identity while still following the driver’s motion. Third, it appears to remain stable when the subject is shown in more challenging full-body or smaller-face compositions, helped by the stitching behavior referenced in the captions and discussion.

In the manuscript, Figure 2 highlights general animation quality from a static portrait input, with emphasis on seamless stitching and control over eyes and lips. Another result figure shows examples under diverse body proportions, reinforcing the paper’s main claim that the DiT-based design is better suited to varying compositions than methods designed mainly for head-only reenactment.

9. User Study

The paper includes a user study with 20 participants from diverse geographic regions. Participants evaluated synthesized videos using the model names explicitly shown to them, and they judged two aspects: motion accuracy and human likeness. Across 100 collected responses, 63% preferred SkyReels-A1. This is presented as evidence that users find the generated videos more faithful to the driving motion and more convincing in perceived identity than the baselines.

10. Limitations

The paper states a clear limitation: despite strong overall performance, the current model still has difficulty with extreme pose variations. This is identified as an open direction for future work. The rest of the manuscript does not provide a more detailed failure taxonomy or a dedicated ablation table, so the main explicit limitation is the robustness gap under very challenging viewpoint changes.

11. Overall Assessment

SkyReels-A1 is presented as a pragmatic DiT-based portrait animation system that combines a stronger motion representation, lightweight identity conditioning, and staged training to improve realism and stability. Its main technical emphasis is not on a new diffusion objective, but on how to adapt a large video generative backbone to portrait animation with better expression transfer and fewer identity artifacts. Based on the reported experiments, the method is especially compelling for visual quality and for handling more compositional variety than head-only systems, while still leaving room for improvement in extreme pose scenarios and in matching the strongest commercial baseline on some motion-fidelity measures.