LivePortrait

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

LivePortrait offers an efficient and controllable system for animating a static portrait image using video-driven motion transfer, uniquely combining large-scale mixed data training and lightweight retargeting modules to achieve real-time performance and high-quality results competitive with diffusion-based methods.

• talking-head
• portrait-animation
• face-animation
• face-reenactment
• head-pose
• blendshape
• realtime
• low-latency

Demos

The demos showcase LivePortrait's capability to efficiently animate both human and animal portraits with high fidelity and expressive detail. Watch for smooth, realistic motion transfer from driving videos to source portraits, and pay attention to the stitching and retargeting controls that preserve identity and enhance animation quality. The examples demonstrate versatility across species and reflect the system's potential for real-time applications and creative portrait manipulation.

Authors: Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, Di Zhang

Categories: cs.CV

Published 2024-07-03 · Updated 2025-02-28

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub18,663 stars147 watchers1,942 forksMIT
• Hugging Face0 downloads0 likes

Impact

• Semantic Scholar225 citations48 influential citations

Abstract

Portrait Animation aims to synthesize a lifelike video from a single source image, using it as an appearance reference, with motion (i.e., facial expressions and head pose) derived from a driving video, audio, text, or generation. Instead of following mainstream diffusion-based methods, we explore and extend the potential of the implicit-keypoint-based framework, which effectively balances computational efficiency and controllability. Building upon this, we develop a video-driven portrait animation framework named LivePortrait with a focus on better generalization, controllability, and efficiency for practical usage. To enhance the generation quality and generalization ability, we scale up the training data to about 69 million high-quality frames, adopt a mixed image-video training strategy, upgrade the network architecture, and design better motion transformation and optimization objectives. Additionally, we discover that compact implicit keypoints can effectively represent a kind of blendshapes and meticulously propose a stitching and two retargeting modules, which utilize a small MLP with negligible computational overhead, to enhance the controllability. Experimental results demonstrate the efficacy of our framework even compared to diffusion-based methods. The generation speed remarkably reaches 12.8ms on an RTX 4090 GPU with PyTorch. The inference code and models are available at https://github.com/KwaiVGI/LivePortrait

---

Introduction

LivePortrait is a video-driven portrait animation system designed to synthesize a realistic animated portrait from a single source image while preserving identity, transferring motion from a driving video, and remaining fast enough for practical interactive use. The paper positions itself against two dominant families of prior work: diffusion-based approaches, which are visually strong but expensive and often less precise to control, and earlier implicit-keypoint warping methods, which are efficient and controllable but typically limited in generalization and fine-grained expression control. LivePortrait’s central thesis is that the implicit-keypoint framework can be pushed much further with better data, a stronger architecture, improved motion parameterization, and lightweight control modules.

The paper’s main contributions are threefold. First, it substantially upgrades an implicit-keypoint portrait animation backbone using large-scale data, mixed image-video training, a new motion transform with explicit scaling, a landmark-guided keypoint optimization term, and cascaded reconstruction/adversarial losses. Second, it introduces the idea that compact implicit keypoints behave like a kind of implicit blendshape basis, and uses this to build a stitching module plus eyes and lip retargeting modules with negligible overhead. Third, it demonstrates strong empirical performance on self-reenactment and cross-reenactment, with a reported PyTorch inference speed of 12.8 ms on an RTX 4090.

Problem setting and positioning

The task is portrait animation: given a source portrait image $I_s$ that provides appearance and a driving video sequence that provides motion, synthesize a target video in which facial expressions, head pose, and related motion signals are transferred to the source identity. The paper explicitly focuses on video-driven animation, while noting that portrait animation can also be driven by audio, text, or generation in broader settings. LivePortrait chooses not to use diffusion as its core generation engine. Instead, it extends a warping-based implicit-keypoint formulation so that the method retains the low computational cost of earlier non-diffusion systems while improving generalization, controllability, and perceptual quality.

In the related-work framing, the authors distinguish the method from diffusion-based systems such as FADM, Face Adapter, AniPortrait, X-Portrait, and MegActor, which tend to be slower and less precise for controllable motion editing. They also position it relative to earlier implicit-keypoint methods such as FOMM, MRAA, Face Vid2vid, IWA, TPSM, DaGAN, and MCNet. The paper’s claimed advantage is that LivePortrait preserves the efficiency and explicit control surface of implicit-keypoint methods while approaching or exceeding the visual quality of diffusion-based competitors in several metrics.

Method overview

LivePortrait is built as a two-stage training system on top of the Face Vid2vid-style implicit-keypoint pipeline. The first stage trains a strong base model for motion transfer and image synthesis. The second stage freezes the base components and learns small MLP-based control modules for stitching and retargeting. At inference time, the modules can be used independently or combined additively through a simple indicator-based control scheme.

Core representation: implicit keypoints as motion carriers

The method uses canonical implicit keypoints and motion parameters to describe the source and driving frames. In the original Face Vid2vid formulation, the source and driving 3D keypoints are written as

$$x_s = x_{c,s} R_s + \delta_s + t_s, \qquad x_d = x_{c,s} R_d + \delta_d + t_d,$$

where $x_{c,s}$ are canonical keypoints, $R$ is pose, $\delta$ is expression deformation, and $t$ is translation. LivePortrait modifies this transformation to include explicit scale factors:

$$x_s = s_s \cdot (x_{c,s} R_s + \delta_s) + t_s, \qquad x_d = s_d \cdot (x_{c,s} R_d + \delta_d) + t_d.$$

The paper argues that leaving scale implicit makes scale effects leak into expression deformation, which increases learning difficulty and can induce texture flickering when transferring across identities. By separating scale from expression deformation, the model trades some flexibility for better drivability and stability.

During inference, the driving keypoints are further transformed relative to the source identity and a reference driving frame:

$$x_{d,i} = s_s \cdot \frac{s_{d,i}}{s_{d,0}} \cdot \big(x_{c,s}(R_{d,i}R_{d,0}^{-1}R_s) + (\delta_s + \delta_{d,i} - \delta_{d,0})\big) + (t_s + t_{d,i} - t_{d,0}).$$

This formulation is intended to normalize motion over time, preserve scale consistency, and stabilize reenactment across frames.

Stage I: base model training

The first stage strengthens the backbone using a large-scale and carefully curated training corpus. The authors use public video datasets such as VoxCeleb, MEAD, and RAVDESS, plus the styled image dataset AAHQ. They also collect a large internal set of 4K portrait videos, about 200 hours of talking-head videos, private LightStage data, and additional styled portrait videos and images. Videos are split into clips shorter than 30 seconds, face tracking and recognition ensure one person per clip, and KVQ filtering removes low-quality clips. The resulting training data contains about 69 million frames from about 18.9K identities and 60K static styled portraits before/after filtering as described in the paper.

A particularly important design choice is mixed image-video training. The authors observe that realistic video data alone does not generalize well to styled portraits such as anime. Because styled portrait videos are scarce while styled portrait images are plentiful, the model treats single images as one-frame video clips and trains jointly on images and videos. This lets the model learn from the diversity of the static portrait corpus while retaining motion learning from video.

The original Face Vid2vid submodules for canonical keypoint detection, head pose estimation, and expression deformation estimation are unified into a single motion module $\mathcal{M}$, implemented with a ConvNeXt-V2-Tiny backbone. The generator is upgraded to a SPADE decoder, which the authors consider more powerful than the original decoder. A PixelShuffle layer is appended so the output resolution can be upsampled from $256\times256$ to $512\times512$.

Training objective design is also expanded. In addition to the original Face Vid2vid losses—equivariance loss $\mathcal{L}_E$, keypoint prior $\mathcal{L}_L$, head-pose loss $\mathcal{L}_H$, and deformation prior loss $\mathcal{L}_\Delta$—LivePortrait adds cascaded perceptual and GAN losses on global, face, and lip regions, plus a face-identity loss $\mathcal{L}_{\text{faceid}}$. The landmark-guided keypoint optimization loss $\mathcal{L}_{\text{guide}}$ uses 10 selected eye and lip landmarks with Wing loss to encourage more precise learning of micro-expressions such as eye motion and winking.

The overall stage-I objective is

$$\mathcal{L}_{\text{base}} = \mathcal{L}_E + \mathcal{L}_L + \mathcal{L}_H + \mathcal{L}_\Delta + \mathcal{L}_{P,\text{cascade}} + \mathcal{L}_{G,\text{cascade}} + \mathcal{L}_{\text{faceid}} + \mathcal{L}_{\text{guide}}.$$

Stage II: stitching and retargeting

The second stage is the paper’s key controllability contribution. The authors observe that compact implicit keypoints can function like a learned expression basis, analogous to a blendshape set. Instead of explicitly designing a large control system, they show that a small MLP can learn useful deformation offsets with negligible computational overhead. They therefore train three lightweight modules: a stitching module, an eyes retargeting module, and a lip retargeting module.

The stitching module is intended to align the animated portrait back into the original image space, especially around the shoulder region, so pasted-back results do not exhibit visible misalignment. This is crucial when the original image is larger than the cropped face region, and it also enables multi-person portraits or full-body images to be handled more gracefully. During stage-II training, the model deliberately uses cross-identity driving motions for stitching, making the task harder and improving generalization.

The stitching module $\mathcal{S}$ takes source and driving keypoints and predicts a deformation offset $\Delta_{st}$, after which the driving keypoints become $x'_{d,st}=x_d+\Delta_{st}$. The predicted image is compared with a self-reconstruction image, and the loss is a consistency term outside the shoulder mask plus an $L_1$ regularizer:

$$\mathcal{L}_{st} = \| (I_{p,st} - I_{p,recon}) \odot (1 - M^{st}(I_s)) \|_1 + w^{st}_{reg} \|\Delta_{st}\|_1.$$

The retargeting modules operate similarly. The eyes module receives the source keypoints $x_s$, a source eye-open condition tuple $c_{s,eyes}$, and a target scalar $c_{d,eyes} \in [0,0.8]$. The lip module receives $x_s$, $c_{s,lip}$, and a target scalar $c_{d,lip} \in [0,0.8]$. Each module predicts a deformation offset that modifies the keypoints so the generated face matches the desired eye-open or lip-open extent. Their losses include a masked consistency term, a condition-matching term that pushes the predicted condition toward the target scalar, and an $L_1$ regularizer on the offset.

During training, the predicted condition is forced toward the sampled target using a term of the form $\|c^p - c_d\|_1$. The paper reports that the chosen target range $[0,0.8]$ works well, but out-of-distribution values can also produce reasonable results in examples, such as $-0.2$ for more extreme eye or lip closure/opening.

The stage-II modules are tiny: the stitching MLP has layer sizes $[126,128,128,64,65]$, the eyes retargeting MLP has $[66,256,256,128,128,64,63]$, and the lip retargeting MLP has $[65,128,128,64,63]$. The paper emphasizes that the compute cost of these modules is negligible compared with the base network.

Inference procedure and module composition

At inference, the source image is encoded once into an appearance feature volume $f_s$ and canonical keypoints $x_{c,s}$. For each driving frame, the system extracts motion parameters and the eye/lip condition scalars. The optional modules are then composed using indicator variables $\alpha_{st}$, $\alpha_{eyes}$, and $\alpha_{lip}$. The paper’s inference logic is deliberately simple:

• If all indicators are off, the driving keypoints are used directly.
• If stitching alone is enabled, the driving keypoints are first adjusted by the stitching module.
• If eyes or lip retargeting is enabled, the corresponding deformation offsets are added to the source keypoints.
• If stitching is also enabled in the retargeting case, stitching is applied after retargeting.

The key point is that the offsets from eyes and lips are decoupled and can be added linearly, which the authors use to argue that the control factors are learned as separate modes of motion.

The method summary figure in the paper shows that the base model itself already yields animated portraits, while the control modules add practical improvements for aligned compositing and fine-grained expression adjustment.

Training details

The paper is unusually explicit about training cost and optimization settings. Stage I is trained from scratch on 8 NVIDIA A100 GPUs for about 10 days. Stage II, in which only the stitching and retargeting modules are optimized, takes about 2 days. Input images are aligned and cropped to $256\times256$, the batch size is 104, and the output resolution is $512\times512$. Training uses Adam with learning rate $2\times10^{-4}$, $\beta_1=0.5$, and $\beta_2=0.999$.

The authors describe the training cost of the control modules as negligible relative to the base model. This matters because LivePortrait’s practical advantage depends not only on quality but on maintaining near-real-time performance.

Evaluation protocol and benchmarks

The paper evaluates both self-reenactment and cross-reenactment. For self-reenactment, the source is the first frame of the test video, and the remaining frames are both the driving sequence and the ground truth. The official test split of TalkingHead-1KH and VFHQ is used. For cross-reenactment, the first 50 images from FFHQ serve as source portraits.

The metrics are broad and cover fidelity, identity preservation, pose/expression accuracy, and motion accuracy:

• PSNR, SSIM, LPIPS, and $L_1$ measure pixel/perceptual similarity to ground truth in self-reenactment.
• FID measures distributional image realism against real FFHQ frames.
• CSIM measures face-identity similarity using ArcFace embeddings.
• AED measures expression-parameter error from SMIRK.
• APD measures pose-parameter error from SMIRK.
• MAE measures eyeball direction error.

For cross-reenactment, the authors note that driving sequence lengths are sampled differently for TalkingHead-1KH and VFHQ, and that X-Portrait is particularly sensitive to insufficient driving length. They therefore use longer driving sequences for X-Portrait comparisons than for the other methods.

Quantitative results

Across the reported benchmarks, LivePortrait is competitive or best on most metrics while keeping inference efficient. A subtle but important nuance is that the model is not universally best on every score: on cross-reenactment it is not the top method on FID for TalkingHead-1KH, and it does not maximize CSIM on the cross-reenactment benchmarks. However, it offers a strong quality-efficiency tradeoff and better temporal stability than diffusion-based baselines.

Self-reenactment on TalkingHead-1KH

Self-reenactment on VFHQ

Cross-reenactment on TalkingHead-1KH

Cross-reenactment on VFHQ

What the metrics suggest

On self-reenactment, the method is particularly strong: it reports the best PSNR, SSIM, LPIPS, $L_1$, CSIM, and MAE on both TalkingHead-1KH and VFHQ among the listed methods. On cross-reenactment, it is also very strong in expression, pose, and motion accuracy: the reported AED, APD, and MAE are the best in both datasets, while FID is best only in some baselines, reflecting the usual realism-versus-control tradeoff in generative evaluation. The paper highlights that diffusion methods can occasionally achieve competitive image realism but often suffer from slower inference and poorer temporal consistency.

Ablations and analysis

The paper’s ablation section is primarily qualitative, but it is carefully targeted to show why each control module matters.

Stitching

The stitching module solves a practical but important problem: the animated face crop may not align with the shoulder region of the source image when pasted back into the original canvas. Without stitching, the driven shoulder position follows the driving frame and can produce obvious spatial misalignment in the original image space. With stitching, the animated person is force-aligned to the cropped source portrait while preserving motion and appearance. This makes the method suitable for larger portraits, full-body images, and multi-person photos.

Eyes retargeting

The eyes retargeting module addresses incomplete eye closure and weak eye motion transfer in cross-identity reenactment. The paper shows that a source portrait can be driven by a user-chosen eye-open scalar even without any driving video, and the motion affects the rest of the face minimally. This demonstrates direct controllability over eye openness rather than merely reproducing the driving person’s eye dynamics. The authors also show that the module works in cross-reenactment where the source eyes are much larger than those of the driver; the retargeting makes the closing motion sufficiently strong to transfer the intended eye behavior.

Lip retargeting

The lip retargeting module is analogous to the eye module, but for mouth openness. It can animate the source portrait from closed lips to fully open lips using a target scalar, again with little effect on other regions. The authors also show that out-of-training-range targets can still yield reasonable outputs, and that wide mouth opening can even produce a tongue. In a second example, the module helps transfer a closing-mouth motion when the source’s mouth openness differs significantly from the driver’s.

Simultaneous eye and lip retargeting

The paper explicitly asks whether the eye and lip retargeting modules can act simultaneously. The answer is yes: the two offsets can be added together at inference, and the resulting portrait can respect both target eye openness and target mouth openness. This is used as evidence that the retargeting factors are reasonably disentangled, even though the modules are trained independently.

Additional capabilities reported in the appendix

The appendix broadens the paper’s scope beyond the core video-driven portrait-animation benchmark and shows how the same framework can be reused in several related settings.

Multi-person portrait animation

Because stitching enables each face to be aligned back into the original image space, the method can animate multiple subjects in a group photo separately. The paper shows examples where a different driving frame is associated with each person, demonstrating that the system is not restricted to a single centered face crop.

Audio-driven portrait animation

The authors state that the model can be extended to audio-driven animation by regressing or generating motion parameters from audio. In the appendix example, they encode audio with Whisper and then use a transformer-based framework, following FaceFormer, to autoregress motions such as expression deformations and head pose. The examples shown indicate that lip motion can be accurately driven by audio input while retaining stitching.

Generalization to animals

The paper claims that the model can generalize to animals such as cats and dogs after fine-tuning on a small animal-portrait dataset combined with the original data. For this setting, the head-pose loss, lip GAN loss, and face-identity loss are removed because the corresponding human-specific assumptions do not transfer cleanly to animal faces. The authors show that human driving videos can still animate these animals after fine-tuning.

Portrait video editing

LivePortrait can also be adapted to edit a source video sequence by re-animating only the head region while preserving the non-head regions of the original video. In this mode, the source and driving keypoints are computed per frame, and the driving expression offsets are temporally smoothed by averaging adjacent driving frames: $0.5\cdot(\delta_{d,i}+\delta_{d,i+1})$. The example shown in the appendix suggests that the edited frames inherit expressions from the driving video but keep the rest of the source video intact.

Limitations and ethics

The paper is explicit about two limitations. First, the current model struggles in cross-reenactment scenarios with large pose variation. Second, significant shoulder motion in the driving video can still cause jitter with some probability. These are important because the stitching module is aimed at spatial alignment, but large geometric changes still remain challenging.

The authors also discuss ethics. Portrait animation can be misused for deepfakes, so responsible use and ethical guidelines are necessary. They note that the current outputs still contain some visual artifacts, which may help in identifying synthesized media, but this is not framed as a defense mechanism.

Conclusion

LivePortrait shows that an implicit-keypoint warping framework, when scaled with better data, stronger motion modeling, a stronger decoder, landmark guidance, and lightweight control heads, can remain highly competitive with diffusion-based portrait animation methods while being much faster and more controllable. The reported performance is strong on both self-reenactment and cross-reenactment, the control modules are tiny, and the system runs at 12.8 ms per frame on an RTX 4090 under PyTorch. The paper’s broader takeaway is that practical portrait animation may not require a full diffusion stack if the motion representation and control surfaces are carefully designed.