CogPortrait

CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning

CogPortrait provides fine-grained eye-region control in portrait animation from high-level labels, using multimodal large language model agents for detailed facial keypoint planning and a keypoint-driven video synthesis method, enabling precise and natural eye and head motion beyond typical emotion-driven methods.

• llm
• multimodal
• agent
• talking-head
• portrait-animation
• gaze
• head-pose
• emotion

Authors: He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su

Categories: cs.CV

Published 2026-05-27 · Updated 2026-05-27

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Portrait animation methods have achieved substantial visual quality and lip synchronization, but fine-grained manipulation of the eye region still faces a trade-off between input granularity and motion accuracy. Existing methods using emotion labels or coarse text prompts are insufficient for describing subtle ocular dynamics, whereas approaches based on Action Units or driving videos provide higher fidelity at the cost of a heavier input burden. These limitations are still restrictive for beyond-emotion states (e.g., thinking) and drowsiness. In light of the above, we propose CogPortrait, a two-stage framework that generates portrait animations from high-level labels. In the first stage, three chain-of-thought Multimodal Large Language Models (MLLMs) agents compile high-level labels into facial keypoints through temporal event planning, prototype retrieval, and composition from a real-behavior library, and semantic-physiological constraint enforcement. In the second stage, a DiT-based video generation backbone synthesizes the final animation conditioned on the keypoints, reference portrait, audio, and text prompt, enhanced by a dynamic classifier-free guidance strategy with eye-region-aware reweighting and KTO-based refinement for boundary cases. We further introduce the EMH benchmark covering diverse emotions and beyond-emotion categories with two AU-level metrics for evaluating fine-grained eye-region and head-motion control. Extensive experiments on HDTF and the EMH benchmark demonstrate that CogPortrait achieves more precise eye-region control than existing methods while maintaining supe- rior visual quality and identity consistency

---

Introduction

CogPortrait addresses a specific but important gap in portrait animation: most systems can produce visually plausible talking-head videos with good lip synchronization, yet they still struggle to control subtle eye-region motion such as gaze aversion, asymmetric eyebrow motion, eyelid droop, rapid blinking, and coordinated head-eye behavior in a way that is both precise and easy to specify.

The paper argues that there is a persistent trade-off between input granularity and motion accuracy. Label-driven or prompt-driven methods are easier to use but tend to stay at coarse semantic levels such as emotion categories or broad head-motion instructions. Motion-driven methods based on Action Units, landmarks, or driving videos can be more accurate, but they require more detailed inputs and are harder to obtain for irregular or beyond-emotion states such as cognitive effort, drowsiness, or evasive response.

To bridge that gap, CogPortrait proposes a two-stage pipeline. Stage 1 uses chain-of-thought MLLM agents to compile a high-level label plus optional fine-grained instructions into executable facial keypoints. Stage 2 uses a DiT-based video generation backbone conditioned on the keypoints, reference portrait, audio, and text prompt, and augments sampling with a dynamic classifier-free guidance (CFG) schedule and Kahneman-Tversky Optimization (KTO) refinement for long-tail cases.

Methodology

Overall problem formulation

The system generates a portrait video $V \in \mathbb{R}^{T \times H \times W \times 3}$ from a reference portrait $I_{\mathrm{ref}}$, driving audio $A$, a text prompt $P$, a high-level label $y$, and optional fine-grained instructions $u$. The output is intended to faithfully realize specified eye-region and head-motion patterns while preserving identity and visual quality.

The backbone is a flow-matching video model initialized from Wan2.2. In the standard flow-matching setup, a 3D VAE encodes the target video into a latent $x_1$, Gaussian noise $x_0 \sim \mathcal{N}(0, I)$ is sampled, and a velocity network $v_\theta$ is trained on interpolated latents $x_t$ with the objective

$$ \mathcal{L}(\theta)=\mathbb{E}_{x_1,c,t\sim\mathrm{Unif}[0,1],x_0\sim\mathcal{N}(0,I)}\left[\left\|v_\theta(x_t,t,c)-(x_1-x_0)\right\|^2\right]. $$

The paper also adopts KTO, which aligns the model with binary desirable/undesirable feedback using an implicit reward $r_\theta(c,x)=\log\frac{\pi_\theta(x\mid c)}{\pi_{\mathrm{ref}}(x\mid c)}$ and the loss

$$ \mathcal{L}_{\mathrm{KTO}}=\mathbb{E}_{c,x}\left[w(y)\cdot\left(1-\sigma\left(\beta\cdot r_\theta(c,x)-z_{\mathrm{ref}}\right)\right)\right]. $$

Prototype library

A central design choice is the construction of a structured prototype library $\mathcal{L}$, used to avoid the over-smoothed trajectories that direct MLLM-generated control signals can produce. Each prototype is represented as $p=\{y,\{c_t\}_{t=1}^{T},\{k_t\}_{t=1}^{T}\}$, where $c_t=[a_t,g_t,r_t]$ contains 17 channels: 10 AU channels, 4 gaze channels, and 3 head-pose channels.

The 10 AU channels are AU1, AU2, AU4, AU5, AU6, AU7, AU9, AU11, AU33, and AU45. The gaze channels are four directional components, and the pose channels are yaw, pitch, and roll. The paper assigns explicit ranges to these channels: AU5 on $[-1,2]$, AU45 on $[0,1]$, gaze channels on $[0,2]$, yaw and pitch on $[-60^\circ,60^\circ]$, and roll on $[-35^\circ,35^\circ]$.

Prototype retrieval is based on a channel-wise distance

$$ D(q,p)=\sum_{j=1}^{17}w_j\,|q_j-\bar{p}_j|, $$

where AU channels are weighted more heavily than gaze and head-pose channels. The intent is to retrieve behaviorally meaningful trajectories rather than arbitrary appearance similarity.

EMH benchmark

The paper introduces the EMH benchmark to evaluate fine-grained eye-region and head-motion pattern generation from high-level labels. It contains 12 categories: six core emotions and six beyond-emotion states. The categories are paired with text prompts and manual eye-region dynamic annotations, and the benchmark releases indices, processed annotations, and evaluation scripts rather than the underlying videos when source licensing applies.

• Core emotions from MEAD: sadness, fear, disgust, contempt, anger, and surprise, with 120, 122, 115, 121, 120, and 124 videos, respectively, from 33, 38, 30, 33, 32, and 34 actors.
• Beyond-emotion categories: laughter from DH-FaceLolVid (133 videos, 62 actors), cognitive effort from UBFC-Phys (101 videos, 20 actors), low-arousal negative state from EAV (112 videos, 21 actors), social engagement from DH-FaceVid-1K (150 videos, 35 actors), evasive response from SEUMLD (111 videos, 36 actors), and drowsiness from UTA-RLDD (125 videos, 20 actors).

For example, the paper describes drowsiness as a sequence of droopy eyelids, prolonged blink, and then a drowsy head nod over time.

Stage 1: compiling labels into keypoints

Stage 1 converts a target label $y$, a reference portrait, the audio duration, and optional instructions into a frame-wise keypoint sequence $K_{1:T}=\{k_t\}_{t=1}^{T}$. The stage is hierarchical and uses two CoT MLLM agents plus a critic agent:

• Planning agent: decomposes the label into a temporally structured sequence of staged events $E=\{e_k\}_{k=1}^{N_e}$, with each event carrying a time span, semantic phase, and local constraints.
• Composition agent: retrieves and composes prototypes from the library to build an initial control sequence $C_{1:T}^{(0)}$.
• Critic agent: checks semantic consistency and physiological plausibility, and can request revision of the composition or re-planning of the event sequence.

The paper illustrates this with cognitive effort. A representative planning output decomposes the behavior into staged onset, gaze shift, and head-lowering phases. The composition agent then maps each stage to target channel ranges and retrieves matching prototypes. The critic validates constraints such as blink duration, inter-blink interval, blink asymmetry, AU co-activation, gaze main sequence, and gaze-head coordination.

The refined control sequence is denoted $\hat{C}_{1:T}=Q(C_{1:T}^{(0)};y,E,u,\mathcal{R})$, where $\mathcal{R}$ denotes semantic and physiological rules.

The mapping layer converts the refined channels into keypoints by applying AU-driven deformation of a FLAME mesh, using face indices and barycentric coordinates to recover eyelid and eyebrow keypoints, adjusting pupil and iris positions via gaze, and applying rigid rotation for yaw, pitch, and roll. The final output is a 62-keypoint sequence $k_t=\Pi(\hat{a}_t,\hat{g}_t,\hat{r}_t)$. Mouth keypoints are not explicitly controlled in Stage 1; they are generated in Stage 2 under audio conditioning.

Stage 2: keypoint-guided video generation

Stage 2 generates the final video from the reference portrait, audio, prompt, and keypoints. The reference image is encoded into a fixed first-frame latent, while the remaining temporal slots are initialized as Gaussian noise. The keypoints are aligned to the portrait space via TPS warping, rendered into a pose sequence, and passed through the VAE encoder and a 3D convolutional pose adapter. These structural features are added to the noisy video token embeddings. The reference latent is patch-embedded as reference tokens and concatenated with the noisy video tokens. Text is encoded by a frozen mT5 encoder, and audio is encoded by multilingual wav2vec; both are projected into the DiT cross-attention blocks.

The model uses a flow-matching sampling process with a DiT backbone, where each block performs layer normalization, self-attention, cross-attention to text and audio conditions, and a feed-forward network.

Dynamic CFG with eye-region-aware reweighting

The paper identifies two issues with a single global CFG scale: it can weaken high-frequency eye-region motion, and it can cause global color shift when the reference image has a uniform background. To fix this, the paper introduces a dynamic CFG schedule with temporal and spatial components.

Temporally, the guidance weight follows a trapezoidal schedule over denoising progress $\rho=i/N$. It stays at $\omega_{\mathrm{hi}}$ while $\rho<\alpha$, decreases linearly from $\omega_{\mathrm{hi}}$ to $\omega_{\mathrm{lo}}$ for $\alpha\leq\rho<\gamma$, and remains at $\omega_{\mathrm{lo}}$ afterwards.

Spatially, a Gaussian weighting map is centered at the midpoint of the two irises. Let $c_{\mathrm{eye}}$ be the eye-center and $d_{\mathrm{eye}}$ the inter-eye distance. Then

$$ G_f(x,y)=\exp\left(-\frac{\|(x,y)-c_{\mathrm{eye}}\|^2}{2(\kappa d_{\mathrm{eye}})^2}\right), $$

and the final position-dependent scale is

$$ \Omega_f(x,y,\rho)=\omega_t(\rho)G_f(x,y)+\omega_{\mathrm{bg}}\bigl(1-G_f(x,y)\bigr). $$

At inference, the paper uses 40 denoising steps and the following values: $\omega_{\mathrm{hi}}=8.0$, $\omega_{\mathrm{lo}}=4.0$, $\alpha=0.25$, $\gamma=0.55$, $\kappa=1.0$, and $\omega_{\mathrm{bg}}=1.0$.

KTO-based refinement for boundary cases

Even with dynamic CFG, the model can still struggle on boundary cases such as asymmetric eyebrow motion, rapid irregular gaze or blink patterns, and identity degradation under large-angle head motion. The paper therefore uses KTO to refine the model on a curated preference dataset.

The preference set includes large irregular head motions from DH-FaceDrasMvVid, filtered by thresholds of yaw $>40^\circ$, pitch $>30^\circ$, or roll $>30^\circ$, and three self-collected categories from 15 volunteers: asymmetric eyebrow motion, erratic gaze drift, and rapid blinking. Each category contains 400 clips, split into 350 training and 50 test clips, with no identity overlap.

The training set is composed of desirable real clips and undesirable clips generated by the current model under the same conditions when the outputs show imprecise eye-region control or identity drift. In the paper, KTO is combined with flow-matching loss at equal weight during refinement.

Experiments

Setup

All experiments were run on 8 NVIDIA H200 GPUs. The DiT backbone was initialized from Wan2.2 and fine-tuned with the pose encoder using AdamW at $10^{-5}$ for 7 GPU days. KTO refinement used a learning rate of $5\times10^{-8}$ with $\beta=625$, and combined the KTO and flow-matching losses with equal weight for 2 GPU days. The CoT agents used Gemini 3.0 in JSON mode.

Training data for the DiT backbone came from TalkVid and DH-FaceVid-1K. For evaluation, HDTF was used for standard portrait-animation metrics, and EMH was used for fine-grained eye-region and head-motion control. The training and test identities were non-overlapping.

Pre-processing standardized all videos to 25 FPS, resampled audio to 16 kHz, and cropped and resized the face region to $512\times512$.

The baselines cover three input regimes: audio-driven methods (Sonic, FLOAT, Hallo3, FantasyTalking), label-driven methods (EDTalk), and video-driven methods (Follow-Your-Emoji, HunyuanPortrait, ACTalker). The paper states that all baselines were evaluated with their official implementations and pretrained checkpoints.

Evaluation metrics

On HDTF, the paper reports FID, FVD, and LPIPS for visual quality; Sync-C for lip synchronization; ID-Sim for identity consistency; and Eye-LMD for eye-region controllability. On EMH, it reports BRISQUE for no-reference visual quality, DSL-FIQA for face quality, ID-Sim, Eye-LMD, AU-F1, and AU-Temp.

For EMH, the AU-based metrics are defined from AU trajectories extracted from generated and ground-truth videos. Let $\hat{A}_u(t)$ and $A_u(t)$ denote the generated and reference AU intensities for AU $u$, and let $S_y$ be the target AU set for category $y$. Binary activations are defined by thresholding the maximum intensity, and AU-F1 is then computed as the harmonic mean of precision and recall:

$$ P=\frac{\sum_u\mathbb{I}(\hat{a}_u=1\wedge a_u=1)}{\sum_u\mathbb{I}(\hat{a}_u=1)},\qquad R=\frac{\sum_u\mathbb{I}(\hat{a}_u=1\wedge a_u=1)}{\sum_u\mathbb{I}(a_u=1)}, $$

$$ \mathrm{AU\text{-}F1}=\frac{2PR}{P+R}. $$

AU-Temp measures temporal fidelity via normalized dynamic time warping:

$$ \mathrm{AU\text{-}Temp}=1-\frac{1}{|S_y|}\sum_{u\in S_y}\frac{\mathrm{DTW}(\hat{A}_u,A_u)}{Z_u}. $$

Quantitative results on HDTF

CogPortrait achieves the best reported results on HDTF for most metrics. It obtains FID 16.68, FVD 32.90, LPIPS 0.0633, ID-Sim 0.9214, and Eye-LMD 0.0107, with Sync-C 7.15, which is close to the best Sync-C score of 7.43 from Sonic. The Eye-LMD improvement is especially important because it directly reflects fine-grained eye-region control.

The paper emphasizes that Follow-Your-Emoji is strong on some quality metrics but suffers from a globally shared CFG strategy that can produce color drift, while HunyuanPortrait and ACTalker are competitive in some dimensions but do not match CogPortrait's eye-region accuracy and identity preservation.

Quantitative results on EMH

EMH is the paper's main benchmark for fine-grained control. The authors report three settings for their method: $\text{Ours (GT)}$, which uses ground-truth keypoints; $\text{Ours (Inverted)}$, which builds the prototype library via AU inversion from ground truth; and $\text{Ours (Full)}$, the complete pipeline with Stage 1 generation. As expected, GT keypoints provide an upper bound, while the full pipeline remains strong despite relying entirely on agent-generated keypoints.

On EMH, the paper notes that video-driven methods generally outperform audio- and label-driven methods on AU-F1 and AU-Temp because these metrics explicitly reward the correct eye-region and head-motion trajectories. Still, CogPortrait's full pipeline remains competitive and much better than the simpler label-driven or audio-driven baselines, confirming that the agent-generated keypoints are useful even without access to ground-truth motion signals.

Qualitative analysis

The qualitative comparisons reinforce the quantitative story. On HDTF, FLOAT is described as lacking fine-grained eye-region control, Follow-Your-Emoji suffers from skin and eye color drift due to its globally shared CFG strategy, and HunyuanPortrait can miss eyebrow precision because its training objective targets full-face control. CogPortrait preserves identity while producing more precise eye-region dynamics.

On EMH, the paper highlights that FLOAT can introduce background distortion and mouth artifacts in beyond-emotion states, Follow-Your-Emoji can create eye-region artifacts such as excessive eyelid pulling and also suffer identity drift during drowsy head nodding, and HunyuanPortrait can show identity drift and edge artifacts during large head motions. CogPortrait is reported to remain stable across social engagement, drowsiness, and cognitive effort.

Ablation studies

The ablation study separates the contribution of Stage 1 and Stage 2 design choices. For Stage 1, the progression is from direct label-to-keypoints, to rule-based retrieval, to agent-based planning and composition, and finally to critic-guided refinement. The strongest performance comes from the full hierarchical agent pipeline, which gives the best AU-F1, AU-Temp, and Eye-LMD.

For Stage 2, the paper compares fixed CFG, temporal reweighting only, spatial reweighting only, and the full dynamic CFG. Temporal reweighting mostly stabilizes overall appearance and improves ID-Sim, while spatial reweighting most directly improves eye-region control. Combining both yields the best result.

These ablations support the paper's interpretation: Stage 1 must model temporal structure and physiologically plausible composition, not just static mappings; Stage 2 needs both temporal and spatial guidance reweighting to balance global appearance and local eye-region controllability.

KTO impact

The KTO study isolates the benefit of preference-based refinement on difficult boundary cases. On the subset of asymmetric eyebrow motion and large head motions, KTO improves both control accuracy and identity consistency. The paper reports a drop in Eye-LMD from 0.0377 to 0.0311 and an increase in ID-Sim from 0.9089 to 0.9394 when KTO is enabled.

User study

The paper also reports a user study comparing CogPortrait with FLOAT, Hallo3, and ACTalker on motion realism, motion diversity, identity consistency, and overall preference. The detailed user-study numbers are not provided in the main paper text, but the existence of the study supports the claim that the method is intended to improve perceived realism in addition to benchmark metrics.

Key takeaways and stated scope

The paper's main novelty is not just a stronger video generator, but the combination of (1) hierarchical agent planning to convert semantic labels into plausible keypoints, (2) keypoint-conditioned portrait synthesis, (3) dynamic guidance reweighting to protect both local eye motion and global appearance, and (4) preference refinement for hard boundary cases.

The paper explicitly states that the agent pipeline is designed to produce executable control signals from high-level intent and that the critic can force revisions when semantic or physiological checks fail. It also frames the EMH benchmark as a way to measure fine-grained eye-region and head-motion control rather than only standard video quality.

While the paper does not include a separate limitations section, it does identify the core failure modes it is trying to solve: direct MLLM generation can over-smooth motion, uniform CFG can wash out local eye motion or shift colors, and rare boundary cases such as asymmetric eyebrow motion or large-angle head turns remain difficult without preference refinement. These are the main residual challenges implied by the reported ablations and boundary-case experiments.

Conclusion

CogPortrait shows that fine-grained eye-region control can be driven from high-level labels if the system first reasons about behavior in a structured way and then renders those plans through a keypoint-conditioned diffusion backbone. On HDTF it sets strong overall quality and identity scores while achieving the best eye-region metric, and on EMH it delivers the strongest or near-strongest results across keypoint, AU, and quality measures, especially when the full agent pipeline, dynamic CFG, and KTO refinement are combined.