GazeHead

Data-driven Head Motion Generation through Natural Gaze-Head Coordination

A data-driven approach to model temporal gaze-head coordination from in-the-wild videos. Unlike prior work treating head pose independently, it captures the intrinsic relationship between gaze and head motion using a generative VAE, enabling realistic head motion synthesis that naturally follows input gaze.

• talking-head
• gaze
• head-pose
• face-animation
• autoregressive

Authors: Xiaohan Liu, Yilin Wen, Yusuke Sugano

Categories: cs.CV

Published 2026-05-25 · Updated 2026-05-25

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

We present the first data-driven approach to model temporal gaze-head coordination from large-scale in-the-wild facial videos. To obtain training data for generalizable learning, we propose an automatic pipeline that extracts natural yet diverse gaze and head motions with off-the-shelf appearance-based gaze estimators. To capture the probabilistic correlation and temporal dynamics of gaze-head coordination, we build our model on a generative conditional Variational Autoencoder for plausible yet diverse gaze-conditioned head motion generations. We further apply our framework to gaze-controlled facial video generation, where we enable video generation with natural and realistic head motion correlated to the input gaze - an aspect that has not been emphasized before. Human evaluation and quantitative comparisons demonstrate our method's effectiveness and validate our design choices, with evaluators showing statistically significant preference for our approach over baseline methods.

---

Introduction and Motivation

Human visual attention is orchestrated through a tight coupling between eye gaze and head movement. When we shift our gaze, the head naturally rotates to complement and support the eye movement, distributing the oculomotor load across both subsystems. Modeling this gaze-head coordination is not merely of academic interest — it underpins a wide range of practical applications in computer vision and multimedia, including virtual avatar animation, talking-head video synthesis, and interactive digital humans. In all these settings, realistic head motion that appropriately follows gaze is essential; without it, synthesized faces appear rigid, robotic, or otherwise unconvincing.

Prior work on gaze redirection and gaze-controlled facial synthesis has made notable strides in modifying the apparent gaze direction in images and videos. However, these systems typically treat head pose as an independent, externally supplied control signal — they do not model the intrinsic relationship between where a person is looking and how their head naturally moves as a consequence. The result is stiff or implausible head dynamics, especially during dynamic gaze transitions.

Previous scientific studies of gaze-head coordination have relied on carefully controlled laboratory settings, mobile eye trackers, or virtual reality headsets. While rigorous, these approaches are constrained by specialized hardware, limited subject pools, and artificial environments. They cannot easily scale to the diversity of conditions encountered in the wild, and the resulting datasets are too small and homogeneous to support generalizable machine-learning models.

This paper makes three interlocking contributions to address these limitations:

• The first data-driven approach for modeling the temporal gaze-head coordination relationship, learning directly from large-scale in-the-wild facial videos rather than laboratory recordings.
• An automatic data-extraction pipeline that leverages off-the-shelf appearance-based gaze estimators to obtain diverse, natural gaze and head motion sequences without any specialized hardware.
• A generative conditional Variational Autoencoder (cVAE) framework that captures both the probabilistic (one-to-many) mapping from gaze to head motion and the temporal dynamics of that mapping, enabling smooth, diverse, and plausible long-form head motion generation.

The framework is additionally applied downstream to gaze-controlled facial video generation, integrating the head motion generator with a pre-trained face image synthesizer to produce videos where head movements naturally follow the input gaze — an aspect that, as the authors note, has not been explicitly addressed before.

Automatic Data Extraction Pipeline

A central challenge is obtaining training data that is both large-scale enough for learning generalizable models and natural enough to reflect real-world gaze-head coordination. The authors address this by applying an automatic extraction pipeline to in-the-wild facial videos, avoiding any reliance on specialized eye-tracking or motion-capture hardware.

Gaze and Head Pose Extraction

The pipeline uses an off-the-shelf appearance-based gaze estimator to extract both gaze directions and head poses from raw video frames. The process operates as follows:

1. Head pose estimation: Facial landmarks are detected using a landmark detector. Head pose is then recovered by fitting a 3D facial model and solving the Perspective-n-Point (PnP) problem, yielding a 3D rotation that describes the orientation of the head in world coordinates.
2. Normalization: Following the procedure of Zhang et al., the detected face region is transformed to a standardized canonical view with fixed virtual camera parameters. This normalization cancels out geometric variations arising from different head poses and camera positions, making gaze estimation more consistent across subjects and scenes.
3. Gaze estimation: The gaze estimator outputs a 2D gaze direction vector (pitch and yaw angles) in the normalized space.

Both the gaze direction and the head pose are parameterized as pitch-yaw angle pairs: gaze $\mathbf{g}_t = [\theta^g_t,\, \phi^g_t]$ and head pose $\mathbf{h}_t = [\theta^h_t,\, \phi^h_t]$, where $\theta$ denotes pitch and $\phi$ denotes yaw. This representation is compact, interpretable, and directly usable as input/output signals for the generative model.

Quality Filtering

Raw in-the-wild video data inevitably contains sequences that are unsuitable for training a gaze-head coordination model. The pipeline applies three automated filters:

• Sunglasses detection: Videos in which the subject is wearing sunglasses are discarded using a dedicated glasses detector, since sunglasses occlude the eyes and make reliable gaze estimation impossible.
• Multi-face and detection instability filtering: Videos containing multiple faces simultaneously, or sequences with frequent face detection failures, are identified via facial landmarks detected with Google MediaPipe and removed.
• Camera motion and scene-change filtering: Videos with unstable camera motion or rapid scene changes — which would introduce spurious apparent head motion — are filtered out using PySceneDetect.

After applying this pipeline to the CelebV-Text dataset, 45,806 training video sequences and 676 test sequences (from subjects unseen during training) are retained. All sequences are resampled to 25 FPS before further processing.

Gaze-Head Coordination Model

Problem Formulation

Given a sequence of gaze directions $\mathbf{g}_{1:T}$, the goal is to generate a corresponding sequence of head poses $\hat{\mathbf{h}}_{1:T}$ that exhibits natural gaze-head coordination. This is an inherently one-to-many mapping: for any given gaze trajectory, there exist multiple plausible head motion trajectories that a person might naturally produce. Deterministic regression approaches cannot capture this distributional nature. A generative model is therefore required.

Conditional VAE Architecture

The core model is a conditional Variational Autoencoder (cVAE) in which both the encoder $\mathcal{E}$ and the decoder $\mathcal{D}$ are built on Gated Recurrent Units (GRUs) to exploit temporal continuity in both the gaze and head motion sequences.

Encoder $\mathcal{E}$ (training only): The encoder receives concatenated features derived from the gaze directions $\mathbf{g}_{1:T}$, the real (ground-truth) head motions $\mathbf{h}_{1:T}$, and the context history $\mathbf{c}$. Each of these inputs is first passed through a fully connected layer before concatenation. The GRU processes the resulting sequence and outputs the parameters $\mu$ and $\sigma$ of a Gaussian distribution in the latent space. The latent variable is then sampled as $\mathbf{z} \sim \mathcal{N}(\mu, \sigma)$. This latent variable encodes the stochastic component of the gaze-head coordination — the particular style or pattern of head motion that the decoder should produce, beyond what is strictly determined by the gaze input.

Decoder $\mathcal{D}$ (training and inference): The decoder takes the latent variable $\mathbf{z}$, the preprocessed gaze inputs $\mathbf{g}_{1:T}$, and the context history $\mathbf{c}$, and generates head motions $\hat{\mathbf{h}}_{1:T}$ in a frame-by-frame autoregressive manner. The GRU's hidden state is iteratively updated at each timestep, propagating information about past generated poses forward in time to maintain temporal consistency across the generated sequence.

Long-Term Autoregressive Generation

A practical challenge for any window-based sequence model is long-term generation: how to produce head motions for sequences longer than the training window length $T$. The authors address this with an autoregressive segment-level scheme: the last two generated head poses from the previous segment, $\hat{\mathbf{h}}_{T-1}$ and $\hat{\mathbf{h}}_{T}$, are used as context vectors $\mathbf{c}_1, \mathbf{c}_2$ for the next segment. This ensures continuity at segment boundaries, preventing abrupt jumps or discontinuities in the long-term trajectory. At the very beginning of generation, the context vectors are initialized to zero.

Training Objective

The loss function combines a reconstruction term and a KL-divergence regularizer:

where the first term penalizes deviation of the generated head motion sequence from the ground-truth, and $L_{\text{KL}}$ regularizes the posterior distribution $q_\mathcal{E}(\mathbf{z} \mid \mathbf{g}, \mathbf{h}, \mathbf{c})$ to stay close to the standard Gaussian prior $\mathcal{N}(0, I)$. The scalar $\lambda$ balances the two terms. This standard VAE objective encourages the model to both reconstruct training sequences accurately and maintain a well-structured latent space from which diverse samples can be drawn at inference time.

Implementation Details

The model processes sequences at 5 FPS with a window of 12 frames, corresponding to approximately 2.4 seconds of motion per segment. The cVAE latent dimension is $d = 128$. Training runs for 60,000 steps with batch size 64 using the Adam optimizer with a learning rate of $5 \times 10^{-5}$. Several regularization strategies are employed for stable training: KL-divergence weight annealing (gradually increasing $\lambda$ during training to prevent posterior collapse), context dropout (randomly zeroing the context vectors during training to prevent over-reliance on context), and feature dropout.

During inference, the encoder is discarded, and the latent variable is sampled directly from the prior $\mathcal{N}(0, I)$. Different samples of $\mathbf{z}$ produce different plausible head motion sequences for the same gaze input, realizing the desired diversity.

Application to Facial Video Generation

To demonstrate the practical utility of the head motion generator, the authors integrate it with ST-ED, a pre-trained face image generator trained on the ETH-XGaze dataset. In this pipeline, the head motion generator first produces a gaze-coordinated head pose sequence from the input gaze signal. The synthesized head poses are then fed as control signals to ST-ED, which renders the corresponding facial video frames. This results in facial videos where head movements are naturally and automatically coordinated with the input gaze — without requiring the user to manually specify or craft head motion trajectories. The authors note that this integration is currently non-end-to-end (the two modules are combined at inference time but not jointly trained), which represents a direction for future work.

Experiments

Dataset

All experiments use the CelebV-Text dataset, a large-scale collection of in-the-wild facial videos covering diverse subjects, lighting conditions, backgrounds, and head orientations. After the quality-filtering pipeline, 45,806 video clips are used for training and 676 clips (from subjects not seen during training) are held out for testing. Frame rates are standardized to 25 FPS.

Evaluation Protocol

For each test gaze sequence, 30 head motion sequences are randomly generated by sampling different latent vectors. This allows evaluation of both plausibility (how close any generation is to the real motion) and diversity (how varied the 30 generations are).

Evaluation Metrics

The following metrics are reported:

• Angular Error (degrees): The 3D angular difference between generated and real head orientations. Reported as Avg (mean over all 30 generations) and Best (the minimum over 30 generations, reflecting the best achievable plausibility).
• Correlation Coefficient: Temporal correlation between generated and real head poses, reported separately for pitch and yaw dimensions. Reported as Best across 30 generations.
• Average Variance Error (AVE, deg²): Measures whether the variance of the generated head pose distribution matches the variance of real head poses. Reported as average over 30 generations.
• Smoothness (deg/frame³): The mean magnitude of the third derivative (jerk) of the head pose sequence — lower values indicate smoother, more natural motion.
• Average Pairwise Distance (APD, deg²): Measures the mean pairwise distance among the 30 generated sequences for the same input, quantifying diversity.

Baselines and Comparisons

Two simple deterministic baselines are used for comparison:

• Constant Head Motion: The head pose is held fixed (at its initial value) throughout the entire sequence. This represents the absence of any head motion model.
• Mirror Gaze Inputs: The gaze direction is directly copied to the head pose at each frame. This is a heuristic that captures the intuition that gaze and head tend to align, but ignores the statistical relationship between them and produces highly exaggerated head movements.

An additional ablation baseline is included: w/o Temporal Modeling, which removes the GRU components and context conditioning from the cVAE, reducing it to a frame-by-frame independent generator.

Quantitative Results

The full model outperforms all baselines across every plausibility metric. Key observations:

• Compared to Mirror Gaze Inputs, the best angular error improves by 55.3% (from 24.227° to 10.835°), and smoothness improves by 72.4% (from 25.287 to 6.987 deg/frame³).
• The temporal correlation with real head motions is substantially higher than both baselines in both pitch (0.509 vs. 0.157) and yaw (0.598 vs. 0.300).
• AVE is lower than all alternatives, indicating that the variance of the generated distribution better matches the variance of real head motion.
• The ablation without temporal modeling achieves higher APD (435.081 vs. 264.463), indicating greater diversity — but at the cost of plausibility, smoothness, and correlation.

Human Evaluation Results

* p < 0.05, ** p < 0.01 (Wilcoxon signed-rank test)

Four expert evaluators assessed randomly ordered pairs of videos on a scale from −2 (strongly prefer baseline) to +2 (strongly prefer the proposed method). Key findings:

• The proposed method is significantly preferred over Mirror Gaze Inputs ($p = 0.016$) and over the version without temporal modeling ($p = 0.008$, the strongest significance among all comparisons).
• There is also significant preference over Driven by Real Motion ($p = 0.031$). This somewhat counterintuitive result likely reflects that the generated motion, which is correlated with gaze and temporally smooth, is perceptually more natural than the occasionally unpredictable real motion in the test set.
• The comparison against Constant Head Motion is not statistically significant ($p = 0.688$, mean = 0.250). The authors note that minimal head movement can be contextually appropriate in some scenarios, so the balanced result is consistent with the method correctly modulating the amount of head motion based on gaze dynamics.

Qualitative Results

The qualitative comparison shows five consecutive frames of synthesized facial videos. The proposed method produces head movements that smoothly follow the gaze direction changes. In contrast, Mirror Gaze Inputs generates exaggerated, jerky head movements (because gaze angles are typically small relative to natural head rotations), while Constant Head Motion appears unnatural during gaze transitions because the head does not move at all. The proposed method also avoids the jittering artifacts observable in the temporal-modeling-free ablation.

Ablation Study: Importance of Temporal Modeling

The ablation study isolates the contribution of the GRU-based temporal modeling and context conditioning by comparing the full model against a variant in which these components are removed. In the ablation variant, head poses are generated frame-by-frame without any recurrent structure or cross-segment context, so each frame is drawn independently from the latent space conditioned only on the instantaneous gaze input.

The results clearly establish the necessity of temporal modeling:

• Angular Error: Best angular error increases by 44.0% (10.835° → 15.600°) without temporal modeling.
• Smoothness (Jerk): Jerk increases by approximately $8.5\times$ (6.987 → 66.267 deg/frame³), reflecting the severe frame-to-frame discontinuities that arise from independent per-frame generation.
• Correlation: Pitch correlation drops from 0.509 to 0.338; yaw correlation drops from 0.598 to 0.433.
• AVE: Increases by 54.8% (89.934 → 139.245 deg²), indicating the variance of the generated distribution is much further from that of real head motion.
• Diversity (APD): The ablation variant achieves higher APD (435.081 vs. 264.463) because frame-by-frame independence means each frame can vary freely. However, this diversity comes entirely at the cost of plausibility and coherence.

Human evaluators exhibited the strongest and most statistically significant preference for the full model over this ablation ($\text{mean} = 1.625$, $p = 0.008$), confirming that temporal consistency is the most perceptually critical component of the system.

Discussion and Limitations

The paper presents a compelling and complete solution to a previously unaddressed problem: generating natural, gaze-coordinated head motions in a data-driven, generalizable manner. The use of appearance-based gaze estimators as a scalable data source is a key enabler, avoiding the bottleneck of specialized hardware while achieving sufficient accuracy for learning coordination patterns across a wide population of subjects.

The cVAE formulation is well-suited to this task: the probabilistic latent space handles the inherent one-to-many nature of the gaze-to-head mapping, while the GRU-based encoder and decoder handle the temporal dynamics. The autoregressive context conditioning ensures smooth transitions across generation segments, which is critical for practical applications where videos can be arbitrarily long.

Several limitations are acknowledged:

• Preliminary video integration: The integration with facial video generation (via ST-ED) is non-end-to-end. The head motion and image generation modules are coupled only at inference time, meaning errors in one module can cascade into the other without any joint optimization to compensate. End-to-end training is identified as a key direction for future work.
• Facial expression: The current framework focuses exclusively on head pose and gaze; it does not model facial expressions (e.g., eyebrow raises, blinks, or mouth movements) that may also co-vary with gaze in natural behavior.
• Gaze estimation noise: Training data is derived from an automatic gaze estimator applied to in-the-wild video, so it inherits whatever noise and biases are present in the estimator's outputs. The quality of the learned coordination model is therefore partially dependent on the reliability of the gaze estimator across diverse conditions.

Conclusion

This work introduces the first data-driven framework for modeling gaze-head coordination from in-the-wild facial videos. By combining an automatic extraction pipeline with a cVAE generative model incorporating GRU-based temporal dynamics, the approach learns to produce diverse, plausible, and temporally smooth head motion sequences conditioned on arbitrary gaze inputs. The framework is validated on the CelebV-Text dataset through extensive quantitative metrics and a human evaluation study, with evaluators showing statistically significant preference for the proposed method over deterministic baselines and an ablation without temporal modeling. The application to gaze-controlled facial video generation demonstrates the framework's practical value for talking-head synthesis, virtual avatar animation, and related conversational AI applications. Future directions include end-to-end integration with advanced facial synthesis methods such as LivePortrait to jointly optimize head motion generation and facial rendering for gaze-controlled facial animation.