KM-Speaker

KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

KM-Speaker is a speech-driven 3D facial animation system that combines global style from full-face keypoints with frame-level control from upper-face keypoints. It enables high-fidelity motion and precise style control, excelling in dialogue localization with accurate lip-sync and expressive performance.

speech-driven
talking-face
face-animation
dialogue
dubbing
lip-sync
3d-avatar

Authors: Arthur Josi, Emeline Got, Abdallah Dib, Luiz Gustavo Hafemann, Rafael M. O. Cruz

Categories: cs.CV, cs.GR, cs.LG

Comment: 20 pages, 14 figures

Published 2026-06-26 · Updated 2026-06-26

Abstract

Speech-driven 3D facial animation methods face significant challenges in simultaneously achieving high-fidelity motion and precise artistic control at production quality. Existing controllable models typically learn global style control by relying on large-scale, low-quality \emph{in-the-wild} datasets that compromise overall animation realism. Furthermore, these frameworks often lack the fine-grained temporal precision required for demanding tasks such as dialogue localization (e.g., dubbing), where matching specific facial expressions is as critical as lip synchronization. We present KM-Speaker (Keypoint-Matching Speaker), a novel keypoint-conditioned flow-based generative framework that provides both global style guidance and frame-level temporal control from reference performances. We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, together with a global style context preservation mechanism to ensure coherent full-face expressiveness. KM-Speaker advances example-based 3D facial animation by achieving high-fidelity motion and flexible controllability in a data-constrained setting, consistently outperforming state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.

Technical overview

KM-Speaker is a speech-driven 3D facial animation system designed to close a practical gap in production workflows: it aims to deliver both high-fidelity motion and controllable style from reference performances, while also supporting dialogue localization (dubbing) where the lips must be re-synced to new speech but the upper face should preserve a target performance. The paper’s central idea is to condition a flow-based generative model on three signals: audio, a global style descriptor extracted from full-face keypoints, and a temporal style descriptor extracted from upper-face keypoints. This yields two operating modes: (1) example-based generation, which uses audio plus global style to produce a new animation that matches the reference’s overall delivery, and (2) dialogue localization, which additionally uses the temporal keypoint signal to preserve frame-level upper-face motion while adapting mouth motion to new audio.

The paper argues that existing controllable methods often trade off realism for control: label-, text-, and image-based conditioning usually gives only coarse style control, while example-based methods are typically trained on large-scale reconstructions of in-the-wild videos, which inherit reconstruction noise and audio-visual misalignment. KM-Speaker instead operates in a data-constrained but high-quality capture setting and uses sparse, semantically meaningful keypoints as the style interface.

KM-Speaker architecture and applications. A source audio signal and two sets of target keypoints are processed independently. Full-face keypoints provide global style features, while upper-face keypoints provide temporal style cues. Conditioning the flow model on all inputs enables dialogue localization, where the target upper-face motion is matched to a new audio clip. Conditioning the same model only on audio and global style enables example-based generation, where the generation's overall style matches the target. — KM-Speaker architecture and applications. A source audio signal and two sets of target keypoints are processed independently. Full-face keypoints provide global style features, while upper-face keypoints provide temporal style cues. Conditioning the flow model $\Psi$ on all inputs enables dialogue localization, where the target upper-face motion is matched to a new audio clip. Conditioning the same model only on audio and global style enables example-based generation, where the generation's overall style matches the target.

Model architecture

The model operates on fixed windows of length $N$ frames. Audio is encoded with a pretrained speech encoder; the paper uses w2v-BERT 2.0, whose hidden features are averaged, projected to the latent dimension $d$, and resampled to the animation frame rate. Reference motion is encoded with two separate keypoint branches:

Global style encoder $E_{\text{glob}}$: consumes full-face 3D keypoints $\mathcal{K} \in \mathbb{R}^{N \times 3K}$, applies convolutional layers plus a transformer encoder, then temporally averages features into a time-invariant style vector $\mathbf{z}_g \in \mathbb{R}^d$.
Temporal style encoder $E_{\text{temp}}$: consumes upper-face keypoints $\mathcal{K}_u \in \mathbb{R}^{N \times 3K_u}$ and produces a framewise style sequence $\mathcal{S}_{\text{temp}} \in \mathbb{R}^{N \times d}$.

The generator is a transformer-based flow model that predicts facial motion in a PCA-compressed latent space. The paper projects per-frame mesh deformations into a low-dimensional code $\mathbf{x} \in \mathbb{R}^{N \times l}$, where the PCA basis is built from vertex displacements relative to each identity’s neutral mesh. The flow is defined via conditional flow matching using linear interpolation between Gaussian noise and data, $\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\mathbf{x}_1$ with $\mathbf{x}_0 \sim \mathcal{N}(0, \mathbf{I})$. Rather than regressing the velocity directly, KM-Speaker predicts the clean target $\mathbf{x}_1$ from $\mathbf{x}_t$ conditioned on $\mathbf{c} = (\mathcal{A}, \mathcal{S}_{\text{glob}}, \mathcal{S}_{\text{temp}})$. At inference, the predicted clean latent is converted to a velocity estimate and integrated with a midpoint solver for 50 steps, then decoded back to mesh space through the PCA basis.

The supplementary implementation details specify an 8-layer transformer flow backbone with self-attention, cross-attention, and adaptive layer normalization (AdaLN) conditioning; a VAE-style global encoder regularized with KL divergence; and a compact MLP temporal encoder. The PCA basis uses $l=128$ components and retains approximately 99.5\% of the variance, which is an important design choice because the paper explicitly prioritizes high-fidelity facial motion over more aggressive compression.

Audio-keypoint disentanglement

The key modeling challenge is disentangling three roles: audio should control lip synchronization and mouth shapes, temporal keypoints should control upper-face motion frame by frame, and global style should preserve the overall expressive intent and keep the full face coherent. The paper adopts a strategy inspired by MeshTalk, but extends it with a global style pathway and explicit training perturbations to separate the three conditioning sources.

During training, for each sample $i$ in a batch, the model starts from the conditioning tuple $\mathbf{c}^i = (\mathcal{A}^i, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^i)$. It then constructs two mismatched variants by swapping either audio or temporal style with another sample $j$ from the same batch:

Audio conditioning: $\mathbf{c}_{\text{audio}}^i = (\mathcal{A}^i, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^j)$, which keeps the correct audio but replaces temporal style.
Temporal conditioning: $\mathbf{c}_{\text{temp}}^i = (\mathcal{A}^j, \mathcal{S}_{\text{glob}}^i, \mathcal{S}_{\text{temp}}^i)$, which keeps temporal style but replaces audio.

The global style signal is never swapped. That asymmetry is important: it forces the model to keep full-face coherence anchored to a style representation that always matches the target animation, even when audio and temporal style disagree.

The paper uses per-vertex masks $\mathcal{M}_{\text{lower}}$ and $\mathcal{M}_{\text{upper}}$ to constrain which face regions are supervised by which conditioning path. The lower face is primarily trained through the audio-swapped pathway, and the upper face through the temporal-swapped pathway. The losses are:

$$ \mathcal{L}_{\text{mse}} = \left\|\mathcal{M}_{\text{lower}} \odot (\mathcal{X}^i - \hat{\mathcal{X}}_{\text{audio}}^i)\right\|_F^2 + \left\|\mathcal{M}_{\text{upper}} \odot (\mathcal{X}^i - \hat{\mathcal{X}}_{\text{temp}}^i)\right\|_F^2 $$

$$ \mathcal{L}_{\text{vel}} = \left\|\mathcal{M}_{\text{lower}} \odot (\Delta \mathcal{X}^i - \Delta \hat{\mathcal{X}}_{\text{audio}}^i)\right\|_F^2 + \left\|\mathcal{M}_{\text{upper}} \odot (\Delta \mathcal{X}^i - \Delta \hat{\mathcal{X}}_{\text{temp}}^i)\right\|_F^2 $$

The total objective is $\mathcal{L} = \mathcal{L}_{\text{mse}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}} + \lambda_{\text{KL}} \mathcal{L}_{\text{KL}}$, with the supplementary values $\lambda_{\text{mse}} = 100$, $\lambda_{\text{vel}} = 100$, and $\lambda_{\text{KL}} = 10^{-6}$.

Disentanglement strategy. We randomly exchange either audio or temporal style signals using the corresponding signal from another sample of the same batch. Coherent lower-face motion is enforced through losses masked by the lower-face mask, while upper-face coherence is encouraged via the upper-face mask. Never swapping the global style signal enforces global facial coherence. Masks are visualized on a neutral face, where greener vertices correspond to mask weights closer to 1. — Disentanglement strategy. We randomly exchange either audio $\mathcal{A}^i$ or temporal style $\mathbf{S}_{\text{temp}}^i$ signals using the corresponding signal from another sample $j$ of the same batch. Coherent lower-face motion is enforced through losses masked by $\mathcal{M}_{\text{lower}}$, while upper-face coherence is encouraged via $\mathcal{M}_{\text{upper}}$. Never swapping the global style signal enforces global facial coherence. Masks are visualized on a neutral face, where greener vertices correspond to mask weights closer to 1.

Global style context preservation

The authors observe an imbalance in the conditioning strengths: temporal style is rich and framewise, so it can dominate the generator, while global style is compact and higher-level. This creates a training bias that can weaken example-based generation at test time, especially when temporal style is absent. Their solution is simple but effective: randomly drop the temporal style signal during training while always keeping global style. This forces the model to rely on global style alone in some updates and makes the full-face style representation useful even when a strong temporal cue is available in other updates.

The paper frames this as a classifier-free-guidance-like mechanism with a dropout probability of 0.1 for the temporal branch. In effect, KM-Speaker learns to produce coherent full-face motion from $\mathcal{S}_{\text{glob}}$ even when the audio and style intent differ, which is especially important for neutral-audio / expressive-style cases.

Dataset and implementation details

The method is trained on a curated capture dataset rather than on large in-the-wild reconstructions. The dataset contains 2.6 hours of high-quality 4D facial capture from 12 professional actors recorded at 60 FPS. Performances span 8 emotional states and 2 intensities (except Neutral), and the authors do not use emotion labels during training so the model does not collapse expressive variation into coarse categories. The split is 8 actors for training, 1 for validation, and 3 for testing. The supplementary material further specifies balanced casting across age, gender, and ethnicity, with capture in both a seated light-stage setup and a head-mounted rig.

The meshes are aligned to a common topology of 13,473 vertices. Keypoints are not detected by a learned landmark detector; instead, because the topology is fixed, they are defined by direct vertex correspondences. The paper uses 55 full-face keypoints and 47 upper-face keypoints for the main method. Keypoint values are computed as vertex displacements relative to each identity’s neutral mesh and are standardized per keypoint using training statistics.

Training uses Adam with batch size 32, learning rate $10^{-4}$, and weight decay $10^{-6}$. The model trains for about 2.3 days on a single RTX 8000 GPU. At inference, example-based experiments are evaluated at 25 FPS and dialogue-localization experiments at 30 FPS to match the comparative setups used by the baselines.

We visually present the keypoints used in Ours_1660_keypoints, Ours_68_keypoints, and Ours (top row), along with the disentanglement masks from our method and from MeshTalk. Green keypoints denote the upper-face keypoints used by the temporal encoder model, while the combination of green and grey keypoints represents the full-face keypoint signal used as style input. Disentanglement masks correspond to per-vertex weights, where grey vertices indicate weights of zero and green vertices indicate non-zero weights, with more intense green representing weights closer to 1.

Experimental protocol

The experiments are organized around three questions: Can the method preserve lip sync and motion fidelity in a matched audio-style context? Can it preserve global style when the audio and target style conflict? And can it localize dialogue by adapting lips to new audio while keeping the target’s upper-face motion? The paper evaluates all three.

For global stylization, the baselines are MIMIC and MSMD, each in original, retrained, and fine-tuned variants where applicable. The paper explicitly notes that these methods were originally learned on large-scale reconstructed datasets and that retraining them on the authors’ high-quality capture data is necessary for a fair comparison. For dialogue localization, the baseline is MeshTalk, which is the prior work most directly aligned with the task. The paper also tests ablations that replace the keypoint style signal with a full-face embedding from MeshTalk, change the keypoint density, or alter the disentanglement strategy.

The main metrics are:

LVE for lip-vertex error.
MSE for global vertex reconstruction error.
MOD for mouth opening difference.
FDD for upper-face dynamic deviation.
For dialogue localization, U-MSE for upper-face motion error and LVE for lip accuracy.

A useful design detail is that example-based evaluation uses the previous window as style context and the current window as audio, with no overlap, to avoid leakage. The dialogue localization setup samples 100 audio / target-animation pairs of matched length.

Matching audio-style context

In the matched-context setting, the target style and audio intent are aligned, so the evaluation focuses on whether the model can reproduce realistic speech-driven motion while respecting the reference style. The paper reports that training on the authors’ high-quality data helps the baselines substantially, but KM-Speaker still comes out ahead. The advantage is clearest when comparing the authors’ explicit keypoint-based style signal to denser or more implicit alternatives, supporting the paper’s claim that a sparse, semantically meaningful interface is better suited to this constrained-data regime than either a full-face embedding or a denser keypoint set.

Model	LVE (mm) ↓	MSE (mm) ↓	MOD (mm) ↓	FDD (mm^-2) ↓
MIMIC	2.48	2.00	0.68	4.68
MIMIC_finetune	2.41	1.97	0.47	4.59
MIMIC_retrain	0.85	0.49	0.50	4.69
MSMD	1.96	1.10	0.91	9.38
MSMD_finetune	1.35	0.73	0.62	6.38
MSMD_retrain	1.11	0.65	0.56	4.51
ours_meshtalk_encoder	1.10	0.65	0.50	5.63
ours_1660_keypoints	0.83	0.49	0.46	5.23
ours_68_keypoints	0.78	0.47	0.45	4.52
ours	0.77	0.45	0.46	4.31

The table shows several important trends. First, retraining or fine-tuning the baselines on the authors’ data improves them, but not enough to close the gap. Second, the ablation using the MeshTalk encoder as the global style source is consistently worse than the keypoint-based versions, indicating that the gain is not merely due to sharing a transformer-like architecture. Third, the dense 1660-keypoint and 68-keypoint variants both help, but the full method performs best overall. The authors interpret this as evidence that explicit, sparse, semantically meaningful keypoints provide the strongest style cue in their setting.

Generation for the different baselines with a desired angry target style sequence while the audio intent is mostly neutral.

Cross audio-style context

The cross-context experiment is the most revealing for the global-style mechanism. Here, the audio intent is intentionally mismatched from the target style, such as a neutral audio line paired with an angry reference style. The paper reports that the context-preservation dropout makes the global style more influential and produces upper-face motion that better reflects the target intent. The ablated version without the preservation mechanism tends to look more neutral or to follow the speech emotion, which weakens style transfer.

This part of the paper is important because it demonstrates that the style encoder is not only a descriptive summary of the reference motion; it is actively used to steer generation under conflicting conditioning signals.

Qualitative comparison of MSMD_retrain, MIMIC_retrain, and Ours in the matching context scenario, where the target style matches the audio intent. We do not display the target to encourage focus on the lip-sync and naturalness.

Dialogue localization

Dialogue localization is the task most directly aligned with dubbing: the model must adapt the lips to new audio while keeping the upper face synchronized to a target reference performance. The paper evaluates this by sampling 100 matched-length audio and facial-animation pairs and reporting U-MSE and LVE. The critical ablations are: using the MeshTalk encoder instead of keypoints, using MeshTalk’s masks, and removing disentanglement altogether.

KM-Speaker achieves the best overall balance. The keypoint temporal signal provides strong supervision for preserving upper-face motion, while the disentanglement strategy prevents that signal from leaking into the lip region. This is a central finding: without disentanglement, the temporal style signal can contaminate mouth motion and worsen lip sync; with it, the model preserves upper-face intent and still adapts the mouth to the new audio. MeshTalk’s own disentanglement helps, but the paper notes that it can still let upper-face dynamics influence jaw motion, which is especially problematic in expressive sequences.

Model	U-MSE (mm^-1) ↓	LVE (mm) ↓
MeshTalk	6.60	1.68
Ours_meshtalk_encoder	5.64	1.42
Ours_meshtalk_masks	2.81	1.22
Ours_no_disentangle	2.83	1.32
Ours	2.77	1.02

The localization table shows that the full method is the best performer on both metrics. Replacing keypoints with a learned full-face encoder is clearly worse, which supports the paper’s emphasis on an explicit style interface. Removing disentanglement worsens LVE, confirming that upper-face conditioning can easily leak into the lower face if the training scheme does not explicitly separate the roles of audio and temporal style. Using MeshTalk-style masks is helpful but still not enough to match the full design.

Qualitative results for two different actors and target temporal style for the dialogue localization task. We compare performance between MeshTalk and our model, and also conduct ablations on MeshTalk components relative to our approach through: Ours_meshtalk_encoder, Ours_meshtalk_masks, and Ours_no_disentanglement. The red rectangles clearly identify mismatches between the target upper face and the corresponding generated frame, or inaccurate lip shapes with respect to the pronounced sound. Our method precisely matches the target animation's upper face, including challenging eyebrow motion and eye closure.

Blending baseline

In the appendix, the authors compare their unified generator against a deterministic compositing baseline that blends target upper-face motion with generated lower-face motion:

$$ \mathcal{X}_{\text{blend}} = \mathcal{M}_{\text{upper}} \odot \mathcal{X}_{\text{target}} + \mathcal{M}_{\text{lower}} \odot \mathcal{X}_{\text{generated}}. $$

This baseline can preserve the upper face, but it introduces visible seams near the mask boundary and geometry inconsistencies that would become shading artifacts in rendered output. The takeaway is that direct compositing is brittle and identity-dependent, whereas KM-Speaker is trained to produce a coherent full face directly.

Qualitative results for two distinct examples, given a target animation, comparing the blending baseline with the model with and without global style conditioning. Zoomed-in visualizations highlight geometry inconsistencies. — Qualitative results for two distinct examples (left and right), given a target animation (first row), comparing the blending baseline with our model with and without global style conditioning. Zoomed-in visualizations are provided in the center to highlight geometry inconsistencies. Best viewed in color.

User studies and perceptual findings

The paper also runs two MUSHRA-like perceptual studies with practitioners from the video-game and animation industries. One study evaluates lip synchronization and the other evaluates style adherence. Each study contains five sets of ten non-overlapping videos, with randomized model order and a hidden negative anchor. The lip-sync study includes audio; the style study is muted so raters focus on expressive delivery rather than synchronization. The evaluation pools contain 83 lip-sync examples and 200 style examples, and the authors report 25 responses for lip sync and 23 responses for style adherence.

The reported outcome is clear: KM-Speaker is preferred for lip synchronization and for style adherence. The supplementary analysis reports mean pairwise Pearson agreement of 0.88 for lip sync and 0.57 for style adherence, which the authors use to argue that lip sync is a more objective perceptual task while style is more subjective yet still consistently judged.

Lip-synchronization and style-adherence user study results. Mean ratings with 95% confidence intervals are reported. — Lip-synchronization (25 responses) and style-adherence user study (23 responses) results. We report mean ratings (95% CI).

Generalization and limitations

The supplementary generalization experiment deliberately pushes the method outside the clean training regime: it uses in-the-wild audio, styles extracted from an off-the-shelf expression capture pipeline, and novel facial geometries from a different asset source. The method remains reasonably robust: it can transfer global style and maintain plausible lip synchronization even under these harder conditions. However, the paper also documents failure modes such as imperfect mouth closure and inner-mouth mesh intersections on some identities, especially for out-of-distribution facial morphologies.

This limitation is consistent with the overall design philosophy: the model is trained on only eight actors, which is enough for a strong production-quality prior in the paper’s target regime, but not enough to eliminate all geometry-specific edge cases. The authors explicitly suggest that few-shot adaptation or identity-specific modeling could help, and they note that the method’s performance depends on the quality of the input keypoint signal. They also emphasize an ethical constraint: example-based generation and dubbing should only be used with appropriate consent from the individuals whose likeness or performance is being reproduced.

KM-Speaker generalization results on in-the-wild audio, style, and varying face geometries. Given a target video providing the desired style, the corresponding facial expressions are first extracted onto an arbitrary mesh using an off-the-shelf expression capture model. A speech-driven animation is then generated using KM-Speaker with the extracted style and a different speech signal. — KM-Speaker generalization results on in-the-wild audio, style, and varying face geometries. Given a target video providing the desired style (left), we first extract the corresponding facial expressions onto an arbitrary mesh using an off-the-shelf expression capture model. We then generate a speech-driven animation (right) using KM-Speaker, conditioned on the extracted style and a different speech signal. Corresponding dynamic results are shown in the supplementary video (7:20–7:50). Best viewed in color.

Bottom line

KM-Speaker’s main contribution is not just a better generator, but a carefully structured control scheme for production-oriented facial animation: audio is reserved for mouth articulation, sparse keypoints supply both global and framewise expressive cues, and a dropout-based preservation mechanism keeps the global style meaningful at inference. The resulting system is a unified framework for high-fidelity example-based generation and dialogue localization, and the reported experiments support the authors’ claim that explicit keypoint conditioning is a strong fit for controllable speech-driven 3D facial animation in a data-constrained setting.