FacePlex
FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars
FacePlex generates speech and facial motion simultaneously in real-time for conversational avatars, overcoming the limitation of systems that choose between speech-only or pre-computed audio-driven animation. Rolling Flow Matching enables streaming motion generation while maintaining tight speech–facial sync.
Links
Paper & demos
Abstract
Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion. Existing systems only partially address this problem: speech-only full-duplex models can generate speech in real time but do not produce facial motion, while audio-driven facial motion models animate a face from already available audio rather than jointly generating speech and motion online. To bridge this gap, we first formalize full-duplex joint speech-facial motion generation, where speech tokens and facial motion tokens are produced together every step. Building on this formulation, we propose FacePlex, a unified streaming framework with two key components. First, Rolling Flow Matching adapts flow matching to online motion generation by committing new motion frames at each streaming step. Second, Rolling Cross-Attention couples the streaming audio queue with the motion queue, allowing speech and facial motion to condition each other as generation progresses. Through extensive experiments, ablation studies, and a user study, we show that FacePlex enables full-duplex joint speech-facial motion generation under online streaming constraints, while achieving stronger lip-sync quality and motion fidelity than audio-driven facial motion baselines.
Introduction and Motivation
Contemporary chatbot systems predominantly operate in a turn-based paradigm: the user speaks, the system waits, then responds. This abstraction diverges from natural human conversation in two fundamental and intertwined ways:
- (P1) Real-time interaction: Human listeners do not wait for a speaker to finish before reacting — they produce interruptions, barge-ins, and backchannels (verbal cues such as "uh-huh," and nonverbal signals such as nods) throughout an ongoing utterance. Inter-turn gaps in natural dialogue cluster tightly around 200 ms across many languages, and many conversational turns are short backchannels rather than full utterances.
- (P2) Joint speech-facial motion generation: Verbal content and facial behavior are not independent streams. Lip motion is phonetically coupled to speech (the McGurk effect, visual speech perception), and prosody and affect are reflected in facial expressions. A system that produces only audio, or only animates a face from pre-given audio, cannot replicate the full richness of face-to-face conversation.
The authors define a system satisfying both (P1) and (P2) as a full-duplex joint speech-facial motion generation system. Following communication-theoretic convention, full-duplex means simultaneous processing of incoming user signals and generation of outgoing responses at every time step, without buffering complete utterances — making the interaction inherently streaming.
Existing systems address only one fragment of this capability. Audio-driven facial motion systems such as UniLS and DualTalk animate faces from pre-given full-utterance audio, leaving both (P1) and (P2) unaddressed. Speech-only full-duplex systems such as Moshi and PersonaPlex enable interruptions, barge-ins, and verbal backchannels within the audio domain, but produce no visual output, leaving (P2) unaddressed. To the authors' knowledge, no prior system jointly produces speech and facial motion tokens while enabling real-time interaction.
Building a system that satisfies both (P1) and (P2) raises two coupled technical challenges:
- (C1) Generating high-quality facial motion fast enough for real-time interaction. Speech-only full-duplex systems emit one speech token per 80 ms (12.5 Hz). To achieve the same level of visual interactivity, facial motion must also be generated every 80 ms. Under streaming conditions the model cannot exploit future context beyond a bounded window, making it much harder to generate fine-grained articulatory details such as precise lip closures, jaw openings, and expressive micro-movements.
- (C2) Aligning speech with facial motion when both stream asynchronously. Speech tokens and motion tokens are produced at different rates and represent different temporal abstractions — a speech token captures sub-word phonetic content, while motion tokens capture continuous articulator trajectories. Coupling them at sub-second granularity is a non-trivial alignment problem that standard cross-attention does not resolve, especially when each queued motion pair carries a different noise level.
Related Work
Full-Duplex Conversational Speech Models
Conventional spoken dialogue models chain ASR, text generation, and TTS under a turn-taking assumption, requiring utterance completion before responding. Full-duplex speech models remove this assumption by emitting speech tokens at fixed time steps, enabling interruptions, barge-ins, and verbal backchannels at sub-second latency. Moshi pioneers this regime with parallel token streams for user and model speech; subsequent works extend this with frozen LLM backbones (Freeze-Omni), progressive text-to-speech conversion (OmniFlatten), and persona control (PersonaPlex). These systems remain audio-only; FacePlex extends their speech-token emission paradigm to joint speech-facial motion generation.
Audio-Driven Facial Motion and Audio-Visual Generation
A long line of work generates facial motion from given audio via lip-sync experts, speaker priors, codebooks, autoregression, or diffusion (VOCA, FaceFormer, CodeTalker, EmoTalk, ARTalk, DiffPoseTalk, SadTalker, and others). Dyadic extensions generate listener reactions (ReactFace, Learning2Listen) or jointly model speaker-listener behavior (DualTalk, UniLS). Across this lineage, audio is treated as input rather than output. OmniResponse jointly generates listener video frames and audio causally, but its language model emits text tokens with word-level timing markers and uses a separate TTS module for batch audio synthesis, falling short of sub-second responsiveness required by (P1). Moreover, only the listener side is modeled, leaving (P2) only partially covered.
Streaming and Rolling Generative Models
Most diffusion and flow-matching methods generate full sequences in a single denoising or transport process, making them incompatible with streaming. Recent work introduces rolling or progressive noise schedules where different sequence positions are denoised by different amounts at the same step (Rolling Diffusion, Diffusion Forcing, stream-friendly audio variants). Rolling Diffusion is the closest formulation, but targets fixed-length offline video. FacePlex's Rolling Flow Matching extends this principle to streaming flow matching where the motion queue evolves with newly arriving audio. Unlike prior unimodal rolling formulations, FacePlex additionally requires cross-modal conditioning between streaming speech tokens and a mixed-noise motion queue — a new design axis the authors study systematically.
Method
FacePlex couples a PersonaPlex speech language model (based on the Helium-7B main transformer) with a FLAME-parameter motion generator through a cross-attention bridge that conditions a rolling motion queue on PersonaPlex hidden states.
Problem Formulation
At each step $T$, the system consumes the latest user-audio chunk. The PersonaPlex backbone emits a hidden state $\mathbf{h}_T \in \mathbb{R}^{d_h}$ ($d_h = 4096$) and the audio branch predicts the next generated audio chunk $a_T$ spanning 80 ms. To match the standard 25 fps frame rate used in talking-head animation, FacePlex generates two audio-aligned FLAME motion frames $\mathbf{X}_T \in \mathbb{R}^{2 \times d_m}$ per chunk, where $d_m = 108$.
FacePlex maintains three rolling queues that advance once per model step, letting the motion branch access a short window of FacePlex's predicted speech before the corresponding audio chunks are emitted:
- Audio queue $\mathbf{A}_T = [a_{T-L+1}, \ldots, a_T]$: stores generated audio chunks predicted but not yet emitted, so each chunk can be released together with its aligned facial motion.
- Hidden-state queue $\mathbf{H}_T = [\mathbf{h}_{T-L+1}, \ldots, \mathbf{h}_T]$: stores recent PersonaPlex hidden states used to condition motion generation.
- Motion queue $\mathbf{M}_T$: stores the corresponding motion-pair states being progressively refined.
After the motion update at step $T$, the front audio chunk $a_{T-L+1}$ and the generated front motion estimate $\hat{\mathbf{X}}_{T-L+1}$ are dequeued together for synchronized output. At step $T+1$, the queues shift forward as the newly predicted audio chunk $a_{T+1}$, hidden state $\mathbf{h}_{T+1}$, and a fresh Gaussian motion state for the new back-slot are appended. This controlled output delay lets motion generation use a short pre-emission speech look-ahead window.
Rolling Flow Matching (RFM)
Standard flow matching generates a fixed sequence by transporting all frames from noise to data under a shared flow-time schedule — mismatched with streaming. RFM maintains a motion queue whose slots are at different generation stages: the front slot is near-clean and ready for emission, the back slot is freshly initialized noise, and intermediate slots interpolate between them.
Let $\mathbf{X}_T$ denote the clean motion pair aligned with audio chunk $a_T$, and let $\boldsymbol{\epsilon}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ denote its Gaussian noise. The standard flow-matching interpolant is:
$$\mathbf{X}_T^\tau = (1 - \tau)\boldsymbol{\epsilon}_T + \tau \mathbf{X}_T, \quad \tau \in [0,1]$$where $\tau=0$ gives pure noise and $\tau=1$ gives the clean motion pair. Rather than denoising all queued pairs at a shared flow time, RFM assigns each slot $i \in \{0, \ldots, L-1\}$ (front-to-back) its own flow time via the uniform rolling schedule:
$$\tau_i = \frac{L-1-i}{L}, \quad \Delta\tau = \frac{1}{L}$$In the pre-update motion queue at step $T$, slot $i$ contains $\mathbf{X}_{T-L+1+i}^{\tau_i}$, yielding:
$$\mathbf{M}_T = \left[\mathbf{X}_{T-L+1}^{\tau_0},\; \mathbf{X}_{T-L+2}^{\tau_1},\; \ldots,\; \mathbf{X}_{T}^{\tau_{L-1}}\right]$$Before the motion update, the queue ranges from a near-clean front slot ($\tau_0 = (L-1)/L$) to a pure-noise back slot ($\tau_{L-1} = 0$). For $L=4$:
$$\mathbf{M}_T = \left[\mathbf{X}_{T-3}^{0.75},\; \mathbf{X}_{T-2}^{0.50},\; \mathbf{X}_{T-1}^{0.25},\; \mathbf{X}_{T}^{0}\right]$$At each model step, the velocity model $f_\theta$ takes the full motion queue, per-slot flow times, and conditioning signal, and predicts a velocity for every slot:
$$\mathbf{V}_T = f_\theta\!\left(\mathbf{M}_T,\, \boldsymbol{\tau},\, \mathbf{c}_T\right) \in \mathbb{R}^{L \times 2 \times d_m}$$where $\boldsymbol{\tau} = (\tau_0, \ldots, \tau_{L-1})$ and $\mathbf{c}_T$ is derived from the hidden-state queue $\mathbf{H}_T$. All slots are then advanced in parallel by a single Euler step:
$$\mathbf{M}_T' = \mathbf{M}_T + \Delta\tau\, \mathbf{V}_T, \quad \boldsymbol{\tau}' = \boldsymbol{\tau} + \Delta\tau\, \mathbf{1}$$After this update, for $L=4$ the queue becomes $\left[\mathbf{X}_{T-3}^{1.00},\; \mathbf{X}_{T-2}^{0.75},\; \mathbf{X}_{T-1}^{0.50},\; \mathbf{X}_{T}^{0.25}\right]$. The front slot reaches $\tau_0' = 1$ and is dequeued as the generated clean estimate $\hat{\mathbf{X}}_{T-L+1}$, emitted jointly with audio chunk $a_{T-L+1}$. The remaining $L-1$ slots are retained. After $\mathbf{h}_{T+1}$ and $a_{T+1}$ are produced, a fresh Gaussian state $\mathbf{X}_{T+1}^0 = \boldsymbol{\epsilon}_{T+1}$ is appended as the new back slot, forming $\mathbf{M}_{T+1}$.
This design emits exactly two motion frames every 80 ms to match the audio token rate, while each frame passes through multiple denoising steps over its lifetime in the queue, yielding smooth transitions without chunk-boundary discontinuities that arise from independent per-chunk generation.
Rolling Cross-Attention (RCA)
A single hidden state $\mathbf{h}_T$ represents only 80 ms of audio — far too sparse for coherent facial motion. RCA is a streaming conditioning scheme that lets each queued motion pair attend to a bounded window of recent and near-future speech context. RCA conditions the rolling motion queue on the rolling hidden-state queue, advancing in lockstep with both.
RCA applies attention at the granularity of motion pairs (the two frames within each pair share the same visible hidden states, keeping the attention mask compact). Visibility from $\mathbf{H}_T$ to $\mathbf{M}_T$ is defined by a binary mask $\mathcal{A} \in \{0,1\}^{L \times L}$, where row $i$ indexes motion slot $\mathbf{X}_{T-L+1+i}$ and column $j$ indexes hidden-state slot $\mathbf{h}_{T-L+1+j}$. The velocity model becomes:
$$\mathbf{V}_T = f_\theta(\mathbf{M}_T,\, \boldsymbol{\tau},\, \mathbf{H}_T,\, \mathcal{A})$$Life-cycle conditioning. Because the motion and hidden-state queues roll together, each motion pair is reconditioned throughout its denoising life cycle against a shifting speech window. Consider a pair $\mathbf{X}_t$ with $L=4$ under full RCA. When it enters the back slot at $\tau=0$, it attends to $\mathbf{h}_{t-3}, \ldots, \mathbf{h}_t$. At the next step ($\tau = 1/4$), the visible window shifts to $\mathbf{h}_{t-2}, \ldots, \mathbf{h}_{t+1}$. By the time the pair reaches the front slot for emission, it attends to $\mathbf{h}_t, \ldots, \mathbf{h}_{t+3}$. Across its life cycle, $\mathbf{X}_t$ integrates evidence from $\mathbf{h}_{t-3}$ through $\mathbf{h}_{t+3}$, corresponding to approximately $\pm 240$ ms of speech context — without requiring access to the full future utterance.
Mask variants. The authors evaluate four RCA masks for the $L=4$ queue:
$$\mathcal{A}^{\text{full}}_{ij} = 1$$ $$\mathcal{A}^{\text{block-diag}}_{ij} = \mathbb{1}[i = j]$$ $$\mathcal{A}^{\text{causal}}_{ij} = \mathbb{1}[j \le i]$$ $$\mathcal{A}^{\text{anti-causal}}_{ij} = \mathbb{1}[j \ge i]$$- Block-diagonal: enforces one-to-one speech–motion correspondence without temporal context.
- Causal: uses block-diagonal plus past hidden states (no look-ahead).
- Anti-causal: uses block-diagonal plus future hidden states (look-ahead only).
- Full: combines past, aligned, and future context, allowing each motion pair to attend to preceding, aligned, and upcoming speech.
Motion Generator Architecture
The velocity model $f_\theta$ is an 8-layer Transformer with hidden size $d = 512$ and 8 attention heads. Each layer contains self-attention, RCA cross-attention, and a feed-forward network. RCA is applied to active rolling motion tokens, which attend to the projected hidden-state queue $\mathbf{H}_T$ (projected from 4096 to 512 dimensions). The main model uses the full RCA mask. A final velocity head predicts 108-dimensional FLAME velocities.
Training and the RFM Objective
The LLM, audio generator, and motion generator are trained jointly end-to-end. PersonaPlex main-transformer hidden states serve as the conditioning stream, and the motion generator is trained on mixed-flow-time rolling queues with RCA conditioning.
Training states are sampled from the rolling schedule. A shared offset $\delta \sim \mathcal{U}(0, 1/L)$ randomizes each slot's flow time. Given a clean motion window $[\mathbf{X}_0, \ldots, \mathbf{X}_{L-1}]$ aligned with hidden-state queue $\mathbf{H}$ and per-slot noise $\boldsymbol{\epsilon}_i \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, slot $i$ is assigned:
$$\tau_i = \frac{L-1-i}{L} + \delta, \quad \mathbf{X}_i^{\tau_i} = (1-\tau_i)\boldsymbol{\epsilon}_i + \tau_i \mathbf{X}_i, \quad \mathbf{U}_i = \mathbf{X}_i - \boldsymbol{\epsilon}_i$$The flow-matching objective is:
$$\mathcal{L}_{\text{RFM}}(\theta) = \mathbb{E}_{(\mathbf{H}, \mathcal{A}, \mathbf{X}_{0:L-1}) \sim p_{\text{data}}} \left[ \frac{1}{L} \sum_{i=0}^{L-1} \left\| f_\theta(\mathbf{M}^{\boldsymbol{\tau}}, \boldsymbol{\tau}, \mathbf{H}, \mathcal{A})_i - \mathbf{U}_i \right\|_2^2 \right]$$Sweeping $\delta$ over $[0, 1/L)$ covers the full flow-time range across rolling slots, exposing $f_\theta$ to the heterogeneous queue states used by the streaming sampler. Conditioning dropout (probability 0.1 each) is applied independently to style, anchor, and PersonaPlex transformer-out conditioning to support classifier-free guidance and improve streaming warm-up robustness.
Optimizer and Training Setup
Training uses AdamW with a learning rate of $10^{-4}$, zero weight decay, and gradient clipping at a global norm of 1.0. The learning rate is warmed up linearly for 500 steps then decayed with cosine schedule to 10% of peak over $10^6$ steps. Models are trained for up to $10^6$ gradient steps with a batch size of 256 on 4 NVIDIA H200 GPUs using Distributed Data Parallel. An exponential moving average (EMA) is maintained with decay 0.9999, but raw checkpoint weights outperform EMA weights on streaming rollout evaluation and are used for all reported results.
Data Construction
The training corpus totals approximately 1,138 hours of paired speech–motion streams from two complementary sources.
Synthetic Self-Play Generation
Synthetic paired speech–motion data is generated by running PersonaPlex in a two-speaker interactive mode. Two virtual personas are instantiated simultaneously, each receiving the other speaker's Mimi-encoded audio tokens via a cross-fed streaming protocol. At every 80 ms inference step, each speaker's backbone emits a hidden state and an audio chunk; the audio is passed to the other speaker at the next step, creating a closed conversational loop. Sessions are initialized with structured situational prompts covering diverse scenarios (planning sessions, collaborative tasks, free-form discussion). The corpus comprises approximately 67,200 two-speaker conversations (~1,000 hours before filtering).
Teacher-Based FLAME Motion Synthesis
The synthetic self-play produces speech but not facial motion. UniLS, a pretrained speech-driven FLAME motion model, is used as a teacher to synthesize 108-dimensional FLAME motion for each speaker. To improve quality, UniLS is run $K=12$ times per (conversation, speaker) pair with different random seeds, producing 12 candidate motion sequences. Each candidate is scored by the PLRS model, and the highest-scoring candidate is retained (best-of-12 selection).
FLAME motion is stored in the 108-dimensional format used by UniLS: 100 expression coefficients, 3 global pose parameters (axis-angle), 1 jaw-open scalar, and 4 eye parameters. At 25 fps, each 80 ms PersonaPlex step aligns to exactly two FLAME motion frames.
PLRS-Based Filtering
The Perceptual Lip-Reading Similarity (PLRS) score is computed for each candidate using the SpeechMeshTransformer model. The scoring pipeline decodes 108D FLAME parameters to a 5023-vertex mesh via FLAME 2020, resamples from 25 fps to 30 fps, partitions into 5-frame windows, and computes cosine similarity of mel-filterbank and mesh vertex embeddings. After best-of-12 selection, only pairs whose PLRS exceeds a minimum threshold are retained. The resulting whitelist contains approximately 59,500 (session, speaker) pairs (~461 hours), with mean PLRS 0.244 ± 0.019.
Real Interaction Data
The Seamless Interaction dataset contributes approximately 677 hours from ~72,500 dyadic sessions (~145,000 per-speaker streams). Processing includes audio gating by speech activity masks (zeroing inactive frames), peak normalization to match synthetic amplitude distribution, Mimi encoding (resampled to 24 kHz, 8 codebooks at 12.5 Hz), text token alignment using Kyutai STT plus Moshi SPM tokenizer, and FLAME parameter conversion from 112D to 108D format by selecting the appropriate dimension subset.
| Source | Sessions | Streams | Avg. Duration | Total Hours |
|---|---|---|---|---|
| Synthetic self-play (PersonaPlex) | ~45,800 | ~59,500 | ~28 s | ~461 h |
| Real interactions (SeamlessInteractionTalk) | ~72,500 | ~145,000 | ~17 s | ~677 h |
| Total | ~118,300 | ~204,500 | ~20 s | ~1,138 h |
Experiments
Experimental Setup
Baselines. FacePlex is compared with two families: full-duplex speech models (Moshi, PersonaPlex, Freeze-Omni) and facial motion models (ARTalk, DualTalk, UniLS). These baselines address only one side of the target problem, whereas FacePlex jointly supports full-duplex speech interaction and facial motion generation.
Evaluation Metrics. Since no existing benchmark directly evaluates full-duplex joint speech-facial motion generation, two complementary protocols are used:
- Full-duplex speech interaction: Following Full-Duplex-Bench — Pause Task-Oriented Rate (Pause TOR ↓), Backchannel Frequency (BC Freq. ↑), Turn-taking Latency (Turn Lat. ↓), Interruption Latency (Intr. Lat. ↓).
- Facial motion generation: Perceptual Lip-Reading Similarity (PLRS ↑) for audio–motion synchronization; Lip Vertex Error (S-LVE ↓), Mean Head Distance (S-MHD ↓), Upper-Face Dynamics Deviation (S-FDD ↓) for speaking; FDD (L-FDD ↓) and Pose Fréchet Inception Distance (L-PFID ↓) for listening.
Streaming Evaluation Protocol. Each conversation is split into 80 ms chunks. At each step, the model observes only the current audio chunk and past context, generates the next facial motion segment, and commits it without future revision. Offline facial motion baselines are adapted to this chunk-wise protocol by running inference chunk by chunk and concatenating generated segments.
Main Quantitative Results
| Method | FDS | M | Pause TOR ↓ | BC Freq. ↑ | Turn Lat. ↓ | Intr. Lat. ↓ | PLRS ↑ | S-LVE ↓ | S-MHD ↓ | S-FDD ↓ | L-FDD ↓ | L-PFID ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Moshi | ✓ | — | 0.990 | 0.001 | 0.283 | 0.258 | N/A | |||||
| Freeze-Omni | ✓ | — | 0.700 | 0.002 | 0.955 | 1.364 | N/A | |||||
| PersonaPlex | ✓ | — | 0.587 | 0.025 | 0.078 | 0.427 | N/A | |||||
| ARTalk | — | ✓ | N/A | 0.186 | 10.318 | 2.325 | 27.254 | 27.368 | 0.109 | |||
| DualTalk | — | ✓ | N/A | 0.208 | 15.364 | 3.406 | 27.272 | 28.481 | 0.110 | |||
| UniLS | — | ✓ | N/A | 0.166 | 16.138 | 3.761 | 34.432 | 36.250 | 0.044 | |||
| FacePlex (Ours) | ✓ | ✓ | 0.584 | 0.028 | 0.078 | 0.399 | 0.239 | 7.896 | 1.784 | 24.629 | 24.567 | 0.031 |
FacePlex is the only evaluated system that supports both full-duplex speech interaction and facial motion generation. On full-duplex speech, it achieves the best Pause TOR and Backchannel Frequency, with turn-taking latency comparable to speech-only full-duplex baselines, demonstrating that adding streaming facial motion generation does not substantially compromise responsiveness. On facial motion, FacePlex improves PLRS, S-LVE, S-MHD, S-FDD, L-FDD, and L-PFID versus all audio-driven baselines.
Qualitative Results
ARTalk tends to produce conservative mouth movements. DualTalk often shows unstable or exaggerated expressions. UniLS is smooth but less phonetically aligned. FacePlex captures clearer mouth articulation and more natural head movements, especially for words with distinctive mouth shapes such as "understand," "question," and "tomorrow." These qualitative differences align with the quantitative gains in PLRS, S-LVE, and S-MHD.
User Study
A perceptual user study was conducted with 25 participants comparing FacePlex against ARTalk, DualTalk, and UniLS. Each participant evaluated 5 conversation sessions, each containing 4 videos from the same input by different models. Model identities were hidden; video order was counterbalanced to reduce position bias. Participants rated each video on a 1–5 MOS scale for four criteria: Lip Synchronization (Sync), Facial Expression & Speech Natural Coherence (Natural & Coherence), Conversational Interaction (Interaction), and Overall Quality (MOS). In total, 500 video-level evaluations were collected (125 ratings per method per criterion).
| Method | Lip Sync | Natural & Coherence | Interaction | MOS |
|---|---|---|---|---|
| ARTalk | 2.080 | 1.864 | 2.832 | 2.240 |
| DualTalk | 3.000 | 2.888 | 3.360 | 3.080 |
| UniLS | 2.776 | 2.816 | 3.344 | 2.992 |
| FacePlex (Ours) | 3.592 | 3.624 | 3.920 | 3.736 |
FacePlex receives the highest ratings across all four criteria. Gains in Sync and Natural & Coherence indicate improved audio-synchronized facial motion; the higher Interaction score suggests more natural timing and responsiveness in conversational settings.
Ablation Studies
| Model | Config. | PLRS ↑ | S-LVE ↓ | S-MHD ↓ | S-FDD ↓ | L-FDD ↓ | L-PFID ↓ |
|---|---|---|---|---|---|---|---|
| A | FacePlex w/o RFM | 0.201 | 11.855 | 2.802 | 33.918 | 33.883 | 0.142 |
| B | FacePlex w/o RCA | 0.238 | 9.032 | 2.051 | 25.743 | 29.160 | 0.043 |
| C | FacePlex w/ real data only | 0.208 | 8.551 | 1.902 | 24.324 | 24.543 | 0.125 |
| D | FacePlex w/ synthetic data only | 0.233 | 9.115 | 2.096 | 27.618 | 32.238 | 0.022 |
| E | FacePlex w/ RCA (block-diag) | 0.235 | 8.137 | 1.815 | 24.728 | 24.283 | 0.080 |
| F | FacePlex w/ RCA (causal) | 0.235 | 7.851 | 1.770 | 25.934 | 25.396 | 0.032 |
| G | FacePlex w/ RCA (anti-causal) | 0.239 | 8.003 | 1.796 | 24.215 | 24.325 | 0.031 |
| H | FacePlex (Ours) | 0.239 | 7.896 | 1.784 | 24.629 | 24.567 | 0.031 |
Component ablation. Removing RFM (A) and replacing it with a chunk-based generator severely degrades all metrics (PLRS drops from 0.239 to 0.201; S-LVE worsens from 7.896 to 11.855), confirming that the rolling motion queue is essential for temporal coherence and lip-sync precision. Removing RCA (B) preserves streaming but degrades lip-sync (S-LVE: 9.032 vs. 7.896) and motion fidelity (S-FDD: 25.743 vs. 24.629; L-FDD: 29.160 vs. 24.567), indicating that bounded-future speech context routed through RCA is what bridges audio and motion at sub-second granularity.
Data composition. Training on real data alone (C) yields the worst lip-sync (PLRS 0.208) because real conversational data is too scarce for precise audio-motion alignment, despite good motion fidelity (S-FDD 24.324). Synthetic-only training (D) recovers lip-sync (PLRS 0.233) but underperforms on motion fidelity (L-FDD 32.238) due to a distribution gap from the teacher model. Combining both (H) achieves the best overall trade-off.
RCA masking strategies. Block-diagonal (E) — strict one-to-one alignment without temporal context — produces the worst lip-sync among RCA variants (PLRS 0.235). Causal (F) recovers lip-sync metrics but limits motion fidelity. Anti-causal (G) achieves the strongest S-FDD (24.215), highlighting the importance of look-ahead. Full RCA (H) combines past, aligned, and future context, matching the best PLRS and delivering the most balanced trade-off, validating bidirectional speech-context conditioning.
Euler Step Ablation
| $N$ | PLRS ↑ | S-LVE ↓ | S-MHD ↓ | S-FDD ↓ | L-FDD ↓ | L-PFID ↓ | Motion Fwd. Latency (ms) ↓ |
|---|---|---|---|---|---|---|---|
| 1 | 0.238 | 7.392 | 1.670 | 28.008 | 26.942 | 0.015 | 4.144 |
| 2 | 0.239 | 7.896 | 1.784 | 24.629 | 24.567 | 0.031 | 7.909 |
| 4 | 0.237 | 7.983 | 1.802 | 22.890 | 23.717 | 0.007 | 15.465 |
| 8 | 0.236 | 8.195 | 1.849 | 23.800 | 24.935 | 0.108 | 30.647 |
| 16 | 0.233 | 8.331 | 1.878 | 22.878 | 24.619 | 0.003 | 62.261 |
$N=2$ Euler steps is chosen as the default because it provides a strong balance of quality and speed. $N=1$ is faster (4.1 ms latency) but yields noticeably worse FDD scores. $N \ge 4$ gives marginal gains in some FDD metrics at the cost of latency that exceeds the 80 ms streaming budget (15.5 ms at $N=4$, growing to 62.3 ms at $N=16$).
Contributions Summary
- New task formalization: Full-duplex joint speech-facial motion generation — speech and facial motion tokens produced jointly at sub-second (80 ms) granularity — is introduced and formalized for the first time, with explicit input-output protocols, frame-rate conventions, and evaluation criteria.
- Rolling Flow Matching (RFM): A streaming flow-matching formulation using a mixed-flow-time motion queue. Each slot maintains its own flow-time state and is advanced by one Euler step per model step; the front slot is committed as output and a fresh noise pair enters at the back, enabling two FLAME motion frames every 80 ms with smooth inter-frame transitions.
- Rolling Cross-Attention (RCA): A conditioning mechanism that couples the rolling motion queue to the rolling hidden-state queue in lockstep, providing each motion pair with approximately ±240 ms of speech context (for $L=4$) throughout its denoising life cycle without requiring future utterances. Four mask variants (full, block-diagonal, causal, anti-causal) are systematically studied.
- FacePlex system: The unified framework achieves competitive full-duplex speech responsiveness (Pause TOR 0.584, BC Freq. 0.028, Turn Lat. 0.078 s) while surpassing all audio-driven facial motion baselines on PLRS, S-LVE, S-MHD, S-FDD, L-FDD, and L-PFID under chunk-wise streaming evaluation, and achieves the highest MOS across all user-study criteria (Sync 3.592, Natural & Coherence 3.624, Interaction 3.920, MOS 3.736).
Limitations
The authors identify several important limitations:
- FLAME-parameter evaluation vs. photorealistic rendering: Evaluation focuses on FLAME-parameter facial motion rather than fully rendered video. Appearance-level factors such as rendering artifacts, identity preservation, lighting, and photorealistic consistency are not assessed.
- One-to-many nature of listener motion: Conversational facial behavior is inherently one-to-many; multiple reactions may be plausible for the same speech context. Distance-based metrics (LVE, MHD, FDD, P-FID) only partially capture conversational appropriateness, affective nuance, and semantic grounding.
- Synthetic training data biases: Part of the training data is constructed using synthetic self-play and teacher-based FLAME motion synthesis, potentially inheriting biases, failure modes, or limited motion diversity from the teacher model.
- Scope of embodied behavior: FacePlex currently focuses on speech-coupled facial motion and does not model the full range of embodied behaviors needed for realistic avatars — gaze control, body gesture, hand motion, scene context, long-term persona consistency, or explicit semantic understanding of conversational intent.
- Broader evaluation: User study and automatic evaluation are limited in scale; larger evaluations across diverse participants, languages, identities, and conversational contexts remain important future work.
Ethical and safety considerations: High-quality speech-synchronized facial motion generation could be misused for deceptive synthetic media, impersonation, or unauthorized digital replicas. The paper recommends deployment with usage restrictions, consent-aware data practices, and watermarking or provenance mechanisms where applicable.