DuoGesture

DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation

DuoGesture decomposes co-speech gesture synthesis into semantic and beat streams with specialized conditioning and biomechanical regularization. It improves semantic expressivity and speech-motion alignment by using motion-grounded semantic representations and stochastic gating instead of purely linguistic embeddings.

• co-speech
• gesture
• speech-driven
• full-body
• multimodal
• talking-head

Authors: Ferdinand Paar, Lanmiao Liu, Aslı Özyürek, Serge Thill, Esam Ghaleb

Categories: cs.CV, cs.SD

Published 2026-05-25 · Updated 2026-06-04

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub0 stars0 watchers0 forksnull

Abstract

Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose \emph{DuoGesture}, a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams. The two streams are coordinated by a \emph{Semantic Variational Information Bottleneck}, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by \emph{Motion-Grounded Semantic Conditioning}, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an \emph{Inertial Beat Prior}, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong holistic baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

---

Introduction and Motivation

Co-speech gestures are a core component of human communication, yet they are not homogeneous. Cognitive neuroscience distinguishes at least two broadly different gesture classes: beat gestures, which align with prosody and speech rhythm (supporting turn-taking and interaction flow), and semantic gestures (deictic, iconic, metaphoric), which are grounded in lexical content and occur much more sparsely. This functional dissociation is reflected neurologically: manual gesture comprehension recruits partially distinct but interacting networks—parietal-premotor circuits associated with visuomotor structure, and inferior-frontal/temporal circuits implicated in communicative meaning and gesture-speech integration.

Despite this, the overwhelming majority of existing holistic co-speech gesture generation models treat all gesture frames as a single homogeneous stream. This design choice leads to three persistent, compounding limitations:

• Misalignment between gesture and speech timing: a single shared network cannot simultaneously optimise for the prosodic timing demands of beat gestures and the lexical timing of semantic gestures.
• Poor semantic expressivity: conditioning on standard linguistic embeddings (e.g., BERT, FastText) encodes what was said rather than what motion should accompany it, leaving a linguistic-kinematic gap—particularly severe for rare, long-tailed lexical triggers.
• Jittery beat motion: without biomechanical constraints, rhythmically-aligned gestures are prone to over-smoothing and acceleration artefacts.

DuoGesture is proposed to address all three limitations simultaneously through a neuro-inspired and biomechanically informed dual-stream architecture. The core design principle is that beat and semantic gesture generation should be modelled as two coupled but distinct processes, each with stream-specific conditioning and regularisation, coordinated by a learned stochastic gating mechanism.

Related Work

Holistic Co-Speech Gesture Generation

The field has been shaped largely by two-stage hierarchical quantisation paradigms. Early VAE-based stochastic models (TalkSHOW, ProbTalk) established category-contingent decoding for multifaceted body dynamics. EMAGE introduced spatially-decoupled tokenisation across facial, manual, and corporal streams. PyraMotion uses a multi-resolution Anchor-based Pyramid VQ-VAE to minimise distributional divergence. However, all these models operate under an assumption of kinematic homogeneity—treating gesticulation as a singular stochastic process, blurring the functional divergence between rhythmic prosody and semantic morphology.

SemTalk introduced frame-level soft gating as a first-order approximation of the beat/semantic distinction, but remains limited by shallow linguistic conditioning and no physical regularisation. DuoGesture addresses all three missing pieces simultaneously.

Cross-Modal Semantic Grounding

Existing approaches to the linguistic-kinematic gap include LLM-based lexical parsing (Semantic Gesticulator), CLIP-based latent alignment (GestureDiffuCLIP), and non-parametric retrieval of motion exemplars at inference time (RAG-Gesture). DuoGesture takes a different approach: rather than performing explicit retrieval, it distils the structured knowledge of a motion-aligned encoder (Text-to-Motion, TMR) into the generative pipeline, achieving representational richness of retrieval-based methods at parametric inference efficiency.

Physically-Consistent Motion Synthesis

Physics-based constraints have been explored in character animation via differentiable simulators and post-hoc projections (DeepMimic, PhysDiff), but their integration as inductive biases inside gesture generators is largely unexplored. DuoGesture avoids expensive simulation by incorporating anthropometric priors—specifically, De Leva's segment mass distributions—directly into the beat stream as a training-time regulariser with no inference overhead.

Ground-Truth Analysis: Beat vs. Semantic Motion

A crucial contribution motivating the architectural design is a controlled empirical analysis of BEAT2 test-split motion. The authors segment sequences into contiguous beat windows (semantic flag $= 0$) and semantic windows (semantic flag $= 1$) using the dataset's per-frame annotations, drawing a matched sample of $\approx 1{,}870$ windows per class across 25 speakers with balanced bootstrap confidence intervals.

Key findings (Table 1 in the paper):

• Beat motion peaks at 1.12 Hz with moderate inter-joint coupling (Phase Locking Value PLV $= 0.31$), consistent with a damped-pendulum model. A constant-velocity (inertial) oscillator explains $R^2 = 0.41$ of beat variance; this improves by $\Delta R^2 = +0.84$ when the mass grouping covers arm + core joints, but degrades by $\Delta R^2 = -0.95$ when all 55 joints are included—directly motivating IBP's arm-chain-only mask.
• Semantic motion peaks at 1.69 Hz with strongly higher inter-joint coupling (PLV $= 0.53$, $\Delta\text{PLV} = +0.22$), indicating holistic co-activation of the arm during a lexical stroke—better captured by shape-conditioned generation than by a mass-weighted smoother.
• Beat spectra are narrow and tonal (half-bandwidth 0.46 Hz, prominence $4.6\times$); semantic spectra are $1.9\times$ broader and less tonal.

This analysis directly validates the dual-stream decomposition and the design of each module.

Method

DuoGesture is a two-stage latent generator. Stage 1 is a regional Residual Vector Quantisation VAE (RVQ-VAE) tokeniser, adopted from prior work. Stage 2, the main contribution, is a dual-stream generator with three novel components: Motion-Grounded Semantic Conditioning (MGSC), Semantic Variational Information Bottleneck (S-VIB), and Inertial Beat Prior (IBP).

Problem Formulation

A motion sequence of length $L$ is decomposed into four body regions $\mathcal{R} = \{\text{hand}, \text{upper}, \text{lower}, \text{face}\}$, with per-region motion $\mathbf{G}^r \in \mathbb{R}^{L \times J_r}$. Stage 2 conditions on:

• HuBERT audio features $e_a \in \mathbb{R}^{L \times 1024}$
• Speaker identity embedding $\text{ID}$
• Motion-grounded semantic features $\mathbf{S}^m \in \mathbb{R}^{L \times 256}$ (output of MGSC)
• A 4-frame seed pose $\tilde{\mathbf{p}}$

The target is the discrete latent code $\mathcal{Z}^q = \{\mathbf{Z}^q_r\}_{r \in \mathcal{R}}$ produced by the Stage 1 quantiser on ground-truth motion.

Stage 1: Regional RVQ-VAE Tokeniser

One encoder-quantiser-decoder triple $(\mathcal{E}^r, \mathcal{Q}^r, \mathcal{D}^r)$ is used per body region, trained with standard reconstruction and codebook commitment losses (codebook size $C = 256$, 4 RVQ levels). Stage 1 is frozen during Stage 2 training, decoupling tokenisation from temporal motion synthesis and enabling the generator to operate in a structured low-dimensional latent space. Representations use 55 SMPL-X joints in rot6d format (330-dim).

DuoGesture: Dual-Stream Architecture Overview

The beat backbone $f_b$ produces per-region beat latents from audio, speaker identity, and seed pose:

The S-VIB module assembles per-frame lexical ($e_s$), motion-style ($e_m$), and emotion ($e_\varepsilon$) embeddings into motion-grounded semantic features, and produces per-frame semantic features and a scalar gate:

The semantic branch then uses $\mathbf{S}^m$ to produce semantic latents:

The fused codebook is computed as $Z_r = f_{\text{fusion}}(Z_r^{\mathrm{b}},\; Z_r^{\mathrm{s}},\; \Psi;\; \theta_{\text{fusion}})$, and the Stage 1 decoder maps it to joint space: $\hat{\mathbf{G}}^r = \mathcal{D}^r(Z_r)$.

Motion-Grounded Semantic Conditioning (MGSC)

MGSC addresses the linguistic-kinematic gap: standard text or vision-language encoders (FastText, BERT, CLIP) are learned from linguistic or image-text supervision rather than body-motion dynamics, providing weak priors for what a gesture should look like—especially for rare lexical triggers.

MGSC produces a per-frame semantic feature $\mathbf{S}^m \in \mathbb{R}^{L \times 256}$ by assembling three streams (all projected to 256 dimensions):

• $e_s \in \mathbb{R}^{256}$: per-frame FastText embedding of the word spoken at frame $l$, aligned via BEAT2's forced-alignment timestamps.
• $e_m$: utterance-level motion-style embedding from Text-to-Motion (TMR), grounding the conditioning in the kinematic manifold.
• $e_\varepsilon$: emotion embedding from TMR, using emotion labels provided by BEAT2.

The motion-style and emotion embeddings are blended via a learned softmax gate into a fused memory $\mathbf{m}$:

Then $e_s$ queries this memory via cross-attention:

By anchoring semantics directly within a kinematic manifold (TMR), MGSC achieves the representational richness of retrieval-based methods at the inference efficiency of a parametric model—critical for the long-tailed lexical distribution of semantic gestures.

Semantic Variational Information Bottleneck (S-VIB)

A naive deterministic gate (as in SemTalk's $\sigma$-gate) is prone to posterior collapse: the gate saturates to $\Psi \to 0.99$ across all frames under strong reconstruction objectives, making it trivially semantic. S-VIB prevents this via a stochastic variational bottleneck.

S-VIB operates on two inputs: the MGSC output $\mathbf{S}^m \in \mathbb{R}^{256}$ (encoding what to gesture) and a low-capacity HuBERT timing projection $\hat{e}_a \in \mathbb{R}^{64}$ (encoding when to gesture, obtained by convolutional bottlenecking of HuBERT features and entirely independent of the word embedding $e_s$).

The Information-Theoretic Bottleneck Head maps the concatenated input (320-dim) to a 16-dimensional mean $\boldsymbol{\mu}_\phi$ and log-variance $\log \boldsymbol{\sigma}^2_\phi$ via separate linear heads. A stochastic sample is drawn via the reparameterisation trick:

The Latent Sparsity Interpreter $\kappa$ (a two-layer MLP: $\kappa: \mathbb{R}^{16} \to \mathbb{R}^2$) outputs two-dimensional beat/semantic logits; the semantic probability is the per-frame gate:

The bottleneck is regularised against a standard Gaussian prior via KL divergence:

A per-dimension free-bits floor ($\lambda_{\mathrm{fb}} = 0.50$ nats, $Z = 16$) is applied: dimensions whose per-dim KL falls below $\lambda_{\mathrm{fb}}$ receive no gradient, preventing the optimiser from over-regularising uninformative dimensions while preserving capacity for semantically informative ones. The gate $\Psi$ is additionally supervised against BEAT2's per-frame semantic annotations $s_\tau \in \{0, 1\}$ via a semantic classification loss $\mathcal{L}_{\mathrm{sem}}$.

Two-Stream Hierarchical Blender

The Two-Stream Hierarchical Blender produces per-region beat residuals $Z_r^{\mathrm{b}}$ and semantic residuals $Z_r^{\mathrm{s}}$. The two streams are structurally parallel but differ in conditioning and regularisation.

Beat Stream (top, yellow): Takes as input the masked seed pose embedding $\tilde{\mathbf{p}} \in \mathbb{R}^{T \times 768}$, with speaker identity $\text{ID}$ and periodic positional encodings added before a self-attention pass. The speech onset projection $\hat{e}_a$ is integrated via cross-attention. Three region-specific MLPs then project to refined beat region latents $Z_u^{\mathrm{b}}$, $Z_l^{\mathrm{b}}$, $Z_h^{\mathrm{b}}$. The face latent $Z_e^{\mathrm{b}}$ comes from a separate face decoder conditioned on full HuBERT features $e_a$.

Inertial Beat Prior (IBP): Beat latents $Z_u^{\mathrm{b}}$, $Z_l^{\mathrm{b}}$, $Z_h^{\mathrm{b}}$ are decoded by the frozen Stage 1 decoder into raw (rot6d) poses $\mathbf{x}_{j,t}$. IBP penalises deviation from constant-velocity motion via:

The per-joint weight $\tau_{j,t}$ is derived from anthropometric masses and the S-VIB gate:

Here $m_j$ is the De Leva body-segment mass fraction for joint $j$, $m_{\max}$ is the fraction of the heaviest segment (spine1/abdomen, $\approx 0.163$ of body mass), so $\sqrt{m_j / m_{\max}} \in (0, 1]$ is a sqrt-compressed relative inertial weight. $\tau_{\mathrm{base}} = 0.5$ is the maximum smoothing for the heaviest arm-chain joint on a pure beat frame.

The two S-VIB-derived factors serve distinct roles: $(1 - \Psi_t)$ switches IBP off on semantic frames (when $\Psi_t \to 1$, $\tau \to 0$ and the loss is suppressed), and $(1 + \alpha\sigma^2_{\phi,t})$ inflates smoothing on uncertain frames. IBP is applied only to the arm-chain (upper body and hands)—not the face—motivated by the GT analysis showing arm-chain pendulum dynamics govern beat motion. This is a training-time-only regulariser with zero inference overhead.

Semantic Stream (bottom, blue): Operates on MGSC output $\mathbf{S}^m$. Masked seed pose embeddings and $\text{ID}$ are added, positional encodings applied, before a self-attention pass. Cross-attention integrates $\mathbf{S}^m$. Three region-specific MLPs project to semantic region latents $Z_u^{\mathrm{s}}$, $Z_h^{\mathrm{s}}$, $Z_l^{\mathrm{s}}$.

A Hierarchical Cross-Attention (HCA) block refines latents by allowing each region to attend to its sibling latents: the hands decoder attends to $Z_u^{\mathrm{s}} + Z_l^{\mathrm{s}}$ as keys/values; the upper decoder attends to $Z_h^{\mathrm{s}} + Z_l^{\mathrm{s}}$; the lower decoder attends to $Z_u^{\mathrm{s}} + Z_h^{\mathrm{s}}$. HCA is applied similarly to beat codebooks.

Fusion: For each body region $r \in \{h, u, l\}$, beat and semantic residuals are blended frame-by-frame:

The fused latent is quantised by nearest-neighbour lookup:

Selected tokens are decoded by the frozen Stage 1 decoder: $\hat{G}_r = \mathcal{D}_r(\hat{c}_r)$. The face region uses $Z_e^{\mathrm{b}}$ directly (no semantic stream for the face).

Training Objective

The full training loss combines five terms:

• $\mathcal{L}_{\mathrm{lat}}$: MSE between predicted continuous latents and Stage 1 VQ targets for all body regions.
• $\mathcal{L}_{\mathrm{cls}}$: Cross-entropy over the four RVQ codebook levels for each region.
• $\mathcal{L}_{\mathrm{sem}}$: S-VIB gate supervised against BEAT2 semantic flags.
• $\mathcal{L}_{\mathrm{kl}}$: VIB bottleneck KL with free-bits floor.
• $\mathcal{L}_{\mathrm{acc}}$: IBP inertia residual, active only on beat frames via $\tau_{j,t}$.

$\beta_{\mathrm{vib}}$ and $\beta_{\mathrm{phys}}$ are warmup-scheduled weights (KL warmup epochs 20–100, $\beta_{\mathrm{target}} = 0.01$; physics warmup epochs 30–80, $\lambda = 0.01$).

Architecture Details

The Stage 2 architecture uses a transformer backbone (hidden size 768, 1 self-attention layer, 3 decoder layers and 1 codebook head per region; 4 attention heads, dropout 0.1). Key component dimensions:

• HuBERT encoder: 2× Conv1d (k=3, p=1), BN, GELU; $1024 \to 256$.
• FastText: Linear $300 \to 256$.
• Seed pose: VQEncoderV6 (3 layers, frozen); $337 \to 768$.
• Speaker ID: Embedding table (25 entries) $\to 768$.
• Shared backbone: Self-attention TransformerEncoder (1 layer, $d_{\mathrm{ff}} = 1536$, $d = 768$); Audio cross-attention TransformerDecoder (8 layers, $d_{\mathrm{ff}} = 1536$, $d = 768$).
• MGSC: TM projections (Linear $256 \to 256$); Softmax memory gate; Semantic cross-attention (TransformerDecoder, 1 layer, $d_{\mathrm{ff}} = 512$, $d = 256$); Output MLP ($256 \to 256$).
• S-VIB: Timing projection (Linear + LN + GELU, $256 \to 64$); Bottleneck heads (2× Linear $320 \to 16$); Latent Sparsity Interpreter (2-layer MLP + softmax, $16 \to 2$).
• Region decoders (per region $r \in \{h, u, l\}$): Region MLP ($768 \to 256$); Temporal downsample (Conv1d, k=2, s=2, $L \to L/2$); HCA (TransformerDecoder, 1 layer, $d_{\mathrm{ff}} = 512$, $d = 256$).
• Face decoder: TransformerDecoder (4 layers, $d_{\mathrm{ff}} = 1536$, $d = 768$).
• Codebook prediction heads: Autoregressive TransformerDecoder (3 layers, $H = 8$, $d_{\mathrm{ff}} = 1024$, $d = 256$); 2-layer MLP classifier ($256 \to 256$).
• Total Stage 2 parameters: $\approx 122$M.

Experiments

Dataset

All experiments use BEAT2, the standard benchmark for co-speech gesture generation containing approximately 76 hours of speech, motion, facial expression, and speaker identity annotations from 30 speakers. Uniquely, BEAT2 provides frame-level annotations of gesture type (beat vs. semantic) and eight emotion categories—crucial for DuoGesture's training signals. Data are split 85%/7.5%/7.5% into training/validation/test at 64-frame clips, 30 fps, stride 20, with a 4-frame seed pose.

Results are reported in two standard settings:

• Single-speaker setting: Speaker 2 (Scott) only.
• Multi-speaker (all-speaker) setting: 25 speakers.

Evaluation Metrics

The primary metric is Fréchet Gesture Distance (FGD), which compares generated and real motion distributions in a learned gesture-feature space. FGD is the only objective metric found in prior work to correlate with subjective human-likeness ratings, and is treated as the primary realism metric throughout.

Secondary diagnostics include:

• Beat Alignment (BA): speech-motion synchrony.
• Diversity: pairwise L1 diversity.
• Facial MSE: facial motion reconstruction error.
• L1 Vertex Difference (LVD): facial motion stability.

A Pareto criterion is adopted: a preferable model should reduce FGD without achieving this gain at the expense of severe degradation in other metrics. High BA can reflect exaggerated beat-like motion; high Diversity can reflect large motion variance rather than plausible gestures.

Comparison Methods

DuoGesture is benchmarked against 14 representative systems: DiffuseStyleGesture (IJCAI 2023), AMUSE (CVPR 2024), SynTalker (ACM MM 2024), HoloGest (3DV 2025), RAG-Gesture (CVPR 2025), Habibie et al. (IVA 2021), DiffSHEG (CVPR 2024), ProbTalk (CVPR 2024), MambaTalk (NeurIPS 2024), TalkSHOW (CVPR 2023), EMAGE (CVPR 2024), SemTalk (ICCV 2025), GestureLSM (ICCV 2025), and PyraMotion (NeurIPS 2025).

Quantitative Results

Overall Comparison

In the single-speaker setting, DuoGesture achieves the lowest FGD (4.101), improving over the next-best SemTalk (4.278) and PyraMotion (4.612). It also improves MSE over PyraMotion (7.103 vs. 7.176) and is competitive on BA (7.557). The trade-off appears on Diversity and LVD, where DuoGesture concedes a small margin to the strongest baselines. Methods with substantially higher BA or Diversity (AMUSE, HoloGest) achieve this at much higher FGD, placing them at a different operating point rather than dominating DuoGesture.

The all-speaker setting follows the same pattern. DuoGesture again reaches the best FGD (4.081), with the closest competitor being GestureLSM (4.268). BA is effectively tied with EMAGE (7.699 vs. 7.707), and DuoGesture achieves the lowest MSE (7.502) among reported methods. Diversity (12.83) sits between EMAGE and SemTalk.

Ablation Study

Three clear findings emerge from the ablation:

• MGSC dominates the FGD gain: removing it raises FGD from 4.081 to 4.803—an effect larger than removing S-VIB (+0.056) or IBP (+0.097). This confirms that motion-grounded semantic conditioning closes the linguistic-kinematic gap that purely lexical embeddings leave open.
• IBP drives beat alignment: adding IBP on top of MGSC and S-VIB (variants c vs. e) raises BA from 7.446 to 7.699—the largest BA delta in the table—confirming that the inertial prior contributes specifically to rhythmic regularity rather than distributional realism.
• S-VIB protects diversity: replacing it with a deterministic σ-gate (variants b and d) lowers Diversity to 12.52 and 12.65 versus 12.83 for the full model, matching the design motivation that the variational bottleneck prevents gate collapse and keeps the per-frame motion distribution wider.

Subjective and Qualitative Evaluation

User Study

A controlled perceptual study was conducted using 35-second clips from the BEAT2 test set. Thirty native English-speaking participants from the UK, balanced by gender (male:female = 1:1; mean age 38.6), evaluated 24 randomly ordered videos from six narrated topics using a five-point Likert scale. Conditions compared: Ground Truth, DuoGesture, SemTalk, and EMAGE. Three evaluation axes: naturalness, motion diversity, and alignment with speech content and timing. Videos were presented in randomised order per participant.

Ground Truth received the highest scores across all axes. Among generated methods, DuoGesture obtained the strongest overall perceptual ratings—higher than SemTalk and EMAGE across all evaluated axes. This is consistent with the quantitative analysis and provides independent evidence that the Pareto-optimal objective profile corresponds to motions users judge as more natural and better aligned with speech.

The study included attention-check questions (participants asked to identify the topic of narrated speech from candidates such as career, helping a friend, or internet); responses from participants who failed attention checks were excluded. The study was administered through the Prolific platform, targeting English-speaking countries.

Qualitative Comparison

For semantic phrases ("to get", "I can share", "more drama"), DuoGesture produces gestures with clearer arm trajectories and more visible phrase-dependent structure. SemTalk and GestureLSM tend to generate weaker or more ambiguous movements; EMAGE produces plausible but less semantically differentiated gestures.

For beat-dominant phrases ("in this way"), generated motion remains temporally coherent with speech rhythm while avoiding excessive smoothing.

Implementation Details

• Data: BEAT2, 85%/7.5%/7.5% split, 64-frame clips at 30 fps, stride 20, 4-frame seed pose.
• Optimiser: Adam ($\text{lr} = 10^{-4}$, no weight decay), step LR decay ($\gamma = 0.3$), 200 epochs.
• Hardware: 4× A100 GPUs via PyTorch DDP, effective batch size 256.
• Motion representation: 55 SMPL-X joints in rot6d (330-dim); all streams projected to 256-dim.
• Audio: HuBERT (1024-dim).
• Text: FastText (300-dim).
• Stage 1: Regional RVQ-VAE, codebook $C = 256$, 4 RVQ levels, frozen during Stage 2.
• MGSC embeddings: TMR (256-dim) for $e_m$ and $e_\varepsilon$.
• S-VIB: bottleneck $Z = 16$, timing projection 64-dim, KL warmup epochs 20–100 ($\beta_{\mathrm{target}} = 0.01$), free-bits $\lambda_{\mathrm{fb}} = 0.5$ nats, semantic boost $\rho = 3.0$.
• IBP: physics warmup epochs 30–80 ($\lambda = 0.01$), $\tau_{\mathrm{base}} = 0.5$, $\alpha = 1.0$, arm-chain mask only.

Conclusion, Limitations, and Future Work

DuoGesture demonstrates that decomposing co-speech gesture generation into an explicitly semantic stream and a biomechanically-regularised beat stream—coordinated by a stochastic variational gate—is an effective design principle. It achieves state-of-the-art FGD in both single-speaker and multi-speaker settings on BEAT2, improves semantic expressivity, and maintains competitive rhythmic alignment and diversity. Ablations confirm that MGSC, S-VIB, and IBP provide complementary gains that are most effective when all three are combined.

Limitations identified by the authors include:

• Generalisation: DuoGesture has only been evaluated on BEAT2. Generalisation to other languages, cultures, speakers, recording conditions, and interaction settings has not been tested due to dataset availability limitations in the co-speech gesture generation domain.
• MGSC dependencies: The module depends on the coverage and biases of the pretrained TMR text-to-motion representation, which may not cover all gesture-relevant language.
• IBP scope: The biomechanical prior is designed for arm-chain beat motion and may not capture full-body gestures, object interactions, or contact-rich motion.
• Metrics: Existing evaluation metrics do not directly capture communicative meaning; better metrics for semantic gesture quality remain to be developed.

Future directions include: cross-dataset and multilingual generalisation, development of metrics that better capture communicative meaning in gesture, and extension of the biomechanical prior to full-body and contact-rich motion scenarios.

bash resources/DuoGesture/scripts/run_best_fgd.sh  # Test best FGD model
bash resources/DuoGesture/scripts/retrain.sh 200   # Retrain for 200 epochs