EMAGE

EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling

EMAGE is a unified framework generating full-body co-speech gestures from audio and partial gestures, covering face, hands, and body. It features a new mesh-level dataset and uses masked gesture modeling to enhance realism and control in synchronized gesture synthesis.

• gesture
• co-speech
• full-body
• talking-head
• face-animation
• audio-driven

Demos

These demos showcase EMAGE's unified holistic co-speech gesture generation, illustrating its capability to generate expressive and natural body gestures synchronized with speech audio across diverse scenarios. Watch for the fluidity and naturalness of the generated gestures, the expressiveness of motion, and how well the system handles diverse body and hand movements consistent with speech content. The videos also highlight EMAGE's large-scale training benefits and its strong generalization to unseen settings.

Authors: Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black

Categories: cs.CV

Comment: Fix typos; Conflict of Interest Disclosure; CVPR Camera Ready; Project Page: https://pantomatrix.github.io/EMAGE/

Published 2023-12-31 · Updated 2024-03-30

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face0 downloads0 likes

Abstract

We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available https://pantomatrix.github.io/EMAGE/

---

1. Problem Setting and Core Contribution

EMAGE targets holistic co-speech gesture generation: given speech audio and optional partial gesture observations, the model generates a complete, audio-synchronized human motion sequence spanning face, upper body, hands, lower body, and global translation. The paper’s central claim is that full-body gesture generation benefits from two changes at once: (1) a unified mesh-level dataset that standardizes body and head motion into SMPL-X and FLAME representations, and (2) a masked-motion pretraining objective that teaches the model to reconstruct missing gesture segments while also solving audio-conditioned generation.

The main practical idea is that partial motion hints are not just a nuisance to tolerate; they are a useful conditioning signal. EMAGE therefore learns from two paths: a masked gesture reconstruction path that forces the network to encode robust motion priors, and an audio-conditioned generation path that uses those priors together with speech cues to synthesize the missing motion. The paper’s design goal is not only realism, but also controllability: the model should accept predefined spatial-temporal gesture fragments and complete the rest coherently.

The paper’s stated contributions are: release of BEAT2, a representation-unified, mesh-level co-speech dataset; a new EMAGE framework for completion from partial gestures plus audio; state-of-the-art results on both body and face motion with only a short seed segment; and a demonstration that EMAGE can also train on additional non-holistic datasets such as Trinity and AMASS to improve fidelity and diversity.

2. BEAT2: A Unified Holistic Co-Speech Dataset

The paper’s dataset contribution is the most structural part of the work. The original BEAT dataset provides Vicon-based body motion and ARKit facial blendshape weights, but not a mesh-level standardized representation. EMAGE converts BEAT into BEAT2 (also referred to as BEAT-SMPLX-FLAME), combining refined SMPL-X body parameters with optimized FLAME head parameters. This creates a community-standard mesh representation for body-animation and talking-head research.

The authors start from the 76-hour, 30-speaker BEAT collection and remove speakers 8, 14, 19, 23, and 29 because of noisy finger data. This leaves 60 hours of data for 25 speakers (12 female and 13 male). The resulting data are divided into BEAT2-standard and BEAT2-additional, with 27 hours and 33 hours respectively. The experiments in the main paper use BEAT2-standard speaker-2 only, with an 85% / 7.5% / 7.5% train/val/test split per speaker.

The dataset contains 1762 sequences with an average duration of 65.66 seconds per sequence. The paper emphasizes that the sequences are continuous responses to daily questions, which matters because the motion is not clipped into isolated gestures.

Two important dataset-quality analyses appear in the paper and supplement. First, compared with TalkSHOW, BEAT2 has much richer motion variation: the supplement reports Beat Constancy of 6.896 versus 6.104, local diversity of 13.074 versus 5.273, and global diversity of 27.541 versus 5.273. Second, the paper asks human raters which dataset better preserves motion quality. In that study, BEAT2 wins against prior datasets in both body and face categories, with the strongest advantage over TalkSHOW for body and a competitive face score relative to VOCA.

In the paper’s broader dataset comparison, BEAT2 is also presented as the only motion-captured dataset in the comparison that provides standardized mesh-level information across the full body and head, making it especially useful for shared benchmarks across talking-head, gesture, and full-body animation research.

2.1 Body fitting and refinement

The body parameters are initialized from BEAT markers using MoSh++. The paper describes the optimization in terms of marker fitting, surface-distance constraints, initialization regularization, pose and shape priors, and velocity constancy, combined into a weighted objective. After MoSh++, the authors refine the result with three hand-coded physical rules: the neck and head should be about one-seventh of the total body length; fingers except the thumb should not bend backward; and outlier motion values are truncated beyond a $3\sigma$ range and blended with adjacent 10 frames to reduce spikes.

The supplement also notes that the authors preserve the SMPL-X facial identity parameters after body fitting.

2.2 Facial parameter conversion

BEAT facial motion is originally provided as 51 ARKit blendshape weights. EMAGE maps these to FLAME via a handcrafted ARKit-compatible blendshape setup built in Blender using the Faceit add-on. The paper first drives an ARKit-aligned FLAME model with the BEAT ARKit parameters, then solves for FLAME expression parameters by minimizing vertex-wise geometric error. The final result is a mesh-level face representation that can be applied directly in SMPL-X-based animation.

3. EMAGE Architecture

EMAGE models gestures in a unified vector that includes 55 joints in Rot6D, 100 FLAME parameters, 4 foot-contact labels, and 3 global translation values. Given an audio clip and optionally masked gesture tokens, the network jointly learns: (1) how to reconstruct missing motion, and (2) how to generate motion from speech while using the reconstructed motion hints as guidance.

3.1 Compositional discrete priors

The model does not encode the entire body with a single latent space. Instead, it uses four separate VQ-VAE priors for face, upper body, hands, and lower body. The paper argues that this decomposition is necessary because different parts of the body correlate differently with speech. For example, the face is strongly tied to rhythm and semantics, while lower-body motion may be more sporadic and can otherwise be washed out if everything is forced into one shared codebook.

The supplement states that each VQ-VAE is CNN-based, uses a codebook size of 256, a feature length of 512, and a downsample rate of 1. The VQ-VAE objective combines reconstruction, velocity, and acceleration losses with the standard VQ commitment and codebook terms:

$$\mathcal{L}_{\mathrm{VQ\mbox{-}VAE}} = \mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{vel}} + \mathcal{L}_{\mathrm{acc}} + \|\operatorname{sg}[\mathbf{z}] - \mathbf{q}\|^2 + \|\mathbf{z} - \operatorname{sg}[\mathbf{q}]\|^2.$$

Here, $\mathbf{z}$ is the encoded motion feature, $\mathbf{q}$ is the nearest codebook vector, and $\operatorname{sg}[\cdot]$ denotes stop-gradient. The paper notes that reconstruction uses geodesic and $L_1$ terms for motion, and that the commitment loss weight is set to 1.

3.2 Content-Rhythm Self-Attention

EMAGE uses a separate audio branch called Content Rhythm Attention (CRA) to fuse two speech cues: rhythm and content. Rhythm is represented using onset and amplitude features, while content comes from pretrained word embeddings from transcripts. The paper’s key point is that gestures are not uniformly tied to both cues in every frame, so the model should decide adaptively whether the current motion should emphasize beat alignment or semantic content.

The audio streams are first encoded into aligned sequences $\mathbf{r}_{1:T}$ and $\mathbf{c}_{1:T}$. The paper then combines them as:

$$\mathbf{f}_{1:T} = \alpha \mathbf{r}_{1:T} + (1-\alpha)\mathbf{c}_{1:T}, \qquad \alpha = \operatorname{softmax}(\mathcal{A}(\mathbf{r}_{1:T}, \mathbf{c}_{1:T})).$$

In words, CRA learns a per-frame preference between rhythm and content rather than simply summing them. The authors apply separate CRA encoders for face and body.

3.3 Masked audio-conditioned gesture modeling

The masked-motion branch replaces hidden motion tokens with a learned mask embedding instead of zeros, because zero can correspond to a valid pose such as a T-pose. Masking is applied both temporally and spatially, and the paper linearly increases the mask ratio from 0% to 95% over training epochs. This schedule is explicitly reported to work better than a fixed mask ratio.

The masked gesture stream first passes through a spatial convolutional encoder and then through temporal self-attention to form body-hint embeddings $\mathbf{h}$. These hints are then used in a cross-attention decoder to reconstruct the latent gesture codes, producing a masked-gesture reconstruction objective:

$$\mathcal{L}_{\mathrm{mg2g}} = \|\hat{\mathbf{q}}_{\mathrm{mg2g}} - \mathbf{q}\|.$$

The same body hints are also injected into the audio-conditioned path. In that branch, EMAGE uses cross-attention to merge audio features and masked-motion hints before reconstructing the gesture latent codes. The paper reports both latent reconstruction and latent class cross-entropy terms for this path, with the goal of improving both correctness and output diversity.

A notable detail from the supplementary material is that, although the architecture is transformer-based, the paper found non-autoregressive training/inference to work better than autoregressive variants when using FLAME-style representations. For that reason, the final system uses a non-autoregressive setup.

3.4 Face decoding and global translation

The face is handled differently from the body. The paper argues that face motion is weakly coupled to body motion, so the body-hint fusion used for the rest of the body is not the best choice for facial decoding. Instead, the face branch directly concatenates the masked body hints with audio features before decoding the facial latent representation.

Once local lower-body motion is decoded, a pretrained Global Motion Predictor estimates the global translation. Its input is the local motion plus predicted foot-contact labels, and the output is $\mathbf{t} \in \mathbb{R}^{T \times 3}$. The supplement reports that this design reduces foot sliding.

4. Training, Metrics, and Reproduction Details

The main EMAGE training schedule is 400 epochs with Adam, learning rate 2.5e-4, and gradient-norm clipping at 0.99. The mask ratio grows linearly over training from 0% to 95%. The supplementary material also reports that a single EMAGE training run on a 4090 takes about 8.0 hours for the 1-speaker setup and 27.1 hours for the 25-speaker setup at the reported best batch size of 64.

The four VQ-VAEs themselves are pretrained for 200 epochs. The supplement states that pretraining the five VQ-related modules used in the full system, including face, hands, upper body, lower body, and global motion components, takes about 22.4 hours on 5×4090 GPUs.

For evaluation, the paper uses the following metrics:

• FGD (Fréchet Gesture Distance) for body realism, computed from latent features of a pretrained skeleton-CNN autoencoder; lower is better.
• Diversity, defined as average $L_1$ distance between generated motion clips; higher is better. The supplement evaluates local diversity with global translation zeroed out.
• BC (Beat Constancy) for audio-motion synchronization; higher is better. Audio beats are derived from speech onset, and gesture beats from local minima in upper-body joint velocity.
• MSE and LVD for face motion quality; lower is better.

The paper’s metric implementation details matter: for FGD, the authors use a Skeleton CNN encoder and a Full CNN decoder pretrained on both BEAT2-standard and BEAT2-additional. They explicitly state that the Skeleton CNN encoder reconstructs better than a Full CNN encoder for their gesture feature extraction.

In baseline reproduction, the authors normalize all comparison methods to output full-body joint rotations in $\mathbb{R}^{T \times (55 \times 6)}$ plus global translations in $\mathbb{R}^{T \times 3}$. They also report that autoregressive variants such as FaceFormer and CodeTalker perform worse than non-autoregressive variants in this representation, and they therefore use non-autoregressive training for their comparisons. Adversarial training from earlier body-gesture work is omitted because it introduces visible jitter.

5. Main Experimental Results

The main quantitative results are reported on BEATv2, with EMAGE compared against a broad set of body-only, face-only, and holistic baselines. The headline result is that EMAGE obtains the best overall body realism and synchronization while also improving facial motion metrics.

The key takeaway is that EMAGE improves the body-side metrics substantially relative to the prior holistic baseline TalkSHOW: FGD drops from 6.209 to 5.512, BC rises from 6.947 to 7.724, and face MSE/LVD both improve. Diversity is slightly below TalkSHOW’s reported value, but still competitive. The authors interpret the FGD gain as evidence that masked body hints help the generator stay closer to the ground-truth gesture distribution.

The user study supports the quantitative findings. In the holistic comparison, EMAGE is preferred over the prior systems in 52.7% of comparisons, with body preference at 44.7% and face preference at 56.0%. The supplement also reports that BEAT2 as a dataset is preferred over TalkSHOW, and is competitive with or better than AMASS and VOCA depending on whether the evaluation focuses on body or face quality.

The paper’s qualitative figures show that EMAGE produces more coherent facial expressions, more expressive upper-body motion, and visibly synchronized gestures compared with previous systems. The facial-motion figure is especially important because it illustrates the paper’s claim that body hints can help facial motion too, even when the face branch is decoded separately.

6. Ablations and What Actually Matters

The ablation study is useful because it reveals that EMAGE’s gains are not coming from one trick alone. The model improves as the authors add discrete priors, split the priors into multiple VQ-VAEs, use CRA for audio fusion, and finally add masked gesture hints. The overall pattern is that each ingredient contributes, but the masked-hint path is the largest single improvement.

A few conclusions follow directly from the numbers. First, a single full-body VQ-VAE is better than no discrete prior, but splitting into four body-part-specific VQ-VAEs is much better, especially for diversity and FGD. Second, CRA improves FGD and synchronization by adaptively balancing rhythm and content. Third, masked hints bring the biggest FGD and diversity gains, confirming that partial-motion conditioning is not just a controllability feature but also a performance booster. One nuance is that the final row slightly worsens LVD relative to the CRA-only ablation, so the best face metric is not uniformly monotonic across every submetric.

The lower-body analysis is especially instructive. The authors point out that lower-body motion can be semantically aligned with speech content, for example walking-like motion for “hiking in nature” or kicking motion for “playing football.” Their ablation shows that conditioning the lower-body decoder on audio plus upper-body and hand latents improves coherence: compared with audio-only decoding, the model with body hints improves FGD from 6.209 to 5.423 and BC from 6.683 to 6.794, while diversity drops slightly.

6.1 Multi-dataset training

A particularly useful engineering result is that EMAGE can be trained on multiple non-holistic datasets. The paper reports joint training with Trinity and AMASS, using separate VQ-VAEs and codebook-classification heads for each dataset. This improves performance on the BEATv1.3 test set, which suggests that the discrete representation and decoupled segment structure transfer well across datasets even when only subsets of the full holistic body are available.

7. Practical Takeaways and Caveats

The paper does not include a separate formal limitations section, but its experiments imply several practical caveats. First, lower-body motion is more weakly related to audio than the face or upper body, so the model benefits from body hints to stay coherent. Second, the strongest reported benchmark numbers are on BEAT2 speaker-2 with a short seed gesture, so the results are best interpreted as evidence that the architecture works well in the paper’s standardized setting. Third, the dataset conversion and facial parameter optimization are non-trivial preprocessing steps; the performance gains are tied to the authors’ careful unification of the raw BEAT annotations into SMPL-X and FLAME meshes.

From a system-design perspective, the most useful lesson for a talking-head or conversational-AI team is that a single architecture can jointly support completion, generation, and motion steering if the latent space is decomposed by body part, the audio branch is split into rhythm/content cues, and masked motion is treated as an auxiliary supervision signal rather than only as missing input.