BEAT

BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis

BEAT is a large multi-modal dataset for conversational gesture synthesis, combining body, facial, audio, text, and emotion data from diverse speakers and languages. It supports advanced gesture generation and introduces a new metric for semantic relevance in gestures, enhancing emotion-aware, realistic synthesis.

• multimodal
• gesture
• co-speech
• emotion
• full-body
• talking-head
• avatar

Demos

These demos showcase the BEAT dataset's rich multi-modal annotations for conversational gestures, illustrating semantic relevance and diverse emotional expressions. Watch how gestures semantically align with speech content and how distinct emotions like happiness, anger, and surprise affect gesture style, reflecting the dataset's fidelity and versatility for synthesizing vivid, human-like gestures. Evaluations should focus on gesture naturalness, emotional expressiveness, and semantic appropriateness.

Authors: Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, Bo Zheng

Categories: cs.CV, cs.CL, cs.GR, cs.LG, cs.MM

Comment: 28 pages, 15 figures, Accepted by ECCV2022

Published 2022-03-10 · Updated 2022-09-20

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face177 downloads0 likes

Impact

• Semantic Scholar236 citations54 influential citations

Abstract

Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, models and standard evaluation metrics. To address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i) 76 hours, high-quality, multi-modal data captured from 30 speakers talking with eight different emotions and in four different languages, ii) 32 millions frame-level emotion and semantic relevance annotations. Our statistical analysis on BEAT demonstrates the correlation of conversational gestures with facial expressions, emotions, and semantics, in addition to the known correlation with audio, text, and speaker identity. Based on this observation, we propose a baseline model, Cascaded Motion Network (CaMN), which consists of above six modalities modeled in a cascaded architecture for gesture synthesis. To evaluate the semantic relevancy, we introduce a metric, Semantic Relevance Gesture Recall (SRGR). Qualitative and quantitative experiments demonstrate metrics' validness, ground truth data quality, and baseline's state-of-the-art performance. To the best of our knowledge, BEAT is the largest motion capture dataset for investigating human gestures, which may contribute to a number of different research fields, including controllable gesture synthesis, cross-modality analysis, and emotional gesture recognition. The data, code and model are available on https://pantomatrix.github.io/BEAT/.

---

Introduction

This paper addresses a core bottleneck in conversational gesture synthesis: realistic gestures are not determined by audio alone, but by a coupled set of signals that include speech, text, speaker identity, facial expression, emotion, and semantic intent. The authors argue that prior work was constrained by either limited motion-capture data, noisy pseudo-labeled in-the-wild data, or a lack of paired multi-modal annotations. In response, they introduce BEAT (Body-Expression-Audio-Text), a large-scale motion-capture corpus designed specifically for conversational gesture synthesis with semantic and emotional supervision, and they use it to motivate both a baseline synthesis model and a new evaluation metric.

The paper’s main thesis is that conversational gestures should be treated as a multi-modal generation problem with richer conditioning than is typical in prior audio-to-gesture or text-to-gesture systems. The dataset was explicitly collected to make controlled comparisons possible: the same topics, repeated content across speakers, multiple languages, and multiple emotional states. This makes it possible to study not only gesture generation, but also cross-modal correlation, speaker-style variation, emotion recognition from motion, and semantic relevance in gesturing.

Dataset: BEAT

Scale, modalities, and recording setup

BEAT contains 76 hours of high-quality synchronized multi-modal data from 30 speakers. The paper states that the corpus is organized into conversation and self-talk sessions, with more than 2500 topic-segmented sequences overall. The authors emphasize that the collection protocol was deliberately controlled to preserve motion quality and to expose meaningful variation in topic, language, emotion, and speaker style.

The capture stack is more detailed than in many gesture datasets. Body motion is recorded with a Vicon-based setup using 16 synchronized cameras at 120 Hz and suits with 77 markers. Facial motion is captured with ARKit on an iPhone 12 Pro, producing 52 blendshape weights at 60 Hz. Audio is recorded as 48 kHz stereo. For downstream use, the released motion representation is based on Euler-angle rotations plus root translation, and the supplementary material also describes motion-retargeted avatars for visualizing the recordings.

Session design and linguistic control

The corpus is split into conversation and self-talk. Conversation sessions are remote discussions with an instructor so that only the speaker’s voice is recorded. The paper describes four gesture categories in conversation: talking, instantaneous reactions to questions, thinking/silence, and asking. Topics come from 20 predefined prompts, designed to cover both debate and description styles.

Self-talk consists of 120 one-minute recordings in which speakers answer daily-life prompts. The answers were written and proofread by three native English speakers, and the phonetic coverage was designed to resemble frequently used vocabulary. The self-talk set is also the mechanism used to collect the eight emotion classes in a controlled way, with instructor feedback during recording and re-recording when the emotional expression was judged insufficient.

The language mix is intentionally diverse: English dominates, but the corpus also includes Chinese, Spanish, and Japanese. The authors further report careful control over accents, gender balance, age range, and ethnicity. They divide speakers into a larger-duration group and a few-shot group: 10 speakers have about 4 hours each, while 20 speakers have about 1 hour each. This design is meant both to improve generalization and to support low-data experiments.

Annotation pipeline

Text alignment is generated with an in-house ASR system, then proofread and temporally aligned to audio using Montreal Forced Aligner. Emotion labels are confirmed for self-talk and frame-level emotion annotations are provided for conversation. Semantic relevance annotation is more elaborate: annotators judge whether a gesture is semantically related to the text, identify the gesture span, and select keywords it corresponds to. The final frame-level semantic relevance score is computed by post-processing the segment-level and keyword-level annotations together with text alignment.

The supplementary material reports that 600 annotators were recruited on Amazon Mechanical Turk, but only 118 passed the qualification phase for the final semantic annotation task. The authors also report about $10 per hour compensation. For emotion annotation, inter-rater agreement is reported as high: around 16 million frames reached 96% agreement when the two annotators provided the same label.

Data release format

The released dataset includes multiple synchronized file formats rather than a single monolithic representation. The paper lists BVH files for body and hand motion, stereo WAV audio, JSON facial blendshape weights, FBX facial meshes for 8 speakers, TextGrid alignment files, and text files containing emotion and semantic annotations. This packaging is useful for conversational-AI and avatar pipelines because it exposes both low-level motion and higher-level linguistic/emotional supervision.

Dataset analysis: what BEAT reveals about gesture structure

Emotion and facial expression are strongly coupled with gesture style

A key contribution of the paper is not just dataset release, but the statistical analysis enabled by the paired modalities. The authors show that conversational gestures correlate with facial expressions, emotions, and semantics, beyond the already known links to audio, text, and speaker identity. In the emotion analysis, they visualize gestures with a t-SNE embedding based on a 2-second rotation representation. The resulting clusters are not perfectly separable, but the paper reports clear emotion-dependent structure: for example, anger and happiness are often distinguishable, and anger is associated with larger and faster gestures.

The authors explicitly note that the clusters are confounded and do not form perfectly clean partitions, which is consistent with the fact that conversational gesture expression is not governed by emotion alone. This is important for modeling: the data suggest that a useful synthesis model must not treat emotion as a simple label injection, but as one factor in a more complex interaction among multiple modalities.

Semantic relevance is sparse and speaker-dependent

The semantic analysis shows that gesture-to-text relevance is highly variable. For the same text content, different speakers may place semantic gestures at different positions, with different frequencies and different content. The paper reports that in the analyzed data, 83% of gestures have low semantic scores of at most 0.2. It also reports that high-frequency words differ in their semantic-score distributions, even when their average semantic relevance looks similar. In particular, be-verbs have lower semantic scores than pronouns and prepositions in the analyzed corpus.

This analysis motivates the semantic component of the model and the evaluation metric: if semantic gestures are sparse and unevenly distributed, then a model should be judged not only on geometric realism or audio synchrony, but also on whether it reproduces the moments where humans actually expect meaning-bearing motion.

Baseline model: Cascaded Motion Network (CaMN)

To demonstrate the utility of BEAT, the authors propose CaMN, a cascaded multi-modal gesture synthesis baseline. The model conditions on text, emotion, speaker identity, audio, and facial blendshape weights, and it synthesizes body and hand gestures in a staged architecture rather than as a single end-to-end fusion block. The design is influenced by the observation that body motion can inform hand motion, and that audio and facial signals should refine the representation after language and identity have already shaped the coarse motion plan.

Encoders

The text stream is first embedded with pre-trained FastText vectors of size 300, then refined by an 8-layer temporal convolution network with skip connections. For each frame, the text encoder aggregates a temporal neighborhood of 34 frames, producing a 128-dimensional latent representation. Speaker identity is represented as a one-hot vector over 30 speakers and embedded into an 8-dimensional latent vector. Emotion is also represented with a one-hot label over 8 classes and encoded with an embedding layer plus a 4-layer temporal convolution network to capture temporal variation.

Audio is represented as raw waveform chunks downsampled to 16 kHz and aligned at 15 FPS, with each frame carrying a 1067-dimensional vector. The audio encoder is a 12-layer temporal convolution network followed by a 2-layer MLP; text, speaker, and emotion features are concatenated into the later audio layers so that the audio representation is refined in the presence of other modalities. Facial expression uses 52-dimensional ARKit blendshape weights and is encoded by an 8-layer temporal convolution network plus a 2-layer MLP, again refined using the other modalities.

Body-hand cascade

The decoder is split into body and hand stages. The body decoder predicts a body latent state, and the hand decoder conditions on the body latent state as well as the same fused multi-modal features. This reflects the assumption that hand movement can be better estimated after the model has already planned the global body motion. The final outputs are a body pose $\hat{\mathbf{v}}^{B}$ in $\mathbb{R}^{27 \times 3}$ and hand pose $\hat{\mathbf{v}}^{H}$ in $\mathbb{R}^{48 \times 3}$.

The paper also states that the first four frames are used as a seed pose. In other words, CaMN is not purely unconditional sequence generation; it is a conditioned sequential predictor that uses prior motion together with the multi-modal context.

Losses and semantic weighting

The model is trained with a gesture reconstruction loss and an adversarial loss. Reconstruction uses an $L_1$ penalty on body and hand coordinates, with a weighting factor to balance the two parts. The adversarial term is applied on the generated gesture sequence itself. The final training objective then scales the reconstruction term by a semantic relevance weight so that clips with higher semantic importance are encouraged to match ground truth motion more closely.

Conceptually, the loss can be read as

$$ \mathcal{L} = \lambda\,\beta_0\,\mathcal{L}_{\mathrm{rec}} + \beta_1\,\mathcal{L}_{\mathrm{adv}}, $$

where $\lambda$ is derived from semantic relevance and $\mathcal{L}_{\mathrm{rec}}$ is the body-plus-hand reconstruction term. The important modeling idea is that semantically meaningful clips should be optimized more conservatively, while low-semantic clips can tolerate more generative freedom.

Semantic evaluation metric: SRGR

The authors argue that standard gesture metrics are not enough for semantically conditioned synthesis. In particular, $L_1$ distance can be a poor proxy for diversity and human preference, because a model can have a larger motion range while still producing motions that humans judge as less diverse or less appropriate. To address this, they propose Semantic-Relevant Gesture Recall (SRGR), which combines a gesture recall criterion with semantic relevance weights.

The metric is based on the usual PCK-style notion of counting joints that are within a threshold distance from the ground truth, but clips with higher semantic relevance contribute more to the score. In effect, SRGR asks whether the model can recall the motion content in the parts of the utterance where semantic gesture matters most. The paper also interprets this as a better proxy for vividness and diversity than raw coordinate distance.

Experiments and results

SRGR validity study

The first experiment checks whether SRGR aligns with human judgments. The authors randomly trim motion sequences into roughly 40-second clips and ask participants to rate two properties on a 5-point Likert scale: diversity, defined in terms of non-repeated gestures, and attractiveness, defined from the motion itself rather than the speech content. They collect judgments from 160 participants, with each participant rating 15 clips. The comparison includes Seq2Seq, S2G, A2G, MultiContext, and ground truth, 40 clips each.

The main finding is that $L_1$-based diversity has high variance and can disagree with human perception. A method with strong semantic relevance but a smaller motion range can receive a lower $L_1$ diversity score while still being perceived by humans as more diverse or attractive. The paper concludes that SRGR better matches subjective evaluation than a simple equal-weight sum of $L_1$ distances.

Dataset quality study

To validate capture quality, the authors compare BEAT against Trinity and S2G-3D by training gesture synthesis systems on the respective datasets and then asking subjects to judge rendered clips. The user study asks about physical correctness, diversity, and synchrony; the paper also notes that body and hand correctness are evaluated separately. The reported outcome is that BEAT is preferred in all tested aspects, with especially large gains for hand motion. The authors attribute this to the stronger capture setup and the presence of hand markers, which are often weak or noisy in older datasets.

For the comparison with Trinity, the paper states that BEAT again receives higher user preference across correctness, diversity, and audio synchrony, with the hand motions showing the clearest advantage. The exact Trinity table values are not present in the supplied excerpt, but the conclusion is that BEAT-trained models are fit to a more physically correct, diverse, and attractive motion distribution.

Baseline training and evaluation protocol

The baseline is trained with Adam at a learning rate of $2 \times 10^{-4}$ on NVIDIA V100 hardware for the 4-speaker setup used in the main experiments. The data split is performed per speaker with a 10:1:1 train/validation/test ratio rather than a speaker-disjoint split, because speaker ID is itself one of the conditioning inputs. The selected model checkpoint for each baseline is chosen by the lowest validation FGD score.

For objective evaluation, the authors use FGD, SRGR, and BeatAlign. FGD measures distance between the latent distributions of generated and real motion using a pretrained LSTM autoencoder. BeatAlign measures audio-gesture beat correspondence via a Chamfer-style distance between gesture beats and audio beats. The paper explicitly notes that $L_1$ is unsuitable as the primary metric for gesture quality.

The baselines include Seq2Seq, S2G, A2G, and MultiContext, alongside the authors’ own end-to-end and cascaded variants. The paper reports that both of the proposed variants achieve state-of-the-art performance across the reported metrics, although the numeric table entries are not fully visible in the supplied excerpt. The key takeaway is that the cascaded design and the extra modalities improve distributional similarity, semantic relevance, and synchrony relative to prior methods.

Ablation findings

The ablation discussion is quite informative about what each modality contributes. The cascaded connection itself improves performance over an end-to-end alternative, because it injects prior structure into feature extraction and motion prediction. Removing audio reduces synchrony substantially, which is expected, although the model still retains some synchrony through aligned text timing and facial/lip motion. Removing the semantic weighting can actually improve synchrony, which the authors interpret as evidence that semantic gestures are not always tightly coupled to audio beats.

The paper further reports that emotion and speaker identity have some effect on synchrony, but speaker identity alone has little effect. In contrast, SRGR is mainly affected by text and speaker ID, and is less sensitive to audio, emotion, or facial expression removal. For FGD, every modality helps, with audio plus facial expression contributing especially strongly. The authors also observe that the more integrated architecture increases feature diversity and therefore helps the generated distribution match the real motion distribution more closely.

Emotional gesture recognition sanity check

To test whether generated gestures preserve emotional information, the paper trains a 1D CNN plus LSTM classifier and asks human subjects to classify clips. The classifier is trained and evaluated on speaker-4 ground truth data. The resulting accuracy gap between real and generated data is 15.85 points on average, which indicates that the generated gestures capture the emotional categories but are still notably weaker than real motion.

Limitations and caveats

The authors are explicit about two important limitations. First, the self-talk sessions may still reflect some acting bias, because the content is pre-scripted and emotional states are elicited in a controlled way. To mitigate inconsistencies, they filtered out about 21 hours of data and six speakers whose styles were not consistent with the desired recording protocol. Second, SRGR currently depends on semantic annotations, which makes it difficult to apply to unlabeled corpora. The paper suggests learning a scoring network or semantic discriminator as a future direction.

A subtle but important implication is that BEAT is intentionally controlled rather than fully in-the-wild. That is a strength for benchmark building, but it also means the dataset is optimized for studying gesture structure, controllable synthesis, and multi-modal analysis rather than noisy unconstrained behavior. The paper therefore positions BEAT as a foundation for both generation and recognition research, not as a direct replacement for naturalistic video corpora.

Conclusion

BEAT is presented as a large, carefully curated dataset for conversational gesture synthesis that combines body motion, facial expression, audio, text, emotion, and semantic relevance. The dataset’s value comes from both scale and design: controlled topic prompts, multiple languages, balanced emotions, paired modalities, and frame-level semantic labeling. On top of BEAT, the authors propose CaMN, a cascaded multi-modal gesture generator, and SRGR, a semantic-aware evaluation metric that is designed to be closer to human judgment than raw coordinate distance. The experiments support three claims: BEAT is a higher-quality benchmark than prior motion-capture datasets, semantic weighting matters for synthesis, and richer modality integration helps gesture generation.

For teams building talking-head or conversational-AI systems, the main practical takeaway is that gesture synthesis should be conditioned and evaluated in a semantically aware multi-modal framework, rather than as a purely audio-driven motion regression problem.