Speech2Gesture

Learning Individual Styles of Conversational Gesture

This paper introduces a person-specific speech-to-gesture model that learns to generate individual hand and arm gestures from audio using unlabeled videos. It uniquely captures a speaker's gesture style by translating speech to temporally aligned poses while ensuring realistic motion with adversarial training.

• speech-driven
• gesture
• co-speech
• audio-driven

Demos

These demos illustrate the model's ability to generate plausible, speaker-specific arm and hand gestures directly from speech audio in natural settings. Watch for how the predicted gestures align with speech rhythm and individual styles across diverse speakers. The overlays demonstrate improvements over baselines by capturing unique conversational gesture patterns from noisy pose data.

Authors: Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, Jitendra Malik

Categories: cs.CV, cs.LG, eess.AS

Comment: CVPR 2019

Published 2019-06-10 · Updated 2019-06-10

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub394 stars23 watchers43 forks

Impact

• Semantic Scholar395 citations67 influential citations

Abstract

Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at http://people.eecs.berkeley.edu/~shiry/speech2gesture .

---

Overview

This paper studies speech-to-gesture translation for conversational, co-speech motion in the wild: given an audio clip of a speaker’s monologue, the model predicts a temporally aligned sequence of the speaker’s arm and hand poses. The key framing is person-specific cross-modal translation: the goal is not to generate generic gestures, but to learn the characteristic motion style of a particular speaker from unlabeled internet video. The setting is deliberately difficult because speech and gesture are often asynchronous, the mapping from words to gesture is multimodal, and human pose supervision is unavailable, so training relies on noisy pseudo labels from an automatic pose detector.

The paper’s main contributions are: (1) a large speaker-specific gesture dataset built from in-the-wild videos; (2) a fully convolutional audio-to-pose model that predicts a whole motion sequence from a whole audio segment; (3) an adversarial term that reduces over-smoothing and encourages plausible, speaker-consistent motion; and (4) an evaluation showing that the method outperforms several baselines on pose regression metrics, while also producing gestures that can be perceptually convincing to humans. The paper also includes an appendix-style analysis of gesture-unit segmentation and unsupervised gesture dictionaries for individual speakers.

Problem Formulation

The paper asks whether the relationship between speech and conversational gesture can be learned from data rather than hand-crafted rules. Formally, the input is an audio segment $\mathbf{s}$ and the output is a temporally ordered stack of 2D pose keypoints $\mathbf{p}$. The predicted pose sequence covers the full time span of the audio clip rather than being generated frame-by-frame with a recurrent decoder. This sequence-level formulation is intended to help with temporal smoothness and with the fact that gestures can begin before, during, or after the speech they accompany.

The authors emphasize that there is no direct one-to-one target between utterance content and gesture: a speaker may produce different valid gestures for the same words, and the same gesture style can appear with many different utterances. Therefore the learning objective must tolerate ambiguity while still extracting a signal from noisy pseudo ground truth pose detections.

Dataset: Person-Specific In-the-Wild Gesture Video

To support speaker-specific modeling, the paper introduces a dataset of 10 speakers and 144 hours of video in total. The speakers are drawn from three broad types of public-speaking contexts: 5 talk show hosts, 3 lecturers, and 2 televangelists. The dataset is intentionally diverse in topics and speaking styles, spanning television commentary, lectures, and religious speaking. The data are assembled by querying YouTube, removing duplicates, and retaining intervals in which a single subject is visible and all keypoints are detected.

The dataset contains 60,000 intervals, with average interval length 8.7 seconds and standard deviation 11.3 seconds. The split is 80% train, 10% validation, and 10% test, with each source video appearing in only one split. The pose representation uses 49 2D keypoints from OpenPose: neck, shoulders, elbows, wrists, and hands. The authors provide the keypoints at 15 fps, using them as pseudo ground truth rather than human annotation.

The dataset analysis in the paper is itself an important result. The heatmaps show that speakers have distinct resting poses and distinct motion habits, supporting the claim that gesture is highly idiosyncratic and that per-speaker models are more appropriate than a single universal model.

Because annotations come from automatic pose detection, the paper explicitly checks whether this pseudo ground truth is accurate enough. On a 200-frame subset labeled by three human annotators, the authors report that the OpenPose detections are close to human labels and that the label noise is small relative to the prediction error. Concretely, the mean distance between OpenPose and the annotation mean is reported as $0.14$, while the mean annotator standard deviation is about $0.06$; the model’s own prediction error on this subset is about $0.25$, which the authors interpret as evidence that the pseudo labels are noisy but usable for their task.

Model Architecture

The proposed system is a fully convolutional audio-to-pose translator with two stages: an audio encoder and a 1D U-Net-style translation network. The audio side takes a 2D log-mel spectrogram and downsamples it into a 1D temporal signal at the same frame rate as the pose stream, 15 Hz. The translation module maps that signal to a full sequence of 2D pose vectors. The use of a U-Net is motivated by two properties that matter for gesture generation: the bottleneck provides large temporal context, and the skip connections preserve high-frequency details that can support fast motion.

The pose target is a temporal stack of keypoint vectors, trained with a direct regression loss:

$$ \mathcal{L}_{L_1}(G) = \mathbb{E}_{\mathbf{s},\mathbf{p}}\big[\lVert \mathbf{p} - G(\mathbf{s}) \rVert_1\big]. $$

This is the only supervision available from the unlabeled video data. Because simple regression tends to average over multiple plausible gesture modes and produce overly smooth motion, the paper adds an adversarial discriminator on motion dynamics. The discriminator is conditioned on a motion-difference sequence, formed by taking frame-to-frame differences of the predicted pose stack, so it focuses on temporal plausibility rather than absolute location alone.

In the paper’s notation, if $\mathbf{m}$ denotes the stacked motion derivatives, the full objective is a GAN-style min-max problem with an $L_1$ term:

$$ \min_G \max_D \; \mathcal{L}_{GAN}(G, D) + \lambda \mathcal{L}_{L_1}(G). $$

Intuitively, the regression term anchors the output to the noisy pseudo labels, while the adversarial term pushes the outputs toward the set of motion sequences that look real for the current speaker.

Implementation Details

The authors make several preprocessing and optimization choices that are important to reproduce the behavior of the model. First, they subtract the neck keypoint from all other keypoints on a per-frame basis to obtain translation invariance. Second, they normalize each keypoint coordinate per speaker by subtracting the speaker mean and dividing by the speaker standard deviation. This further adapts the representation to individual body scale and camera framing.

During training, the network consumes roughly 4 seconds of audio and predicts 64 pose vectors, corresponding to about 4 seconds at 15 fps. At test time, the model can process arbitrary-length audio clips. Optimization uses Adam with batch size 32 and learning rate 10^{-4}. The model is trained for 300k iterations without the adversarial loss and 90k iterations with it; the best validation model is selected.

Baselines and Evaluation Protocol

The paper compares against a set of simple and not-so-simple baselines designed to probe different aspects of the task.

• Median pose: always predict the speaker’s median pose, motivated by the fact that speakers often rest in a neutral position.
• Random gesture: randomly sample a gesture sequence from the same speaker’s training set, ignoring the input audio.
• Nearest neighbor: retrieve the training utterance whose audio is nearest to the test audio under pretrained VGGish features, then transfer its motion.
• RNN baseline: an adapted version of the audio-to-motion model from Shlizerman et al., using a 1-layer LSTM with MFCC input and an extra hidden layer so that the comparison is based on pose-space $L_1$ error.
• Ours, no GAN: the full translation network without the adversarial term, used as an ablation.

The paper uses two main metrics. The first is the average $L_1$ regression loss over keypoint coordinates, which is the primary metric because the training signal itself is regression to noisy pose pseudo labels. The second is PCK (percent of correct keypoints), evaluated with thresholds $\alpha = 0.1$ and $0.2$ relative to the person bounding box size. The authors note that PCK is less natural for this cross-modal task than $L_1$ because it imposes a hard threshold and can be sensitive to large motion spans, but it is included as a familiar pose-estimation metric.

Quantitative Results

The main table in the paper reports test-set performance on 2,048 randomly chosen intervals per speaker. Overall, the no-GAN model gives the strongest average quantitative performance, while the GAN version tends to trade some $L_1$ accuracy for more realistic motion dynamics. The paper interprets this as a classic regression-versus-plausibility trade-off: adversarial training pushes the generator toward a sharper mode of the motion distribution, which can move it away from the average target pose under $L_1$.

The reported averages show that the method without GAN is best on average in both $L_1$ and PCK: $0.67$ and $44.62$, respectively. The GAN version still improves over the simple baseline methods and remains competitive with the RNN baseline, but the paper’s own analysis suggests that the discriminator makes the output more sharply realistic at the cost of some regression accuracy.

Human perceptual study

To test whether the generated motion looks realistic to humans, the authors run a two-alternative forced-choice study on Amazon Mechanical Turk. Participants are shown pairs of skeletal wire-frame videos, one real and one generated, and asked to choose the one that matches the speech. The study uses two speakers: Oliver, who is dynamic, and Meyers, who is comparatively stationary. Video clips are shown at 4 seconds and 12 seconds, with ground-truth mouth and face keypoints included for context.

The main takeaway is that for the dynamic speaker, methods that produce realistic motion can fool people at similar rates even when they are not well aligned to the input audio. For the stationary speaker, the real motion is more often chosen because there is less motion to get wrong and prediction noise becomes more noticeable.

The authors note that for Oliver, their GAN-based model is perceptually competitive with real motion and with the randomly selected or nearest-neighbor motion from the same speaker, underscoring the point that plausible motion and audio alignment are not the same thing. This is an important nuance for conversational AI systems: realistic body motion can look convincing even when the exact speech-movement correspondence is imperfect.

Qualitative examples

The paper provides visual examples for a lecturer and a show host, showing the input audio spectrogram and predicted pose sequences overlaid on the original video. These examples are intended to convey the temporal character of the predictions, which is harder to assess from static frames or scalar metrics alone.

What the Ablations and Auxiliary Analyses Show

Person-specificity

The paper tests whether the learned mapping is speaker-specific by applying models trained on one speaker to the speech of every other speaker. The resulting confusion matrix has a strong diagonal structure: the model performs best when the training speaker and test speaker match. Off-diagonal performance is better than random motion but much worse than the median pose of the target speaker, confirming that the network learns speaker identity and not just a generic audio-to-gesture prior.

Does speech add information beyond the initial pose?

The authors also ask whether audio contributes useful predictive information once the initial pose of a gesture sequence is known. They compare a simple repetition baseline, an initial-pose-only model, and a model that conditions on both speech and initial pose. The results show that the initial pose is already informative, but adding speech improves performance further, particularly for speakers who make larger and more expressive gestures.

The key result is that adding speech to an initial-pose-conditioned model improves average $L_1$ to $0.47$ and PCK to $62.39$, with the authors reporting statistical significance at $p < 10^{-3}$ under a two-sided $t$-test. This supports the paper’s central claim that the audio signal contains useful cues beyond motion continuity alone.

Qualitative interpretation of the generated motion

The qualitative examples in the paper highlight two recurring behaviors. First, the model can reproduce broad motion style, such as whether a speaker tends to make expansive gestures or remain relatively still. Second, the generated output can still drift away from exact temporal alignment, which is expected because the same audio can correspond to multiple valid gestures and because the training target is noisy. The authors explicitly note that the adversarial discriminator often produces motion that looks more realistic, even when the regression score is not optimal.

Appendix: Gesture Units and Individual Gesture Dictionaries

Beyond the main speech-to-gesture model, the appendix describes an unsupervised pipeline for mining a speaker’s gesture repertoire. The idea is to segment motion into gesture units, define a compact descriptor for each unit, and cluster the units to build an individual gesture dictionary.

A gesture unit is defined as a sequence that starts from a rest position and returns to rest only after the last gesture. The segmentation procedure is unsupervised and is based on prediction error: given motion from time $t_0$ to $t_T$, the model predicts the next pose $t_{T+1}$, and a high error is treated as evidence that a new gesture unit has begun. This is a per-speaker segmentation model and is not intended to generalize across speakers.

For dictionary learning, the paper uses the first 5 principal components of static keypoint frames, which capture 93% of the variance. Similarity between gesture units is measured with dynamic time warping to account for temporal differences. The authors then precompute pairwise distances for a sample of 1,000 training gesture units and perform hierarchical clustering.

The resulting clusters are interpreted as a data-driven, individual-specific gesture lexicon. For each cluster, the paper defines a central representative gesture by selecting the unit closest on average to all others in the cluster. The figures for Jon Stewart illustrate how a person’s recurring gestures can be organized into a small number of prototypical motion patterns.

Limitations and Takeaways

The paper is explicit about several limitations. Most importantly, audio does not encode high-level linguistic semantics as directly as text, so the method may miss gesture types that are tied to meaning rather than prosody, such as metaphorical gestures. The model also does not separate the speaker’s voice from background sounds, so laughter or other audio events can interfere with the mapping. A further limitation is the reliance on pseudo ground truth from pose detection, which is noisier than human annotation and is particularly problematic for fingers and fine hand detail.

Methodologically, the paper shows that person-specific gesture generation is feasible from raw audio-visual correspondences alone, without explicit text, gesture categories, or handcrafted gesture rules. At the same time, the results suggest an important trade-off: regression losses favor smooth and accurate average motion, while adversarial training favors sharper, more plausible motion. The work therefore functions both as a practical speech-to-gesture system and as a study of what can and cannot be inferred about conversational gesture from speech audio alone.

For conversational-AI systems, the broader implication is that co-speech motion should likely be modeled as a speaker-conditioned, temporally extended, multimodal prediction problem, not as a simple framewise regression task. The dataset and analyses in this paper are positioned as a starting point for that line of work.