TalkSHOW

This cross-conditioning is the main mechanism for body-hand coherence: the body predicts the hand, and the hand history is also part of the conditioning context. The autoregressive prior is implemented with a masked Gated PixelCNN and a teacher-forcing training scheme with cross-entropy loss. At inference time, the model samples code indices from multinomial distributions, looks up the corresponding codes, and decodes them into continuous body and hand motion.

The authors emphasize that this design is different from a deterministic encoder-decoder: it enables diverse outputs for the same speech input while preserving synchronization across body parts.

Training and implementation details

The paper provides separate training settings for the face generator, the VQ-VAEs, and the autoregressive model. The face generator is trained with SGD with momentum and learning rate $0.001$ for 100 epochs, using batch size 1 and full-length audio-motion pairs. The face encoder uses zero-mean, unit-variance normalized audio and interpolates extracted features to the target frame rate.

The body-and-hand VQ-VAEs are trained with Adam ($\beta_1 = 0.9$, $\beta_2 = 0.999$) and learning rate $10^{-4}$, with commitment weight $\beta = 0.25$, batch size 128, sequence length 88 frames, and 100 epochs. Their encoders use three residual stages with temporal convolutions, batch normalization, and Leaky ReLU activations. The autoregressive prior uses the same optimizer and batch size, a 15-layer Gated PixelCNN, and the same 88-frame training length for 100 epochs.

The paper also reports that the face and body branches use different audio features for a reason: wav2vec is better suited to lip synchronization, while MFCC is adequate for rhythm-driven gesture generation and converges more reliably for body motion.

Experiments

The paper evaluates motion generation on the dataset it constructed. Clips longer than 3 seconds are split into train/validation/test with an 80%/10%/10% split. The authors compare against the prior SOTA speech-to-motion method of Habibie et al. and against several baselines for the body-and-hand generation branch.

Evaluation metrics

• L2: landmark distance for facial motion, including jaw and lip landmarks.
• LVD: landmark velocity difference, used to assess temporal synchronization between speech and facial motion.
• RS: a learned realism score for body and hand motions based on a binary real/fake classifier.
• Variation: diversity measured from the variance across 16 generated motion samples for the same audio.
• FGD: Frechet Gesture Distance, reported in the appendix for motion realism.
• BC: beat consistency, used to measure alignment between generated body motion and the audio beat.

Main quantitative comparison

The main comparison shows that TalkSHOW improves both face accuracy and body/hand realism over Habibie et al. The strongest gains are in body/hand realism and diversity, consistent with the paper’s claim that discrete compositional latent codes and cross-conditional prediction are essential for multimodal gesture synthesis.

On body and hand motion, the paper also compares against three generative baselines. The deterministic audio encoder-decoder achieves zero diversity, the Audio VAE improves diversity slightly, and the Audio+Motion VAE improves further. TalkSHOW’s VQ-VAE-based body/hand branch is the strongest overall in realism and diversity trade-off.

The appendix adds more metrics and compares TalkSHOW with Habibie et al., Audio VAE, Audio+Motion VAE, and Audio2Gesture. Under these additional metrics, the cross-conditional model achieves the strongest overall trade-off: FGD is lowest at $74.88$, variation remains high at $0.821$, and beat consistency is $0.872$, close to the reported ground truth value of $0.868$.

Qualitative results

The qualitative examples emphasize two main behaviors. First, the face branch produces lip motion consistent with phonemes such as /f/, /t/, /b/, and /æ/. Second, the body-and-hand generator produces multiple plausible gesture variants for the same speech, including different hand selections and different motion amplitudes. The paper highlights that the generated motions can reflect prosodic emphasis, such as the stressed word “But”.

The authors also report that the face generator generalizes to unseen languages and audio types, including French speech and songs, although the paper still frames the face branch as mainly a lip-synchronization model rather than a full emotion model.

Ablation studies

The paper’s ablations isolate three core design decisions: wav2vec features for the face generator, compositional VQ-VAEs for body and hands, and cross-conditional body-hand prediction.

Compositional codebooks. The capacity experiment shows that two separate codebooks for body and hands consistently outperform a single VQ-VAE with a larger codebook, indicating that the compositional latent space better captures the diversity of motion patterns. For example, at codebook size 1024 the single-codebook VQ-VAE has capacity distance $6.2518$, while the compositional VQ-VAEs reduce this to $5.0106$. The same pattern holds across all reported sizes.

Cross-conditional modeling. Removing body-hand conditioning slightly increases raw diversity but hurts realism and synchronization. The reported values are RS $0.4093$ vs. $0.4137$ and SLVD $0.4628$ vs. $0.3445$ for without vs. with cross-conditioning, respectively. In the extended metrics table, the same trend appears in FGD and beat consistency.

Motion diversity vs. realism. The paper’s ablations show that purely deterministic or weakly stochastic models produce motion that is either too rigid or not realistic enough. The VQ-VAE approach provides the most useful balance: it keeps the motion plausible while allowing multiple samples for the same audio input.

User studies

The authors complement the quantitative evaluation with two user studies using Google Forms. For reconstruction, 10 participants assess 40 randomly sampled videos. For generation, participants perform A/B comparisons over 40 sampled outputs per method.

In generation A/B testing, participants prefer TalkSHOW over Habibie et al. with very high margins: $0.888$ for face, $0.910$ for body-and-hands, and $0.913$ for the holistic body. When compared to the p-GT, TalkSHOW is preferred less often than the real data, but still obtains substantially better scores than the prior method.

Applications

Because TalkSHOW outputs a temporally coherent SMPL-X mesh sequence, it can feed downstream neural renderers. The paper demonstrates this with SMPLpix, where the generated mesh vertices and vertex colors are projected onto the image plane and rendered into photo-realistic human avatars. This example is important because it shows that the motion generator is not limited to abstract pose prediction: it can act as a front-end for controllable talking-head or talking-avatar synthesis.

Limitations, risks, and conclusion

The paper is explicit about several limitations. On the reconstruction side, the pipeline remains sensitive to severe hand deformation and heavy occlusion, and it currently assumes a static camera. On the generation side, the face branch focuses mainly on speech-linked mouth motion and does not fully model complex emotion-driven facial behavior. The authors also note that the precise correspondence between words and gestures remains open.

There is also a stated misuse risk: because the system can generate realistic 3D talking humans from audio, it could be used to fabricate convincing motion for speech that never occurred. The paper therefore frames the work as a research tool that should be used responsibly.

In summary, the paper’s main technical lesson is that hierarchical modeling matters: use a high-quality whole-body dataset, reconstruct connected SMPL-X pseudo ground truth carefully, model face and body with different inductive biases, and use discrete compositional latent spaces plus cross-conditioning to obtain diversity without sacrificing coherence. The experiments and user studies consistently support that design choice.