Audio-to-Photoreal Embodiment
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
Generates photorealistic avatars that gesture naturally in conversations from speech audio. Combines vector quantized pose sampling with diffusion for detailed motion, using a new multi-view dataset for training and evaluating full body, face, and hand gestures in dyadic interactions.
Demos
These demos showcase the system's ability to synthesize photorealistic full-bodied avatars from conversational audio, capturing diverse and expressive face, hand, and body gestures. Viewers should note the naturalness and dynamic range of motions, including nuanced expressions like wrist flicks and smirks, as well as the ability to generate multiple plausible gesture samples from the same audio input. Comparisons against prior state-of-the-art and generalization to arbitrary audio further highlight the method's fidelity and practical application.
Links
Paper & demos
Impact
Abstract
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online.
Overview and central contribution
This paper tackles audio-driven conversational gesture synthesis for fully embodied photorealistic avatars in a dyadic setting. Given the speech audio of a conversation, the system generates motion for one participant, including face, body, and hands, and then renders that motion with a learned photorealistic avatar renderer. The main technical idea is a coarse-to-fine generation pipeline for the body: an autoregressive vector-quantized transformer first samples diverse low-rate guide poses, and a diffusion model then infills high-frequency details. For facial motion, the paper uses a separate audio-conditioned diffusion model augmented with a pretrained lip regressor. The authors argue that photoreal rendering is not just a visualization choice: it exposes subtle motion errors and makes evaluation of conversational gestures more faithful.
Why photorealism matters for conversational motion
A recurring claim in the paper is that coarse meshes and skeletal renderings can hide the very details that matter most in conversation: gaze shifts, smirks, grimaces, and other micro-expressions that strongly affect perceived mood and intent. The authors contrast this with photoreal avatars, where the same coarse pose can read as very different social signals depending on facial detail and texture. This motivates both the dataset design and the evaluation protocol: the paper explicitly studies whether human raters judge motion differently when it is rendered as a mesh versus as a photoreal avatar.
Photorealistic conversational dataset
The paper introduces a new multi-view dyadic conversational dataset designed specifically to support photorealistic reconstruction and motion learning. The dataset contains 8 hours of video from 4 participants, with each participant contributing about 2 hours of paired conversational data. The conversations are non-scripted and cover a variety of situations, including selling, interviews, difficult scenarios, and everyday discourse, so the motion distribution is more diverse than a narrow task-specific setting.
The capture setup uses two simultaneous domes: one participant stands in a full-body dome, while the other sits in a head-only dome. During the interaction, both participants view live screens of the other person, enabling simultaneous capture of conversational dynamics. This setup lets the authors reconstruct high-fidelity renderings of the face for one person and the full face, body, and hands for the other, using learning-based avatar rendering. The paper states that this is, to the authors' knowledge, the first dataset to provide photorealistic renderings for dyadic conversational agents with simultaneous face, body, and hand reconstructions.
The dataset release includes audio, video, precomputed joint angles, face expression codes, and trained personalized avatar renderers.
Problem formulation and representations
The input to the model is conversational audio from both speakers. The paper extracts Wav2Vec features for the agent being synthesized and for the conversation partner, writing the audio input as $\mathbf{A} = (\mathbf{a}_{\mathit{self}}, \mathbf{a}_{\mathit{other}}) \in \mathbb{R}^{2 \times d_a \times T}$. This is important because the task is dyadic rather than monologic: the generator must account for both the speaker's audio and the partner's audio context.
The output motion is decomposed into two parts. Face motion is represented as a sequence of latent expression codes $\mathbf{F} = (f_1, \dots, f_T)$ with each $f_t \in \mathbb{R}^{256}$. Body motion is represented as joint rotations $\mathbf{J} = (j_1, \dots, j_T)$, where each $j_t \in \mathbb{R}^{d_j \times 3}$ contains per-joint rotation angles in a forward-kinematic skeleton. The final avatar is produced by a personalized renderer that maps the generated expression and pose for a given view to a photorealistic image sequence.
Model architecture
The full system consists of two generative motion models plus a photorealistic avatar renderer. The face and body are modeled separately because they differ in both dynamics and representation: facial motion is strongly tied to audio synchronization, while body motion has weaker audio correspondence and therefore admits more diverse plausible trajectories. Separating the modalities allows each model to specialize rather than forcing one network to cover everything at once.
Face motion diffusion model
Facial motion is generated with a diffusion model conditioned on audio and on a pretrained audio-to-lip regressor. The regressor predicts a sequence of lip vertices $\mathbf{L} = (l_1, \dots, l_T)$ from audio, which supplies an explicit lip-synchronization cue. The paper defines the forward noising process in the usual DDPM form:
$$ q(\mathbf{F}^{(\tau)} \mid \mathbf{F}^{(\tau-1)}) = \mathcal{N}\big(\sqrt{\alpha_\tau}\,\mathbf{F}^{(\tau-1)},\, (1-\alpha_\tau)\mathbf{I}\big), $$
and learns a denoiser $\mathcal{F}$ that predicts the clean facial sequence from a noisy input:
$$ \mathbf{F}^{(0)} \approx \mathcal{F}(\mathbf{F}^{(\tau)};\, \tau,\, \mathbf{A},\, \mathbf{L}). $$
Training uses the simplified diffusion objective described in the paper as $\mathcal{L}_{\mathit{simple}}$. The model also uses classifier-free guidance during training by randomly dropping either audio or lip conditioning with low probability. Audio and lip information are injected via cross-attention, and the diffusion timestep is incorporated with FiLM. The paper reports that conditioning on both audio and lip predictions improves lip-sync quality compared with audio-only conditioning.
Body motion: coarse guide poses plus diffusion infilling
The body-hand generator is also diffusion-based, but it is not run directly from audio alone. The authors found that audio-only body diffusion produces motion that is less diverse and can feel temporally uncanny. Instead, they use a two-stage design: first, an autoregressive transformer samples guide poses at a low frame rate; second, a diffusion model infills high-frequency motion between those guide poses.
During training, the guide poses are obtained by subsampling the ground-truth 30 fps pose sequence down to 1 fps. The diffusion body model is then conditioned on audio and on these sparse guide poses. At inference time, the guide poses must be generated, so the authors train a transformer to predict them autoregressively from audio.
This design is the paper's main mechanism for combining diversity and detail. The transformer produces coarse, editable trajectories that can vary substantially across samples, while the diffusion model adds realistic intermediate motion and local dynamics. The resulting samples include expressive gestures such as pointing, clapping, itching, and other more extreme pose changes that a pure diffusion model or a pure VQ model may miss or smooth out.
Guide pose generation via residual VQ and autoregressive Transformer
To make the coarse pose sequence amenable to autoregressive modeling, the paper first quantizes it using a residual VQ-VAE. Residual quantization recursively encodes the residual errors of previous quantization stages, which improves reconstruction quality over a single-codebook VQ. The resulting token sequence is flattened and modeled as next-token prediction by an audio-conditioned Transformer.
If $\mathbf{Z} = (z_1, \dots, z_K)$ denotes the quantized guide-pose codes, the transformer models $$ p(\hat z_k \mid \hat z_{1:k-1}, \mathbf{A}) = \mathcal{P}(\hat z_{1:k-1}; \mathbf{A}), $$ and is trained with standard cross-entropy teacher forcing: $$ \mathcal{L}_{\mathcal{P}} = -\sum_{k=1}^{K \cdot N} \log \Pr\big[\mathcal{P}(z_{1:k-1}, \mathbf{A}) = z_k\big]. $$
At test time, the paper uses nucleus sampling to control diversity. Larger cumulative probability yields more varied samples, while smaller values tighten the distribution around more likely guide-pose continuations. The appendix reports a residual length of 4, codebook size of 1024, embedding size of 64, 1D convolutions with kernel size 2, total receptive field 8, and training for about 300k steps. The guide-pose Transformer uses 2 masked self-attention layers and 6 cross-attention layers with 8 heads, and is trained for roughly 100k iterations per subject.
Photorealistic avatar rendering
After motion generation, the system renders the avatar with a learned, subject-specific neural renderer based on prior work on drivable avatars. For each frame, the renderer takes the facial expression code, body pose, and view direction, and outputs registered geometry and view-dependent texture. Images are then synthesized by rasterization. The renderer is a conditional variational autoencoder trained end-to-end on multi-view images of each subject. The paper trains a personalized renderer for each participant in the dataset.
Training and implementation details
The paper trains the models separately for each subject. Sequences are up to 600 frames at 30 fps, corresponding to 20-second windows. During training, the authors randomly sample sequence lengths between 240 and 600 frames and pad variable-length sequences so the models can later generate arbitrarily long sequences. All networks were trained on a single A100 GPU. The appendix reports approximate training times of 8 hours for the face diffusion model, 5 hours for the VQ plus coarse pose predictor, and 8 hours for the pose diffusion model.
Evaluation protocol
The paper evaluates motion realism, diversity, lip sync quality, and human preference. To assess motion quantitatively, the authors use a mix of distributional and temporal metrics adapted from prior gesture-generation work. They also run Mechanical Turk A/B tests to measure perceptual quality and to study whether photoreal rendering changes the way humans judge motion.
The main motion metrics are: $\mathrm{FD}_g$, geometric Frechet distance between generated and ground-truth static expressions or poses; $\mathrm{FD}_k$, the same distance computed on motion velocities; $\mathrm{Div}_g$, average pairwise L2 distance between randomly sampled static pose or expression pairs within a sequence; $\mathrm{Div}_k$, temporal variance within a sequence; and $\mathrm{Div}_{\mathit{sample}}$, diversity across multiple samples generated from the same audio. Lower is better for the FD metrics, while higher is better for the diversity metrics.
Quantitative results
The paper compares against Random, KNN retrieval, SHOW, LDA, and several ablations. The strongest qualitative theme in the results is that the full model achieves a better balance between realism and diversity than either a VQ-only or diffusion-only approach. In particular, adding guide-pose conditioning substantially improves diversity and realism, while adding audio conditioning on top of guide poses yields the best overall scores.
| Method | $\mathrm{FD}_g$ | $\mathrm{FD}_k$ | $\mathrm{Div}_g$ | $\mathrm{Div}_k$ | $\mathrm{Div}_{\mathit{sample}}$ |
|---|---|---|---|---|---|
| GT | -- | -- | 3.09 | 2.50 | -- |
| Random | 9.37 | 1.44 | 3.10 | 2.49 | 3.97 |
| KNN | 8.44 | 0.62 | 2.13 | 1.21 | 1.96 |
| SHOW | 4.97 | 2.60 | 2.10 | 0.77 | 2.82 |
| LDA | 5.08 | 1.04 | 2.45 | 1.88 | 2.68 |
| Ours uncond | 8.45 | 1.53 | 2.74 | 2.06 | 2.94 |
| Ours w/o guide poses $\mathbf{P}$ | 5.08 | 1.13 | 2.47 | 1.67 | 2.06 |
| Ours w/o audio $\mathbf{A}$ | 3.94 | 0.98 | 2.69 | 2.16 | 2.71 |
| Ours | 2.94 | 0.96 | 2.98 | 2.36 | 3.58 |
The quantitative table shows three especially important trends. First, the full model achieves the lowest Fréchet distances among the learned methods, indicating better alignment with the ground-truth motion distributions. Second, it also achieves the highest diversity among the learned methods, especially in sample-to-sample variation. Third, the ablation without guide poses performs much worse than the full model, showing that the coarse-to-fine decomposition is not just an implementation detail but a core source of improved plausibility and variation. The ablation without audio also improves over unconditional generation, but the best results require both audio and guide poses.
| Method | Horizontal L2 error | Vertical L2 error | Mesh L2 |
|---|---|---|---|
| SHOW | 2.76 | 2.15 | 2.25 |
| Ours w/o lip regressor $\mathbf{L}$ | 2.62 | 2.43 | 2.24 |
| Ours | 2.29 | 1.89 | 1.76 |
The lip-specific evaluation shows that the pretrained lip regressor materially improves synchronization and mouth shape accuracy. The paper measures vertical and horizontal mouth opening via keypoint distances and also reports mesh-vertex error on the lip region. Compared with the version without lip conditioning, the full model reduces all three errors. The qualitative explanation in the paper is that lip conditioning reduces random opening/closing when the speaker is not talking and improves the overall mouth dynamics.
Perceptual evaluation and the role of photorealism
The paper's perceptual study is designed to answer two questions: whether the proposed model is preferred over a strong diffusion baseline, and whether evaluation results change when the same motion is rendered as a mesh versus photorealistically. The authors run A/B tests on Amazon Mechanical Turk, generating 50 tests per comparison condition and using 3 evaluators per test, for a total of 600 evaluators. Each test contains 14 questions. Participants are shown video pairs and asked which motion looks more plausible given the conversational audio, with an option to indicate slight or strong preference.
The result is that the proposed method is preferred over LDA in both mesh and photoreal settings, with about 70% of raters preferring the paper's method in each case. However, the confidence of preference changes: when motion is rendered photorealistically, evaluators move from slightly preferring the paper's method to strongly preferring it more often. This suggests that photorealistic rendering makes subtle motion differences more visible and therefore makes the assessment more discriminative.
When comparing against ground truth, the paper reports an important asymmetry: the model looks fairly competitive in mesh form, but in the photoreal setting raters more often prefer the ground truth, with 43% strongly preferring GT over the model. The authors interpret this as evidence that meshes can hide motion mistakes that are obvious once the avatar is rendered photorealistically. In other words, photorealism does not merely improve appearance; it changes the evaluation regime by revealing fine-grained motion errors.
Qualitative findings and additional behavior
The paper repeatedly emphasizes synchronized dynamics between motion and speech. In listening segments, the system tends to produce still, attentive motion; in speaking segments, it produces motion peaks and expressive gestures aligned with audio. The appendix also notes that the model can generalize, without retraining, to some unseen conversational audio such as a movie clip, and that guide poses can be swapped for a form of video editing or reanimation. These behaviors are presented as qualitative evidence that the model is not merely memorizing training sequences but has learned a useful audio-to-motion prior.
Limitations and ethical considerations
The paper is unusually explicit about limitations. First, the model operates on short-range audio, so it cannot generate gestures that require longer-range language understanding, such as counting or other discourse-level behaviors. Second, the dataset and rendering setup are limited to four subjects, so the approach is not presented as a general arbitrary-human renderer. Third, the photorealistic setting is intentionally tied to consenting participants, which the authors frame as an ethical advantage over systems that might render arbitrary non-consenting humans.
The authors position the dataset release as an ethically grounded way to study photoreal motion synthesis: the participants consented to the captures, and the released personalized avatars are specific to those people. In the paper's framing, this makes the work useful both as a technical benchmark and as a safer research setting for photorealistic human motion synthesis.
Bottom line
The paper's main contribution is a photorealistic conversational avatar pipeline that combines a VQ-based coarse pose prior with a diffusion-based refinement stage, plus a learned photoreal avatar renderer and a new dyadic multi-view dataset. Empirically, the method improves both realism and diversity over VQ-only, diffusion-only, retrieval, and unconditional baselines, and the perceptual study makes a strong case that photorealism is essential for judging the subtle motion details that matter in conversation.