EchoAvatar

EchoAvatar: Real-time Generative Avatar Animation from Audio Streams

EchoAvatar is a real-time system that generates continuous full-body avatar motion from streaming speech and music audio without explicit mode switching. It combines audio-driven motion synthesis with semantic control from large language models to enable expressive, interactive humanoid avatars.

• audio-driven
• full-body
• gesture
• avatar
• realtime
• streaming
• llm

Demos

These demos show EchoAvatar's real-time full-body avatar animation driven by streaming audio inputs, covering both conversational speech and rhythmic music without explicit mode switching. Watch for the seamless audio-to-motion synchronization, natural gesture variety reflecting speech dynamics, and the model's generalization across domains. The ablation studies highlight the benefits of joint training on speech and music datasets, and the critical role of reward models in maintaining motion quality and audio alignment.

Authors: Bohong Chen, Yumeng Li, Yinglin Xu, Youyi Zheng, Yanlin Weng, Kun Zhou

Categories: cs.CV

Comment: SIGGRAPH 2026; Project Page: https://robinwitch.github.io/EchoAvatar-Page

Published 2026-05-27 · Updated 2026-05-27

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar0 citations

Abstract

Real-time synthesis of high-fidelity 3D character motion from audio is a pivotal component for next-generation interactive avatars and virtual assistants. However, most existing approaches are limited to offline processing of complete audio sequences or are constrained to specific domains, rarely handling both speech and music effectively. In this paper, we introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency. Central to our approach is a unified streaming architecture capable of synthesizing continuous motion from incremental audio inputs. We employ a robust training strategy that enforces strong audio dependency, allowing the model to seamlessly generalize across conversational speech and rhythmic music without requiring explicit domain labels or mode switching. Additionally, we explored Reinforcement Learning to refine the quality of online generation. Furthermore, we bridge reactive animation with intent-driven behavior via a tool-call interface that allows upstream Large Language Models to inject explicit semantic control. By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars. Extensive experiments demonstrate that our method outperforms state-of-the-art realtime baselines in motion quality and synchronization while maintaining the flexibility required for live deployment. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/EchoAvatar-Page.

---

Overview

EchoAvatar addresses the problem of generating high-fidelity, full-body avatar motion from streaming audio with low latency. The paper is motivated by a gap between the quality of modern voice agents and the responsiveness of their visual embodiment: many prior audio-to-motion methods either require the full audio sequence before synthesis or are specialized to a single acoustic domain such as speech or music. EchoAvatar proposes a unified streaming system that can synthesize continuous motion from incremental audio inputs, handle both speech and music without explicit domain labels or mode switching, and expose an additional tool-call interface so that upstream large language models can inject semantic control.

The main technical claim is not just that the model is audio-driven, but that it is causal, streaming, and unified across domains. In the authors' framing, this makes the system a plug-and-play module for voice agents: audio streams in, motion streams out, and optional semantic signals or motion exemplars can steer the animation when needed.

Problem Setup and Core Contributions

The paper targets two coupled requirements: real-time responsiveness and domain-unified motion synthesis. The first requirement rules out offline pipelines that wait for complete utterances; the second rules out narrow models that only work for either conversational gesture or rhythmic dance. EchoAvatar's design resolves both by combining:

• a causal motion tokenizer that can encode and decode motion without future-frame look-ahead;
• a pre-trained language model repurposed as a motion generator, trained through a three-stage curriculum;
• a hierarchical corruption strategy that prevents the autoregressive motion prior from overwhelming the audio condition when multiple tasks are merged into one latent space;
• a reinforcement-learning alignment stage, using either proxy rewards (GRPO) or preference labels (DPO), to improve perceived online generation quality;
• a tool-call interface for explicit semantic control and motion-example conditioning.

The paper also contains a deployment-oriented appendix: a distributed real-time system, a face-animation branch, online inverse-kinematics post-processing, and a latency analysis showing the pipeline stays within the per-chunk budget.

Motion Tokenizer: Causal, Attention-Based, and RVQ-Backed

The tokenizer is one of the paper's central innovations. It replaces the causal convolutions used by earlier streaming tokenizers with stacked attention blocks under a causal mask, so each motion token can only attend to the preceding $p$ frames. The authors argue that pure convolutional causality can be too restrictive and can introduce reconstruction artifacts; attention is used to increase expressiveness while preserving strict temporal causality.

Motion is parameterized as root velocity, root height, and 6D joint rotations. The motion sequence $$ is encoded to a latent trajectory, then discretized using Residual Vector Quantization (RVQ) with $Q$ codebooks. Each residual layer refines the error of the previous layer, so the final latent representation is the sum of the quantized residuals. Reconstruction is trained with a combination of pose-level reconstruction, commitment loss, and forward-kinematics-based geometric losses:

$$ \mathcal{L} = \|\hat{\mathbf{m}} - \mathbf{m}\|_1 + \eta \sum_{q=0}^{Q-1} \|\mathbf{z}^q - \operatorname{sg}[\hat{\mathbf{z}}^q]\|_2^2 + \Phi\big(\operatorname{FK}(\hat{\mathbf{m}}), \operatorname{FK}(\mathbf{m})\big) $$

Here, $ \Phi$ bundles auxiliary losses on global joint positions, velocities, accelerations, and foot contact consistency. This is meant to reduce physically implausible artifacts such as jitter and foot sliding. The tokenizer also uses a dual-path temporal resampling strategy inspired by DC-AE: downsampling combines temporal pooling with feature concatenation followed by an MLP, and upsampling combines temporal replication with channel expansion.

Another important detail is anatomical partitioning: the system maintains separate codebooks for the upper body, lower body, and hands, which the authors use to decouple part-specific dynamics.

Tokenizer Ablation Results

The tokenizer ablation table compares reconstruction quality and downstream generation quality. The key trend is that each proposed component matters, and the full attention-based, body-part-aware tokenizer is best overall.

The strongest result is the body-part-aware attention tokenizer: it reduces reconstruction FID to 1.306, lowers MPJPE to 184.1, and improves downstream generation FID to 9.465. The paper also reports that removing the dual-path resampling or the auxiliary FK losses causes large degradations, and that the CausalConv baseline is consistently worse than the attention-based design.

Audio-to-Motion Generation with a Repurposed LLM

EchoAvatar adapts a pre-trained language model, specifically Qwen2.5-0.5B-Instruct, as the generator. The adaptation follows a three-stage curriculum:

1. Embedding space alignment to project audio and motion codebooks into the LLM's latent manifold.
2. Acoustic-kinematic alignment to condition the generator on streaming audio and produce motion tokens.
3. Exemplar-driven control to accept motion examples as explicit stylistic directives.

For audio, the system uses a causal version of EnCodec so that speech and music can share the same discrete interface. For motion, it uses the causal RVQ tokenizer described above. Importantly, the paper does not just model the first RVQ layer: it flattens and interleaves all RVQ layers into a single autoregressive stream, following the MusicGen-style serialization pattern. Because the RVQ layers have different information density, the training objective uses hierarchical loss scaling, with deeper layers assigned monotonically smaller weights so that the model learns coarse structure before fine detail.

The generator also supports motion-example control. The paper states that the conditioning mechanism extracts control tokens from the first-level codebook of the reference sequence, which is treated as the most semantically dense level.

Why Hierarchical Token Corruption Matters

The authors identify a key failure mode when mixing speech-gesture and music-dance into one motion space: the autoregressive motion prior becomes so strong that the model can ignore the audio condition. This is described as conditional collapse. Their theoretical analysis frames the issue as a context-logit floor: shared motion history accumulates enough probability mass to dominate competing trajectories, so the model can become overconfident in motion-history continuation rather than audio-following behavior.

To counter this, they introduce Hierarchical Token Corruption. During training, they stochastically corrupt context tokens in a structure-aware way. For each selected timestep, they sample a random RVQ layer depth and replace tokens from that layer onward, while leaving coarser layers intact. The intuition is that this reduces the model's dependence on the autoregressive context and forces it to use the audio condition more strongly, while also teaching the model to recover from local token errors during long-form inference.

In the appendix, the authors formalize the effect as a reduction in the context penalty by a factor of approximately $1-\rho$, where $\rho$ is the corruption rate. In other words, corruption weakens the self-reinforcing motion-history path and widens the expected logit gap between the correct trajectory and spurious trajectories.

Unified-Training Ablation and Cross-Domain Synergy

The unified benchmark table shows that the corruption strategy is crucial. Without it, the model achieves very high diversity but poor fidelity, which is exactly what one would expect from a system that has collapsed away from the audio condition and is producing unrelated, high-variance motion. With corruption, fidelity improves sharply.

The paper's interpretation is that joint training across gesture and dance is beneficial only when the corruption strategy preserves conditioning. Without corruption, the model can drift into implausible but diverse motion. With corruption, the joint dataset becomes useful and can even transfer rhythmic priors from dance into speech-gesture generation.

Reinforcement Learning Alignment: GRPO and DPO

EchoAvatar explores two RL-based alignment strategies to improve online generation quality beyond maximum likelihood training.

• GRPO (Group Relative Policy Optimization) is used when explicit human labels are not available. The paper constructs proxy rewards from motion quality and audio-motion synchronization.
• DPO (Direct Preference Optimization) is used when human preference labels are available, so the model can be optimized directly against winning and losing samples.

The GRPO objective is written as:

$$ \mathcal{L}_{\text{GRPO}} = -\frac{1}{G} \sum_{i=1}^{G} \rho_i \hat{A}_i + \beta_G \mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}}) $$

where $\rho_i$ is the importance ratio, $\hat{A}_i$ is the group-normalized advantage, and the KL term keeps the updated policy close to the reference model. For DPO, the paper uses the standard preference objective:

$$ \mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\left(\beta_D \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta_D \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right)\right] $$

where $y_w$ and $y_l$ are the preferred and dispreferred motion sequences. The preference data is constructed with Best-of-N sampling: for each audio input, eight candidate motions are generated and human annotators pick the best and worst, with low-contrast pairs filtered out.

Reward Models Used for GRPO

The reward model design is practical and data-efficient. For motion quality, the authors create a synthetic degradation curriculum by corrupting RVQ tokens at varying rates and mapping corruption severity to a quality score using FID ordering. The reward network mirrors the motion tokenizer but uses bidirectional attention and outputs a scalar score in $[0,1]$. The appendix reports a Pearson correlation of $0.9977$ between the motion quality model and the degradation levels on held-out data, which the authors use to argue that the proxy reward is reliable.

For audio-motion synchronization, they train a contrastive audio-motion model. The audio encoder is pretrained BEATs; the motion encoder is a Transformer. Positive pairs are segments from the same source and time window, and mirrored motion variants are also treated as positives. At inference, the reward is the cosine similarity between the normalized audio and motion embeddings. The appendix reports that BEATs performs better than Wav2CLIP on the retrieval task, with much higher R@1, lower median rank, and much better behavior than random retrieval.

RL Results and Trade-offs

The main finding is that RL can improve human preference, but it can also hurt distributional fidelity. The paper explicitly notes a trade-off: FID worsens after RL, and GRPO causes a larger FID shift than DPO. In the unified test set, the no-RL model achieves FID 9.465, while DPO yields 12.39 and GRPO yields 24.13. The authors interpret this as the familiar mode-seeking effect of reward optimization.

In user studies, DPO and GRPO behave differently by domain: GRPO is more effective for dance, while DPO is more effective for conversational gesture. The paper recommends GRPO when the application is dance-heavy and DPO when it is gesture-heavy.

Datasets, Preprocessing, and Auxiliary Branches

The paper evaluates across two primary motion corpora and several auxiliary collections:

• ZeroEGGS: about 2 hours of stylized speech-gesture data from a single speaker across 19 expressive styles.
• Motorica: about 6 hours of rhythmic dance data from five performers across eight genres.
• RL alignment corpus: about 1 hour of synthesized conversational speech, generated from Gemini 3 Pro scripts and rendered with ElevenLabs TTS, plus 100 music tracks collected from YouTube.
• Facial animation data: about 1 hour of synchronized speech and ARKit blendshape capture recorded with an iPhone 12 and Live Link Face, using Harvard Sentences.

To make the motion sources compatible, the paper applies the standardized skeletal representation proposed by Holden and retargets everything to a unified digital character in Autodesk Maya. Because Motorica contains finger artifacts, the authors train a motion inpainting model using high-fidelity finger motion from ZeroEGGS, then smooth the result with a Savitzky-Golay filter.

The face branch is distinct from the body-motion pipeline. It outputs 52-dimensional ARKit blendshape coefficients from streaming audio using a pretrained multilingual HuBERT encoder, a causal GPT backbone, and a flow-matching diffusion head. In the appendix, the face model is trained with 10% audio dropout for classifier-free guidance, uses AdamW with learning rate $2 \times 10^{-4}$, and autoregressively generates 8 frames per step during inference.

Training Configuration and Deployment

The paper reports fairly detailed system settings. The body-motion system synthesizes native motion at 30 FPS and interpolates to 60 FPS for rendering. The RVQ-VAE uses a temporal downsampling factor of 4, latent rate 7.5 Hz, codebook size 512, latent dimension 512, and quantization depth 6. It is trained with batch size 256, commitment weight $\eta = 0.1$, learning rate $4 \times 10^{-4}$, and 64-frame motion windows.

The generator fine-tuning stage uses Qwen2.5-0.5B-Instruct, batch size 256, and learning rate $5 \times 10^{-5}$. The RL stage reduces the batch size to 16 and uses learning rate $6 \times 10^{-5}$. For GRPO, the KL penalty is $\beta_G = 0.01$ with 30 rollouts per prompt. For DPO, the deviation penalty is $\beta_D = 0.1$. The whole pipeline trains on two NVIDIA H200 GPUs in about 30 hours.

At inference, the optimized pipeline reaches roughly 300 tokens per second. The real-time deployment uses a distributed system with three tiers: a cloud or local voice agent, a rendering client, and a GPU inference server. The fixed-context model is run with a sliding window, generating motion in 0.266-second chunks, i.e., 8 frames per step. CUDA Graphs are used to reduce kernel-launch overhead.

Latency Breakdown

The appendix reports latency on both NVIDIA H200 and RTX 4090, averaged over 20 intermediate inference steps. The important point is that the total computational latency stays below the 266 ms audio chunk budget on both platforms.

The paper also states that the end-to-end system can include an additional 100 ms client-side buffering delay, and that a cloud voice-agent demo can incur roughly 300 ms of network latency.

Quantitative Evaluation

The paper benchmarks the unified system on both speech-to-gesture and music-to-dance tasks. It uses FID and diversity as distributional metrics, and uses domain-specific beat-alignment measures: $\text{BA}_G$ for speech-gesture and $\text{BA}_D$ for music-dance.

Unified Test Set

On the unified test set, the full method outperforms the domain-specific baselines in FID while keeping synchronization competitive. Relative to MECo and EDGE, the method substantially improves FID, though the exact beat-alignment numbers depend on task and are not uniformly dominant across every metric.

Key values reported in the paper are:

• MECo: FID 14.73, diversity 23.13, $\text{BA}_G = 7.507$, $\text{BA}_D = 2.622$.
• EDGE: FID 18.06, diversity 19.71, $\text{BA}_G = 8.190$, $\text{BA}_D = 2.668$.
• Ours: FID 9.465, diversity 20.70, $\text{BA}_G = 8.277$, $\text{BA}_D = 2.603$.

The no-corruption variant is especially informative: it gets much worse FID (25.92) but very high diversity (29.58), supporting the paper's claim that the model can drift into diverse but poorly conditioned motion if Hierarchical Token Corruption is removed.

BEAT2 Benchmark

On the BEAT2 test set, the paper reports a new best FID, while diversity remains highest for MECo. The reported numbers are:

So the strongest BEAT2 claim is not synchronization but fidelity: the method gets the lowest FID in the table. MECo still has the highest diversity, so the result is a real fidelity-vs-diversity trade-off rather than a universal dominance across all metrics.

Subjective Evaluation

The user study compares dance and gesture quality using pairwise judgments on three axes: human likeness, beat matching, and overall preference. The evaluation protocol uses 10-second clips, identical audio conditions, a 5-point Likert scale mapped to $[-2,2]$, and a total of 1,680 pairwise judgments.

The important qualitative takeaway is that the merged training set beats single-task training, and the RL choice should depend on the domain:

• Dance: GRPO produces stronger preference gains than DPO.
• Gesture: DPO produces stronger preference gains than GRPO.

This is consistent with the authors' interpretation that dance tolerates more aggressive optimization, while conversational gesture benefits from more conservative preference learning.

Discussion, Limitations, and Risks

The paper is unusually explicit about limitations. The current system decouples facial and body motion, and it does not model gaze or richer non-verbal interaction cues. The authors also note that acoustic-only conditioning plus limited speaker diversity can cause domain confusion, such as misidentifying male speech as singing and generating dance-like motion. Another failure mode appears when music stops abruptly, because the system does not yet have a dedicated transition policy for sudden acoustic termination.

From a broader interaction standpoint, the system only models the speaker role. The authors identify the lack of listener-side behavior, such as backchannels or reactive listening motion, as an important future direction.

Finally, the paper includes an ethical-risk note: because the method improves motion fidelity in synthesized videos, it could be misused for deepfakes or non-consensual impersonation.

Bottom Line

EchoAvatar is a well-scoped, engineering-heavy system paper with three notable technical ideas: a causal attention-based RVQ motion tokenizer, a hierarchical corruption strategy to stabilize unified audio-to-motion training, and a real-time deployment stack that connects streaming voice agents to streaming motion generation. Its strongest empirical result is a new low FID on BEAT2, while its main system-level contribution is making unified speech-and-music motion synthesis work in a low-latency setting that can plug into voice-agent pipelines.