DiscoForcing

DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing

DiscoForcing is a real-time audio-driven system for full-body character control that operates with strict causality and low latency. It balances responsiveness with long-term stability under changing audio streams, enabling interactive avatar animation and humanoid robot control in real time.

• audio-driven
• full-body
• realtime
• streaming
• low-latency
• avatar

Demos

The demos highlight DiscoForcing's real-time, causal streaming synthesis of full-body dance motion tightly aligned with live audio inputs. Watch for the model's ability to maintain smooth, beat-synchronized motion, robustly adapt to abrupt audio changes and tempo shifts, and uphold long-horizon temporal coherence under low-latency constraints. The pipeline diagram explicates the unified framework integrating causal audio encoding and diffusion forcing for both avatar animation and humanoid control deployment.

Authors: Kaiyang Ji, Bingsheng Qian, Binghuan Wu, Kangyi Chen, Ye Shi, Jingya Wang

Categories: cs.CV

Comment: accepted by ICML 2026

Published 2026-05-27 · Updated 2026-05-27

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar0 citations

Abstract

We study real-time audio-responsive character control as a deployment-faithful problem: strictly causal, bounded-latency streaming that must generate coherent full-body motion at interactive frame rates while the audio condition can change abruptly, including tempo shifts, drops, or user edits. Prior music-to-motion systems are largely optimized for offline generation with global context, and degrade in streaming rollouts where conditioning history becomes stale or unreliable. We introduce DiscoForcing, a streaming audio-driven diffusion framework that combines a causal music encoder that captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels across the temporal horizon. Building on this, we design a hybrid temporal schedule and a history-guided streaming sampler to explicitly trade off responsiveness against long-horizon consistency under non-stationary audio. Implemented in an end-to-end real-time interactive system with online avatar playback and humanoid deployment workflows, DiscoForcing delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints while maintaining real-time throughput.

---

1. Problem setting and high-level idea

DiscoForcing studies real-time audio-responsive character control as a deployment-faithful streaming problem. The target setting is stricter than conventional music-to-motion generation: the system must operate causally, respect a bounded latency budget, emit coherent full-body motion at interactive frame rates, and remain responsive when the audio stream changes abruptly because of tempo shifts, drops, or user edits. The paper argues that many prior methods are optimized for offline clip generation with access to global context, which makes them brittle once they are forced into online rollouts.

The core design combines three ideas:

• a causal music encoder that extracts rhythm- and phase-aware conditioning from a sliding audio buffer;
• a diffusion-forcing sequence model trained with heterogeneous noise levels across the motion horizon so it can learn from imperfect histories;
• a streaming sampler with hybrid temporal scheduling and history-guided temporal guidance to balance quick reaction against long-horizon stability.

The paper’s thesis is that audio-driven control is not just a generation problem but a closed-loop online system problem: the model must remain usable when previous outputs become stale, when conditions shift mid-rollout, and when the downstream executor needs motion that can be consumed immediately without retroactive refinement.

2. System overview

The system operates at 30 Hz on both the music-conditioning side and the motion side. At each streaming step $t$, it consumes a causal music feature $c_t$ extracted from a fixed-length sliding audio window and a finite motion history buffer, then predicts the next short motion chunk. The generated motion is then routed to two deployment front ends: a Unity avatar platform for user-facing visualization and interaction, and a physics-based humanoid pipeline that retargets motion into robot-compatible joint commands.

2.1 Causal music processing

To preserve strict causality, DiscoForcing does not use future audio context. Instead, it builds a compact feature from a sliding buffer using a Vector-Quantized Periodic Autoencoder (VQ-PAE). The resulting conditioning vector is decomposed as $\mathbf{c}_t = [\mathbf{f}^{\text{vq}}_t; \mathbf{f}^{\text{pae}}_t]$, where the first branch encodes discrete rhythmic structure and the second branch captures continuous periodic alignment dynamics.

The paper’s motivation is that streaming music is non-stationary, so the encoder must react to local changes while still representing longer rhythmic structure. In the described pipeline, the shared causal latent is produced by a dilated 1D convolutional encoder, then passed through residual vector quantization for rhythmic codes. In parallel, frequency-domain analysis and a lightweight phase predictor reconstruct a periodic alignment signal. Intuitively, this gives the generator both a discrete beat-aware cue and a smooth phase cue that are suitable for low-latency control.

2.2 Streaming motion representation

DiscoForcing uses a canonicalized incremental motion representation to reduce drift under autoregressive rollout. Instead of modeling motion in a global frame, it expresses motion in the root frame with velocity-like increments, making the conditioning distribution more stationary. The main text contrasts this with the widely used 263-D HumanML3D-style representation, which reconstructs joint positions but does not directly expose joint rotations needed for immediate retargeting.

The paper instead adopts a 272-D per-frame representation: $$ \mathbf{m}_t = \{\dot r^x_t,\ \dot r^z_t,\ \dot r^a_t,\ \mathbf{j}^p_t,\ \mathbf{j}^v_t,\ \mathbf{j}^r_t\} \in \mathbb{R}^{272}, $$ where $\dot r^x_t$ and $\dot r^z_t$ are root planar velocities, $\dot r^a_t$ is a 6D root angular velocity, and $\mathbf{j}^p_t$, $\mathbf{j}^v_t$, and $\mathbf{j}^r_t$ are local joint positions, velocities, and rotations in the root frame. For SMPL, the paper uses $K=22$ joints.

This representation is important for deployment because it can be reconstructed directly by forward kinematics without a costly per-frame inverse-kinematics fitting stage. The authors explicitly frame this as a practical advantage for real-time avatar playback and humanoid retargeting.

2.3 Motion primitive learning with a latent VAE

To reduce computation, DiscoForcing first compresses the 272-D motion stream into a latent sequence using a motion VAE. The paper’s stated reason is twofold: the latent bottleneck lowers the cost of the sequence model, and it acts as a low-pass filter that removes high-frequency jitter. Let $\mathbf{z}_\mathcal{T} = \mathcal{E}(\mathbf{m}_\mathcal{T})$ and $\hat{\mathbf{m}}_\mathcal{T} = \mathcal{D}(\mathbf{z}_\mathcal{T})$ denote the encoder and decoder outputs. The VAE objective is a standard reconstruction-plus-KL loss: $$ \mathcal{L}_{\text{VAE}} = \|\mathbf{m}_\mathcal{T} - \hat{\mathbf{m}}_\mathcal{T}\|_2^2 + \lambda D_{\text{KL}}\big(q(\mathbf{z}_\mathcal{T}\mid \mathbf{m}_\mathcal{T}) \parallel \mathcal{N}(\mathbf{0}, \mathbf{I})\big). $$

2.4 Diffusion forcing over a sequence horizon

The generative backbone is a latent diffusion-forcing model adapted to streaming motion. At time $t$, the model predicts the next $n$ motion latents conditioned on the history buffer and music context. Each token $t$ has its own diffusion time $k_t \in [0,1]$, which lets the method train on heterogeneous noise states rather than a single global diffusion level.

The forward process is defined token-wise as $$ p(\mathbf{x}^{k_t}_t \mid \mathbf{x}^0_t) = \mathcal{N}(\mathbf{x}^{k_t}_t; \alpha_{k_t}\mathbf{x}^0_t, \sigma^2_{k_t}\mathbf{I}), $$ with boundary conditions $\alpha_0=1$, $\sigma_0=0$, $\alpha_1=0$, and $\sigma_1=1$. The model learns a vector field $\mathbf{v}_\theta(\mathbf{x}^{k_\mathcal{T}}_\mathcal{T}, k_\mathcal{T}, \mathbf{c})$ whose token-wise target is $$ \mathbf{v}_t = \dot{\alpha}_{k_t}\mathbf{x}^0_t + \dot{\sigma}_{k_t}\boldsymbol{\epsilon}_t. $$ The loss is a masked flow-matching objective: $$ \mathcal{L}_{\text{DF}} = \mathbb{E}\big[\|\mathbf{v}_\theta(\mathbf{x}^{k_\mathcal{T}}_\mathcal{T}, k_\mathcal{T}, \mathbf{c}) - \mathbf{v}_\mathcal{T}\|^2_{\mathcal{K}}\big], $$ where the norm only sums over tokens with $k_t>0$.

Conceptually, this means the model is not trained only on fully clean prefixes and fully noisy futures. Instead, it sees a mixture of clean, partially corrupted, and highly corrupted tokens, which is important because streaming inference often has to cope with stale or imperfect history rather than ideal context.

2.5 Hybrid temporal schedule

A key novelty in the paper is the hybrid temporal schedule, which is used to bridge training and streaming inference. Rather than relying on a single schedule, the method samples among three variants during training:

• Random schedule: each token receives an independent $k_t \sim \mathcal{U}(0,1)$;
• Monotonic schedule: the schedule linearly transitions from clean history to noisy future around a current time index $\tau$ and a window length $l$;
• Trapezoid schedule: distant history is additionally corrupted, so older context is weakened more aggressively than recent context.

The paper’s interpretation is that random noise preserves flexibility, monotonic noise matches the structure of streaming rollouts, and trapezoid corruption helps the model learn to avoid over-trusting outdated history.

2.6 Streaming inference with temporal guidance

During inference, DiscoForcing performs bounded-latency rollout with a fixed-size FIFO denoising window. The sampler denoises the active window jointly, emits the leftmost token once it becomes clean, then shifts the buffer and appends a fresh noise token. This establishes a one-token-at-a-time online generation process rather than offline whole-sequence refinement.

The paper also introduces temporal guidance, a history-aware analogue of classifier-free guidance. It combines a history-only prediction with a condition-aware prediction using a guidance scale $\omega$: $$ \mathbf{v}^{\text{guided}} = \mathbf{v}_{\text{hist}} + \omega\big(\mathbf{v}_{\text{cond}} - \mathbf{v}_{\text{hist}}\big). $$ The conditional branch uses the audio-conditioned model with a trapezoid-corrupted history, while the history branch uses a monotonic schedule and no music condition. The intended effect is to retain responsiveness to fresh audio while suppressing motion lock-in and drift from stale autoregressive state.

The main paper frames this as the central stability-versus-responsiveness trade-off for non-stationary music streams. If the sampler trusts history too much, it becomes sluggish to react; if it ignores history, it can become jittery or discontinuous. Temporal guidance is presented as a way to interpolate between those extremes at inference time.

2.7 Real-time interactive system and deployment

The paper is not limited to offline metrics: it also describes an end-to-end communication-based runtime that connects audio ingestion, online generation, and playback. The two front ends are a Unity avatar platform and a physics-based humanoid deployment stack. The avatar pathway renders SMPL-based motion in real time, while the humanoid pathway retargets the generated motion to the Unitree G1 using GMR-based mapping, interpolation, and whole-body control.

This deployment framing matters because it forces the system to respect the practical constraints the paper is targeting. In particular, the model must run causally, produce motion that is directly usable by downstream controllers, and remain coherent through abrupt audio transitions.

3. Experimental setup

The evaluation uses two public dance datasets with synchronized music and 3D motion:

• FineDance: 7.7 hours of music-paired optical motion capture at 30 FPS;
• AIST++: 5.2 hours of motion sequences reconstructed from multi-view videos at 60 FPS.

The paper states that it follows the official splits for both datasets and converts all examples to 30 FPS. This keeps the streaming setting consistent with the model’s 30 Hz motion rate.

The reported metrics fall into three categories:

• Motion quality: $\mathrm{FID}_k$, $\mathrm{FID}_g$, and foot-skating ratio $\mathrm{FSR}$;
• Motion diversity: $\mathrm{Div}_k$ and $\mathrm{Div}_g$;
• Music-dance correlation: beat alignment score $\mathrm{BAS}$.

The paper uses $\mathrm{FID}_k$ and $\mathrm{FID}_g$ on kinetic and geometric features to measure realism, $\mathrm{FSR}$ to penalize foot sliding, and $\mathrm{BAS}$ to quantify beat synchronization.

4. Quantitative results

The main comparisons are against representative music-to-dance systems: FACT, Bailando, EDGE, Lodge, and MEGADance. The paper’s central claim is not that DiscoForcing dominates every metric, but that it provides a strong balance of fidelity, synchronization, and deployability under streaming constraints.

4.1 FineDance

On FineDance, DiscoForcing reports the best $\mathrm{FID}_k$ and $\mathrm{FID}_g$ among the listed methods, indicating a strong match to the real motion distribution. Its $\mathrm{BAS}$ is competitive but not the highest, and its $\mathrm{FSR}$ is better than most baselines but not as low as Lodge. This is consistent with the paper’s emphasis on a balanced trade-off rather than optimization of one metric alone.

4.2 AIST++

On AIST++, DiscoForcing achieves the best $\mathrm{FID}_k$ and the best $\mathrm{BAS}$ among the compared methods. However, Bailando remains better on $\mathrm{FID}_g$ and on both diversity metrics. The result suggests that DiscoForcing is especially strong on the kinetic and synchronization aspects that matter for streaming responsiveness, while still staying competitive on the rest.

4.3 Qualitative streaming behavior

The paper emphasizes that the qualitative advantage of DiscoForcing is most visible in online rollouts rather than in static clip metrics. In particular, the system can remain still during silence and react immediately when audio resumes, which is an important property for user-facing interactive control. The authors also stress smoothness over long horizons and resilience to multiple music transitions within one stream.

5. Ablation study

The ablations are reported on AIST++ and are especially informative because they show how the paper’s design choices affect both quality and runtime. The authors ablate the motion representation, the music encoder, the inference guidance strategy, and the number of denoising steps.

The ablation trends are clear:

• Motion representation: the 272-D design is chosen because it preserves joint rotations and enables direct retargeting, even though the 263-D variant has slightly better geometric FID. The paper explicitly prefers deployability over a small geometric gain.
• Music encoder: VQ-PAE improves realism and overall motion quality relative to Librosa features, while keeping latency roughly unchanged.
• Guidance: temporal guidance outperforms classifier-free guidance, improving $\mathrm{FID}_k$, $\mathrm{FID}_g$, $\mathrm{FSR}$, and $\mathrm{BAS}$ simultaneously.
• Sampling steps: 10 denoising steps are the paper’s chosen operating point because 5 steps is too crude and 100 steps is too slow for real-time use despite slightly better offline scores.

6. What the paper claims as contributions

• A streaming, end-to-end framework for real-time audio-responsive character control under strict causality and latency constraints.
• A causal music encoder that decomposes streaming audio into rhythmic and periodic conditioning features.
• A latent diffusion-forcing generator trained with heterogeneous noise schedules to handle imperfect histories.
• A hybrid temporal schedule and temporal guidance sampler designed to trade off responsiveness and long-horizon consistency under non-stationary audio.
• A real-time interactive system that supports both avatar visualization and physics-based humanoid deployment within a unified runtime.

7. Limitations and scope

The paper’s own conclusion points to several open directions: stronger robustness under distribution shift, richer user control signals, stronger physical feasibility constraints, broader real-world testing, and tighter integration with downstream controllers for safer deployment. In other words, DiscoForcing is presented as a strong step toward deployment-faithful streaming motion generation, not as a fully solved final system.

From the reported experiments, the evaluation is still centered on two standard dance datasets and standard offline metrics, plus qualitative online demonstrations. The main text does not provide a dedicated quantitative benchmark for humanoid execution quality or a comprehensive user study in the reported sections, so the deployment claims are primarily supported by the described system design and qualitative rollout behavior.

Another practical trade-off visible in the ablations is that more aggressive denoising improves metrics but quickly violates latency budgets. The method’s final configuration is therefore a compromise between fidelity and real-time feasibility, which is exactly the problem the paper is trying to address.

8. Takeaway

DiscoForcing’s main contribution is not simply a better music-to-dance generator; it is a streaming-native formulation of audio-driven character control. By combining causal audio encoding, diffusion forcing with heterogeneous noise, and a history-guided sampler, the paper shows that one can obtain motion that is both responsive to live music and stable enough for long-horizon online playback and downstream robot-style execution.