StreamChar

StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration

StreamChar enables real-time streaming character animation by decoupling transcript orchestration from audio-video denoising. An LLM orchestrator maintains fidelity while a joint diffusion transformer handles efficient synthesis, eliminating error accumulation from autoregressive chunk generation.

• streaming
• realtime
• multimodal
• audio-driven
• talking-head
• lip-sync
• low-latency

Authors: Linrui Tian, Qi Wang, Bang Zhang

Categories: cs.CV

Published 2026-05-25 · Updated 2026-05-25

Links

Paper & demos

• arXiv
• Project page

Impact

• Semantic Scholar0 citations

Abstract

Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.

---

Introduction and Motivation

Real-time streaming joint audio-video generation for characters — where a system must simultaneously speak a requested transcript, maintain consistent visual identity, and operate within a strict playback latency budget — sits at the challenging intersection of multimodal learning, efficient inference, and interactive systems. Recent advances in latent diffusion and Diffusion Transformers (DiT) have enabled high-quality short-clip generation, and unified text-audio-video backbones have pushed joint modeling further. However, the transition from generating clips to streaming minutes-long content interactively exposes two tightly coupled but fundamentally distinct difficulties.

The first difficulty is long-horizon coherence. In chunk-wise autoregressive generation, local decoding decisions can drift from the textual plan, leading to omitted content, repetitions, or semantic misalignment with the source script. Beyond transcript-audio fidelity, the model must simultaneously preserve speaker identity and visual appearance across segment boundaries, and maintain frame-accurate lip-phoneme alignment as the sequence lengthens. In monolithic multimodal DiT designs, the same backbone shoulders all three responsibilities — semantic understanding, cross-chunk memory, and local spatiotemporal denoising — creating competition for model capacity. Errors in global context propagate directly into local generation, manifesting as semantic drift, identity shifts, and degraded synchronization after the first few chunks. The autoregressive feedback loop rapidly diverges from the intended content.

The second difficulty is interactive inference speed. Real-time streaming requires each chunk to be generated faster than its playback duration, yet diffusion models typically need tens to hundreds of denoising steps for quality. Aggressive step reduction via distillation is necessary but introduces distillation-induced mode collapse: the student model, deprived of iterative refinement, collapses to stereotyped spatial behaviors and reduced diversity. Moreover, errors accumulate across chunks, causing progressive video drifting.

Critically, these two challenges are not independent. Quality degradation from aggressive distillation amplifies error accumulation, while long-horizon instability makes it harder to train a robust few-step student. To address them jointly, the authors propose StreamChar, which coordinates design at two levels: an architecture that distributes responsibility across specialized components, and an optimization strategy that sequentially resolves step reduction and rollout consistency.

Overall Architecture

StreamChar separates long-horizon orchestration from short-window audio-video denoising. The system has two primary components:

• LLM Orchestrator: A causal language model that reads the prompt/transcript and reference/history audio, producing a frame-aligned continuous audio condition $\mathbf{c}_a$ for the active chunk. This separates global transcript planning from local synthesis.
• Joint Audio-Video DiT: A short-window denoiser that performs bidirectional denoising of video and audio latents conditioned on prompt embeddings, $\mathbf{c}_a$, and visual conditions (reference and motion frames).

Preprocessing and Flow Matching

Ground-truth video is mapped to VAE latents $\mathbf{z}_v$ with shape $C_v \times T_v \times H' \times W'$; audio is represented as audio-VAE latents $\mathbf{z}_a \in \mathbb{R}^{T_a \times C_a}$. The prompt for visual semantics is encoded by a frozen T5 into context vectors $\mathbf{h}_{\text{T5}}$.

Training follows a latent flow setup. Given time $t \in [0,1]$, intermediate noisy states are formed as:

with $\boldsymbol{\epsilon}_v, \boldsymbol{\epsilon}_a \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, so $t=0$ recovers clean latents and $t=1$ yields pure noise. The same $t$ is used for timestep conditioning in both the Orchestrator and the DiT.

The DiT is trained to regress the flow-matching velocity target $\mathbf{v} = \boldsymbol{\epsilon} - \mathbf{z}$ for both modalities:

LLM Orchestration

The Orchestrator is a causal language model that does not solve diffusion itself. It reads the prompt and script, optionally uses reference audio/text to anchor speaker timbre, encodes history from long-term generated clips, and outputs a frame-aligned continuous audio condition $\mathbf{c}_a$ for the DiT.

Crucially, the system avoids autoregressing discrete neural-codec or speech tokens. Instead, the Orchestrator consumes a single causal sequence of embedding vectors:

where:

• $\mathbf{e}_{\mathrm{ref}}$ and $\mathbf{e}_{\mathrm{hist}}$: reference and history waveforms passed through the audio VAE to obtain frame-aligned vectors, linearly projected into the LLM embedding space.
• $\mathbf{E}_{\mathrm{txt}}$: prompt and transcript tokenized by the original text tokenizer.
• $\mathbf{E}_{\mathrm{cond}}(t)$: the conditioning tail for the current denoise step, formed by the noisy audio latent $\mathbf{x}_t^a$ and the timestep $t$.

After a causal forward pass, the final-layer hidden states at positions corresponding to $\mathbf{E}_{\mathrm{cond}}(t)$ are collected to form $\mathbf{c}_a$. This audio condition is learned end-to-end jointly with the DiT through the diffusion flow loss. The Orchestrator is intentionally coupled to the denoising state: it receives the noisy audio latent $\mathbf{x}_t^a$ and the shared timestep $t$.

Joint Audio-Video Diffusion

Within each diffusion step, the reference/motion visual conditions and noisy audio-video latents are patchified into token streams and concatenated as:

where $\mathbf{x}^{t}_{v}$ and $\mathbf{x}^{t}_{a}$ denote noisy video and audio tokens at diffusion step $t$.

Audio Alignment and Joint Attention

Before entering the main DiT blocks, audio condition features $\mathbf{c}_a$ are fused with the noisy audio latents through a lightweight Transformer-based audio encoder, producing aligned audio tokens. In the main DiT blocks, the denoiser applies shared self-attention over noisy video and noisy audio tokens, allowing lip motion, prosody, and scene dynamics to interact directly at the token level. After the main blocks, the audio tokens are passed through an audio decoder to obtain the final output $\mathbf{f}_\theta^{a}$.

Modality-Aware Mixture-of-Experts (MoE)

To balance cross-modal interaction with modality-specific feature learning, the transformer blocks employ a shared attention mechanism coupled with a modality-aware Mixture-of-Experts feed-forward network. While attention projections are shared across modalities to facilitate robust audio-visual communication, video and audio tokens are dynamically routed to distinct FFN experts within a two-expert architecture. This preserves modality-specific dynamics within one joint backbone.

Timestep-Invariant Conditioning and Asymmetric Masking

For condition frames serving as reference and motion controls, the system enforces timestep invariance by using clean-state embeddings ($t=0$), so these tokens represent static guidance independent of the diffusion noise schedule. An asymmetric attention mask is introduced: noisy latent tokens attend to condition tokens, whereas condition tokens are masked from attending to noisy inputs. This unidirectional flow prevents stochastic noise from corrupting control signals, making key-value states of condition tokens invariant after the initial denoising step. Consequently, these states can be pre-computed and cached throughout the sampling trajectory, reducing redundant computation during multi-step inference.

Modality-Aware RoPE

Cross-modal attention requires precise temporal alignment despite differing latent sampling rates. Video (24 fps, 4× VAE compression) and audio (49,152 Hz, 2048× VAE compression) yield latent rates of 6 fps and 24 Hz respectively — a 4:1 token density ratio. The RoPE base frequency for audio is scaled by $1/4$ relative to video, ensuring tokens from the same physical timestamp share identical rotational phases.

For chunk-wise streaming, global temporal continuity is maintained via offset-aware indexing: instead of resetting positions at chunk boundaries, newly generated frames are anchored at index 0, and motion-frame latents are assigned negative temporal position offsets (e.g., $-K, \dots, -1$ for $K$ motion frames). This preserves a consistent global timeline across chunks.

Streaming Inference and Distillation

Bidirectional Architecture for Chunk-wise Streaming

Many streaming video generators adopt causal temporal attention, where each token depends only on past frames. The authors observe that causal training systematically degrades generation quality because the model never accesses future context within the active clip during denoising. StreamChar instead retains bidirectional self-attention both during training and inference within each chunk.

The DiT applies full bidirectional attention over all noisy video and audio tokens in the current window, enabling the denoiser to leverage global temporal context within the segment. Long-form continuity is handled across chunks through explicit conditioning on historical information (motion frame latents from previously generated video), rather than through architectural causality constraints. At inference, the student rolls forward in time by feeding motion latents from prior decoded output as $\mathbf{z}_{\mathrm{mot}}$, while bidirectional attention remains confined to the current chunk.

The system generates 9 latent video frames per chunk, which decode to 33 RGB frames (at 24 fps) under the temporal packing scheme (VAE stride). The theoretical per-chunk latency is approximately $33/24 \approx 1.38$ seconds before codec and orchestration overhead — short enough for responsive streaming while still exploiting bidirectional context.

Two-Stage Decoupled Distillation

Streaming inference over extended sequences introduces error accumulation from autoregressive generation. Naive distillation that jointly optimizes for both step reduction and long-horizon consistency leads to severe instability. StreamChar addresses this through a two-stage distillation strategy that decouples step compression from rollout consistency training.

Stage I: Step Reduction via Distribution Matching Distillation (DMD)

In the first stage, the pretrained joint backbone is distilled into a four-step generator using distribution matching distillation (DMD). This stage compresses the original 50-step sampler into a few-step generator while preserving single-chunk generation quality. On a single H100 GPU, the distilled student achieves approximately 24 fps generation at the clip level (including the reduced diffusion loop). The resulting four-step model serves as initialization for Stage II.

Stage II: Online Rollout for Autoregressive Consistency

Starting from the four-step initialization, training continues with an online rollout procedure that simulates the chunk-by-chunk generation process used at inference. During this phase, the student autoregressively generates multiple consecutive chunks, and the motion latents for later chunks are taken from the student's own forward passes rather than solely from teacher or ground-truth crops. This exposes the optimization to the same autoregressive pipeline encountered during deployment, narrowing the train-test gap.

Two key designs enable stable Stage II training:

1. Progress-Aware Pointer (PAP) for transcript truncation.
2. Sink-chunk memory for suppressing long-horizon drift.

After generating multiple chunks, the last several chunks are concatenated and fed into real-score and fake-score branches to compute the DMD loss.

Progress-Aware Pointer (PAP)

PAP is integrated into the Orchestrator and trained jointly during the pretraining stage. It determines the transcript truncation point for each chunk, which is essential for the online rollout: without knowing how far the audio has progressed through the transcript, accurate DMD loss computation across chunks is impossible.

Given transcript hidden states $\mathbf{A}$ (final-layer LLM states of $N$ transcript tokens) and audio conditions $\mathbf{c}_a$, PAP computes cross-attention to derive frame-wise soft positions $p_j$. These are refined by a learnable offset $\delta_j$ and aggregated via confidence weights $w_j$ to predict the spoken endpoint index $\hat{s}$:

where $\hat{s}$ is clamped to $[0, N]$. The module is supervised using ground-truth end indices derived from ASR timestamps via smooth $\ell_1$ loss. This ensures precise alignment between the generated audio span and the transcript, allowing accurate transcript truncation for DMD loss computation in Stage II.

Sink-Chunk Memory

The sink-chunk mechanism provides persistent long-range temporal memory to suppress video drifting over extended rollouts. The first chunk generated by the student serves as a persistent visual anchor: all subsequent chunks attend to it, in addition to their local motion frames. This gives every downstream chunk a stable reference to the early appearance and layout, preventing progressive identity drift and mode collapse that otherwise accumulates across boundaries.

Implementation Details

Architecture Backbones

• DiT: Built upon the WAN 2.2-5B architecture, with feed-forward modules replicated to construct dedicated audio experts.
• Orchestrator: Initialized from the Qwen2.5-3B architecture.

Pretraining

Pretraining Stage 1: The Orchestrator is pretrained on the Emilia dataset for 80k steps, using a batch size of 640 and a learning rate of $6 \times 10^{-5}$.

Pretraining Stage 2: An audio-video dataset is curated by combining SpeakerVid-5M, TalkVid, and OpenHumanVid. The Orchestrator and DiT are jointly trained for 100k steps with a batch size of 128 and a learning rate of $1 \times 10^{-5}$. During generation, the model produces 33 frames per chunk at 24 fps. The number of motion frames is aligned with the chunk size (33 frames). The maximum duration of historical audio input to the Orchestrator is set to 15 seconds; the oldest audio segments and their transcripts are truncated during inference to ensure training-inference consistency.

Distillation

Distillation Stage I (step compression): 600 steps to compress the sampler into a 4-step generator.

Distillation Stage II (online rollout consistency): 400 steps to refine autoregressive stability.

For both distillation stages, the student network uses a learning rate of $2 \times 10^{-6}$, while the fake score network uses a learning rate of $4 \times 10^{-7}$.

Inference Efficiency

On a single H100 GPU, generating a 33-frame chunk ($512 \times 512$, 4 steps) in bfloat16 takes 0.96 s (LLM + DiT). The pipeline adds VAE decoding (~0.30 s), preprocessing (~0.05 s), and stream writing (0.025 s). Motion frame latents are directly reused from the preceding chunk, bypassing VAE encoding. The next chunk's preprocessing initiates immediately after generation, overlapping with VAE decoding. The total per-chunk latency (~1.34 s) remains within the playback budget ($33/24 \approx 1.38$ s).

Experiments

Compared Methods

Due to limited open-source methods capable of long-horizon real-time streaming joint audio-video generation, a two-tier comparison strategy is adopted:

Tier 1 — Real-time audio-driven streaming baselines:

• SoulX-FlashTalk: Real-time infinite streaming via self-correcting bidirectional distillation.
• SoulX-FlashHead: Lightweight 1.3B model capable of 96 fps on a single consumer GPU.
• LiveAvatar: Streaming real-time avatar animation with controllable expressions.

These methods do not synthesize speech from text, making StreamChar's generated audio serve as the common driver when comparing video synthesis.

Tier 2 — State-of-the-art offline joint audio-video generators:

• LTX-2: Efficient joint audio-video generation architecture.
• OVI: Twin-backbone cross-modal joint generation.
• MagiHuman: Human-centric audio-visual synthesis with strong identity preservation.

Evaluation Benchmarks

Results are reported on two protocols derived from the EMTD dataset:

• Standard short-clip set: 150 clips generating 10 s audio-video pairs from original transcripts and first frames.
• Long-horizon set: 50 clips paired with randomly sampled transcripts (>300 words) to produce 5-minute continuous streams.

Evaluation Metrics

• Sync-C / Sync-D: Audio-visual synchronization.
• FID / FVD: Perceptual fidelity (image and video).
• VBench-2.0 Human Anatomy / Human Identity: Human-centric quality dimensions.
• WER (Word Error Rate): Speech intelligibility and transcript alignment (not applicable to audio-driven baselines).
• VBench Dynamic score: Motion diversity (long-horizon).
• Quality Drift: Following rolling-forcing, every 30 s the absolute quality difference between that segment's final 5 s and the video's initial 5 s is computed; the maximum difference over the full video is reported.

Quantitative Results

Analysis of Quantitative Results

Speech intelligibility and transcript alignment: StreamChar demonstrates a clear advantage in WER. The base model (3.54%) outperforms all evaluated joint audio-video generators, while the chunk-wise distilled variant maintains near-identical performance (3.65%). This suggests the LLM orchestrator preserves fine-grained phonetic alignment during continuous, multi-chunk streaming — by offloading global script understanding and acoustic intent planning to the LLM, the DiT can focus on short-window denoising while receiving chunk-level acoustic guidance.

Visual fidelity: Despite employing a more compact 5B-parameter video foundation model, StreamChar achieves perceptual quality comparable to recent generators that typically scale beyond 14B parameters (e.g., LTX-2, MagiHuman, SoulX-FlashTalk, LiveAvatar). Among joint audio-video baselines, StreamChar achieves a leading Human Anatomy score and the lowest FID (17.99). The lower FID of specialized audio-driven pipelines may reflect their more constrained motion scope, where dynamics are concentrated around facial and hand regions while preserving spatial consistency with the reference frame.

Audio-visual synchronization: Sync-C/D scores are competitive but do not dominate the specialized audio-driven baselines, which directly condition on waveforms rather than generating speech from text.

Post-distillation trade-offs: Following aggressive distillation (50 → 4 steps), a mild increase in FVD is observed, reflecting the expected quality-efficiency trade-off. Synchronization and speech intelligibility (WER) remain stable, and anatomical quality is preserved.

Long-horizon stability: StreamChar achieves a negligible Quality Drift of 0.0067, demonstrating robust suppression of error accumulation over extended sequences. Crucially, it maintains the maximum VBench Dynamic score (1.0), indicating that stability is achieved without sacrificing motion diversity or anchoring to the reference frame. LiveAvatar, despite achieving a high Dynamic score, suffers from visible oscillatory artifacts.

Qualitative Comparisons

Short-Clip Comparison

A common tendency observed in recent streaming baselines is that generated frames tend to remain closely anchored to the initial reference image — motion is often confined to localized facial expressions or hand gestures, while torso posture remains static or exhibits minimal variation. In comparison, StreamChar exhibits comparatively less reliance on reference anchoring, yielding more varied upper-body poses and natural hand-object interactions, with motion that frequently extends beyond the immediate reference neighborhood.

Long-Horizon Comparison

Sink-Chunk Ablation

User Study

A GSB (good-same-bad) user study is conducted for the streaming setting, recruiting 24 participants each presented with 50 randomly sampled cases. Each case shows two randomly selected results; participants judge their relative quality in terms of motion naturalness, lip-sync accuracy, and motion richness. StreamChar achieves more favorable preferences than competing streaming baselines and ablated variants.

Ablation Studies

Sink Chunk for Error Accumulation and Mode Collapse

Removing the sink chunk increases Quality Drift from 0.0067 to 0.0304 — a roughly 4.5× increase. As visualized in the long-horizon comparison figures, this metric reflects severe error accumulation across chunks, manifesting as noticeable color shifts and appearance degradation over time. Without sink conditioning, the distilled student exhibits stereotyped spatial behaviors and persistent spatial offsets, suggesting a collapse toward low-diversity motion patterns. The sink mechanism mitigates these issues by providing a stable long-range reference, effectively suppressing quality drift and mode collapse while preserving motion diversity (Dynamic score 1.0).

Single-Chunk vs. Multi-Chunk in Stage II

Comparing online rollout over multiple consecutive chunks against training on isolated chunks only (Ours stage-2 one-chunk): training on isolated chunks degrades WER to 35.4% and reduces Sync-C/D scores significantly. Shorter sequences lack sufficient cross-chunk acoustic context for correct transcript alignment. Concatenating multiple chunks allows the distillation objective to capture consistent prosody and long-range phonetic transitions.

Two-Stage vs. Single-Stage Distillation

Ablating the pipeline by skipping Stage I (DMD step compression) and training Stage II directly from the 50-step teacher (Ours distill stage-2 only): while several short-clip metrics remain competitive, qualitative results reveal issues such as motion suppression and reference-frame anchoring. Directly combining step reduction with autoregressive rollout training places competing pressure on the student — it must simultaneously learn a low-step sampler and recover from its own rollout errors. Decoupling the stages first stabilizes the few-step mapping (Stage I) and then refines cross-chunk consistency (Stage II), thereby preserving both efficiency and visual dynamics.

Related Work

Diffusion Models for Audio-Video Generation

Denoising diffusion probabilistic models and latent diffusion, particularly Diffusion Transformers (DiT), form the backbone of modern generative media. Recent works unify text, audio, and visual tokens in monolithic DiTs for joint generation. While effective for single short clips, these monolithic designs face challenges when scaled to long-form streaming: the shared backbone must simultaneously handle semantic understanding, cross-segment memory, and local spatiotemporal denoising, leading to capacity competition and error propagation. Recent real-time streaming methods are typically confined to short temporal windows of a few seconds.

LLMs as Planners and Conditioners

Large language models have demonstrated capabilities in high-level semantic understanding and structured reasoning, increasingly used as planners in generative AI. In visual synthesis, LLMs decompose complex prompts into structured layout specifications or storyboards. In the audio domain, LLMs bridge textual semantics and acoustic realization. These approaches highlight an emerging paradigm where LLMs handle long-range contextual consistency and global script semantics while generative backbones focus on local fidelity — StreamChar formalizes and extends this paradigm for joint streaming audio-video generation.

Audio-Driven Video Generation

Audio-driven video generation has evolved from offline quality-oriented methods (e.g., EMO, Wan2.2-S2V) requiring dozens of denoising steps, to real-time streaming approaches achieving sub-second latency through knowledge distillation. These streaming systems operate in the audio-driven paradigm where video is synthesized from a given waveform — this simplifies synchronization but does not address the harder text-to-audio-video setting where speech content and visual motion must be generated together. StreamChar targets this joint setting.

Knowledge Distillation for Efficient Diffusion

Progressive distillation, consistency models, and distribution matching distillation (DMD) compress multi-step samplers into few-step generators. When deployed in chunk-wise streaming settings, these distilled models encounter two intertwined failure modes: distillation-induced mode collapse and error accumulation. Prior work largely treats step reduction and rollout consistency as separate problems. StreamChar shows these challenges are deeply coupled in streaming scenarios, and proposes a two-stage strategy that sequentially resolves them.

Contributions Summary

• A decoupled LLM orchestrator + short-window DiT architecture that addresses long-horizon coherence by offloading global semantics from the denoising backbone, with motion frame conditioning for cross-chunk continuity and modality-aware MoE for preserving audio-visual dynamics.
• A two-stage distillation recipe (Stage I: DMD step compression → Stage II: online rollout fine-tuning) that decouples the competing objectives of few-step generation quality and autoregressive consistency.
• A progress-aware pointer (PAP) that predicts the spoken transcript endpoint from audio conditions and transcript hidden states, enabling accurate transcript truncation during rollout training.
• A sink-chunk memory mechanism that uses the first generated chunk as a persistent visual anchor for all subsequent chunks, suppressing long-horizon drift and mode collapse.
• Demonstration of real-time streaming on a single H100 GPU with a total per-chunk latency of ~1.34 s within the 1.38 s playback budget, along with quantitative and qualitative evaluation on short-clip and long-horizon protocols showing competitive or leading performance across transcript fidelity, audio-visual synchronization, visual quality, and streaming stability.

Limitations

Several limitations of StreamChar are acknowledged:

• Evaluation protocol asymmetry: Audio-driven streaming baselines are evaluated using StreamChar's own generated audio as the common driver, making them controlled video references rather than full text-to-audio-video competitors — a fair but inherently asymmetric comparison.
• Hardware dependency: Real-time performance is measured on a single H100 GPU with a 33-frame chunk budget; lower-end deployment may require additional optimization.
• Training data length: Training data contains no videos/transcripts longer than 20 seconds, requiring truncation of oldest audio segments during inference for very long streams, which may limit adaptation to very long-form content statistics.