MoCha

MoCha: Towards Movie-Grade Talking Character Synthesis

MoCha generates movie-quality full-portrait talking character videos from speech and text, enabling natural lip sync, body motions, and multi-character conversations. It uses a novel speech-video window attention and structured prompts to sync speech and dialogue without auxiliary inputs.

• speech-driven
• audio-driven
• lip-sync
• full-body
• gesture
• dialogue
• multimodal
• talking-face

Authors: Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, Animesh Sinha, Peter Vajda, Wenhu Chen

Categories: cs.CV

Comment: https://congwei1230.github.io/MoCha/

Published 2025-03-30 · Updated 2025-03-30

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub16 stars0 watchers2 forks
• Hugging Face0 downloads4 likes

Impact

• Semantic Scholar29 citations2 influential citations

Abstract

Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.

---

Introduction

MoCha addresses a gap between modern video generation systems and the needs of character-driven filmmaking and animation. The paper argues that existing video models can generate visually impressive motion, but they usually do not support speech-driven, full-portrait talking characters with expressive body motion, synchronized dialogue, and controllable scene context. To formalize this gap, the authors introduce the task of Talking Characters: generating one or more characters from text and speech, where the output is not limited to the face, but spans the full portrait and can include close-up through wide-shot framing, body motion, facial affect, and multi-character interaction.

The core claim of the paper is that MoCha is the first end-to-end diffusion transformer for this task. Unlike prior talking-head systems that depend on reference images, keypoints, skeletons, or other auxiliary controls, MoCha is trained directly on text and speech. The model is designed to support cinematic generation, including natural lip synchronization, expressive facial motion, realistic body gestures, and even multi-character turn-based conversation. The authors position this as a step toward automated film and animation generation, as well as downstream applications such as virtual assistants and avatars.

The paper’s main technical contributions are threefold: (1) a speech-video window attention mechanism for better temporal alignment between audio and video tokens, (2) a joint training strategy that mixes speech-labeled video with much larger text-labeled video data, and (3) a structured prompt and character-tagging scheme that makes multi-character conversational generation feasible without extra control signals.

Task Definition: Talking Characters

The task accepts two inputs. First, a text prompt describes the character(s), the environment, the action, optional facing direction, optional position in the frame, and optional camera framing. Second, a speech audio clip drives mouth motion, facial expression, and body motion. The desired output is a video featuring one or more speaking characters, which may be human, cartoon, or animal.

The paper evaluates systems along five axes:

• Lip-sync quality: how accurately mouth motion matches speech.
• Facial expression naturalness: whether expressions look natural and contextually coherent.
• Action naturalness: whether body motion and gestures are realistic and speech-aligned.
• Text alignment: whether the scene and actions match the prompt.
• Visual quality: whether the video is temporally coherent and artifact-free.

Model Overview

MoCha is built as a diffusion transformer (DiT) that generates video frames from joint text and speech conditioning. The architecture operates on a latent video representation produced by a 3D VAE. A video clip $\mathbf{v} \in \mathbb{R}^{T \times H \times W \times 3}$ is encoded into a latent tensor $\mathbf{x}_0 \in \mathbb{R}^{\tau \times h \times w \times c}$, where the temporal downsampling ratio is $r = T / \tau$. The latent is flattened into a token sequence and processed by the DiT.

Each DiT block first performs self-attention over video tokens and then sequential cross-attention to the text tokens $\mathbf{c}$ and the audio tokens $\boldsymbol{\alpha}$. The audio stream is extracted from raw waveforms using Wav2Vec2 and passed through a single-layer MLP so that its feature dimension matches the video latent tokens. The model is trained with a flow-matching objective rather than a standard discrete denoising objective.

Flow Matching Objective

Let $\mathbf{x}_1$ denote the clean latent video, $\boldsymbol{\epsilon} \sim \mathcal{N}(0, I)$ denote Gaussian noise, and $t \in [0,1]$ be a continuous time variable. The intermediate latent is defined by linear interpolation:

$$\mathbf{x}_t = (1-t)\boldsymbol{\epsilon} + t\mathbf{x}_1.$$

The model predicts the velocity field

$$\mathbf{v}_t = \frac{d\mathbf{x}_t}{dt} = \mathbf{x}_1 - \boldsymbol{\epsilon},$$

and is optimized with the loss

$$\mathcal{L} = \mathbb{E}_{\boldsymbol{\epsilon},\mathbf{x}_1,\mathbf{c},\boldsymbol{\alpha},t}\left\| f_\theta(\mathbf{x}_t, \mathbf{c}, \boldsymbol{\alpha}, t) - (\mathbf{x}_1 - \boldsymbol{\epsilon}) \right\|_2^2.$$

This formulation lets the model learn a continuous-time generative flow for video latents conditioned on both language and speech.

Speech-Video Window Attention

A key contribution of the paper is the speech-video window attention mechanism. The authors identify two issues with applying standard DiT cross-attention to speech-driven video: (1) temporal compression in the 3D VAE reduces video length to $\tau = T/r$, while the audio remains at full resolution, and (2) parallel generation allows each video token to attend to all audio tokens, which can mix phonetic content from irrelevant timesteps and weaken lip-sync.

To preserve local synchrony, each latent video frame $\mathbf{x}^{(i)}$ attends only to a bounded audio window. For latent index $i \in \{1,\dots,\tau\}$, attention is restricted to audio tokens with indices

$$j \in \left[\max(1,(i-1)r-1),\,\min(T,ir+1)\right].$$

This window covers the $r$ audio tokens corresponding to the latent frame plus one token on either side, yielding $r+2$ audio tokens in total. The intent is to capture short-term phonetic cues for lip motion while still preserving continuity between adjacent latents. The paper’s ablation shows that removing this mechanism materially worsens synchronization metrics.

Multi-Character Conversation and Structured Prompting

MoCha is designed to generate multi-clip, multi-character conversations without architectural changes. The model uses self-attention across video tokens to preserve character identity and environmental consistency across clips. The paper assumes that only one character speaks at a time, so changes in the audio stream can implicitly guide clip transitions.

The structured prompt template is important because plain visual descriptions become verbose and ambiguous when multiple characters recur across clips. The paper’s solution is to introduce fixed prompt scaffolding: a top-level declaration of the number of video clips, a character list with unique tags such as Person1 and Person2, and then clip-by-clip descriptions that refer only to those tags. This reduces redundancy, helps keep prompts within token limits, and improves binding between character attributes and actions.

Training Strategy

The paper emphasizes that large-scale speech-labeled video is scarce compared with text-labeled video, so training on speech data alone would be too narrow. MoCha therefore uses a joint ST2V and T2V training strategy. In each training batch, about 80% of samples come from speech-conditioned video data and 20% from text-only video data. For the text-only subset, audio embeddings are replaced by zero vectors, which teaches the model to continue working when speech is absent.

The second training idea is a multi-stage curriculum over shot types. The model begins with close-up shots, where speech has the strongest effect on motion. As training proceeds, the authors reduce the previous stage’s data by 50% and introduce more difficult settings with weaker speech conditioning, such as medium shots and broader body motion. The 80/20 ST2V/T2V ratio is maintained across stages. Stage 0 is text-only pretraining, which establishes a strong generative prior before speech conditioning is introduced.

Data Processing Pipeline

To build training data, the authors describe a multi-stage filtering and annotation pipeline. First, videos are segmented into scenes using PySceneDetect. Speech detection is then used to remove non-speech intervals and segments dominated by background noise or music. Remaining segments are cleaned by music and noise removal, and Wav2Vec2 is used to extract speech embeddings.

The pipeline then filters for prominent characters using an LLM-based scene analysis, keeping only scenes with a central human character. Additional motion and lip-sync filters remove scenes that do not contain meaningful human expression or coherent action. Finally, each processed scene is captioned with an LLM into a structured description that includes character appearance, position, speech activity, emotion, and body language.

After this pipeline, the paper reports a curated speech-conditioned video dataset of about 300 hours, corresponding to roughly 500K samples. For text conditioning, the model uses a much larger dataset of around 100M samples. Training is carried out on 64 nodes, and the model is based on a pretrained 30B DiT backbone. The model is trained at approximately 720×720 spatial resolution, supports multiple aspect ratios, and generates 128-frame videos at 24 fps, corresponding to roughly 5.3 seconds per sample.

Benchmark and Evaluation Protocol

The paper introduces MoCha-Bench, a benchmark tailored to the talking-character setting. It contains 150 examples, each with a text prompt and a speech clip. The benchmark includes both close-up and medium-shot scenes so that evaluation covers lip-sync and facial expression on the one hand, and gestures and body motion on the other. The prompts were manually curated and then enriched with help from LLaMA-3.

Since MoCha generates video directly from text and speech while the baselines operate in an image-to-video setting, the comparison protocol provides the baseline methods with the first frame of MoCha’s own generated video as the reference input. When necessary, that first frame is cropped or resized to match the baseline model’s requirements.

Results

Human Evaluation

Human evaluation uses five axes and a 1-to-4 scale, where 4 indicates performance close to real video or cinematic production. Each model output receives 5 independent ratings per example, yielding over 750 responses per model on the 150-example benchmark. MoCha is reported to outperform all baselines on every axis.

These human scores show a clear gap in favor of MoCha. The strongest relative gains are on action naturalness and facial expression naturalness, which is consistent with the model’s full-portrait design and text-guided motion generation. The paper also reports that SadTalker and AniPortrait are limited to head motion, which explains their action scores of 1.0.

Automatic Synchronization Metrics

The paper additionally reports synchronization metrics on MoCha-Bench: Sync-C is better when larger, and Sync-D is better when smaller. MoCha achieves the best score on both metrics, indicating improved lip-sync quality relative to the baselines.

The automatic results align with the human studies: MoCha improves synchronization over Hallo3 by a substantial margin in Sync-C and also lowers Sync-D. The paper uses these metrics primarily to validate the effect of the proposed alignment mechanism.

Ablations

The ablation study isolates the two key design choices: speech-video window attention and joint ST2V+T2V training. Both are shown to matter for synchronization and generalization.

Removing joint training reduces Sync-C and increases Sync-D, indicating that the larger text-only corpus materially improves coverage and robustness beyond the speech-labeled set alone. Removing speech-video window attention hurts synchronization even more, which supports the claim that local audio windows are important for precise lip alignment under parallel latent-frame generation.

What the Paper Claims as Novel

• New task formulation: talking characters, extending beyond traditional talking-head generation to full-portrait, multi-character, speech-driven synthesis.
• End-to-end generation: no auxiliary reference image, keypoints, or skeleton control is required.
• Speech-video window attention: local audio conditioning improves temporal alignment under DiT-style parallel generation.
• Joint speech-text training: combines scarce speech-labeled video with large text-labeled video to improve diversity and generalization.
• Structured prompt templates with character tags: enable multi-character, turn-based dialogue with coherent identity binding across clips.
• MoChaBench: a benchmark specifically designed for this problem setting.

Scope and Stated Limitations

The paper does not include a dedicated limitations section. From the described method and evaluation protocol, the main scope boundaries are clear: the system is designed for speech-plus-text conditioning; the multi-character setup assumes one active speaker at a time; and the evaluation is centered on MoChaBench, which contains 150 curated examples rather than a large-scale open-world benchmark. The model also depends on a substantial pretrained DiT backbone and a curated speech-video dataset, so the reported gains are tied to a relatively heavy training setup.

Conclusion

MoCha is presented as the first diffusion-transformer system for movie-grade talking-character synthesis. Its technical recipe is straightforward but effective: a strong video foundation model, joint conditioning on text and speech, local audio windows for synchronization, a curriculum that mixes speech-labeled and text-labeled data, and prompt engineering for multi-character dialogue. Across the paper’s human studies, synchronization metrics, and ablations, the model is consistently reported to outperform the talking-face baselines on the benchmark they define for this task.