MultiTalk

Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation

MultiTalk enables audio-driven multi-person conversational video generation by binding multiple audio streams to corresponding people, producing synchronized lip motion and coherent interactions. It advances single-person talking-head methods to dynamic multi-person scenarios with improved instruction-following.

• audio-driven
• talking-head
• full-body
• lip-sync
• multimodal
• dialogue

Authors: Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, Wenhan Luo

Categories: cs.CV

Comment: Homepage: https://meigen-ai.github.io/multi-talk Github: https://github.com/MeiGen-AI/MultiTalk

Published 2025-05-28 · Updated 2025-05-28

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub2,958 stars66 watchers490 forksApache-2.0
• Hugging Face387 downloads198 likes

Impact

• Semantic Scholar58 citations11 influential citations

Abstract

Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.

---

Introduction

MultiTalk addresses a new task introduced in this paper: audio-driven multi-person conversational video generation. The goal is to generate a video of multiple people interacting conversationally, conditioned on a prompt and multiple audio streams, while keeping lip motion synchronized to the correct speaker and preserving visual quality. The paper positions this task as a step beyond conventional talking-head and talking-body generation, which typically assume a single person and a single audio stream.

The core difficulty is not only generating plausible human motion, but also solving three coupled problems that arise in multi-person dialogue scenes: (1) the model must accept multi-stream audio, (2) each audio stream must be bound to the intended person rather than driving everyone simultaneously, and (3) the person regions are dynamic across frames, so localization must be adaptive rather than fixed to the initial image. The authors argue that existing single-person audio-driven animation methods do not solve these issues and are further limited in instruction following when prompts require larger motion or richer interaction.

The paper's main claims are that MultiTalk can (i) bind multiple audio streams to different people in a scene, (ii) preserve the instruction-following ability of the underlying video diffusion model through carefully chosen training strategies, and (iii) perform competitively on talking-head, talking-body, and two-person conversational benchmarks.

Problem Formulation and High-Level Design

MultiTalk is built on a DiT-based image-to-video diffusion backbone with a 3D VAE. The base model takes a reference image and a text prompt, then generates a video by denoising latent tokens over time. The method extends this backbone with an audio pathway and a multi-person binding mechanism.

The design philosophy is to keep the visual backbone largely intact and add only the components needed for audio conditioning and binding. This is reflected in the training choice to update only the audio cross-attention module and the audio adapter, while freezing the rest of the network. The authors report that this partial fine-tuning helps preserve the base model's instruction-following behavior and reduces distortions that appear with full-parameter training.

Method

Backbone video model

The foundational model is a DiT-based video diffusion model with a 3D VAE that compresses both spatial and temporal dimensions. A text encoder produces the text-conditioned input $c_{\text{text}}$, and a CLIP image encoder provides a global visual context that is injected through decoupled cross-attention. MultiTalk starts from this image-to-video setting and adds audio conditioning rather than redesigning the entire generator.

Single-person audio conditioning

To support audio-driven animation, the authors insert an additional audio cross-attention layer after the text cross-attention in each DiT block. Queries come from video latents, while keys and values come from audio embeddings. Audio is extracted using Wav2Vec. Because speech motion depends on neighboring audio, the method concatenates nearby audio features with context length $k$:

$$a_i = \operatorname{Concat}\big(a_{i-\lfloor k/2 \rfloor}, \ldots, a_i, \ldots, a_{i+\lfloor k/2 \rfloor}\big).$$

Since the audio sequence is longer than the temporally compressed video latent sequence, the paper introduces an audio adapter to compress the audio condition to match the video latent timeline. The first audio frame is processed separately from the remaining frames, the latter are downsampled, and the outputs are fused through MLP layers to form the compressed audio condition $c_a$.

$$c_a = \operatorname{MLP}\Big(\operatorname{Concat}\big(\operatorname{MLP}(a_1),\, \operatorname{MLP}(\operatorname{Down}(a_{[2:l]}))\big)\Big).$$

Multi-stream audio injection

Multi-person conversational generation requires the model to inject multiple audio streams without confusing the speaker identity. The paper evaluates four strategies, summarized in the figure below. The first two are simple but fail to bind each audio stream to the correct person. The third strategy splits the latent video spatially into left and right halves and works only when motion is limited. The final strategy, Label Rotary Position Embedding or L-RoPE, is the proposed adaptive solution.

In the failed baselines, multi-stream audio embeddings are either concatenated and attended jointly, or attended separately and then added. These choices do not reliably create person-specific binding. The spatial split works better but lacks generalization when people move widely or occupy irregular regions.

Adaptive person localization

L-RoPE depends on knowing which latent tokens correspond to which person. The paper therefore proposes an adaptive person localization procedure. Given a reference image containing two people, the method first obtains masks for person 1, person 2, and the background, denoted as $M = \{M_{p1}, M_{p2}, M_b\}$. It then uses the reference-image-to-video self-attention map from the DiT model to compute similarity between each video latent token and the reference-image regions.

If $A_{r2v}$ denotes the reference-image-to-video attention map, the paper forms a similarity matrix $S \in \mathbb{R}^{f h w \times 3}$, where each row measures how strongly a video token aligns with each of the three regions. The token is assigned to the region with the highest similarity. This gives a dynamic token-to-person/background label for the current frame sequence, rather than relying on a fixed spatial prior.

L-RoPE for audio-person binding

The central idea of L-RoPE is to attach a label to each query token and each audio token so that tokens with related labels more strongly interact in audio cross-attention. The paper adapts rotary position embedding by turning the rotation angle into a function of the token label $l_i$:

$$\theta_i = l_i \cdot \theta_{\text{base}},$$ $$\hat{q}_i = \operatorname{LRoPE}(q_i, l_i) = q_i e^{l_i \theta_i}.$$

For person tokens, the label is assigned from a person-specific range based on the normalized similarity score from the adaptive localization step. The paper uses distinct label ranges for different people; in the final configuration, the first person uses labels in $[0, 4]$, the second person uses labels in $[20, 24]$, and the background is assigned the static label $12$. For audio tokens, the two streams are labeled with values matching their intended correspondences, specifically $2$ and $22$ in the reported setup. The claim is that these labels activate localized regions in the audio cross-attention map and reduce incorrect binding.

Importantly, the ablation suggests that L-RoPE is not highly sensitive to the exact label range as long as the relative assignment structure is preserved.

Training strategy

The paper emphasizes that training recipe matters as much as architecture for retaining general instruction-following capability. Three strategy choices are highlighted: two-stage training, partial-parameter training, and multi-task training.

• Two-stage training: stage 1 focuses on single-person audio-driven animation; stage 2 introduces dual-stream audio and two-person conversation data.
• Partial parameter training: only the audio cross-attention layer and the audio adapter are fine-tuned; all other parameters stay frozen.
• Multi-task training: training alternates between audio + image to video ($\mathrm{AI2V}$) and image to video ($\mathrm{I2V}$) tasks. In the $\mathrm{I2V}$ branch, audio embeddings are zeroed out. The $\mathrm{I2V}$ data is drawn from multi-event clips containing human-object and human-environment interactions, which the authors say is important for preserving instruction following.

The figure above supports the paper's key training claim: when the model is trained only on talking-head/talking-body audio-animation data, instruction following degrades; adding the $\mathrm{I2V}$ branch with multi-event data helps retain the ability to follow prompt-described motion and interaction.

The appendix further compares full-parameter training with cross-attention-only tuning and reports that full training can degrade instruction-following behavior and produce hand/object distortions, whereas the restricted tuning path better preserves the base model.

Long-video inference

The model can generate only a short clip directly, so the authors add an autoregressive inference strategy for long videos. The last $5$ frames of the previously generated video are reused as additional conditions; after 3D VAE compression these become $2$ frames of latent noise. The method then pads the remaining frames with zeros and concatenates the latent conditions and a video mask before feeding them into the DiT sampler. This enables longer rollouts without changing the core model.

Datasets and Experimental Setup

The experiments use three training/data sources and three evaluation settings. The first-stage training corpus contains about 2K hours of videos with a single talking person, covering either the face or body. The authors also collect about 200K video clips with multiple events and human-object or human-environment interactions, with an average clip duration of about 10 seconds. For the second stage, they collect about 100 hours of videos consisting of conversations between two people.

For evaluation, the paper uses the following datasets:

• Talking head: HDTF and CelebV-HQ.
• Talking body: EMTD.
• Two-person conversational body generation: the authors' collected MTHM dataset, consisting of 40 internet-sourced videos.

The evaluation metrics are standard generation and synchronization measures: Fréchet Inception Distance $\mathrm{FID}$, Fréchet Video Distance $\mathrm{FVD}$, Expression-FID $\mathrm{E\text{-}FID}$, Sync-C, and Sync-D. Sync-C is better when higher, while the other metrics are better when lower.

Implementation-wise, the authors use Wan2.1-I2V-14B as the backbone, train with AdamW at a constant learning rate of $2\times 10^{-5}$ plus warm-up, and use 64 NVIDIA H800-80G GPUs. Batch size is 64 in stage 1 and 32 in stage 2. Sampling uses 40 steps, and the appendix notes that Qwen-VL is used for reference image captioning.

Quantitative Results

The paper compares MultiTalk with leading talking-head and talking-body baselines. The authors report both a single-person variant, which is evaluated with standard single-person benchmarks, and a multiple-person variant, which is evaluated on the multi-person setting.

Talking-head results

On talking-head benchmarks, the single-person MultiTalk variant is strongest on most metrics: it achieves the best Sync-C and the lowest E-FID, FID, and competitive FVD on HDTF, while also performing best or near-best on CelebV-HQ. The multi-person variant is similarly strong and in several cases improves FVD on CelebV-HQ, suggesting that multi-stream capability does not catastrophically damage the base animation quality.

Talking-body results

On EMTD, MultiTalk again improves synchronization and image quality compared with the reported baselines. The multi-person variant achieves the best Sync-C, E-FID, and FID, while the single-person variant slightly outperforms it on Sync-D and FVD. Overall, the results support the paper's claim that the proposed audio-conditioning and training recipe transfer well from heads to bodies.

Qualitative Analysis

The qualitative comparisons reinforce two central claims: better prompt following and more reliable motion consistency. In the single-person setting, the paper shows that when a text prompt specifies motion or interaction, MultiTalk responds more faithfully than the compared methods and produces fewer visible artifacts.

For multi-person scenes, the paper compares against a simple video-concatenation baseline that generates the left and right parts separately and then stitches them together. MultiTalk avoids the inconsistency that can arise at the seam and is able to generate more coherent interactive scenes. The attention visualization also suggests that the adaptive localization step finds the active speaker region more precisely, which helps audio binding.

Ablations and Additional Analyses

The paper includes an ablation of L-RoPE label ranges on the MTHM dataset. Two variants are compared: one using tighter ranges and one using the final broader person-specific ranges. The reported scores are close, indicating that the method is not strongly sensitive to the exact chosen label interval.

The accompanying analysis of multi-stream versus single-stream training shows that moving from single-person to multi-person modeling does not inherently degrade performance on the standard head/body benchmarks. This is an important result because it suggests that the binding mechanism can be added without sacrificing the original single-person capability.

The appendix also reports a qualitative comparison between full-parameter training and cross-attention-only tuning. Full training is said to reduce instruction-following ability, especially for motion and interaction descriptions, and can introduce hand distortion. The restricted tuning strategy is therefore not just computationally cheaper but also functionally safer for preserving the base model's generalization.

Limitations and Societal Impact

The paper states one explicit limitation: the method performs better with real audio than with synthesized audio in facial expression quality, likely because the training data uses real audio. The authors say they will study the gap between real and synthesized audio in future work.

In the societal-impact discussion, the authors note that the method could be misused to generate fake videos of celebrities or other misleading content. This concern is framed as a general risk shared by human-animation systems rather than a problem unique to MultiTalk.

Takeaways

• MultiTalk formalizes a new problem setting: generating conversational videos with multiple speakers from multiple audio streams.
• The main technical contribution is L-RoPE, which uses label-aware rotary embedding to improve audio-person binding in cross-attention.
• An adaptive localization step based on reference-image-to-video attention is used to determine which latent tokens correspond to each person and the background.
• Training recipe matters substantially: partial-parameter tuning and multi-task training help preserve the base model's instruction-following ability.
• Across the reported benchmarks, MultiTalk improves synchronization and overall video quality, and the multi-person version remains competitive with the single-person variant on standard head/body datasets.