InterTalk

Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

InterTalk enables real-time generation of multi-person conversational talking faces without participant constraints. It balances flexibility, naturalness, and efficiency through motion disentanglement and an iterative generation strategy that progressively refines mutual feedback among participants.

• talking-face
• dialogue
• audio-driven
• realtime
• face-animation
• co-speech
• avatar
• talking-head

Authors: Baiqin Wang, Sen Chen, Jiankuo Zhao, Xiangyu Liu, Zhen Lei, Xiangyu Zhu

Categories: cs.CV

Comment: 17 Pages,8 figures

Published 2026-06-30 · Updated 2026-06-30

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

Conversational talking face generation has recently attracted increasing attention, aiming to synthesize interactive talking videos where characters speak, listen, and respond dynamically to each other. This task presents three core challenges: 1) Flexibility: enabling multi-round dialogues with an arbitrary number of participants; 2) Naturalness: maintaining coherent motion and appropriate non-verbal feedback throughout the interaction; and 3) Efficiency: achieving real-time generation and low computation overhead for long-term continuous online conversation. Despite recent advances, existing methods still fall short in balancing all three requirements. To bridge this gap, we introduce InterTalk, a novel and efficient framework designed for highly interactive conversational talking face generation. Built upon a motion-based architecture, InterTalk supports real-time conversation synthesis. Our method achieves strong flexibility by explicitly modeling multi-round conversational dynamics among each participant, eliminating constraints on their numbers. To enhance interactivity, we incorporate motion feedback from multiple participants and introduce an iterative generation strategy for more natural behaviors. Besides, we disentangle motion into several facial components, enabling targeted refinements for natural response such as precise lip sync and realistic eye blinking. Finally, we construct a new multi-person conversational dataset and enrich it with 3D face-based data augmentation. Extensive experiments demonstrate that InterTalk achieves superior interaction quality while maintaining real-time performance at 30 FPS.

---

Introduction and Motivation

Talking face generation — the task of synthesizing expressive facial animations whose lip movements are synchronized to speech — has seen substantial progress in recent years. However, the dominant line of work focuses exclusively on single-speaker scenarios, leaving largely unexplored the richer challenge of conversational talking face generation, where multiple participants speak, listen, and react to one another in a shared scene.

The authors identify three orthogonal but mutually necessary requirements that any practical conversational system must simultaneously satisfy:

• Flexibility: the system must support multi-round dialogues among an arbitrary number of participants, with no hard constraint on participant count and seamless role switching between speaker and listener.
• Naturalness: generated motions must exhibit temporally coherent articulation for the active speaker and appropriate non-verbal feedback (head nods, eye blinks, expression changes) for the listener(s), with smooth transitions between states.
• Efficiency: the system must achieve real-time throughput (targeting 30 FPS) with low computational overhead, making it viable for long-running interactive applications such as virtual companions, live-streaming, or remote meetings.

Existing approaches fall into two broad categories, each missing at least one requirement. Motion-explicit methods such as DIM, INFP, and ARIG model dynamics directly but either require manual role switching, restrict themselves to single-character animation, or provide only weak partner-feedback modeling. Video generation methods such as MultiTalk (built on Wan 2.1) achieve end-to-end multi-person synthesis but are computationally prohibitive and cap participant count at two. Closed systems like Veo3 and Sora2 are similarly costly and inaccessible.

To close this gap, the paper introduces InterTalk, a motion-based framework that explicitly models multi-participant conversational dynamics and achieves real-time generation at 30 FPS while outperforming prior work on both speaking and interactive quality metrics.

Related Work

Single-Person Talking Face Generation

Early GAN-based work such as Wav2Lip synchronizes lip motion to audio but leaves head pose and expression fixed. More recent diffusion-based methods (EchoMimic, Hallo2, Sonic) produce temporally coherent videos with richer motion but are too slow for real-time use. A middle ground is occupied by two-stage methods (SadTalker, PC-AVS, LivePortrait) that first generate an intermediate representation — facial landmarks or implicit keypoints — and then render the final frame conditioned on an identity image, achieving a good balance between quality and speed.

A related sub-field is listening head generation, which focuses on non-verbal feedback driven by another speaker's audio. L2L uses a codebook model; ELP adds emotional cues; DIM can switch between speaking and listening but cannot generate both simultaneously, limiting interactivity.

Conversational Talking Face Generation

INFP was the first to generate interactive videos from dual audio streams without manual role switching. ARIG extends this with an auto-regressive model. Both remain limited to single-person animation and provide weak partner-feedback. DualTalk models dialogue feedback in 3D but is restricted to mesh-based output, preventing direct 2D video synthesis. MultiTalk applies Wan 2.1 with a human-binding strategy for two-person animation but cannot scale further and requires lengthy inference times.

Method: InterTalk Framework

InterTalk consists of three tightly coupled components:

1. Responsive Context Encoder (RCE) — aggregates all participants' audio and motion into a unified interactive latent feature.
2. Interactive Motion Generator (IMG) — conditioned on self-audio and the RCE feature, produces temporally coherent facial motions with disentangled refinement.
3. Rendering Pipeline — animates each participant via implicit keypoints and composites them into a shared scene.

Responsive Context Encoder (RCE)

The RCE takes as input the audio $A_i$ and motion $M_i$ of all $N$ participants and produces a unified interactive context feature $F_{\text{RCE}}$:

$$F_{\text{RCE}} = \text{RCE}\!\left(\{A_i, M_i\}_{i=1}^{N}\right)$$

For each participant $i$, audio is encoded via a frozen Wav2Vec 2.0 encoder to obtain $F_{\text{a}}^{i}$. Motion is disentangled into three facial components — lip motion $M_{i}^{\text{lip}}$, eye motion $M_{i}^{\text{eye}}$, and head pose $M_{i}^{\text{pose}}$ — each encoded separately and then combined into $F_{\text{m}}^{i}$:

$$F_{\text{a}}^{i} = \text{Enc}_{\text{a}}(A_i), \qquad F_{\text{m}}^{i} = \text{Enc}_{\text{m}}(M_{i}^{\text{lip}}, M_{i}^{\text{eye}}, M_{i}^{\text{pose}})$$

A cross-attention layer then aligns and fuses the audio and motion feature domains for participant $i$. A bidirectional LSTM captures past and future conversational context within a ~3-second window (chosen for practical deployment), yielding the per-participant interactive context feature $F_{\text{RCE}}^{i}$:

$$F_{\text{RCE}}^{i} = \text{biLSTM}\!\left(\text{CrossAttn}(F_{\text{a}}^{i}, F_{\text{m}}^{i})\right)$$

Finally, the RCE aggregates features across all participants via adaptive attention pooling. A reference feature $\bar{F} = \frac{1}{N}\sum_{i=1}^{N} F_{\text{RCE}}^{i}$ is computed by averaging. Attention weights $\alpha_i$ are assigned based on the cosine-like similarity of each participant's feature to this reference, scaled by feature dimension $d$:

$$\alpha_i = \operatorname{softmax}\!\left(\frac{{F_{\text{RCE}}^{i}}^{\!\top} \bar{F}}{\sqrt{d}}\right), \qquad F_{\text{RCE}} = \sum_{i=1}^{N} \alpha_i F_{\text{RCE}}^{i}$$

This design ensures that participants whose features are most consistent with the global conversational context receive higher weights, producing a single global interactive representation that scales naturally to any number of participants.

Interactive Motion Generator (IMG)

The IMG generates temporally coherent facial motions conditioned on both the target participant's own audio $A_{\text{self}}$ and the global interactive feature $F_{\text{RCE}}$:

$$M_{\text{refine}} = \text{IMG}(A_{\text{self}}, F_{\text{RCE}})$$

$A_{\text{self}}$ is encoded using the same Wav2Vec 2.0 encoder as in the RCE (ensuring feature-space consistency) to obtain $F_a$. This is added to $F_{\text{RCE}}$ and fed into a four-block generation network:

$$M_{\text{coarse}} = \text{GenBlock}(F_{\text{RCE}} + F_a)$$

The four sequential blocks are:

1. A fused Transformer Encoder that integrates audio and interactive features into a unified latent space.
2. A temporal alignment attention module adapted from FaceFormer, which applies a causal mask to ensure temporal consistency and prevent future-frame leakage.
3. A Transformer Decoder that captures long-range dependencies and contextual transitions.
4. A feed-forward projection layer that projects representations into separate subspaces for lip, eye, and head-pose components.

Refinement on Disentangled Motion

Because different facial components follow distinct behavioral patterns (lip motion is tightly coupled to audio; eye motion is more stochastic and periodic; head pose depends on spatial arrangement), InterTalk refines each component independently with three dedicated modules:

$$\begin{cases} M_{\text{refine}}^{\text{eye}} = \text{BlinkEnhancer}(M_{\text{coarse}}^{\text{eye}},\, f,\, z), \\ M_{\text{refine}}^{\text{lip}} = \text{SyncEnhancer}(M_{\text{coarse}}^{\text{lip}},\, A_{\text{self}}), \\ M_{\text{refine}}^{\text{pose}} = \text{PoseInitializer}(M_{\text{coarse}}^{\text{pose}},\, x_{\text{self}},\, y_{\text{self}}). \end{cases}$$

• SyncEnhancer: A lightweight module trained on a large-scale single-speaker dataset. It takes the initial coarse lip motion and the self-audio as input and predicts local deformations for refined lip–audio synchronization. It is trained independently after all other modules are pretrained, ensuring compatibility with the full system.
• BlinkEnhancer: Regulates blink frequency $f$ and a noise variable $z$ to produce realistic, non-repetitive eyelid closures.
• PoseInitializer: Adjusts the initial head orientation based on the relative facial position $(x_{\text{self}}, y_{\text{self}})$ in the shared scene, maintaining mutual gaze alignment at the start of interaction.

The three refined components are then combined to form the final motion output $M_{\text{refine}}$.

Rendering Pipeline

InterTalk adopts 3D implicit keypoints as intermediate representations, a paradigm inspired by LivePortrait. Three submodules extract keypoints from reference images $I_{\text{ref}}$:

• A pose estimator computing rotation $R$, translation $t$, and scale $s$.
• An expression estimator predicting deformation $\delta$.
• A canonical keypoint detector extracting canonical keypoints $K_c$.

Original keypoints are constructed as: $$K_{\text{ori}} = s \cdot (K_c \cdot R + \delta) + t$$

The refined motion $M_{\text{refine}}$ is decomposed into rotation $R_d$ and expression deformation $\delta_d$, which drive the final keypoints: $$M_{\text{refine}} \Rightarrow \{R_d, \delta_d\}, \qquad K_d = s \cdot (K_c \cdot R_d + \delta_d) + t$$

Each participant's face region is cropped from $I_{\text{ref}}$ and encoded into an appearance embedding $f_a$ via an identity encoder. A warping decoder fuses $f_a$ with the driven keypoint sequence $K_d$ to estimate a dense flow field, which is applied to warp the appearance embedding and render the final frame: $$I_{\text{res}} = \text{Decoder}\!\left(\text{Warp}(f_a, K_{\text{ori}}, K_d)\right)$$

A stitching module then spatially aligns and blends all individual renderings back into the original shared scene using fine-grained keypoint deformations and a spatial blending mask, producing seamless multi-participant composite outputs.

The rendering pipeline components are initialized from LivePortrait weights to leverage its robust pretrained rendering quality.

Iterative Generation Strategy

At inference time, motions must be generated sequentially participant-by-participant, meaning later participants can only use previously generated (rather than truly simultaneous) motion feedback from others — potentially weakening interactive dynamics. The iterative generation strategy addresses this by running multiple passes over the participant set, each time feeding the prior iteration's output motions as context:

$$M_{ij} = \text{IMG}\!\left(A_i,\; \text{RCE}\!\left(M_{(j-1)}^{\neg i},\; A_{(j-1)}^{\neg i}\right)\right)$$

Here, $M_{ij}$ is the motion of participant $i$ at iteration $j$; the superscript $\neg i$ denotes all participants except $i$. Initial motions $M_{i0}$ and audio features $A_0^{\neg i}$ are zero-initialized. Across several iterations, each participant's motion gradually incorporates the refined reactions of all others, yielding progressively more responsive and natural interactions without introducing latency in the rendered output. The three target-refinement modules (SyncEnhancer, BlinkEnhancer, PoseInitializer) are disabled during iterative passes. Because the IMG is lightweight, the extra compute per iteration is negligible relative to the rendering pipeline, preserving real-time throughput.

Video Dubbing and Single-Role Generation

InterTalk can process video inputs by treating each frame as $K_{\text{ori}}$, enabling video dubbing while preserving natural background changes and body motion. For single-role interactive head generation, pose initialization and the iterative strategy are disabled; the model directly animates from multi-track audio and available motion feedback.

Data Construction

Existing public interactive datasets are limited in quality and conversational depth. ViCo covers only single-round dialogues; DualTalk provides only 3D FLAME coefficients; most multi-person datasets concatenate separate single-speaker recordings rather than capturing participants in a shared scene.

2D Data Collection

The authors collect multi-speaker conversation videos from the Internet and manually segment them to ensure that each clip includes at least two participants continuously appearing together in the same scene. After cleaning, the dataset contains 143 identities and 205 clips. Individual audio tracks are separated from mixed recordings using a visual-audio separation network (IIANet) that leverages lip movements to infer each speaker's voice. Additional TTS-synthesized audio segments are added to enhance evaluation diversity.

3D Face-Based Data Augmentation

To further diversify training data, the authors convert DualTalk's 3D FLAME coefficient data into motion signals compatible with InterTalk's implicit-keypoint representation. Two converter networks are trained on a 2D talking-face dataset where FLAME parameters are annotated using EMOCA.

A 3D mesh is reconstructed from jaw pose $\alpha_{\text{jaw}}$ and expression coefficients $\alpha_{\text{exp}}$, with all other coefficients $\alpha_{\text{other}}$ set to the template $T$. Vertex offsets are computed as: $$\Delta = \text{FLAME}(\alpha_{\text{exp}}, \alpha_{\text{jaw}}, \alpha_{\text{other}}) - T$$

Eye and lip region masks $R_{\text{eye}}$ and $R_{\text{lip}}$ extract localized deformations, which are fed into component-specific converters: $$M_c^{*} = \text{Converter}_c\bigl(R_c(\Delta)\bigr), \quad c \in \{\text{eye},\, \text{lip}\}$$

Head-pose information from FLAME coefficients is directly mapped to $M_{\text{pose}}^{*}$. This augmented data substantially improves inter-participant responsiveness and motion coherence, as confirmed by ablation results.

Training Details

Videos are preprocessed to 25 FPS and audio is resampled at 16 kHz. The RCE and IMG are trained with the Adam optimizer at a learning rate of $1 \times 10^{-4}$ on eight RTX 4090 GPUs for three days.

Loss Functions

Motion supervision combines an L2 reconstruction loss $\mathcal{L}_{\text{rec}}$ and a velocity loss $\mathcal{L}_{\text{vel}}$: $$\mathcal{L}_{\text{motion}} = \lambda_{\text{rec}} \mathcal{L}_{\text{rec}} + \lambda_{\text{vel}} \mathcal{L}_{\text{vel}}$$

The reconstruction loss ensures motion fidelity; the velocity loss promotes temporal smoothness between frames. The SyncEnhancer is trained separately on single-person talking-face data with a synchronization loss $\mathcal{L}_{\text{sync}}$ (following Wav2Lip) to refine lip–audio alignment independently of conversational data limitations.

The rendering pipeline components (MAP) are initialized from LivePortrait and fine-tuned. During inference, participants are animated in parallel across multiple GPUs to maximize throughput. Output resolution is 512×512 before stitching.

Experiments and Results

Evaluation Metrics

The paper evaluates along two axes:

• Speaking quality: Sync-C and Sync-D (SyncNet lip synchronization), FID (Fréchet Inception Distance), NIQE (no-reference image quality), and FVD (Fréchet Video Distance for temporal consistency).
• Interactive quality: FLAME coefficients are extracted from generated videos and used to compute Fréchet Distance (FD), Paired Fréchet Distance (P-FD), Mean Squared Error (MSE), Similarity Index for Diversity (SID), and Residual Pearson Correlation Coefficient (RPCC) for both expression and pose sub-dimensions.
• Efficiency: Frames per second (FPS).

Talking Face Generation (Single-Speaker Baseline)

To verify speaking quality in isolation, interactive audio and motion feedback are set to zero and the model is evaluated on HDTF. Results are shown below.

InterTalk achieves the best Sync-C, FID, NIQE, and FVD scores. The strong lip synchronization relative to Sonic is attributed to the SyncEnhancer module. The lower FID and FVD reflect the quality and temporal consistency of the LivePortrait-based rendering pipeline.

Interactive Head Generation

InterTalk is compared against DIM, INFP, ARIG, and DualTalk on the ViCo dataset using FLAME-based interaction metrics:

InterTalk achieves best or tied-best performance across all metrics. Qualitatively, DIM and INFP display relatively rigid expressions, ARIG maintains nearly static head pose, while InterTalk generates context-aware gestures — gentle nods, natural smiles — demonstrating strong responsiveness.

Multi-Participant Conversations

InterTalk is evaluated against MultiTalk (the only open-source multi-person talking face method) on the newly collected InterTalk dataset:

InterTalk outperforms MultiTalk on all metrics except SID, where the large video-generation model's stronger generalization ability gives MultiTalk a slight edge. More critically, InterTalk runs at 31.23 FPS vs. MultiTalk's 0.61 FPS — a roughly 50× efficiency advantage — and imposes no constraint on participant count, while MultiTalk is capped at two participants.

User Study

A perceptual user study was conducted under the Mean Opinion Score (MOE) protocol with 23 participants, rating each method on four aspects using a 1–5 Likert scale:

InterTalk achieves the highest scores on every dimension. Particularly notable are the Motion Naturalness (4.55) and Interaction Fluency (3.96) scores, validating the iterative generation strategy, and the Lip-sync Accuracy (4.03) score, confirming the SyncEnhancer's effectiveness.

Ablation Study

A systematic ablation study examines four components on the multi-person InterTalk dataset:

Key findings from the ablation:

• Motion Feedback in RCE: Removing it degrades FD from 22.67 to 29.72 and SID from 4.28 to 2.71, confirming that cross-participant motion context is critical for realistic interaction modeling.
• Iterative Generation Strategy: Without it, FD rises to 24.26 and SID drops to 4.12. Speed improves slightly (34.67 FPS), confirming that the extra iteration overhead is indeed small. The iterative strategy introduces no output latency.
• Disentanglement on Motion: Without disentanglement (and thus without the SyncEnhancer), Sync-C drops noticeably from 8.30 to 7.58, illustrating that training the SyncEnhancer on large single-speaker data compensates for the limited diversity of conversational datasets.
• 3D Face-based Data Augmentation: This is the single most impactful component: removing it collapses Sync-C to 7.01, FD to 38.67, and SID to 1.95 — showing that augmenting with FLAME-derived motions is essential for learning rich conversational dynamics.

The analysis of iteration count (Figure: Variation of FD and SID across iterations) confirms monotonic improvement: FD steadily decreases and SID steadily increases with more iterations, indicating progressively more realistic and diverse motions. This empirically validates the convergence of the iterative feedback mechanism.

Stated Contributions and Novelty

• InterTalk framework: A unified, motion-based architecture for flexible (arbitrary participant count), natural (coherent motion with non-verbal feedback), and efficient (real-time 30 FPS) conversational talking face generation — the first method to simultaneously satisfy all three requirements.
• Responsive Context Encoder (RCE): Encodes multi-participant audio and disentangled motion into a unified interactive context feature via cross-attention, bidirectional LSTM, and adaptive attention pooling. Scales naturally to any number of participants.
• Interactive Motion Generator (IMG) with Disentangled Refinement: Generates coarse conversational motions with a causal Transformer and refines each facial component (lip, eye, pose) independently using targeted enhancement modules, leveraging large-scale single-speaker data to improve lip-sync beyond what conversational data alone can provide.
• Iterative Generation Strategy: A training-free inference procedure that progressively incorporates mutual feedback across participants with negligible computational overhead, enabling realistic group interaction dynamics.
• New Multi-Person Conversational Dataset: 143 identities, 205 clips of genuine shared-scene multi-speaker conversation video, with automated audio source separation.
• 3D Face-Based Data Augmentation: Converts FLAME 3D facial coefficients (from DualTalk's dataset) into implicit-keypoint motion signals via trained converter networks, substantially enriching the training distribution for interaction learning.

Limitations and Discussion

The paper does not include an explicit limitations section, but several can be inferred from the reported results and design choices:

• SID trade-off with video generation models: On multi-person generation, MultiTalk (backed by Wan 2.1) achieves a higher SID (5.14 vs. 4.28), suggesting that large-scale video generative models retain a generalization advantage in motion diversity. InterTalk's motion-explicit design trades some generative diversity for efficiency.
• Dataset scale: The newly collected dataset contains 205 clips — modest compared to the massive corpora used to train video generative baselines. The FLAME-based augmentation partially compensates, but this remains a potential bottleneck for generalization to rare interaction styles.
• ~3-second context window: The RCE operates over a limited look-ahead (~3 seconds) rather than full conversation history, which may limit long-range conversational coherence.
• No explicit emotion modeling: Non-verbal feedback is modeled through motion statistics rather than explicit emotion labels, which may limit controllability of affective responses.
• Rendering fidelity dependency: The pipeline relies on LivePortrait's implicit keypoint renderer; very large head motions, extreme expressions, or heavy occlusions may produce artifacts not attributable to the motion generation components.

Conclusion

InterTalk presents a comprehensive solution to conversational talking face generation that achieves a rare combination of flexibility (arbitrary participants, multi-round dialogue), naturalness (disentangled motion refinement, iterative feedback), and efficiency (31+ FPS real-time synthesis). The framework's core innovations — the Responsive Context Encoder, the Interactive Motion Generator with component-wise refinement, and the training-free iterative generation strategy — are each validated by ablation experiments showing significant, complementary contributions. The introduction of a new multi-person conversational dataset and 3D face-based augmentation further strengthens the training regime. Extensive quantitative and perceptual evaluations confirm state-of-the-art performance across both speaking quality and interactive naturalness dimensions, positioning InterTalk as a strong foundation for applications in virtual companions, remote meetings, live-streaming, and emotionally intelligent digital humans.