MEMO
MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
MEMO generates expressive talking videos by combining memory-guided temporal diffusion and emotion-aware audio conditioning. It improves long-term identity consistency, audio-lip sync, and emotional expression by integrating memory and dynamic emotion detection directly into the diffusion model.
Demos
These demos showcase MEMO's state-of-the-art ability to generate highly expressive, identity-consistent talking videos driven by various audio types across multiple languages. Viewers should note the natural audio-lip synchronization, smooth and vivid motion, diverse head poses, and the system's robustness over long video durations with minimal error accumulation. The comparison demos highlight MEMO's clear advantages over recent baselines in expression alignment, motion realism, and temporal consistency.
Links
Paper & demos
Code & resources
Impact
Abstract
Recent advances in video diffusion models have unlocked new potential for realistic audio-driven talking video generation. However, achieving seamless audio-lip synchronization, maintaining long-term identity consistency, and producing natural, audio-aligned expressions in generated talking videos remain significant challenges. To address these challenges, we propose Memory-guided EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) a memory-guided temporal module, which enhances long-term identity consistency and motion smoothness by developing memory states to store information from a longer past context to guide temporal modeling via linear attention; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion adaptive layer norm. Extensive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment.
Problem Setting and Motivation
MEMO addresses audio-driven talking video generation: given a reference face image and driving audio, the model must synthesize a realistic talking video that simultaneously preserves identity, keeps lip motion synchronized with speech, produces smooth head motion, and reflects the emotional tone of the audio. The paper frames these as a coupled set of hard constraints rather than independent subproblems. In particular, it argues that existing diffusion-based approaches often rely on cross attention for audio conditioning and on only a small number of recent generated frames for autoregressive temporal conditioning, which makes them vulnerable to temporal error accumulation, identity drift, weak audio-video interaction, and overly static or unnatural expressions.
The paper’s central claim is that long-range temporal memory and emotion-aware audio conditioning should be built directly into the diffusion model. MEMO therefore combines a memory-guided temporal module for longer-horizon coherence and an emotion-aware audio module for stronger audio-video interaction and more expressive facial synthesis.
Overall Architecture
MEMO is an end-to-end latent diffusion system built around two components: a Reference Net and a Diffusion Net. The Reference Net carries identity information from the reference image, while the Diffusion Net synthesizes the target video clip. The two major contributions live inside the Diffusion Net:
- Memory-guided temporal module: replaces short-context temporal self-attention with a memory-based linear attention scheme that can incorporate a much longer past context efficiently.
- Emotion-aware audio module: replaces traditional audio cross attention with multi-modal attention, then adds emotion-aware conditioning through an automatically detected audio emotion label and emotion-adaptive layer normalization.
Implementation-wise, both the Reference Net and the spatial module of the Diffusion Net are initialized from Stable Diffusion 1.5, while the temporal module is initialized from AnimateDiff. The paper also replaces text cross attention with image cross attention in both the Reference Net and the Diffusion Net, and adds two projection modules to map audio and image embeddings to the dimensions required by the attention blocks. Audio conditioning uses all hidden states from Wav2Vec 2.0.
Diffusion Objective and Training Formulation
The method is formulated in latent diffusion space. Given an image $I$, an encoder produces a latent code $z_0 = \mathcal{E}(I)$. Diffusion training injects noise and learns to predict it from noisy latents conditioned on audio, reference-image, and temporal context. The paper adopts the rectified-flow style objective used by Stable Diffusion 3:
$$ \mathcal{L} = \mathbb{E}_{z_t, c, \epsilon \sim \mathcal{N}(0,1), t}\big[\lambda(t)\,\|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2\big], $$
with $\lambda(t) = 1/(1-t)^2$ and reparameterization $z_t = (1-t)z_0 + t\epsilon$. The conditioning variable $c$ can include audio, reference-image, and motion-frame context depending on the stage and module being trained.
Memory-Guided Temporal Module
The paper argues that existing talking-video diffusion methods usually condition on only the last 2 to 4 generated frames. That is efficient, but it creates two problems: first, the model cannot fully leverage longer motion history; second, if the most recent frames contain artifacts, those artifacts get fed back into later generations and the errors accumulate over time. MEMO’s temporal module is designed specifically to mitigate this issue.
The key idea is to replace standard temporal self-attention with linear attention, and to augment it with a persistent memory state that stores information from older frames using a causal decay factor $\gamma$.
For a frame index $i$, the linear-attention output is computed as:
$$ \operatorname{out}_i = \frac{\phi(Q_i)^\top \big(\sum_{j=1}^{f} \phi(K_j) V_j^\top\big)}{\phi(Q_i)^\top \sum_{j=1}^{f} \phi(K_j)}, $$
where $f$ is the number of frames and $\phi$ is the activation used in the attention kernel; the paper uses softmax in this work.
To store a longer history without growing GPU memory with the number of past frames, the module maintains two memory statistics for the historical clip:
$$ M^f_{KV} = \sum_{i=1}^{f} \gamma^i \phi(K_{m,i}) V_{m,i}^\top, \qquad M^f_{K} = \sum_{i=1}^{f} \gamma^i \phi(K_{m,i}), $$
where $0 < \gamma < 1$ is the decay factor. Recent frames have more influence because of the exponential weighting. The memory is updated incrementally when a new chunk of frames arrives:
$$ M^{a+b}_{KV} \leftarrow \gamma^a M^b_{KV} + \sum_{j=1}^{a} \gamma^j \phi(K_{h,j})V_{h,j}^\top, $$
$$ M^{a+b}_{K} \leftarrow \gamma^a M^b_{K} + \sum_{j=1}^{a} \gamma^j \phi(K_{h,j}). $$
At generation time, the current clip is guided by both the current-frame statistics and the stored memory:
$$ \operatorname{out} = \frac{\phi(Q_h)^\top (H_{KV} + M_{KV})}{\phi(Q_h)^\top (H_{K} + M_{K})}, $$
with $H_{KV} = \phi(K_h)V_h^\top$ and $H_K = \phi(K_h)$. The paper emphasizes that this design allows MEMO to use possibly all past frames in a computationally efficient way, rather than being restricted to the latest few frames.
Emotion-Aware Audio Module
The second major contribution is an audio conditioning module that is explicitly designed to make the video more expressive and more tightly coupled to the audio. The paper identifies two limitations of common audio-conditioning schemes. First, standard cross attention conditions video features on fixed audio embeddings, which can limit the richness of audio-video interaction. Second, a single human-assigned emotion label for an entire clip cannot capture dynamic changes in the emotional tone of speech or song.
Multi-modal attention
Instead of using audio only as an external conditioning signal, MEMO uses multi-modal attention to jointly process video and audio features. The paper contrasts this with the standard conditional diffusion loss under audio-only conditioning, and writes the multi-modal objective as:
$$ \mathcal{L}_{\theta_{va}} = \mathbb{E}_{t,\epsilon \sim \mathcal{N}(0, I)} \big[ \lambda(t)\,\|\epsilon_\theta(v_t, a) - \epsilon\|_2^2 \big]. $$
This is meant to improve the depth of interaction between the modalities during denoising, not just attach audio features to a video backbone through a standard cross-attention layer.
Audio emotion-aware diffusion
MEMO also predicts an emotion label directly from the audio and uses that label to modulate generation. The emotion detector recognizes eight classes: angry, disgusted, fearful, happy, neutral, sad, surprised, and others. Emotion detection is performed at the subsegment level for robustness. For each frame, the detector looks at audio from a 3-second sliding window centered on that frame; then the subsegment label is taken as the most frequent emotion across its frames.
The resulting emotion embedding is injected into every layer via emotion-adaptive layer normalization, yielding the emotion-conditioned objective:
$$ \mathcal{L}_{\theta_{va|e}} = \mathbb{E}_{t,\epsilon \sim \mathcal{N}(0, I)} \big[ \lambda(t)\,\|\epsilon_\theta(v_t, a \mid e) - \epsilon\|_2^2 \big]. $$
At inference, classifier-free guidance is applied to control the strength of the emotion condition:
$$ \tilde{\epsilon}_\theta(v_t, a \mid e) = (1+w)\epsilon_\theta(v_t, a \mid e) - w\epsilon_\theta(v_t, a), $$
where $w$ is the guidance scale. The paper is explicit that the reference image largely determines the overall emotional tone of the face, and the audio emotion acts as a subtle adjustment rather than an override.
Emotion decoupling training
To encourage the model to separate the expression in the reference image from the emotion expressed by the audio, the paper introduces an emotion decoupling strategy for MEAD clips. When the training clip is from MEAD, the reference image is not taken from the same clip; instead, a different image of the same person with a different emotion is sampled. This forces the model to disentangle reference-expression cues from audio-driven emotion cues. The method also allows manual replacement of the detected emotion label at inference time if a user wants to specify a target emotion explicitly.
Data Collection, Filtering, and Emotion Detection
The paper makes a strong point that the quality of training data is critical for talking-head diffusion. It aggregates open-source datasets such as HDTF, VFHQ, CelebV-HQ, MultiTalk, and MEAD, plus additional collected data. The raw corpus exceeds 2,200 hours, but the authors report that many clips contain severe issues: audio-lip misalignment, missing heads, multiple heads, subtitle occlusions, extremely small face regions, and low resolution. Training directly on this raw pool led to unstable optimization and poor generation quality.
To address that, the paper proposes a five-stage preprocessing pipeline:
- Scene transition detection using TransNet V2, then trimming clips to under 30 seconds.
- Face detection with Grounding DINO, filtering clips with no faces, partial faces, or multiple heads, then cropping talking heads with a scale factor of 1.1.
- Quality filtering with HyperIQA, using a threshold of 40 on the first frame to remove low-quality or low-resolution videos.
- Audio-lip synchronization filtering with SyncNet, using a Sync-C threshold of 5.
- Manual verification on a subset of the data for higher-confidence filtering.
After filtering, the paper reports about 660 hours of high-quality processed training video.
For the emotion detector itself, the authors collect a separate large-scale training set spanning both speech and music emotion recognition. The speech side is based on EmoBox, which aggregates 23 datasets across 12 languages. The music side uses RAVDESS-song and MTG-Jamendo, including songs with and without background music. All audio is converted to mono at 16 kHz, long clips such as MTG-Jamendo are split into 30-second segments, and the datasets are split 3:1 into training and testing sets.
The original emotion label spaces are merged into the paper’s eight-class taxonomy. The classifier is built on a frozen Emotion2vec backbone with a 5-layer MLP head. The paper adds gradient clipping at an $\ell_2$ norm of 1.0 and contrastive learning to improve generalization. The reported overall accuracy is 78.26%, compared with 68.78% for the original Emotion2vec baseline used with a linear head.
Training Strategy and Inference
MEMO is trained in two stages.
Stage 1: Face domain adaptation
In the first stage, the Reference Net and the spatial attention modules of the Diffusion Net are adapted to the face domain using the rectified-flow loss. This stage also adapts the original text cross-attention module. The goal is to make the backbone strong at representing facial appearance before introducing the more specialized temporal and emotion modules.
Stage 2: Emotion-decoupled robust training
In the second stage, the memory-guided temporal module and the emotion-aware audio module are introduced. The paper first performs a warm-up phase for the newly added modules while keeping Stage 1 modules fixed; after that, all modules are trained jointly. During this stage, the model uses the emotion-conditioned flow loss, and the full processed dataset is used.
The paper also adds a robustness trick: samples whose loss suddenly exceeds a large threshold are filtered out during training. The threshold used in the paper is 0.1, while the loss is said to typically settle around 0.03. This is meant to suppress noisy examples that would otherwise destabilize diffusion training and bias optimization.
The stage-2 training uses a fixed temporal context of 16 past frames to compute memory states, although the authors note that the memory update mechanism with causal history decay means that dynamically varying the past-frame count during training is not necessary. During classifier-free training, emotion embeddings, reference images, audio embeddings, and past frames are each randomly dropped with probability 5%.
At inference, the system runs at 30 FPS, generates 16 frames per iteration, and uses classifier-free guidance with scale 3.5. The paper also reports that increasing the guidance scale makes emotional expressions more pronounced.
Experimental Setup
The paper evaluates generalization using two out-of-distribution test sets. The first contains 150 clips sampled from the VoxCeleb2 test set. The second also contains 150 clips and is assembled to be more diverse in audio, backgrounds, ages, genders, and languages.
The evaluation metrics are:
- FVD for overall video distribution quality.
- FID for per-frame visual quality.
- Sync-D for audio-lip synchronization.
The compared baselines include both two-stage methods and end-to-end diffusion methods: SadTalker, AniPortrait, V-Express, Hallo, Hallo2, and EchoMimic. The paper’s human study also evaluates quality, motion smoothness, expression-audio alignment, audio-lip synchronization, and identity consistency.
Quantitative Results
The main quantitative result is that MEMO achieves the best reported scores across both OOD test sets on all three metrics. The paper interprets this as evidence that the method improves not only lip synchronization but also general visual quality and robustness to unseen identities and audio conditions.
| Method | VoxCeleb2 FVD ↓ | VoxCeleb2 FID ↓ | VoxCeleb2 Sync-D ↓ | OOD FVD ↓ | OOD FID ↓ | OOD Sync-D ↓ |
|---|---|---|---|---|---|---|
| SadTalker | 397.0 | 71.7 | 8.6 | 288.7 | 48.3 | 10.6 |
| AniPortrait | 333.2 | 45.5 | 11.0 | 238.7 | 31.2 | 10.5 |
| V-Express | 418.9 | 58.9 | 8.2 | 315.2 | 46.7 | 9.5 |
| Hallo | 330.4 | 41.6 | 8.0 | 231.1 | 31.9 | 9.3 |
| Hallo2 | 302.0 | 41.6 | 8.0 | 223.1 | 29.8 | 9.3 |
| EchoMimic | 293.9 | 43.8 | 10.1 | 223.9 | 39.9 | 9.8 |
| MEMO | 254.3 | 31.7 | 7.4 | 161.1 | 24.9 | 9.2 |
On the VoxCeleb2 test set, MEMO reports 254.3 FVD, 31.7 FID, and 7.4 Sync-D. On the collected OOD dataset, it reports 161.1 FVD, 24.9 FID, and 9.2 Sync-D. These are the best values among the compared methods in the paper.
The human study supports the same trend: MEMO receives the highest preferences across all five evaluated dimensions, including overall quality, motion smoothness, expression-audio alignment, audio-lip synchronization, and identity consistency.
Qualitative Findings and Ablations
The qualitative section emphasizes three kinds of generalization: stronger head-motion diversity, broader audio robustness, and better reference-image robustness.
Motion diversity. The paper reports that MEMO can generate a wider range of smooth head movements than prior methods, which is visible in both video samples and motion heatmaps. This is linked to the memory-guided temporal design, which preserves coherence without forcing motion into a narrow, short-context regime.
Audio robustness. The paper tests speeches, songs, and raps. MEMO maintains lip synchronization across these audio types, including fast or highly expressive audio where synchronization is especially difficult.
Reference-image robustness. The model is also evaluated on reference images that differ strongly from the training distribution, including portraits, sculpture, and digital art, and the paper reports that MEMO still produces stable videos without obvious artifacts.
Multilingual audio. Even though most training data is English, the system is shown to generalize to English, Chinese, Spanish, Japanese, and Korean audio.
Head pose robustness. The method also handles reference images with different head poses, including frontal and side views, while preserving identity and expression consistency.
Emotion-guided generation. The paper shows that the generated expression depends on both the reference-image expression and the detected audio emotion. When a fixed reference image is paired with different emotion labels, MEMO can refine the output expression accordingly; when the same person is shown in neutral, angry, happy, or sad reference images, it can generate emotionally consistent results after matching the audio emotion label.
Long-duration generation. The appendix reports that MEMO can generate long videos with thousands of frames while alleviating error accumulation and keeping identity stable.
Core ablation findings
- Longer memory helps: the human evaluation on past-frame length shows that increasing the temporal context improves temporal coherence, overall quality, motion smoothness, identity consistency, and audio-lip alignment.
- Multi-modal attention helps: human preference comparisons favor multi-modal attention over cross attention for overall quality and synchronization.
- Emotion-aware training helps: the emotion-aware module together with emotion decoupling better aligns facial expression with the specified emotion label.
- Guidance scale matters: higher classifier-free guidance leads to more pronounced emotional expressions.
Stated Contributions, Limitations, and Future Directions
The paper’s stated contributions are: (1) a memory-guided temporal module that mitigates error accumulation and improves long-range identity consistency; (2) an emotion-aware audio module that improves audio-video interaction and emotional alignment; (3) a high-quality data processing pipeline for talking-head training data; and (4) a two-stage training strategy that combines face-domain adaptation with emotion-decoupled robust training.
The paper does not include a separate formal limitations section. The closest thing to a future-work statement is in the conclusion: the authors say it would be interesting to explore a Diffusion Transformer with better identity preservation strategies for talking video generation. They also point out that because MEMO does not depend on face-specific inductive biases, it could be extended to broader tasks such as talking body generation.
Overall, MEMO is presented as a strong end-to-end diffusion baseline for expressive talking video synthesis, with its main novelty coming from the combination of long-memory temporal modeling and emotion-aware audio conditioning rather than from any external post-processing or hand-crafted face constraints.