A Brief History of Audio-Driven Talking-Avatar Research (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.
Conversational AI Research Digest
Daily summaries for ML researchers and engineers about conversational AI.
Featured
Posts
Emotion-Aware Avatars, Smarter TTS, and Tool-Calling Speech Agents
Today's digest spans real-time emotion-controllable portrait animation, reinforcement-learned TTS, environment-aware voice synthesis, and spoken LLMs that reason with external tools — a broad sweep across talking avatars, voice generation, and speech intelligence.
Empathetic Voice Agents, Precise Speech Editing, and Real-Time Gestures
Today's digest covers full-duplex empathetic dialogue from ByteDance, XML-driven speech editing with fine-grained prosody control, streaming co-speech gesture generation with drift-free anchoring, and a novel Latent Softmax output layer for multilingual ASR across tonal and non-tonal languages.
Emotion at the Center: Expressive TTS and Talking Faces Get Personal
Today's digest focuses on emotion in speech and video synthesis — from geometry-guided talking face generation and hallucination-free TTS decoding, to personalized emotional speech adapted to individual cultural perception.
Stable Speech Synthesis, Ultra-Fast ASR, and Single-Image 3D Avatars
Today's digest covers stable long-horizon autoregressive TTS with co-designed codecs, a parallelized LLM-based ASR system hitting real-time factors below 0.01, a new full-duplex dialogue benchmark from Sony, and two Gaussian Splatting methods for reconstructing detailed 3D avatars — hands and garments included — from a single image.
Real-Time Avatars, Ultra-Low Bitrate Video, and Score-Native Singing
Today's digest covers a cluster of advances in real-time talking-head avatars — from Gaussian splatting to generative video coding — plus a novel approach to singing voice synthesis that reads directly from musical scores.
From Face to Voice, Text to 4D: Synthesizing Richer Human Presence
Today's digest spans three frontiers of human synthesis: generating dynamic 4D avatars directly from text, handling multi-speaker noisy dialogue with turn-aware speech models, and cloning a voice from nothing but a face photo.
Turn-Taking, Memory-Augmented ASR, Federated SpeechLLMs, and Unified Audio Generation
Today's digest covers human-calibrated dialogue turn-taking, agentic ASR with editable voice memory, federated learning for SpeechLLMs, a latency-busting talking head model, and ByteDance's unified multi-speaker audio generation system.
Real-Time Digital Humans, Expression-Aware TTS, and the Phase Problem in Neural Vocoders
Today's digest covers real-time audio-video digital human generation, video diffusion-based portrait animation, facial-expression-driven conversational TTS, and a deep-dive into why phase reconstruction bottlenecks time-frequency neural vocoders.
Talking Avatars, Dynamic Hair, and On-Device TTS
Today's digest covers fast audio-driven avatar generation, steerable conversational avatars, physics-based hair reconstruction, unified portrait mesh estimation, and two new speech synthesis systems from Alibaba and Apple targeting controllability and mobile efficiency.
Real-Time Avatars, Multilingual Speech LLMs, and Singlish TTS
Today's digest covers real-time multi-modal avatar generation, an open-source multilingual projector bridging Whisper and LLMs across 28 European languages, and fine-tuning zero-shot TTS to authentically capture Singapore English.
One-Shot Avatars, Low-Latency TTS, and Chain-of-Thought for Spoken LLMs
Today's digest covers animatable 3D Gaussian head generation from a single image, a sparse-embedding approach to real-time mobile TTS, and efficient chain-of-modality reasoning that brings CoT gains to spoken language models.
Streaming S2ST, Animatable Avatars, and 3D Face Reconstruction
Today's digest covers data-efficient streaming speech-to-speech translation with a Thinker-Talker architecture, plus a trio of advances in 3D human and face reconstruction — from occlusion-robust Gaussian avatars to UV-space face fusion and a generative anthropometric head model.
Context-Aware Expressive TTS via LLM-Driven Control Layers
Today's digest features Harness TTS from Alibaba Group, a novel approach that wraps existing TTS engines with an LLM planner and prompt-tool registry to achieve context-aware, expressive, and auditable speech synthesis — no model retraining required.
Real-Time Speech Translation and Compact TTS for Low-Resource Languages
Today's digest covers a real-time multi-speaker speech-to-speech translation system with streaming ASR stabilization, and a staged depth-pruning distillation approach for building compact Hindi TTS models under low-data constraints.
SALMONN-2 Pushes General-Purpose Audio Language Models Further
Today's digest spotlights SALMONN-2 from Tsinghua University, an audio language model that leverages self-supervised representations and multi-layer fusion to achieve broad, balanced hearing abilities across speech, audio, music, and paralinguistics.
Authentic Emotion in Conversational Speech Synthesis
Today's digest focuses on emotionally expressive conversational TTS, featuring AuEmoChat from NUS — a system that moves beyond discrete emotion categories to learn continuous emotion tokens from large-scale speech data.
Benchmarking Voice AI and Building Real-Time Conversational Agents
Today's digest covers two fronts in voice and conversational AI: Hume AI's multidimensional real-world benchmark exposing critical gaps in voice systems, and Alibaba's unified video model enabling sub-second multimodal conversational agents.
Controllable TTS, Efficient Dialog Synthesis, and Streaming Speech Translation
Today's digest covers fine-grained style control in TTS, memory-efficient dialog speech synthesis via latent flow matching, cross-domain voice conversion from music models, and data-driven simultaneous speech translation with unmodified LLMs.
Animating Avatars: Nodding, Vocal-Tract Motion, and Non-Humanoid Blendshapes
Today's digest covers three papers pushing the boundaries of avatar animation — from real-time listener nodding and synchronized 3D vocal-tract articulation, to feed-forward blendshape registration for non-humanoid heads.
Synthetic L2 Speech for Automated English Proficiency Assessment
Today's digest focuses on a Cambridge study using TTS and voice cloning to synthesize second-language learner speech, showing that proficiency-matched synthetic data significantly boosts automated speaking assessment models.
Dyadic Talking Heads and AI Interview Coaches
Today's digest covers two papers pushing the boundaries of conversational avatar systems: a new framework for dyadic talking-head generation that modulates social interaction over monologic priors, and an LLM-powered mock interview platform with a lip-synced digital interviewer and multimodal coaching feedback.
Tokenizer-Free Turkish TTS and Video as World Models
Today's digest covers FreyaTTS, a production-ready Turkish speech synthesis system that ditches phonemizers for direct character-level Diffusion Transformers, alongside a Hugging Face trending paper reconceptualizing video as a joint world-state and event stream.
Visual Speech, Multimodal Alignment, and RL-Driven ASR Adaptation
Today's digest covers viseme-guided event-based lip reading, optimal-transport alignment for audio-visual LLM-based ASR, and reinforcement learning as a superior alternative to supervised fine-tuning when only synthetic speech is available.
Full-Duplex Dialogue via RL and Robust Contextual ASR
Today's digest covers two papers from National Taiwan University: one decouples conversational timing from reasoning in full-duplex spoken models using reinforcement learning, and another introduces COALA, a framework for robust contextual biasing in speech-augmented ASR.
Word-Level TTS Control, KV Cache Compression, and Universal ASR Alignment
Today's digest covers fine-grained prosodic control in LLM-based TTS, distributional training objectives, a Taiwanese code-switching system, KV cache compression for speech LLMs, orthogonal audio-language connectors, and a gradient-based alignment method that works across every ASR family.
Unified Audio LLMs, Full-Duplex Dialogue Data, and Test-Time Scaling for ASR
Today's digest covers unified audio-text LLMs, benchmarks for streaming conversational naturalness, large-scale full-duplex dialogue data construction, controllable speaker synthesis from text profiles, and test-time scaling for ASR — a broad sweep across the speech AI stack.
Streaming Talking Heads & the Road to Photorealistic Digital Humans
Today's digest covers real-time talking-head generation at higher resolution without sacrificing latency, plus a sweeping survey of controllable 3D avatar creation from body priors to photorealistic animation.
Smarter Speech Tokens: Diffusion TTS, Accent Normalization, and Voice Anonymization
Today's digest explores three papers reshaping how discrete speech tokens are generated, converted, and protected — from a diffusion-based TTS that breaks the autoregressive bottleneck, to RL-guided accent normalization and hierarchical VAE voice anonymization.
Real-Time Avatars, Emotional Voice Control, and Proactive Dialogue
Today's digest covers speech-driven real-time avatar generation at 42 FPS, zero-shot emotional voice conversion via natural-language instructions, and a proactive thinking framework that cuts LLM response latency without extra training.
Dyadic Avatars, Efficient SpeechLLMs, and Fine-grained TTS Pronunciation
Today's digest covers end-to-end dyadic conversation generation, real-time 3D facial expression synthesis, parameter-space composition for instruction-following speech LLMs, high-throughput audio inference pipelines, and per-word pronunciation control in zero-shot TTS.
Emotion Control Across Avatars, TTS, and Speech-LLMs
Today's digest covers three papers tackling emotional control and representation: real-time emotional talking heads via 3D Gaussian Splatting, geometric analysis of emotion steering in TTS models, and a speech-text interleaving strategy that keeps LLM priors intact during ASR training.
Flexible Speech Models, Compact Avatars, and Brain-Driven Faces
Today's digest covers dynamic frame-rate spoken LLMs, compact 3D human Gaussian splatting, domain-aware TTS evaluation, and the first EEG-conditioned facial action-unit editing framework.
Real-Time Avatars, Speech Synthesis, and Conversational AI
Today's digest covers real-time conversational avatars with joint speech-facial motion, fast audio-driven portrait animation, feed-forward 4D head reconstruction, culturally-aware gesture generation, speech-to-speech LLMs, and unified voice attribute editing — eight papers pushing the frontier of expressive, interactive digital humans.
This Week in Conversational AI Research: June 29–July 05, 2026
A dense week spanning real-time multimodal agents, TTS alignment via reinforcement learning, and a wave of single-image avatar reconstruction systems — plus a sobering audit of how well production voice AI actually uses the emotions it perceives.
Real-Time Avatars and Noise-Robust Speech Recognition
Today's digest covers ray-traced shadows for 3DGS avatars, high-resolution feedforward human reconstruction from sparse video, and a variational bottleneck approach to noise-robust audio-visual speech recognition.
Bridging Synthetic and Real Speech for LLM-Based ASR
Today's digest examines how LLM-based ASR systems perceive synthetic speech and how to close the gap with real data — achieving real-data parity with just 25% genuine recordings through smart augmentation and pooling strategies.
Expressive Avatars, Emotional Voice, and Smarter ASR
Today's digest covers speech-driven 3D facial animation, relightable avatar reconstruction from a single image, expressive human motion disentanglement, hierarchical reward optimization for emotional TTS, and edit-flow refinement for non-autoregressive ASR — a broad sweep of lifelike digital human and voice research.
Faces in 3D, Guided by Geometry and Preferences
Today’s digest explores two ways to improve AI-generated faces in 3D: geometry-constrained multi-view diffusion for consistent head generation, and human-preference fine-tuning that sculpts face GAN geometry without explicit surface supervision.
Voice AI Faces the Emotion Test
Today’s digest spans faster-adapting text-to-speech, low-resource voice cloning, and a harder question for speech systems: whether they can preserve emotion and expression. It also spotlights a new benchmark and evidence that realtime voice agents still miss vocal cues even when they hear them.
This Month in Conversational AI Research: June 2026
June 2026 was defined by the industrialization of real-time avatar pipelines and full-duplex voice agents, as the field moved decisively from proof-of-concept generation toward production-grade streaming systems — while a parallel wave of interpretability work exposed how audio LLMs actually process acoustic signals.
Real-Time Voices, Agents, and Avatars
Today’s digest spans end-to-end interactive models, better speech and TTS synthesis, and instant 3D avatar generation. The common thread: more natural, controllable, low-latency conversational AI across voice and visual embodiment.
Controlling speech, audio, and avatars
Today’s digest spans new ways to shape how machines speak, listen, and move: from universal audio generation and controlled TTS to stronger ASR alignment and adaptive speech encoders. It also includes a fresh step toward smoother digital-human motion from sparse sensor input.
Voice, Avatars, and Spoken Reasoning
Today’s digest spans real-time talking avatars, relightable digital humans, universal speech synthesis, and a new look at how speech-language models reason internally. Together they point to more natural, controllable, and interactive conversational AI systems.
This Week in Conversational AI Research: June 22–28, 2026
This week pushed conversational AI toward native-streaming systems: full-duplex audio-visual agents, streaming TTS, and real-time avatars all moved closer to a single interactive loop. At the same time, speech generation papers focused on controllability, prosody, voice cloning robustness, and representation alignment for deployable voice agents.
From speech to face, code-switching, and pronunciation
Today's digest spans full-duplex talking avatars, stronger multilingual ASR for code-switching, and LLM-based grapheme-to-phoneme benchmarking for Japanese TTS. Together they point to voice systems that sound more natural, adapt more flexibly, and animate more convincingly.
Real-Time Speech for Conversational AI
Today’s digest spotlights low-latency speech generation, prosody-aware voice conversion, and robust spoken-language agents. The common thread: speech systems that sound better, respond faster, and work more reliably in live conversation.
Speech synthesis, voice conversion, and interactive avatars push toward real-time realism
Today’s digest spans transcript-free text-to-speech, diffusion-guided speech generation, streaming voice conversion, and speech codecs built for cleaner identity control. It also features physically interactive 3D avatars that deform realistically under contact and motion.
Speech systems move toward richer dialogue, controllable voices, and stronger robustness
Today’s digest spans full-duplex spoken dialogue, multi-speaker scene generation, controllable TTS, and improved ASR representations. The papers focus on making speech systems more natural, more editable, and more reliable in noisy or complex settings.
Reliable voice AI for multi-speaker dialogue and long-form synthesis
Today’s digest spans speech LLMs, voice agents, and TTS: from diarization-aware multi-speaker grounding and persona-driven speech role-play to more robust, faster, and longer-form speech generation. The common thread is making voice systems more controllable, consistent, and reliable.
Speech systems tighten context, accents, and avatars
Today’s digest spans context-aware spoken dialogue, multilingual and accent-aware speech models, faster ASR decoding, and dynamic 3D head editing. Together, these papers push conversational systems toward more faithful, natural, and controllable interaction.
Talkers Move and Speak Better
Today’s digest spans expressive talking heads, stylized co-speech gesture generation, and code-mixing speech tools for better downstream ASR. The common thread: tighter control over how synthetic voices, faces, and motions carry meaning.
Prosody and Stress Take Center Stage
Today’s digest focuses on speech fidelity at the syllable level, from dynamic prosody prediction and duration-based watermarking in LLM TTS to stress-preserving speech-to-speech translation. Together, these papers push synthesized and translated speech closer to natural, speaker-faithful delivery.
Speech Agents Learn to Listen
Today’s digest spans full-duplex spoken dialogue, emotional voice synthesis, and ASR that catches hesitation and disfluency. Together, these papers push voice agents toward more natural, responsive, and expressive interaction.
Streaming Speech, Lip Sync, Turn-Taking
Today’s digest spans talking avatars, 4D human reconstruction, and speech agents that anticipate endpoints and manage multi-party turns. It also includes expressive TTS with finer emotion control for more natural spoken output.
Prosody Powers Spoken AI
Today’s digest spotlights two directions in speech-native AI: an empathetic multi-agent dialogue system that uses prosody for better emotional alignment, and a study of speech-token design that improves how frozen LLMs reason over spoken input.
Talking Agents Get More Human
Today’s digest spans real-time lip sync and facial animation, stereoscopic digital humans, and speech models that handle turn-taking, interruptions, and paralinguistic cues more naturally. It also includes progress in multilingual TTS and higher-fidelity voice generation.
Speech Models Get Leaner
Today’s digest spotlights faster, simpler speech systems: streaming TTS, end-to-end discrete-token training, interpretable voice control, and new ways to plug speech directly into LLMs. The common thread is practical speech intelligence with lower latency, cleaner architectures, and more controllable outputs.
Voice Gets More Real-Time
Today’s digest spans holistic video dubbing, streaming speech LLMs, expressive and waveform-native TTS, and low-latency voice conversion. The common thread: better control, faster inference, and more natural-sounding speech across generation and recognition.
Faces and Hair Get Rigged
Today’s digest spotlights advances in digital humans: fully automatic face rigging with inner-mouth geometry and blendshapes, plus strand-level facial hair capture for editable beards, brows, and lashes. Together they push avatars closer to animation-ready realism.
Smarter Voices, Fewer Errors
Today’s digest spotlights more expressive speech synthesis, emotion conversion, and better ASR reliability for voice agents. From image-based TTS and continuous latent speech models to hallucination steering in Whisper, the focus is on making spoken AI sound better and fail less.
Avatars Get More Social
Today’s digest spans social digital humans, efficient 3D avatar creation, noise-robust spoken dialogue, and more controllable voice synthesis. From theory-of-mind avatars to stronger zero-shot TTS, the common thread is making conversational systems feel more natural and expressive.
Voice AI Goes Real Time
Today’s digest spans streaming speech agents, emotional text-to-speech control, robust audio-visual recognition, and a new reference-free way to evaluate ASR. Together, these papers push conversational systems toward more responsive, expressive, and reliable voice interaction.
Talking Avatars Get Real
Today’s digest spans full-duplex speech-motion avatars, portrait animation, outfit-personalized 3D humans, speech-LLM reasoning fixes, and raw-waveform zero-shot TTS. A strong day for more natural voices, more expressive bodies, and better spoken reasoning.
Avatars Speak and See Clearly
Today’s digest spans photorealistic 3D human avatars, identity-preserving video generation, and a unified audio-language model for speech, sounds, and music. Together they point to richer multimodal agents that can see, hear, and render people more naturally.
Avatars Voices And Edge Speech
Today’s digest spans real-time talking portraits, single-photo 3D face avatars, unified speech-singing synthesis, interpretable emotion control in TTS, and efficient neuromorphic speech recognition. It’s a strong mix of expressive generation and practical speech systems for interactive AI.
Speech AI Across Empathy, Scale, and Timing
Today’s digest spans emotionally adaptive voice assistants, a 100+ language speech benchmark, and a new way to evaluate talking-head generation with better temporal alignment. Together, these papers push speech systems toward richer conversation, broader coverage, and fairer assessment.
Speech models that talk, adapt, and translate in real time
Today’s digest spans expressive voice synthesis, low-latency speech systems, and talking avatars. From zero-shot long-form TTS to latent reasoning ASR and streaming translation, the focus is on models that sound more natural and respond faster.
Talking avatars and voice models push toward more natural, controllable generation
Today’s digest spans fine-tuning-free talking-face synthesis, semantically grounded gesture generation, unified digital human models, and faster streaming TTS. Across speech and avatar generation, the theme is better alignment between meaning, motion, and voice with less latency and more control.
Real-Time Avatars, Smarter Speech, and More Expressive Faces
Today's digest spans talking heads, full-body digital humans, and interactive speech systems. Highlights include text-guided head stylization, real-time avatar animation, relightable telepresence capture, and agentic ASR that revises transcripts in dialogue.
Grounding Audio LLMs, Smarter TTS, and Cinematic Multi-Talker Benchmarks
Today's digest covers seven papers tackling the hardest grounding problems in conversational AI: audio LLMs that ignore acoustics, TTS systems that punch above their data weight, event-camera-driven speech synthesis, and a new cinematic benchmark for multi-talker video generation.
Lifelike Avatars, Expressive Voices, and Smarter Gesture Generation
Today's digest covers eight papers pushing the boundaries of audio-driven avatar animation, 3D head reconstruction, co-speech gesture synthesis, and expressive TTS — from production-ready streaming avatars to privacy-aware speaker unlearning.
Dyadic Avatars, Speech LLMs, and Fine-grained TTS Pronunciation
Today's digest covers five papers spanning full dyadic audio-visual conversation generation, text-driven 3D facial expression control, parameter-space composition for instruction-following speech LLMs, high-throughput audio LLM serving, and per-word pronunciation control in zero-shot TTS.
Talking Faces, Smarter TTS, and Speech LLMs at Scale
Today's digest covers dyadic audio-visual conversation generation, text-driven 3D facial expressions, instruction-following speech LLMs, efficient vLLM-based audio inference, fine-grained pronunciation control in TTS, and a wave of industrial TTS technical reports trending on Hugging Face.
A Brief History of Audio-Driven Talking-Avatar Research (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.
How Avatars Learned to Talk: Six Years of Audio-Driven Human Animation (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.