A Brief History of Audio-Driven Talking-Avatar Research (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.
Conversational AI Research Digest
Daily summaries for ML researchers and engineers about conversational AI.
Featured
Posts
From Deepfake Defense to End-to-End Waveforms: Speech & Avatar Research, Nov 10
Today's digest spans talking-avatar control, deepfake voice protection, compact multilingual TTS, end-to-end waveform synthesis, flow-matching guidance fixes, beat-adaptive rap synthesis, speech reward modeling, and cascaded spoken-dialogue optimization.
Real-Time Speech: Lip Sync, Accent Conversion, and Low-Latency Dialogue
Today's digest covers real-time advances across the speech pipeline: emotion-aware universal lip sync, timing-precise accent conversion for dubbing, latency-masking turn-taking for voice agents, and a stable-prefix streaming ASR model.
Expressive Dubbing, Streaming ASR, and Smarter Speech Translation
Today's digest covers expressive speech-to-speech translation and video dubbing, real-time streaming ASR and speech LLM benchmarks, and pronunciation-controlled TTS for logographic languages.
Dubbing, Digital Humans, and Spoken LLMs: Eight Papers Pushing Audio-Visual AI Forward
Today's digest covers a wave of advances in video dubbing, talking-head generation, speech-to-speech translation, oral animation evaluation, generative spoken language models, real-time voice benchmarking, and Arabic voice anonymization — a broad sweep across the audio-visual AI stack.
Voice Agents Level Up: Unified Speech-Reasoning, Enrollment-Free Speaker Lock, and a New ASR Benchmark
Today's digest covers three advances in voice agent infrastructure: a 27B open model unifying speech, reasoning, and tool use; enrollment-free target-speaker extraction for live calls; and a new benchmark reframing ASR as grounded generative reformulation.
TTS Control, Smarter Voice Agents, and Identity-Preserving Video
Today's digest covers emphasis control and prompt selection for TTS, interpretability of voice-cloning models, turn detection and end-of-turn training for voice agents, real-time speaker anonymization, identity-preserving face video super-resolution, and efficient spoken dialogue via compressed speech tokens.
Streaming S2ST and a Speech LLM for Basque
Today's digest covers two advances in speech-to-speech systems: a semantically supervised streaming translation model from Alibaba DAMO Academy, and the first open Speech Language Model for the low-resource Basque language.
Full-Duplex Dialogue, Expressive TTS, and Anime Blendshapes
Today's digest covers turn-taking mechanics in full-duplex speech models, expressive and efficient TTS advances, early tool-calling in spoken dialogue, multilingual ASR expansion, and neural blendshape generation for anime characters.
3D Avatars, Full-Duplex Failures, and the Long-Horizon Speech Challenge
Today's digest spans audio-driven 3D avatar generation, cross-lingual dubbing, simultaneous translation, co-speech gesture synthesis, unified streaming speech LLMs, full-duplex conversational failures, and a new multi-speaker TTS benchmark — a broad sweep of where spoken-language AI is pushing hardest right now.
Scene-Aware Voice Cloning, Face-Driven Emotion TTS, and Smarter Real-Time Voice Agents
Today's digest covers multimodal voice synthesis — from scene-adaptive cloning and face-conditioned emotion TTS to a massive multilingual speech LLM and a speculative "heartbeat" reasoning trick for low-latency voice agents.
Voice Control, Timbre Localization, and Adaptive Codecs in Speech Synthesis
Today's digest digs deep into speaker controllability across TTS, voice conversion, and cross-lingual cloning — plus new findings on where timbre actually lives in speech signals and a variable-rate codec that budgets tokens intelligently.
Synced Edits, Steered Prosody, and Smarter ASR
Today's digest covers unified audio-visual editing with lip-sync, RL-driven TTS improvements, intonation steering, a Sanskrit recitation corpus, full-duplex dialogue recovery, Pareto-optimal speech LLM training, and two fresh approaches to hotword-aware contextual ASR.
Lip Sync, Talking Avatars, Single-Step TTS, and Smarter Voice Agents
Today's digest spans real-time talking avatars, single-step TTS breakthroughs, deep dives into Qwen3-TTS failure modes, and novel architectures for grounding audio in LLMs — including a clinical audio model that reasons beyond transcripts.
Making Spoken Dialogue More Human: Full-Duplex Avatars, Faster Tokens, and Smarter Turn-Taking
Today's digest covers three advances in spoken dialogue systems: a full-duplex multimodal avatar that jointly generates speech, gestures, and expressions; a post-hoc token compression method for faster speech LLM inference; and a unified turn-taking model that handles end-of-turn, backchannels, and speech validity in a single forward pass.
Talking Avatars, Smarter TTS, and the Challenge of Multi-Party Dialogue
Today's digest covers open-source audio-driven avatar generation, a wave of TTS advances tackling long-form synthesis, prosody cloning, noise adaptation, and masked diffusion refinement, plus new benchmarks for visual speech and multi-party floor control.
Full-Duplex Dialogue, Smarter TTS, and Cross-Species Portrait Animation
Today's digest covers a wave of full-duplex spoken language model advances from NVIDIA, MIT-IBM, and UIUC, alongside fresh TTS work on realism and drama synthesis, and portrait animation that crosses species boundaries.
Expressive TTS, Smarter Speech Agents, and Faster ASR
Today's digest spans emotional TTS controllability, on-device quantization, turn-taking benchmarks for speech LLMs, co-speech portrait video generation, long-form motion synthesis, and lossless ASR acceleration — eight papers pushing the boundaries of conversational AI.
Real-Time Avatars, Steerable Duplex Dialogue, and Expressive TTS
Today's digest covers real-time Gaussian head animation, single-pass talking-head GANs, fine-grained TTS voice control, and a pair of papers pushing full-duplex spoken dialogue toward controllability and speed — plus new evaluation frameworks for co-speech gestures and 3D head avatars.
Emotion Steering, Gaussian Avatars, and Smarter TTS — Oct 8
Today's digest spans emotion control via activation steering in TTS and full-duplex speech models, advances in 3D Gaussian head avatar quality and efficiency, a new unified lip-sync metric, and cross-modal audio-visual editing from Meta AI.
Full-Duplex Dialogue, Smarter TTS, and Faster Voice Agents
Today's digest covers eight papers spanning full-duplex spoken dialogue systems, emotion-aware and cross-lingual speech LLMs, intelligibility-boosting TTS, and speculative execution for on-device voice agents.
Real-Time Avatars, Drift-Free TTS, and Empathetic Spoken Dialogue
Today's digest covers streaming talking-head reenactment, person-specific speaking style imitation, synchronized audio-video generation, flow-matching TTS fixes, expressive voice control, empathetic spoken dialogue reasoning, and LLM-based streaming ASR.
Language-Guided Avatars, Few-Step TTS, and Real-Time Speech Translation
Today's digest covers language-driven editing of 3D Gaussian head avatars, a distillation-free few-step TTS approach, spatially grounded gesture generation for VR dialogue, and end-to-end simultaneous speech translation with confidence-based commit timing.
Real-Time Avatars, Latent Reasoning, and Physically Grounded Speech
Today's digest spans real-time 3D avatar animation, lip-sync evaluation, latent-space reasoning for speech LLMs, voice agent grounding benchmarks, encoder-free speech understanding, articulatory TTS, and training-free voice conversion.
Controllable Avatars, Faster Voice Cloning, and Full-Duplex Benchmarking
Today's digest covers controllable talking avatars with full-body pose guidance, transcript-free voice cloning 10× faster than autoregressive baselines, codebook-aware expressive TTS, one-step voice conversion, unified multilingual dubbing, and a new benchmark exposing timing failures in full-duplex speech systems.
Identity-Aware Avatars, Faster TTS, and Smarter Spoken Dialogue
Today's digest covers identity-preserving lip-sync animation, semantic-aware gesture generation, full-duplex multi-party speech models, emotionally grounded dialogue, and a wave of TTS efficiency advances from reward alignment to on-device speculative decoding.
Adaptive Avatars, Full-Duplex Agents, and Smarter Voice Synthesis
Today's digest spans adaptive 3D talking heads, a wave of full-duplex voice agent architectures, and advances in controllable TTS and speech translation — with highlights from both arXiv and the Hugging Face Daily tab.
Streaming Avatars, Full-Duplex Dialogue, and Smarter Speech Synthesis
Today's digest spans expressive streaming avatars, real-time speech-to-speech translation, full-duplex dialogue memory, and content representations for voice synthesis — eight papers pushing the state of the art in conversational AI from lab to production.
Real-Time Lip Sync, Adaptive Full-Duplex Dialogue, and Controllable TTS
Today's digest spans real-time lip sync in unconstrained scenes, three complementary takes on full-duplex spoken dialogue (adaptive turn-taking, synthetic training data, and backchannel control), physiology-aware head avatars, sign language avatars, and two new flow-based TTS frameworks.
RL for Expressive TTS, Speaker Unlearning, and Smarter Dialogue Evaluation
Today's digest covers reinforcement learning for prosodic emphasis in TTS, a speaker unlearning framework for voice cloning privacy, one-step voice conversion via flow matching, adversarial robustness for full-duplex dialogue agents, and a rethinking of turn-taking evaluation grounded in speaker intent.
Talking Avatars, Agentic Voice, and Smarter TTS Control
Today's digest covers unified human-centric video generation, lip-sync dubbing, full-duplex agentic voice assistants from Alibaba, spoken LLMs that reason aloud, and a trio of TTS advances spanning style adaptation, learnable CFG embeddings, and low-resource adaptation.
Real-Time Avatars, Reasoning TTS, and Voice Agent Benchmarks
Today's digest spans the full conversational AI stack: open real-time avatar systems, singing head synthesis datasets, relightable 3D faces, chain-of-thought and RL-refined TTS, speaker embedding geometry, and new benchmarks stress-testing long-context and multi-speaker voice agents.
Full-Duplex Voice Agents, Proactive AudioLLMs, and Streaming Gestures
Today's digest covers a wave of real-time conversational AI research: full-duplex speech agents with tool-calling and turn-taking, native audio-visual dialogue, proactive AudioLLM assistance, on-device audio models, and streaming co-speech gesture generation.
Agentic Voice, Faster ASR, and Multilingual Talking Faces
Today's digest spans tool-calling full-duplex speech agents from Google, streaming ASR advances from IBM and CAS, multilingual G2P and prosody evaluation from Alibaba and Tsinghua, and a new Korean-English 3D talking face benchmark from Seoul National University.
Reinforcement Learning Meets Speech: Smarter Avatars, Reasoning Models, and Efficient SpeechLLMs
Today's digest covers reinforcement learning applied to talking avatars and spoken math reasoning, alongside two efficiency-focused advances for adapting language models to speech — token-level alignment and encoder-targeted fine-tuning.
TTS System Design, Long-Form Reliability, and Smarter Dialogue Benchmarks
Today's digest covers principled TTS architecture retrospectives, inference-time fixes for long-form speech, self-distilled pronunciation control, layer-wise codec compression, and two new benchmarks probing the limits of spoken dialogue agents in role-playing and turn-taking.
Full-Duplex Voice Agents, Smarter TTS, and Interactive Digital Avatars
Today's digest covers a dense wave of advances in full-duplex speech LLMs, real-time voice agents, robust long-form TTS, cross-lingual voice cloning, and interactive digital avatar simulation — plus Tsinghua's Vidu S2 from the HuggingFace Daily tab.
Full-Duplex Dialogue, Hallucination-Free TTS, and Mesh-Bound Avatars
Today's digest spans the full voice-AI stack: unified audio generation with LLM backbones, smarter TTS alignment via DPO, streaming mel decoders, lightweight vocoders, speech-to-speech translation, full-duplex dialogue datasets, multiparty voice-agent benchmarking, and template-free head avatars.
Duet Animation, Self-Listening Dialogue, and Acoustic Bottlenecks in LLMs
Today's digest covers 3D duet singing animation with explicit interaction dynamics, a self-listening mechanism for full-duplex speech models, and a diagnostic study on why audio-conditioned LLMs fail to act on the acoustic information they encode.
Full-Duplex Dialogue, Brain-to-Speech, and Smarter Voice LLM Serving
Today's digest spans Meta AI's 3B-parameter alignment-free dubbing and full-duplex dialogue model, real-time brain-to-speech synthesis, low-resource Thai TTS, scalable voice context orchestration for LLMs, and object-grounded co-speech gesture generation.
Full-Duplex Dialogue, Smarter TTS, and Open Speech Foundations
Today's digest covers full-duplex spoken dialogue with decoupled turn-taking, streaming speaker-attributed ASR, direction-following TTS, grounded speech planning evaluation, and two major open-source releases from Tencent Hunyuan — an omni interaction agent and a unified speech generation model.
Smarter Speech Pipelines: TTS-Aligned LLMs, Audio-Grounded DST, and Multi-Talker ASR
Today's digest covers three papers tightening the full spoken-language pipeline: aligning LLMs to generate TTS-friendly text, grounding dialogue state tracking in raw audio, and softly injecting speaker posteriors into Whisper for multi-talker recognition.
Avatars, Gestures, and the Inner Life of SpeechLLMs
Today's digest spans adversarial defense for talking-face generation, environment-aware co-speech gesture synthesis, RL alignment in TTS, and mechanistic dissections of how audio-language models handle paralinguistic information — plus a trending HF paper on joint speech-and-motion dialogue.
Fine-Grained Emotional Control in Instruction-Following TTS
Today's digest spotlights a new approach to multi-emotion modeling in TTS, enabling both sequential emotion trajectories and simultaneous emotion blending via reinforcement learning with Group Relative Policy Optimization.
Closed-Loop TTS: AudioLLM Diagnosis Meets Fine-Grained Prosody Correction
Today's digest spotlights a novel closed-loop TTS pipeline from NUS that uses an AudioLLM to diagnose prosodic defects and a fine-grained refiner to correct them — moving beyond the limitations of single-pass generation and standard quality metrics.
Beyond Transcripts: Speech Grounding, Voice Privacy, and Cross-Lingual Prosody
Today's digest probes what gets lost — or hidden — when speech is reduced to text: benchmarks exposing transcript bias in speech LLMs, a voice cloning model repurposed for anonymization, and the first fine-grained cross-lingual prosody analysis for expressive translation.
Voice Agents, Gesture Control, and Smarter ASR Augmentation
Today's digest covers reinforcement learning for audio-native voice agents, two new frontiers in co-speech gesture generation for streaming and humanoid robots, adaptive contrastive decoding for noisy audio-visual ASR, and a phoneme-guided TTS pipeline for multilingual ASR augmentation.
Unified Audio LLMs and Expressive TTS with Non-Verbal Vocalizations
Today's digest covers two advances in speech synthesis: FireRedAudio's decoupled continuous representations for unified audio understanding and generation in a 9B LLM, and a preference optimization framework for synthesizing non-verbal vocalizations like laughter and sighs in expressive TTS.
Compressing Avatars, Transitioning Emotions, and Peeking Inside Audio LLMs
Today's digest covers semantic-aware compression for 3D Gaussian head avatars, intra-utterance emotion transitions in TTS, and a new probe into the hidden multilingual reasoning of audio language models.
Training-Free Watermarking for Flow Matching TTS
Today's digest spotlights AudioNoisePrints, a model-free watermarking technique that embeds provenance signals directly into flow matching TTS synthesis by exploiting spatial correlations in the initial noise — no training, no quality loss.
Real-Time Digital Humans: Instant Avatars, Streaming Gestures, and Memory-Augmented Voice
Today's digest focuses on real-time digital human pipelines — from sub-second animatable 3D avatar reconstruction and streaming co-speech gesture generation to a dual-brain memory architecture for emotionally aware speech language models.
Real-Time Lip Sync, Streaming TTS, and Prosody-Aware Assistants
Today's digest covers real-time lip synchronization at 166 FPS, causal streaming TTS with 15.8 ms latency, backchanneling in AI clones, and a new benchmark probing whether LLMs act on prosody.
Streaming Avatars, Unified TTS, and the State of Multimodal Dialogue
Today's digest covers a fix for dynamic collapse in real-time avatar synthesis, a unified speech generation and editing system stabilized by semantic supervision, and a sweeping survey of multi-turn conversational AI from text to multimodal interaction.
Singing Avatars, Rolling-Shutter Gaussians, and Empathetic Robots
Today's digest spans audio-driven avatar generation, 3D Gaussian reconstruction from real-world video, speaker-normalized speech tokenization, and emotion-aware empathetic dialogue agents — a broad sweep across embodied AI and speech research.
Dubbing, Cloning, and Dialogue: Speech Synthesis Goes End-to-End
Today's digest covers end-to-end video dubbing, zero-shot voice cloning, expressive TTS via iterative self-learning, realistic dialogue speech synthesis, and animation-free 3D lip sync for arbitrary characters.
Smarter 3D Talking Heads: Continuous Emotion Control and Human-Preference Alignment
Today's digest focuses on audio-driven 3D facial animation, covering continuous valence-arousal emotion control for expressive talking heads and a new human preference reward model that moves evaluation beyond ground-truth metrics.
Streaming Speech Synthesis: Real-Time Dialogue and Immersive Soundscapes
Today's digest covers two advances in speech synthesis: Meta AI's low-latency, interruption-aware TTS for interactive agents, and Zhejiang University's unified vocalized audio model that weaves speech into environmental soundscapes via autoregressive flow matching.
Flow Matching, Diffusion, and Real-Time Avatars: August 19 Roundup
Today's digest covers four papers pushing the frontiers of real-time avatar generation, zero-shot TTS, controllable voice design, and unified audio scene generation — with Flow Matching emerging as a common thread across labs at ByteDance, Alibaba, Google, and Xiaomi.
Voice Agents, Parallel TTS, and Gesture Benchmarks
Today's digest covers parallel diffusion-based TTS, transcript-free cross-lingual voice cloning, a unified speech+singing model, a new voice agent benchmark, frame-synchronous turn-taking, 3D speech animation, and the GENEA 2026 gesture generation challenge.
Unified Audio Scene Generation and Temporally-Aware Animatable Avatars
Today's digest covers two papers pushing the boundaries of synthesis: SonicWeave unifies speech, music, and sound effects into coherent audio scenes via chunk-routed mixture-of-experts, while DSAR models cloth dynamics autoregressively for photorealistic animatable avatars that generalize to novel motions.
Emotional Intelligence for Spoken LLMs and Faster Zero-Shot TTS
Today's digest covers two papers pushing the frontier of spoken AI: a theory-grounded framework for benchmarking and aligning emotional intelligence in spoken language models, and an efficient autoregressive TTS system that cuts voice-cloning latency by 23%.
Dialect Speech Dialogue and Fine-Grained Prosodic Control in TTS
Today's digest covers end-to-end spoken dialogue for low-resource Chinese dialects and a coarse-to-fine approach to expressive TTS that gives word- and phoneme-level control over pitch, loudness, and duration.
Smarter Voice Dialogue, Cleaner Speech Synthesis, and Streaming Avatars
Today's digest covers a middleware fix for full-duplex LLM voice agents, a hybrid semantic-continuous TTS architecture, and two Qwen streaming avatar systems for real-time identity swapping and human animation.
Real-Time Talking Avatars, Unified Audio-Visual Models & ASR Evaluation
Today's digest is dominated by a wave of audio-visual generation systems pushing toward real-time, long-form, and multi-task synthesis — plus a sharp empirical look at rare-word ASR and a counterfactual audit of audio language model judges.
3D Avatars, Phoneme-Precise Lip Sync, and Spoken Function Calling
Today's digest covers phoneme-driven talking heads, empathetic 3D avatar chatbots, multi-view upper-body animation, discrete diffusion for speech editing, spoken function calling for audio LLMs, and an omni-modal dialogue model that generates coordinated text, speech, and video.
Multilingual ASR Distillation and Melody-Preserving Singing Voice Editing
Today's digest covers two focused advances: a multi-teacher RL distillation strategy that resolves cross-lingual conflicts in LLM-based ASR, and a continuous-latent autoregressive model for editing lyrics in singing voice synthesis without disrupting melody.
Emotion-Aware Avatars, Smarter TTS, and Tool-Calling Speech Agents
Today's digest spans real-time emotion-controllable portrait animation, reinforcement-learned TTS, environment-aware voice synthesis, and spoken LLMs that reason with external tools — a broad sweep across talking avatars, voice generation, and speech intelligence.
Empathetic Voice Agents, Precise Speech Editing, and Real-Time Gestures
Today's digest covers full-duplex empathetic dialogue from ByteDance, XML-driven speech editing with fine-grained prosody control, streaming co-speech gesture generation with drift-free anchoring, and a novel Latent Softmax output layer for multilingual ASR across tonal and non-tonal languages.
Emotion at the Center: Expressive TTS and Talking Faces Get Personal
Today's digest focuses on emotion in speech and video synthesis — from geometry-guided talking face generation and hallucination-free TTS decoding, to personalized emotional speech adapted to individual cultural perception.
Stable Speech Synthesis, Ultra-Fast ASR, and Single-Image 3D Avatars
Today's digest covers stable long-horizon autoregressive TTS with co-designed codecs, a parallelized LLM-based ASR system hitting real-time factors below 0.01, a new full-duplex dialogue benchmark from Sony, and two Gaussian Splatting methods for reconstructing detailed 3D avatars — hands and garments included — from a single image.
Real-Time Avatars, Ultra-Low Bitrate Video, and Score-Native Singing
Today's digest covers a cluster of advances in real-time talking-head avatars — from Gaussian splatting to generative video coding — plus a novel approach to singing voice synthesis that reads directly from musical scores.
From Face to Voice, Text to 4D: Synthesizing Richer Human Presence
Today's digest spans three frontiers of human synthesis: generating dynamic 4D avatars directly from text, handling multi-speaker noisy dialogue with turn-aware speech models, and cloning a voice from nothing but a face photo.
Turn-Taking, Memory-Augmented ASR, Federated SpeechLLMs, and Unified Audio Generation
Today's digest covers human-calibrated dialogue turn-taking, agentic ASR with editable voice memory, federated learning for SpeechLLMs, a latency-busting talking head model, and ByteDance's unified multi-speaker audio generation system.
Real-Time Digital Humans, Expression-Aware TTS, and the Phase Problem in Neural Vocoders
Today's digest covers real-time audio-video digital human generation, video diffusion-based portrait animation, facial-expression-driven conversational TTS, and a deep-dive into why phase reconstruction bottlenecks time-frequency neural vocoders.
Talking Avatars, Dynamic Hair, and On-Device TTS
Today's digest covers fast audio-driven avatar generation, steerable conversational avatars, physics-based hair reconstruction, unified portrait mesh estimation, and two new speech synthesis systems from Alibaba and Apple targeting controllability and mobile efficiency.
Real-Time Avatars, Multilingual Speech LLMs, and Singlish TTS
Today's digest covers real-time multi-modal avatar generation, an open-source multilingual projector bridging Whisper and LLMs across 28 European languages, and fine-tuning zero-shot TTS to authentically capture Singapore English.
One-Shot Avatars, Low-Latency TTS, and Chain-of-Thought for Spoken LLMs
Today's digest covers animatable 3D Gaussian head generation from a single image, a sparse-embedding approach to real-time mobile TTS, and efficient chain-of-modality reasoning that brings CoT gains to spoken language models.
Streaming S2ST, Animatable Avatars, and 3D Face Reconstruction
Today's digest covers data-efficient streaming speech-to-speech translation with a Thinker-Talker architecture, plus a trio of advances in 3D human and face reconstruction — from occlusion-robust Gaussian avatars to UV-space face fusion and a generative anthropometric head model.
Context-Aware Expressive TTS via LLM-Driven Control Layers
Today's digest features Harness TTS from Alibaba Group, a novel approach that wraps existing TTS engines with an LLM planner and prompt-tool registry to achieve context-aware, expressive, and auditable speech synthesis — no model retraining required.
Real-Time Speech Translation and Compact TTS for Low-Resource Languages
Today's digest covers a real-time multi-speaker speech-to-speech translation system with streaming ASR stabilization, and a staged depth-pruning distillation approach for building compact Hindi TTS models under low-data constraints.
SALMONN-2 Pushes General-Purpose Audio Language Models Further
Today's digest spotlights SALMONN-2 from Tsinghua University, an audio language model that leverages self-supervised representations and multi-layer fusion to achieve broad, balanced hearing abilities across speech, audio, music, and paralinguistics.
Authentic Emotion in Conversational Speech Synthesis
Today's digest focuses on emotionally expressive conversational TTS, featuring AuEmoChat from NUS — a system that moves beyond discrete emotion categories to learn continuous emotion tokens from large-scale speech data.
Benchmarking Voice AI and Building Real-Time Conversational Agents
Today's digest covers two fronts in voice and conversational AI: Hume AI's multidimensional real-world benchmark exposing critical gaps in voice systems, and Alibaba's unified video model enabling sub-second multimodal conversational agents.
Controllable TTS, Efficient Dialog Synthesis, and Streaming Speech Translation
Today's digest covers fine-grained style control in TTS, memory-efficient dialog speech synthesis via latent flow matching, cross-domain voice conversion from music models, and data-driven simultaneous speech translation with unmodified LLMs.
Animating Avatars: Nodding, Vocal-Tract Motion, and Non-Humanoid Blendshapes
Today's digest covers three papers pushing the boundaries of avatar animation — from real-time listener nodding and synchronized 3D vocal-tract articulation, to feed-forward blendshape registration for non-humanoid heads.
Synthetic L2 Speech for Automated English Proficiency Assessment
Today's digest focuses on a Cambridge study using TTS and voice cloning to synthesize second-language learner speech, showing that proficiency-matched synthetic data significantly boosts automated speaking assessment models.
Dyadic Talking Heads and AI Interview Coaches
Today's digest covers two papers pushing the boundaries of conversational avatar systems: a new framework for dyadic talking-head generation that modulates social interaction over monologic priors, and an LLM-powered mock interview platform with a lip-synced digital interviewer and multimodal coaching feedback.
Tokenizer-Free Turkish TTS and Video as World Models
Today's digest covers FreyaTTS, a production-ready Turkish speech synthesis system that ditches phonemizers for direct character-level Diffusion Transformers, alongside a Hugging Face trending paper reconceptualizing video as a joint world-state and event stream.
Visual Speech, Multimodal Alignment, and RL-Driven ASR Adaptation
Today's digest covers viseme-guided event-based lip reading, optimal-transport alignment for audio-visual LLM-based ASR, and reinforcement learning as a superior alternative to supervised fine-tuning when only synthetic speech is available.
Full-Duplex Dialogue via RL and Robust Contextual ASR
Today's digest covers two papers from National Taiwan University: one decouples conversational timing from reasoning in full-duplex spoken models using reinforcement learning, and another introduces COALA, a framework for robust contextual biasing in speech-augmented ASR.
Word-Level TTS Control, KV Cache Compression, and Universal ASR Alignment
Today's digest covers fine-grained prosodic control in LLM-based TTS, distributional training objectives, a Taiwanese code-switching system, KV cache compression for speech LLMs, orthogonal audio-language connectors, and a gradient-based alignment method that works across every ASR family.
Unified Audio LLMs, Full-Duplex Dialogue Data, and Test-Time Scaling for ASR
Today's digest covers unified audio-text LLMs, benchmarks for streaming conversational naturalness, large-scale full-duplex dialogue data construction, controllable speaker synthesis from text profiles, and test-time scaling for ASR — a broad sweep across the speech AI stack.
Streaming Talking Heads & the Road to Photorealistic Digital Humans
Today's digest covers real-time talking-head generation at higher resolution without sacrificing latency, plus a sweeping survey of controllable 3D avatar creation from body priors to photorealistic animation.
Smarter Speech Tokens: Diffusion TTS, Accent Normalization, and Voice Anonymization
Today's digest explores three papers reshaping how discrete speech tokens are generated, converted, and protected — from a diffusion-based TTS that breaks the autoregressive bottleneck, to RL-guided accent normalization and hierarchical VAE voice anonymization.
Real-Time Avatars, Emotional Voice Control, and Proactive Dialogue
Today's digest covers speech-driven real-time avatar generation at 42 FPS, zero-shot emotional voice conversion via natural-language instructions, and a proactive thinking framework that cuts LLM response latency without extra training.
Dyadic Avatars, Efficient SpeechLLMs, and Fine-grained TTS Pronunciation
Today's digest covers end-to-end dyadic conversation generation, real-time 3D facial expression synthesis, parameter-space composition for instruction-following speech LLMs, high-throughput audio inference pipelines, and per-word pronunciation control in zero-shot TTS.
Emotion Control Across Avatars, TTS, and Speech-LLMs
Today's digest covers three papers tackling emotional control and representation: real-time emotional talking heads via 3D Gaussian Splatting, geometric analysis of emotion steering in TTS models, and a speech-text interleaving strategy that keeps LLM priors intact during ASR training.
Flexible Speech Models, Compact Avatars, and Brain-Driven Faces
Today's digest covers dynamic frame-rate spoken LLMs, compact 3D human Gaussian splatting, domain-aware TTS evaluation, and the first EEG-conditioned facial action-unit editing framework.
Real-Time Avatars, Speech Synthesis, and Conversational AI
Today's digest covers real-time conversational avatars with joint speech-facial motion, fast audio-driven portrait animation, feed-forward 4D head reconstruction, culturally-aware gesture generation, speech-to-speech LLMs, and unified voice attribute editing — eight papers pushing the frontier of expressive, interactive digital humans.
This Week in Conversational AI Research: June 29–July 05, 2026
A dense week spanning real-time multimodal agents, TTS alignment via reinforcement learning, and a wave of single-image avatar reconstruction systems — plus a sobering audit of how well production voice AI actually uses the emotions it perceives.
Real-Time Avatars and Noise-Robust Speech Recognition
Today's digest covers ray-traced shadows for 3DGS avatars, high-resolution feedforward human reconstruction from sparse video, and a variational bottleneck approach to noise-robust audio-visual speech recognition.
Bridging Synthetic and Real Speech for LLM-Based ASR
Today's digest examines how LLM-based ASR systems perceive synthetic speech and how to close the gap with real data — achieving real-data parity with just 25% genuine recordings through smart augmentation and pooling strategies.
Expressive Avatars, Emotional Voice, and Smarter ASR
Today's digest covers speech-driven 3D facial animation, relightable avatar reconstruction from a single image, expressive human motion disentanglement, hierarchical reward optimization for emotional TTS, and edit-flow refinement for non-autoregressive ASR — a broad sweep of lifelike digital human and voice research.
Faces in 3D, Guided by Geometry and Preferences
Today’s digest explores two ways to improve AI-generated faces in 3D: geometry-constrained multi-view diffusion for consistent head generation, and human-preference fine-tuning that sculpts face GAN geometry without explicit surface supervision.
Voice AI Faces the Emotion Test
Today’s digest spans faster-adapting text-to-speech, low-resource voice cloning, and a harder question for speech systems: whether they can preserve emotion and expression. It also spotlights a new benchmark and evidence that realtime voice agents still miss vocal cues even when they hear them.
This Month in Conversational AI Research: June 2026
June 2026 was defined by the industrialization of real-time avatar pipelines and full-duplex voice agents, as the field moved decisively from proof-of-concept generation toward production-grade streaming systems — while a parallel wave of interpretability work exposed how audio LLMs actually process acoustic signals.
Real-Time Voices, Agents, and Avatars
Today’s digest spans end-to-end interactive models, better speech and TTS synthesis, and instant 3D avatar generation. The common thread: more natural, controllable, low-latency conversational AI across voice and visual embodiment.
Controlling speech, audio, and avatars
Today’s digest spans new ways to shape how machines speak, listen, and move: from universal audio generation and controlled TTS to stronger ASR alignment and adaptive speech encoders. It also includes a fresh step toward smoother digital-human motion from sparse sensor input.
Voice, Avatars, and Spoken Reasoning
Today’s digest spans real-time talking avatars, relightable digital humans, universal speech synthesis, and a new look at how speech-language models reason internally. Together they point to more natural, controllable, and interactive conversational AI systems.
This Week in Conversational AI Research: June 22–28, 2026
This week pushed conversational AI toward native-streaming systems: full-duplex audio-visual agents, streaming TTS, and real-time avatars all moved closer to a single interactive loop. At the same time, speech generation papers focused on controllability, prosody, voice cloning robustness, and representation alignment for deployable voice agents.
From speech to face, code-switching, and pronunciation
Today's digest spans full-duplex talking avatars, stronger multilingual ASR for code-switching, and LLM-based grapheme-to-phoneme benchmarking for Japanese TTS. Together they point to voice systems that sound more natural, adapt more flexibly, and animate more convincingly.
Real-Time Speech for Conversational AI
Today’s digest spotlights low-latency speech generation, prosody-aware voice conversion, and robust spoken-language agents. The common thread: speech systems that sound better, respond faster, and work more reliably in live conversation.
Speech synthesis, voice conversion, and interactive avatars push toward real-time realism
Today’s digest spans transcript-free text-to-speech, diffusion-guided speech generation, streaming voice conversion, and speech codecs built for cleaner identity control. It also features physically interactive 3D avatars that deform realistically under contact and motion.
Speech systems move toward richer dialogue, controllable voices, and stronger robustness
Today’s digest spans full-duplex spoken dialogue, multi-speaker scene generation, controllable TTS, and improved ASR representations. The papers focus on making speech systems more natural, more editable, and more reliable in noisy or complex settings.
Reliable voice AI for multi-speaker dialogue and long-form synthesis
Today’s digest spans speech LLMs, voice agents, and TTS: from diarization-aware multi-speaker grounding and persona-driven speech role-play to more robust, faster, and longer-form speech generation. The common thread is making voice systems more controllable, consistent, and reliable.
Speech systems tighten context, accents, and avatars
Today’s digest spans context-aware spoken dialogue, multilingual and accent-aware speech models, faster ASR decoding, and dynamic 3D head editing. Together, these papers push conversational systems toward more faithful, natural, and controllable interaction.
Talkers Move and Speak Better
Today’s digest spans expressive talking heads, stylized co-speech gesture generation, and code-mixing speech tools for better downstream ASR. The common thread: tighter control over how synthetic voices, faces, and motions carry meaning.
Prosody and Stress Take Center Stage
Today’s digest focuses on speech fidelity at the syllable level, from dynamic prosody prediction and duration-based watermarking in LLM TTS to stress-preserving speech-to-speech translation. Together, these papers push synthesized and translated speech closer to natural, speaker-faithful delivery.
Speech Agents Learn to Listen
Today’s digest spans full-duplex spoken dialogue, emotional voice synthesis, and ASR that catches hesitation and disfluency. Together, these papers push voice agents toward more natural, responsive, and expressive interaction.
Streaming Speech, Lip Sync, Turn-Taking
Today’s digest spans talking avatars, 4D human reconstruction, and speech agents that anticipate endpoints and manage multi-party turns. It also includes expressive TTS with finer emotion control for more natural spoken output.
Prosody Powers Spoken AI
Today’s digest spotlights two directions in speech-native AI: an empathetic multi-agent dialogue system that uses prosody for better emotional alignment, and a study of speech-token design that improves how frozen LLMs reason over spoken input.
Talking Agents Get More Human
Today’s digest spans real-time lip sync and facial animation, stereoscopic digital humans, and speech models that handle turn-taking, interruptions, and paralinguistic cues more naturally. It also includes progress in multilingual TTS and higher-fidelity voice generation.
Speech Models Get Leaner
Today’s digest spotlights faster, simpler speech systems: streaming TTS, end-to-end discrete-token training, interpretable voice control, and new ways to plug speech directly into LLMs. The common thread is practical speech intelligence with lower latency, cleaner architectures, and more controllable outputs.
Voice Gets More Real-Time
Today’s digest spans holistic video dubbing, streaming speech LLMs, expressive and waveform-native TTS, and low-latency voice conversion. The common thread: better control, faster inference, and more natural-sounding speech across generation and recognition.
Faces and Hair Get Rigged
Today’s digest spotlights advances in digital humans: fully automatic face rigging with inner-mouth geometry and blendshapes, plus strand-level facial hair capture for editable beards, brows, and lashes. Together they push avatars closer to animation-ready realism.
Smarter Voices, Fewer Errors
Today’s digest spotlights more expressive speech synthesis, emotion conversion, and better ASR reliability for voice agents. From image-based TTS and continuous latent speech models to hallucination steering in Whisper, the focus is on making spoken AI sound better and fail less.
Avatars Get More Social
Today’s digest spans social digital humans, efficient 3D avatar creation, noise-robust spoken dialogue, and more controllable voice synthesis. From theory-of-mind avatars to stronger zero-shot TTS, the common thread is making conversational systems feel more natural and expressive.
Voice AI Goes Real Time
Today’s digest spans streaming speech agents, emotional text-to-speech control, robust audio-visual recognition, and a new reference-free way to evaluate ASR. Together, these papers push conversational systems toward more responsive, expressive, and reliable voice interaction.
Talking Avatars Get Real
Today’s digest spans full-duplex speech-motion avatars, portrait animation, outfit-personalized 3D humans, speech-LLM reasoning fixes, and raw-waveform zero-shot TTS. A strong day for more natural voices, more expressive bodies, and better spoken reasoning.
Avatars Speak and See Clearly
Today’s digest spans photorealistic 3D human avatars, identity-preserving video generation, and a unified audio-language model for speech, sounds, and music. Together they point to richer multimodal agents that can see, hear, and render people more naturally.
Avatars Voices And Edge Speech
Today’s digest spans real-time talking portraits, single-photo 3D face avatars, unified speech-singing synthesis, interpretable emotion control in TTS, and efficient neuromorphic speech recognition. It’s a strong mix of expressive generation and practical speech systems for interactive AI.
Speech AI Across Empathy, Scale, and Timing
Today’s digest spans emotionally adaptive voice assistants, a 100+ language speech benchmark, and a new way to evaluate talking-head generation with better temporal alignment. Together, these papers push speech systems toward richer conversation, broader coverage, and fairer assessment.
Speech models that talk, adapt, and translate in real time
Today’s digest spans expressive voice synthesis, low-latency speech systems, and talking avatars. From zero-shot long-form TTS to latent reasoning ASR and streaming translation, the focus is on models that sound more natural and respond faster.
Talking avatars and voice models push toward more natural, controllable generation
Today’s digest spans fine-tuning-free talking-face synthesis, semantically grounded gesture generation, unified digital human models, and faster streaming TTS. Across speech and avatar generation, the theme is better alignment between meaning, motion, and voice with less latency and more control.
Real-Time Avatars, Smarter Speech, and More Expressive Faces
Today's digest spans talking heads, full-body digital humans, and interactive speech systems. Highlights include text-guided head stylization, real-time avatar animation, relightable telepresence capture, and agentic ASR that revises transcripts in dialogue.
Grounding Audio LLMs, Smarter TTS, and Cinematic Multi-Talker Benchmarks
Today's digest covers seven papers tackling the hardest grounding problems in conversational AI: audio LLMs that ignore acoustics, TTS systems that punch above their data weight, event-camera-driven speech synthesis, and a new cinematic benchmark for multi-talker video generation.
Lifelike Avatars, Expressive Voices, and Smarter Gesture Generation
Today's digest covers eight papers pushing the boundaries of audio-driven avatar animation, 3D head reconstruction, co-speech gesture synthesis, and expressive TTS — from production-ready streaming avatars to privacy-aware speaker unlearning.
Dyadic Avatars, Speech LLMs, and Fine-grained TTS Pronunciation
Today's digest covers five papers spanning full dyadic audio-visual conversation generation, text-driven 3D facial expression control, parameter-space composition for instruction-following speech LLMs, high-throughput audio LLM serving, and per-word pronunciation control in zero-shot TTS.
Talking Faces, Smarter TTS, and Speech LLMs at Scale
Today's digest covers dyadic audio-visual conversation generation, text-driven 3D facial expressions, instruction-following speech LLMs, efficient vLLM-based audio inference, fine-grained pronunciation control in TTS, and a wave of industrial TTS technical reports trending on Hugging Face.
A Brief History of Audio-Driven Talking-Avatar Research (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.
How Avatars Learned to Talk: Six Years of Audio-Driven Human Animation (2019–2026)
From speech-driven 3D meshes to real-time diffusion portraits to video foundation models — the fifty-five papers that defined how machines make a face (and a body) speak.