From Face to Voice, Text to 4D: Synthesizing Richer Human Presence
Today's digest spans three frontiers of human synthesis: generating dynamic 4D avatars directly from text, handling multi-speaker noisy dialogue with turn-aware speech models, and cloning a voice from nothing but a face photo.
4DHumanDiff directly generates 4DGS-based 360-degree dynamic humans within one minute from user-provided text prompts. From Harbin Institute of Technology.
Today's papers share a common thread: pushing the boundaries of how we represent, animate, and give voice to digital humans. Whether it's rendering a fully articulated 3D figure from a text prompt, navigating a noisy cocktail party as a speaking agent, or synthesizing someone's voice from their photograph alone — the field is converging on richer, more holistic models of human presence.
Digital Humans & 3D Avatars
Text-driven generation of dynamic, view-consistent 3D humans — without the video detour.
Most text-to-4D pipelines lean on an intermediate video generation step, inheriting its temporal artifacts and view inconsistencies. 4DHumanDiff from Harbin Institute of Technology takes a more direct route: it generates 360-degree dynamic human assets as 4D Gaussian Splatting (4DGS) end-to-end, with a temporal-aware diffusion model operating directly in 4D space. The result is dramatically better view consistency and temporal coherence — and a 10× reduction in inference time compared to video-mediated approaches.
4DHumanDiff: Direct Text-to-4DGS Generation for Consistent 360-Degree Dynamic Humans
This work directly generates 360-degree dynamic human assets from text as 4D Gaussian Splatting, bypassing intermediate video synthesis. By modeling 4D space end-to-end with temporal-aware diffusion, it achieves superior view consistency and temporal coherence while reducing inference time by over 10x.
Abstract
Generating high-quality 360-degree dynamic human assets from text prompts is challenging. Existing methods usually synthesize monocular or multi-view videos first and then fit a 4D representation, which is expensive and often causes incomplete geometry or view-inconsistent renderings. We present 4DHumanDiff, a diffusion framework that directly generates dynamic humans represented by 4D Gaussian Splatting (4DGS) from text prompts. By modeling the structured 4D representation space end-to-end, 4DHumanDiff avoids video pre-generation and per-scene reconstruction, making it better suited for view-consistent and temporally coherent asset generation. The model uses a 3D U-Net backbone with temporal attention for motion-aware generation. We further construct a large-scale text-to-4DGS dataset with 60,000 high-quality pairs, and introduce 2D regularization and training-free 4D interpolation to improve rendering quality and motion smoothness. Experiments show that 4DHumanDiff generates consistent 360-degree dynamic humans within one minute, achieves better temporal and multi-view consistency, and reduces inference time by more than 10x.
SpeechLLMs & Spoken Dialogue
Teaching a spoken dialogue model to know when — and whether — to speak.
Real-world conversations are messy: multiple speakers, background noise, and the constant question of whether a given utterance is even addressed to you. Cocktail-Talker from Columbia University tackles this head-on by first predicting a discrete turn action — respond, listen, or ignore — before generating any speech. This action-aware formulation is trained using GRPO on synthetic multi-speaker dialog data, and it uniquely models selective participation: the system decides whether to engage at all before it decides what to say. It's a meaningful step toward voice agents that behave naturally in social, multi-party environments.
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Cocktail-Talker handles multi-speaker conversations in noisy environments by predicting discrete turn actions—respond, listen, or ignore—before generating responses. It uniquely addresses selective participation in social settings through action-aware speech generation trained on synthetic multi-speaker dialogs.
Abstract
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
TTS & Voice Synthesis
Voice cloning when you only have a photo to go on.
What if there's no reference audio — only a face? This independent research effort tackles zero-shot face-to-speech synthesis by training a lightweight adapter that maps facial image embeddings into the style latent space of a frozen, pre-trained diffusion-based TTS model. The frozen backbone keeps speech quality high while the adapter learns a language-agnostic face-to-voice mapping — a finding that suggests the learned correspondence captures something genuinely cross-lingual about how faces relate to vocal identity.
Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model
This work generates natural speech directly from a facial image, enabling voice cloning for cases where only photos are available. A lightweight adapter aligns face embeddings to a frozen TTS model's style space, achieving high-quality synthesis while discovering a language-agnostic face-to-voice mapping.
Abstract
Zero-shot text-to-speech (TTS) clones a voice from a short audio prompt, but this reliance on reference audio is a barrier when only visual information is available, e.g. for historical figures or video-game characters. In this work, we propose a Face-to-Speech (F2S) framework that predicts a plausible voice from a static facial image. A lightweight Face Adapter, together with soft-tuning of the face encoder's upper blocks, aligns face-recognition features with the style space of a frozen StyleTTS 2 model, kept frozen during training. We evaluate on held-out identities from LRS3, a large-scale audiovisual corpus of English TED-talk videos. The synthesized speech is highly natural (UTMOS 3.7-4.0, matching or exceeding the 3.61 of ground truth), face-to-voice retrieval is consistently above chance, and the generated voice is consistent with the target speaker. Without any retraining, an English-trained adapter also produces fluent Spanish speech, indicating that the face-to-style mapping is largely language-agnostic.
Trending on Hugging Face
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.