Streaming S2ST, Animatable Avatars, and 3D Face Reconstruction
Today's digest covers data-efficient streaming speech-to-speech translation with a Thinker-Talker architecture, plus a trio of advances in 3D human and face reconstruction — from occlusion-robust Gaussian avatars to UV-space face fusion and a generative anthropometric head model.
Qualitative comparison of Ours with ExAvatar under full-body, upper-body, and head-only inputs. Across all settings, our method yields noticeably sharper geometry, improved texture fidelity, and superior high-frequency detail reconstruction. From University of British Columbia.
Today's papers span two active frontiers: low-latency speech translation that decouples reasoning from acoustic generation, and a cluster of 3D human reconstruction systems pushing fidelity, topology consistency, and generative modeling of the human head. A Google paper from today's Hugging Face Daily tab rounds out the avatar theme.
SpeechLLMs & Streaming Speech Translation
Making simultaneous speech-to-speech translation practical — without the massive data bill.
Simultaneous speech-to-speech translation (S2ST) has long required large paired corpora and careful latency-quality engineering. SimulS2ST-Omni from the National University of Singapore rethinks this with a Thinker-Talker architecture that explicitly decouples linguistic reasoning (the Thinker) from acoustic prediction (the Talker). The key insight is joint text-code trajectory supervision: both intermediate text tokens and discrete speech codes are jointly optimized, giving the model a richer supervisory signal that compensates for scarce paired data. The whole system trains on only ~2k hours of paired speech, yet achieves competitive quality-latency trade-offs on standard benchmarks.
National University of Singapore · Jul 2026
SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
This work enables data-efficient streaming speech-to-speech translation via joint text-code trajectory supervision using only ~2k hours of paired data. A Thinker-Talker architecture decouples linguistic reasoning from acoustic prediction, achieving competitive quality-latency trade-offs on benchmark datasets.
Abstract
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Digital Humans & 3D Face Avatars
Reconstructing humans and faces from partial observations — occlusion-robust, topology-consistent, and generative.
Building photorealistic, animatable digital humans from casual video is hard enough when the whole body is visible. FlexiAvatar (University of British Columbia) tackles the harder problem of arbitrary body visibility — portraits, half-body shots, action clips where limbs disappear — by only optimizing 3D Gaussian splats for observed regions and leaning on diffusion-based synthesis to fill in plausible geometry and appearance for unseen areas. Occlusion-robust SMPL-X tracking keeps the underlying body model consistent, and the result is higher-fidelity reconstruction across a wide range of partial and full-body capture scenarios compared to prior methods that naively attempt to model everything.
University of British Columbia · Jul 2026
FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility
FlexiAvatar reconstructs animatable 3D avatars from monocular video by optimizing only visible regions, avoiding artifacts from unobserved areas. It combines occlusion-robust SMPL-X tracking with diffusion-based synthesis for unseen areas, delivering higher fidelity across partial and full-body capture.
Abstract
Reconstructing animatable 3D human avatars from monocular video is a fundamental problem in computer vision with broad applications in AR/VR and digital content creation. Existing approaches typically couple parametric body models with neural rendering or 3D Gaussian splatting and optimize all body regions jointly from short videos, which often degrades fidelity in the visible areas. To overcome this limitation, we introduce FlexiAvatar, a unified framework that explicitly optimizes only the visible body regions, effectively eliminating artifacts arising from unobserved limbs. Our method integrates occlusion-robust SMPL-X tracking with part-specific residual refinement to capture high-frequency geometric and appearance details. To complete entirely unseen regions (e.g., back views), we leverage a diffusion-based approach to generate texture consistent with the observed appearance. Experiments on full-body (NeuMan, ZJU-MoCap, WildAvatar), upper/half-body (talk-show clips), and head-only (INSTA) inputs show that FlexiAvatar delivers consistently higher reconstruction quality, outperforming state-of-the-art methods by an average PSNR improvement of approximately 3% across datasets. Finally, by restricting optimization to observed regions, our method reduces the effective number of Gaussians that must be optimized and rendered, leading to reduced runtime and memory overhead in partial-visibility scenarios.
Multi-view face reconstruction is a well-studied problem, but existing pipelines often rely on slow, heuristic optimization that struggles to generalize outside controlled settings. UVFaceFusion (Tsinghua University) replaces that bottleneck with learnable fusion directly in canonical UV space, recovering high-fidelity face meshes with fixed topology from in-the-wild multi-view images in a single forward pass. Operating in UV space enforces topological consistency across captures without per-instance optimization, yielding state-of-the-art geometric accuracy and fast inference — a practical combination for downstream avatar pipelines.
Tsinghua University · Jul 2026
UVFaceFusion: Fast Multi-view Topologically Consistent Face Reconstruction in the Wild via UV-space Neural Fusion
UVFaceFusion reconstructs high-fidelity face meshes with fixed topology from multi-view images via learnable fusion in canonical UV space. By replacing heuristic optimization with direct geometry fusion, it achieves state-of-the-art accuracy with fast inference and strong generalization to in-the-wild captures.
Abstract
Reconstructing high-fidelity facial geometry with an assigned topology is essential for digital avatar creation and animation, yet existing automated methods often trade off geometric fidelity and in-the-wild generalization. We present UVFaceFusion, a feed-forward framework for multi-view, fixed-topology face reconstruction from daily images. Our key idea is to replace heuristic topological optimization with learnable neural fusion in a canonical UV space. Given multi-view images, we first obtain dense point maps and facial UV correspondences of each view using VGGT and Pixel3DMM, respectively. Then, the view-specific point maps are lifted into the canonical UV domain and fused with a novel mask-aware neural fusion network. The network predicts a complete UV-space point map, from which a fixed-topology mesh is directly sampled. Although trained only on Ava-256, UVFaceFusion generalizes well to multiple public benchmarks and in-the-wild captures, benefiting from its canonical UV-space geometry-to-geometry fusion that reduces dependence on dataset-specific appearance and capture conditions. Experiments on various benchmarks show that UVFaceFusion achieves state-of-the-art reconstruction accuracy while reconstructing a mesh from 16 input views in less than 3 seconds on a single RTX 4090. Code is available at https://github.com/grignarder/UVFaceFusion.
From today's Hugging Face Daily picks, Google introduces GNM Head, a Generative aNthropometric Model of the human head. Rather than relying on fixed statistical shape spaces, GNM Head learns a generative model grounded in anthropometric measurements, enabling controlled synthesis of diverse, anatomically plausible head geometries. This kind of structured generative prior is a natural complement to reconstruction systems like UVFaceFusion and FlexiAvatar, providing a principled shape space to regularize fitting or bootstrap novel identities.
Google · Jul 2026↑3★ 1,220
GNM Head: A Generative aNthropometric Model of the human head
Abstract
Parametric models of the human head are essential tools traditionally used in computer vision and graphics for animation, rendering, and reconstruction. More recently, they serve as crucial conditioning signals within generative large vision models, allowing for tight spatial control of generated imagery. However, existing publicly available models are typically limited in anatomical scope, modeling only outer geometry while ignoring intra-oral and ocular structures, and frequently suffer from reduced geometric quality stemming from low-fidelity input datasets. In this report we introduce a new parametric model dubbed Generative aNthropometric Model (GNM), named as a homophone of the human genome. GNM encompasses the head, face, neck, eyeballs, teeth, and tongue, and it is built on an extensive database of high-resolution 3D scans combined with high-quality anatomy specific artist-made samples. This report details the data provenance, the model architecture including the specialized sub-models for the ocular and intra-oral structures, and shows its SotA performance on fitting target 3D face scans. To foster community innovation, the complete GNM framework is made publicly available.
Trending on Hugging Face
Qwen · Jan 2026↑775 comments★ 12,638
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Microsoft Research · Aug 2025↑17510 comments★ 50,672
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Apr 2026
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
MiniCPM-o 4.5 enables real-time full-duplex interaction where perception and response happen simultaneously across vision, audio, and speech. Its Omni-Flow framework aligns multimodal signals on a shared timeline, supporting both reactive and proactive behaviors in continuous, streaming conversation.
Oct 2024↑171 comment★ 61,614
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.