Akapulu Labs logo Akapulu Labs Research

Streaming S2ST, Animatable Avatars, and 3D Face Reconstruction

Today's digest covers data-efficient streaming speech-to-speech translation with a Thinker-Talker architecture, plus a trio of advances in 3D human and face reconstruction — from occlusion-robust Gaussian avatars to UV-space face fusion and a generative anthropometric head model.

Streaming S2ST, Animatable Avatars, and 3D Face Reconstruction

Qualitative comparison of Ours with ExAvatar under full-body, upper-body, and head-only inputs. Across all settings, our method yields noticeably sharper geometry, improved texture fidelity, and superior high-frequency detail reconstruction. From University of British Columbia.

Today's papers span two active frontiers: low-latency speech translation that decouples reasoning from acoustic generation, and a cluster of 3D human reconstruction systems pushing fidelity, topology consistency, and generative modeling of the human head. A Google paper from today's Hugging Face Daily tab rounds out the avatar theme.

SpeechLLMs & Streaming Speech Translation

Making simultaneous speech-to-speech translation practical — without the massive data bill.

Simultaneous speech-to-speech translation (S2ST) has long required large paired corpora and careful latency-quality engineering. SimulS2ST-Omni from the National University of Singapore rethinks this with a Thinker-Talker architecture that explicitly decouples linguistic reasoning (the Thinker) from acoustic prediction (the Talker). The key insight is joint text-code trajectory supervision: both intermediate text tokens and discrete speech codes are jointly optimized, giving the model a richer supervisory signal that compensates for scarce paired data. The whole system trains on only ~2k hours of paired speech, yet achieves competitive quality-latency trade-offs on standard benchmarks.

National University of Singapore

National University of Singapore · Jul 2026

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

This work enables data-efficient streaming speech-to-speech translation via joint text-code trajectory supervision using only ~2k hours of paired data. A Thinker-Talker architecture decouples linguistic reasoning from acoustic prediction, achieving competitive quality-latency trade-offs on benchmark datasets.

Abstract

Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.

Digital Humans & 3D Face Avatars

Reconstructing humans and faces from partial observations — occlusion-robust, topology-consistent, and generative.

Building photorealistic, animatable digital humans from casual video is hard enough when the whole body is visible. FlexiAvatar (University of British Columbia) tackles the harder problem of arbitrary body visibility — portraits, half-body shots, action clips where limbs disappear — by only optimizing 3D Gaussian splats for observed regions and leaning on diffusion-based synthesis to fill in plausible geometry and appearance for unseen areas. Occlusion-robust SMPL-X tracking keeps the underlying body model consistent, and the result is higher-fidelity reconstruction across a wide range of partial and full-body capture scenarios compared to prior methods that naively attempt to model everything.

University of British Columbia

University of British Columbia · Jul 2026

FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility

FlexiAvatar reconstructs animatable 3D avatars from monocular video by optimizing only visible regions, avoiding artifacts from unobserved areas. It combines occlusion-robust SMPL-X tracking with diffusion-based synthesis for unseen areas, delivering higher fidelity across partial and full-body capture.

Abstract

Reconstructing animatable 3D human avatars from monocular video is a fundamental problem in computer vision with broad applications in AR/VR and digital content creation. Existing approaches typically couple parametric body models with neural rendering or 3D Gaussian splatting and optimize all body regions jointly from short videos, which often degrades fidelity in the visible areas. To overcome this limitation, we introduce FlexiAvatar, a unified framework that explicitly optimizes only the visible body regions, effectively eliminating artifacts arising from unobserved limbs. Our method integrates occlusion-robust SMPL-X tracking with part-specific residual refinement to capture high-frequency geometric and appearance details. To complete entirely unseen regions (e.g., back views), we leverage a diffusion-based approach to generate texture consistent with the observed appearance. Experiments on full-body (NeuMan, ZJU-MoCap, WildAvatar), upper/half-body (talk-show clips), and head-only (INSTA) inputs show that FlexiAvatar delivers consistently higher reconstruction quality, outperforming state-of-the-art methods by an average PSNR improvement of approximately 3% across datasets. Finally, by restricting optimization to observed regions, our method reduces the effective number of Gaussians that must be optimized and rendered, leading to reduced runtime and memory overhead in partial-visibility scenarios.

Multi-view face reconstruction is a well-studied problem, but existing pipelines often rely on slow, heuristic optimization that struggles to generalize outside controlled settings. UVFaceFusion (Tsinghua University) replaces that bottleneck with learnable fusion directly in canonical UV space, recovering high-fidelity face meshes with fixed topology from in-the-wild multi-view images in a single forward pass. Operating in UV space enforces topological consistency across captures without per-instance optimization, yielding state-of-the-art geometric accuracy and fast inference — a practical combination for downstream avatar pipelines.

Tsinghua University

Tsinghua University · Jul 2026

UVFaceFusion: Fast Multi-view Topologically Consistent Face Reconstruction in the Wild via UV-space Neural Fusion

UVFaceFusion reconstructs high-fidelity face meshes with fixed topology from multi-view images via learnable fusion in canonical UV space. By replacing heuristic optimization with direct geometry fusion, it achieves state-of-the-art accuracy with fast inference and strong generalization to in-the-wild captures.

Abstract

Reconstructing high-fidelity facial geometry with an assigned topology is essential for digital avatar creation and animation, yet existing automated methods often trade off geometric fidelity and in-the-wild generalization. We present UVFaceFusion, a feed-forward framework for multi-view, fixed-topology face reconstruction from daily images. Our key idea is to replace heuristic topological optimization with learnable neural fusion in a canonical UV space. Given multi-view images, we first obtain dense point maps and facial UV correspondences of each view using VGGT and Pixel3DMM, respectively. Then, the view-specific point maps are lifted into the canonical UV domain and fused with a novel mask-aware neural fusion network. The network predicts a complete UV-space point map, from which a fixed-topology mesh is directly sampled. Although trained only on Ava-256, UVFaceFusion generalizes well to multiple public benchmarks and in-the-wild captures, benefiting from its canonical UV-space geometry-to-geometry fusion that reduces dependence on dataset-specific appearance and capture conditions. Experiments on various benchmarks show that UVFaceFusion achieves state-of-the-art reconstruction accuracy while reconstructing a mesh from 16 input views in less than 3 seconds on a single RTX 4090. Code is available at https://github.com/grignarder/UVFaceFusion.

From today's Hugging Face Daily picks, Google introduces GNM Head, a Generative aNthropometric Model of the human head. Rather than relying on fixed statistical shape spaces, GNM Head learns a generative model grounded in anthropometric measurements, enabling controlled synthesis of diverse, anatomically plausible head geometries. This kind of structured generative prior is a natural complement to reconstruction systems like UVFaceFusion and FlexiAvatar, providing a principled shape space to regularize fitting or bootstrap novel identities.

Google

Google · Jul 2026↑3★ 1,220

GNM Head: A Generative aNthropometric Model of the human head

Abstract

Parametric models of the human head are essential tools traditionally used in computer vision and graphics for animation, rendering, and reconstruction. More recently, they serve as crucial conditioning signals within generative large vision models, allowing for tight spatial control of generated imagery. However, existing publicly available models are typically limited in anatomical scope, modeling only outer geometry while ignoring intra-oral and ocular structures, and frequently suffer from reduced geometric quality stemming from low-fidelity input datasets. In this report we introduce a new parametric model dubbed Generative aNthropometric Model (GNM), named as a homophone of the human genome. GNM encompasses the head, face, neck, eyeballs, teeth, and tongue, and it is built on an extensive database of high-resolution 3D scans combined with high-quality anatomy specific artist-made samples. This report details the data provenance, the model architecture including the specialized sub-models for the ocular and intra-oral structures, and shows its SotA performance on fitting target 3D face scans. To foster community innovation, the complete GNM framework is made publicly available.