Flexible Speech Models, Compact Avatars, and Brain-Driven Faces
Today's digest covers dynamic frame-rate spoken LLMs, compact 3D human Gaussian splatting, domain-aware TTS evaluation, and the first EEG-conditioned facial action-unit editing framework.
Visualization of EEG Attention–Sensitive Spatial Regions in Generated Images From MindAU.
Today's papers push the boundaries of what it means to control and evaluate conversational AI systems — from speech models that dynamically adapt their processing rate, to faces animated directly from brain signals. Across four categories, researchers are questioning long-held assumptions (does naturalness actually matter for TTS?), eliminating redundancy (in both speech tokens and 3D Gaussian representations), and opening entirely new input modalities for expression control.
SpeechLLMs & Spoken Dialogue
Dynamic token rates let a single spoken LLM punch well above its weight.
Speech contains highly variable information density — silences and steady vowels demand far fewer tokens than rapid consonant clusters — yet most spoken language models operate at a fixed frame rate, wasting compute on low-density regions. FlexiSLM tackles this head-on by introducing the first spoken language model that supports dynamic and controllable frame rates on both the input and output sides. A single 7B-parameter model spans the range of 4.0–12.5 Hz without any retraining, exploiting variable information density to achieve roughly 2× inference speedup while matching or outperforming fixed-rate baselines on quality benchmarks.
FlexiSLM · 2026-06
FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model
The Chinese University of Hong Kong, Shenzhen
The first spoken language model supporting dynamic and controllable frame rates on both input and output, exploiting variable information density in speech. A single 7B model operates from 4.0–12.5 Hz without retraining, outperforming fixed-rate baselines while achieving roughly 2× speedup with maintained quality.
Abstract
Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .
Digital Humans & 3D Avatars
Human-centric prediction sidesteps the per-view redundancy that bloats conventional Gaussian splatting.
Real-time streaming of photorealistic human avatars demands compact representations, but standard 3D Gaussian Splatting methods predict primitives per view and accumulate heavy inter-view redundancy. PointSplat takes a fundamentally different approach: it predicts Gaussian primitives directly in 3D space using a human-centric prior, yielding representations that are naturally compact from the start. The result is high-fidelity reconstruction well-suited to real-time streaming scenarios without the overhead of redundant multi-view fusion.
PointSplat · 2026-06
PointSplat: Compact Gaussian Splatting via Human-Centric Prediction
State Key Lab of CAD&CG, Zhejiang University
A method for compact 3D human reconstruction that directly predicts Gaussian primitives in 3D space rather than per-view, eliminating inter-view redundancy. This human-centric approach produces naturally compact representations ideal for real-time streaming while maintaining high fidelity.
Abstract
Producing 3D human representations from input views on the fly is essential for immersive live streaming systems, where representation compactness is as critical as high fidelity given limited computational power and transmission bandwidth. Although recent feed-forward reconstruction methods achieve impressive quality through the view-centric prediction of 3D representations, they repeatedly encode the same subject content across multiple views, leading to significant inter-view redundancy. Our key insight is to perform predictions directly in 3D space, enabling the network to learn and produce a highly compact representation. To this end, we propose PointSplat, a novel human-centric approach that directly infers Gaussian primitives from an input point set. The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points and establish explicit 2D--3D correspondences. Subsequently, it employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass. This design restricts predictions to foreground regions of interest, substantially reducing the total number of Gaussians while improving novel-view rendering quality. Extensive experiments demonstrate that PointSplat achieves higher efficiency and quality while exhibiting strong robustness to variations in view count and image resolution across multiple datasets.
TTS & Voice Synthesis
A perceptual study asks whether chasing naturalness scores is the right goal for TTS in the wild.
The field has long used naturalness as its primary evaluation signal for text-to-speech systems, but does a universally high naturalness score actually translate to a good listener experience across every deployment context? A new perceptual study on context-aware TTS evaluation demonstrates that naturalness and appropriateness diverge substantially depending on the target application domain — a voice that scores well on one axis can fail on the other. The findings challenge the assumption that a single universal metric adequately reflects real-world TTS performance, and call for domain-sensitive evaluation protocols.
Context-Aware TTS Evaluation · 2026-06
Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation
Iconic
A perceptual study revealing that text-to-speech quality depends critically on the target application domain, with naturalness and appropriateness diverging substantially across contexts—challenging the assumption that universal naturalness scores adequately reflect real-world TTS performance.
Abstract
Text-to-speech (TTS) evaluation is an open challenge. While the primary target was "naturalness," recent fidelity gains shifted focus toward "appropriateness" and whether speech is correct for its context. In this work, we examine how perception changes when the expected downstream use varies. We measure the appropriateness and human-likeness of five SOTA TTS systems across five domains: AI assistant, reader, actor, animated character, and spontaneous speaker. Results show appropriateness varies across domains independently of naturalness. While systems shine at reading, expressive domains remain challenging, and optimizing for one can degrade others. Furthermore, naturalness scores tend to penalize stylized speech while rewarding spontaneity. Finally, our study also highlights blind spots in one-size-fits-all evaluation metrics across more expressive domains. We demonstrate that TTS performance is not "solved" but depends on the target domain, requiring context-aware evaluation.
Apache-2.0
Face Animation & Expression Control
Bridging EEG brain signals and semantic facial expressions opens a new channel for assistive communication.
Controlling facial expressions typically requires visual or audio inputs, but MindAU opens an entirely new modality: EEG brain signals. Through dual-stream manifold alignment, the framework bridges neural activity patterns with a semantic expression space defined by facial action units (AUs), enabling fine-grained AU-level edits while preserving subject identity. This is the first framework to achieve AU-level control directly from brain activity, with clear implications for assistive communication interfaces where conventional input channels are unavailable.
MindAU · 2026-07
MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment
Binghamton University
MindAU enables facial action-unit editing from EEG brain signals through dual-stream manifold alignment that bridges neural patterns with semantic expression spaces while preserving identity. It is the first framework for AU-level control from brain activity, with applications toward assistive communication interfaces.
Abstract
Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.