Closed-Loop TTS: AudioLLM Diagnosis Meets Fine-Grained Prosody Correction
Today's digest spotlights a novel closed-loop TTS pipeline from NUS that uses an AudioLLM to diagnose prosodic defects and a fine-grained refiner to correct them — moving beyond the limitations of single-pass generation and standard quality metrics.
Overview of the pipeline. Stage~1 (Filter): coarse-grained metrics (WER, UTMOS) discard catastrophically degraded outputs and trigger full re-generation. Stage~2 (Judge): an AudioLLM evaluates prosody naturalness and emotional fidelity, producing an overall score (1--10) along with structured refine instructions that specify both global attributes (emotion, speed, pitch) and local operations (stress, pause). Utterances meeting the threshold are accepted as final output. Stage~3 (Refiner): the Refiner takes the initial utterance together with the refine instructions and performs guided expressive re-synthesis, producing corrected speech with optional iterative refinement for residual defects. From National University of Singapore.
Most text-to-speech systems treat synthesis as a one-shot process: generate an utterance and hope for the best. Today's digest highlights a compelling departure from that paradigm — a closed-loop architecture that audits its own output and iteratively corrects prosodic defects, courtesy of researchers at the National University of Singapore.
TTS & Voice Synthesis
From single-pass to self-correcting: rethinking TTS quality control
Standard TTS evaluation pipelines lean heavily on automatic metrics like MOS predictors and word error rate, but these can miss subtle prosodic failures — awkward phrasing, misplaced stress, unnatural rhythm — that are immediately obvious to human listeners. The question is: can a system catch and fix its own mistakes?
NUS tackles exactly this challenge with a diagnose-then-refine paradigm. An AudioLLM acts as a structured diagnostic module, identifying specific prosodic defects in a generated utterance and producing correction instructions. A fine-grained refiner then consumes those instructions and revises the speech accordingly, forming a closed feedback loop rather than a linear pipeline. Critically, the system is designed to recover low-quality utterances that conventional automatic metrics would either pass or fail to flag at all — making the pipeline especially valuable for production scenarios where silent quality degradation is the real risk.
Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
A closed-loop TTS system that diagnoses prosodic defects via AudioLLM and corrects them with a fine-grained refiner, moving beyond single-pass generation. The diagnostic-then-refine pipeline recovers low-quality utterances that standard metrics miss by providing structured correction instructions.
Abstract
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
The architectural insight here is clean: by decoupling diagnosis (what is wrong and why) from correction (how to fix it), the system can leverage the rich audio-language understanding of modern AudioLLMs without requiring them to directly control the acoustic generation process. The refiner handles the low-level signal manipulation guided by structured, interpretable instructions — a principled separation of concerns that should make the system easier to audit, tune, and extend.
Trending on Hugging Face
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
VoiceMem introduces a dual-brain streaming memory architecture for speech language models that improves retrieval accuracy, emotional personalization, and real-time efficiency.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.