Context-Aware Expressive TTS via LLM-Driven Control Layers
Today's digest features Harness TTS from Alibaba Group, a novel approach that wraps existing TTS engines with an LLM planner and prompt-tool registry to achieve context-aware, expressive, and auditable speech synthesis — no model retraining required.
Overview of the Harness TTS architecture. The offline stage constructs the tool registry. The online stage consists of three steps: observation, planning, and execution. The planner selects a tool from the registry given the structured observation, and the executor synthesizes speech using the corresponding prompt. From Alibaba Group.
Today's digest is a focused one — a single paper from Alibaba Group that tackles a practical and underexplored problem in production TTS: how do you make a voice assistant expressive and context-aware without retraining or modifying the underlying synthesis model?
TTS & Voice Synthesis
Steering expressiveness through orchestration, not architecture.
Getting TTS systems to respond naturally to nuanced instructions — speaking softly, adding urgency, mirroring conversational affect — is notoriously hard when the control interface is bolted on as an afterthought. Most approaches either retrain the model or rely on brittle prompt heuristics. Harness TTS takes a different path entirely: it wraps any existing TTS engine with a lightweight Harness Layer that intercepts expressive requests and routes them intelligently.
The core of the system is an LLM planner that parses user intent and maps it to a prompt-tool registry — a structured catalogue of expressive controls (prosody adjustments, style prompts, paralinguistic cues, etc.). Rather than modifying model weights, the planner selects and composes the right tools at inference time, making the system both auditable and modular. This architecture is particularly well-suited for voice assistant deployments where transparency and low latency are non-negotiable.
Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer
Harness TTS wraps TTS engines with a control layer that uses an LLM planner to route expressive requests to a prompt-tool registry. This enables context-aware, auditable speech synthesis for voice assistants without modifying the underlying model, achieving strong instruction-following gains with minimal latency.
Abstract
Expressive speech synthesis for voice assistants requires flexible style control that adapts to explicit requests and broader interaction context. We propose Harness TTS, a lightweight control layer that wraps around a TTS engine to externalize and govern its expressive behavior. It reformulates style control as closed-set prompt-tool routing: offline, a compact registry of stylistic prompt tools is constructed with structured metadata; online, an LLM planner selects the appropriate tool based on a priority-aware observation schema, and the TTS executor synthesizes speech using the corresponding prompt audio. We evaluate Harness TTS on both routing and synthesis tasks. In routing, Qwen3-4B achieves Top-1 accuracies of 74.3%, 43.0%, and 64.6% on explicit, implicit, and conflict subsets. For synthesis, experiments on CosyVoice3 and VoxCPM2 show that Harness TTS outperforms instruction-only control, achieving higher instruction-following win rates (margins of 23.1-35.6 points on CosyVoice3 and 13.8-20.0 points on VoxCPM2) and improving UTMOSv2 scores by 0.11-0.38. Moreover, the 4B planner delivers its first tool recommendation in under 50 ms in standard mode, introducing negligible latency for real-time interaction. These results demonstrate that equipping TTS engines with a dedicated Harness layer offers a practical, auditable, and context-aware solution for voice assistant expression control.
Key highlights from the paper:
- Model-agnostic design — the Harness Layer sits atop any TTS backend, meaning teams can upgrade their synthesis engine independently of the control logic.
- Instruction-following gains — the system achieves strong improvements in following expressive speech directives compared to baseline TTS pipelines.
- Minimal latency overhead — the LLM planning step is kept lean enough for real-time voice assistant use cases.
- Auditability — because all routing decisions go through an explicit registry, the system's expressive choices can be inspected and debugged, unlike end-to-end neural approaches.
This "wrap, don't retrain" philosophy echoes broader trends in LLM tool-use and agentic systems — applying the same orchestration patterns that have proven powerful in text to the speech synthesis stack.
Trending on Hugging Face
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.
Moshi: a speech-text foundation model for real-time dialogue
A speech-text foundation model for real-time dialogue via end-to-end speech-to-speech generation instead of cascaded components. By jointly modeling overlapping audio streams with text-token prediction, it achieves 200ms latency while preserving emotion and handling natural conversational dynamics like interruptions.
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
MiniCPM-o 4.5 enables real-time full-duplex interaction where perception and response happen simultaneously across vision, audio, and speech. Its Omni-Flow framework aligns multimodal signals on a shared timeline, supporting both reactive and proactive behaviors in continuous, streaming conversation.