Smarter Speech Tokens: Diffusion TTS, Accent Normalization, and Voice Anonymization
Today's digest explores three papers reshaping how discrete speech tokens are generated, converted, and protected — from a diffusion-based TTS that breaks the autoregressive bottleneck, to RL-guided accent normalization and hierarchical VAE voice anonymization.
Speed--quality Pareto front on Seed-TTS test-en (both axes log-scale). DELTA-TTS at $16$ steps reaches the bottom-left corner, achieving the lowest WER and RTF among all NAR baselines and the AR baseline. The result shows that DELTA-TTS lies on the Pareto front, offering the most favorable speed–quality trade-off among the evaluated systems. From Kakao Enterprise.
Today's papers share a common thread: rethinking what we do with discrete speech tokens. Whether the goal is faster, hallucination-free synthesis, accent-to-standard conversion without synthetic data, or generating entirely novel speaker identities for privacy, all three works push on the frontier of token-level speech modeling.
TTS & Voice Synthesis
From left-to-right to confidence-first — rethinking the decoding order in autoregressive TTS.
Autoregressive TTS models have delivered impressive naturalness, but their strictly causal decoding order creates two well-known pain points: slow inference and a susceptibility to hallucinations, since early token decisions are made without awareness of the full target text. DELTA-TTS from Kakao Enterprise tackles both problems at once by adapting a pretrained autoregressive TTS backbone into a diffusion language model that decodes speech tokens by confidence order rather than left-to-right. Low-confidence tokens are deferred and refined in later diffusion steps, while full-text awareness is maintained throughout — breaking the causal bottleneck without discarding the knowledge already baked into the autoregressive prior. The result is 3.3× faster inference alongside a meaningful reduction in hallucinations.
DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
DELTA-TTS adapts pretrained autoregressive TTS into a diffusion model that decodes speech tokens by confidence order rather than left-to-right. This breaks the causal bottleneck, achieving 3.3× faster inference while reducing hallucinations through full-text awareness and deferred low-confidence refinement.
Abstract
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness by propagating local errors and hallucinations. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, as the full input text is available before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a $1/t$-weighted training objective and a time-shifted inference schedule that defer low-confidence positions to later steps. Trained on only $585$ hours of LibriTTS, DELTA-TTS achieves a $\textbf{1.75}\%$ WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens $\textbf{3.3}\times$ faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates hallucinations observed in AR generation.
Voice Conversion & Speaker Modeling
Controlling accent, identity, and privacy through learned token spaces.
Accent normalization is a challenging voice conversion task because it demands changing phonetic realization while leaving speaker timbre intact — and real paired data (accented vs. standard utterances from the same speaker) is scarce. TokAN from the National University of Singapore sidesteps this data problem entirely by operating in the space of self-supervised discrete speech tokens. An autoregressive token-to-token conversion model is trained without any synthetic paired data, then refined with RL optimization to improve intelligibility. A duration-aware synthesis stage rounds out the pipeline, ensuring natural prosody in the converted output while preserving the original speaker's identity.
TokAN: Accent Normalization Using Self-Supervised Speech Tokens
TokAN converts accented speech to standard speech using self-supervised discrete tokens without synthetic training data. It combines autoregressive token conversion with RL optimization and duration-aware synthesis to improve intelligibility while preserving speaker identity.
Abstract
Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
On the privacy side of speaker modeling, most voice anonymization systems work by perturbing features in an existing speaker embedding space — which limits how far the anonymized voice can stray from real speakers. NouveauVoice from the University of Rochester takes a generative approach instead: a hierarchical VAE is used to synthesize novel pseudo-speaker embeddings that don't correspond to any real individual. This design achieves strong identity concealment while maintaining speech quality and, notably, preserving emotional expressiveness — a dimension that feature-space modification methods often degrade.
NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization
NouveauVoice generates diverse pseudo-speaker embeddings using a hierarchical VAE for voice anonymization while preserving speech quality. Unlike feature-space modification approaches, this framework synthesizes novel speakers to achieve strong identity concealment while maintaining emotional expressiveness.
Abstract
Advanced neural technologies in speech synthesis and voice conversion (VC) have introduced severe risks to personal privacy, necessitating robust Speaker Anonymization Systems (SAS). Existing SAS approaches modify voice characteristics in the hand-crafted feature space or speaker embedding space, often struggling to provide sufficient identity variance across generated voices. In this paper, we propose NouveauVoice, a novel pseudo-speaker generation framework based on a Hierarchical Deep Variational Autoencoder (NVAE). Integrated as a standalone plug-in module on top of state-of-the-art architectures (FACodec and CosyVoice2), our approach leverages tractable sampling and the Evidence Lower Bound (ELBO) objective to synthesize highly expressive pseudo-speaker embeddings with significantly enhanced speaker diversity. Evaluating our framework under a protocol similar to the VoicePrivacy Challenge alongside Maximum Mean Discrepancy (MMD) analysis, we demonstrate that NouveauVoice achieves strong identity concealment, yielding an Equal Error Rate (EER) exceeding 38% against an automatic speaker verification attacker model. Our system shows a reasonable trade-off between strict anonymity, rich pseudo-speaker diversity, and downstream speech utility, such as intelligibility and emotional expressiveness.
Trending on Hugging Face
Vidu S1: A Real-Time Interactive Video Generation Model
Vidu S1 is a real-time interactive video generation model that supports voice-controlled digital character animation with infinite-length output and high frame rate on consumer hardware.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
Fish Audio S2 Technical Report
Fish Audio S2 is an open-source text-to-speech system with multi-speaker capabilities, multi-turn generation, and instruction-following control through natural-language descriptions, utilizing a multi-stage training approach and production-ready inference engine.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.