EmoSteer-TTS

EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering

EmoSteer-TTS achieves fine-grained, continuous emotion control in text-to-speech without training by steering activations of pretrained models. This method uniquely enables flexible, interpretable emotion manipulation in speech synthesis through activation modulation at inference time.

• tts
• emotion
• prosody
• speech-to-speech

Demos

These demos showcase EmoSteer-TTS, a training-free method for fine-grained emotion control in TTS via activation steering. Evaluate how smoothly emotions like anger, disgust, and happiness transition with natural and expressive synthesis across different pretrained models. The framework image illustrates how internal activations are modulated to control emotion without retraining.

Authors: Tianxin Xie, Shan Yang, Chenxing Li, Dong Yu, Li Liu

Categories: cs.SD, cs.AI, eess.AS

Comment: 25 pages, 9 figures, 3 tables

Published 2025-08-05 · Updated 2025-10-25

Links

Paper & demos

• arXiv
• Project page

Code & resources

• Hugging Face961 downloads77 likes

Impact

• Semantic Scholar13 citations1 influential citation

Abstract

Text-to-speech (TTS) has shown great progress in recent years. However, most existing TTS systems offer only coarse and rigid emotion control, typically via discrete emotion labels or a carefully crafted and detailed emotional text prompt, making fine-grained emotion manipulation either inaccessible or unstable. These models also require extensive, high-quality datasets for training. To address these limitations, we propose EmoSteer-TTS, a novel training-free approach, to achieve fine-grained speech emotion control (conversion, interpolation, erasure) by activation steering. We first empirically observe that modifying a subset of the internal activations within a flow matching-based TTS model can effectively alter the emotional tone of synthesized speech. Building on this insight, we then develop a training-free and efficient algorithm, including activation extraction, emotional token searching, and inference-time steering, which can be seamlessly integrated into a wide range of pretrained models (e.g., F5-TTS, CosyVoice2, and E2-TTS). In addition, to derive effective steering vectors, we construct a curated emotional speech dataset with diverse speakers. Extensive experiments demonstrate that EmoSteer-TTS enables fine-grained, interpretable, and continuous control over speech emotion, outperforming the state-of-the-art (SOTA). To the best of our knowledge, this is the first method that achieves training-free and continuous fine-grained emotion control in TTS. Demo samples are available at https://emosteer-tts-demo.pages.dev/.

---

1. Problem Setting, Motivation, and Core Claim

EmoSteer-TTS addresses a specific weakness in current emotion-controllable text-to-speech (EC-TTS): most systems expose only coarse control, usually through discrete labels or carefully designed emotion prompts. The paper argues that such interfaces are either too rigid for fine-grained manipulation or unstable because they depend on linguistic prompt variability and large training sets. The authors target a stronger capability: training-free, continuous, fine-grained emotion control over synthesized speech, including conversion, interpolation, erasure, and composite controls such as replacement and multi-emotion blending.

The main technical idea is to steer internal activations of pretrained flow matching-based TTS models rather than training new emotion predictors or prompting mechanisms. The paper’s empirical observation is that only a subset of internal activations in a DiT-based TTS backbone strongly governs the emotional tone of the output. By extracting a steering direction from pairs of neutral and emotional reference speech, then applying that direction at inference time, the method can move output speech toward or away from a target emotion without fine-tuning the backbone.

The paper claims three primary contributions: first, it is the first fine-grained and training-free EC-TTS method in the authors’ view; second, it offers interpretability by exposing a steerable emotion subspace inside pretrained TTS models; and third, it demonstrates that the same steering strategy transfers across multiple pretrained flow matching TTS backbones, including F5-TTS, CosyVoice2, and E2-TTS.

2. Model Family and Architectural Target

The method is designed for zero-shot TTS systems that synthesize mel-spectrograms with a diffusion transformer (DiT) and flow matching / conditional flow matching (CFM). In the main experiments, the authors steer three pretrained models: F5-TTS, E2-TTS, and CosyVoice2. The steering is applied to the first residual stream inside selected DiT blocks, using hooks registered during inference.

The authors make a key empirical claim: if you perturb a carefully chosen subset of internal activations in a flow matching TTS model, the emotional tone of the synthesized speech changes in a controlled and predictable way. This is demonstrated in the paper’s observation figure, where adding a sadness steering vector to several DiT layers of F5-TTS substantially increases the predicted sadness probability of the generated speech.

3. Method: Activation Steering for Emotion Control

3.1 Overview

EmoSteer-TTS has three stages. First, it computes activation differences between neutral and emotional reference speech. Second, it searches for the most emotion-relevant token positions and constructs a weighted steering vector. Third, during inference, it injects that steering vector into selected layers and flow steps of the TTS model, using a strength parameter to control how strongly the synthesized speech is moved toward the target emotion.

3.2 Activation extraction

Let $\mathbf{x}^l_{a,i}$ and $\mathbf{x}^l_{b,j}$ denote first residual activations at layer $l$ for synthesized speech conditioned on neutral reference sample $A_i$ and emotional reference sample $B_j$, respectively. The paper defines a difference-in-means direction:

$$\mathbf{u}^l = \frac{1}{N}\sum_{j=1}^{N}\mathbf{x}^l_{b,j} - \frac{1}{M}\sum_{i=1}^{M}\mathbf{x}^l_{a,i}$$

The vector is then normalized to unit length, $\mathbf{u}^l \leftarrow \mathbf{u}^l / \|\mathbf{u}^l\|_2$, producing a robust layer-wise emotion direction. The authors align variable-length activation sequences to a fixed average length using nearest interpolation so that each activation has shape $[\text{avg\_seq\_length}, \text{hidden\_dim}]$.

3.3 Top-$k$ token search and steering-vector construction

The paper does not steer every token equally. Instead, it evaluates which token positions in the activation difference are most emotion-relevant. For each token candidate, the model perturbs a neutral reference activation using the extracted direction and measures the target emotion probability with a pretrained speech emotion recognition model, emotion2vec. The top-$k$ positions by emotion probability become the emotionally salient subset.

Formally, the token mask keeps only selected indices $\mathcal{I}_{\text{top-}k}$, giving a sparse steering vector $\mathbf{s}^l = \mathbf{u}^l \odot \mathbf{m}$. A weight vector is then built from the emotion probabilities on the selected positions via softmax:

$$\mathbf{w}^l = \operatorname{softmax}(\{P_{\text{emotion}}(\hat{A}_i)\}_{i\in \mathcal{I}_{\text{top-}k}})$$

and the final weighted steering vector is a weighted sum of the selected token directions:

$$\hat{\mathbf{s}}^l = \sum_{i=1}^{k} w_i^l \mathbf{s}_i^l$$

The paper emphasizes that the steering vector lives in a subspace associated with emotional tone and that the search can be made efficient by using the same top-$k$ token indices across the selected layers, reducing complexity from $\mathcal{O}(|\hat{\mathcal{L}}|\times \text{avg\_seq\_length})$ to $\mathcal{O}(\text{avg\_seq\_length})$.

3.4 Inference-time steering rules

For emotion conversion and interpolation, the modified activation is

$$\hat{\mathbf{x}}^l = f_r(\mathbf{x}^l + \alpha\hat{\mathbf{s}}^l),$$

where $f_r$ renormalizes the result to preserve the original $\ell_2$ norm. The scalar $\alpha$ controls the steering strength: $\alpha=0$ preserves the original tone, $\alpha>0$ pushes toward the target emotion, and $\alpha<0$ steers in the opposite direction.

For emotion erasure, the paper removes the target emotional component by subtracting the projection of the activation onto the steering vector:

$$\hat{\mathbf{x}}^l = f_r\big(\mathbf{x}^l - \beta (\hat{\mathbf{s}}^l \cdot \mathbf{x}^l)\hat{\mathbf{s}}^l\big)$$

Here $\beta$ controls erasing strength. The dot product measures how strongly the target emotion is expressed in the reference sample, so the subtraction is intended to preserve speaker and content while removing the target emotion.

The paper also defines composite controls. Emotion replacement first erases emotion $\text{emo}_1$ and then adds $\text{emo}_2$:

$$\hat{\mathbf{x}}^l = f_r\big(\mathbf{x}^l - \beta(\hat{\mathbf{s}}^l_{\text{emo}_1}\cdot \mathbf{x}^l)\hat{\mathbf{s}}^l_{\text{emo}_1} + \alpha\hat{\mathbf{s}}^l_{\text{emo}_2}\big)$$

Multiple-emotion steering is the additive extension

$$\hat{\mathbf{x}}^l = f_r\big(\mathbf{x}^l + \alpha_1\hat{\mathbf{s}}^l_{\text{emo}_1} + \alpha_2\hat{\mathbf{s}}^l_{\text{emo}_2} + \cdots + \alpha_E\hat{\mathbf{s}}^l_{\text{emo}_E}\big)$$

which the authors use to synthesize blended affect such as pleasant surprise or contempt-like mixtures.

3.5 Implementation details from the appendix

The appendix states that the steering is implemented as forward hooks on the first residual stream of selected DiT blocks. The same code path supports conversion/interpolation, erasure, replacement, and multi-emotion steering. The implementation rescales the activation back to its original norm after steering, which the authors present as a stabilizing step.

4. Emotional Speech Data and Experimental Setup

4.1 Curated steering-vector dataset

To build steering directions, the authors curate a dataset of emotionally salient speech from multiple corpora: MSP-Podcast, IEMOCAP, RAVDESS, CREMA-D, TESS, SAVEE, ASVP-ESD, CASIA, M3ED, ESD, and Emo-Emilia. After filtering, the dataset contains 6,900 utterances and covers six basic emotions from Ekman’s set — anger, happiness, sadness, disgust, surprise, and fear — plus neutrality. The paper reports balanced speaker diversity and gender balance, and notes that the dataset spans both English and Chinese speech.

The filtering pipeline removes overly short or long clips, clips with too much silence or low signal-to-noise ratio, and low-confidence emotion labels according to emotion2vec; a manual inspection of half the data is also performed. The resulting set is used only to compute activation differences, not to train the TTS backbones.

For token search, the paper synthesizes speech from 10 random neutral ESD reference samples, split across English and Chinese.

4.2 Inference datasets and evaluation protocol

For in-distribution evaluation, the authors sample neutral and emotional reference speech from MSP-Podcast and ESD, excluding those samples from steering-vector construction. For out-of-distribution evaluation, neutral speech is drawn from the SeedTTS test sets and emotional speech from EMNS. This design tests whether the steering vectors generalize beyond the construction corpora.

The paper evaluates four main aspects: word error rate (WER), speaker similarity (S-SIM), emotion similarity (E-SIM), and naturalness MOS (N-MOS) for conversion; Emotion Interpolation MOS (EI-MOS) for interpolation; and Emotion-Erasure MOS (EE-MOS) for erasure. E-SIM is computed with both emotion2vec and SenseVoice embeddings to reduce the chance of overfitting to a single SER model.

4.3 Backbone configurations

The hardware setup for the experiments is an 8× NVIDIA RTX 6000 Ada server with 256 GB RAM. The paper also reports Ubuntu 20.04.6 LTS and Conda-based execution environments.

5. Main Quantitative Results

5.1 Emotion conversion, interpolation, and erasure

The central results are reported on both in-distribution and out-of-distribution evaluation sets. The paper compares the steered models against label-based baselines with adjustable control strength (EmoSphere++, EmoDubber, HED-TTS), description-based baselines (EmoVoice, CosyVoice2, FleSpeech), and the unsteered backbones.

For conversion the authors use $\alpha=2.0$ and $k=200$; for erasure they use $\beta=2.5$ and $k=200$. Baseline metrics are computed on their demo samples for fairness, because reproduced quality is not guaranteed. The paper additionally includes an appendix table with reproduced open-source baseline results, which is reported only as a reference.

The strongest headline result is that the steered models achieve continuous control without training, while maintaining competitive speech quality. In particular, EmoSteer-TTS + F5-TTS reaches the best EI-MOS of 4.00 and the best EE-MOS of 4.02 in the main table, and the CosyVoice2 variant reaches the best N-MOS among the steered models at 3.65. Emotion similarity is also markedly improved over the unsteered backbones: the unsteered F5-TTS and E2-TTS have average E-SIM scores around 0.055 and 0.065, while the steered variants rise to roughly 0.25–0.28 depending on the backbone and metric.

The paper’s interpretation is that label-based systems have poor flexibility and weaker generalization, while description-based systems are better at preserving linguistic quality but remain bounded by prompt expressivity and the instability of language-model-generated descriptions. EmoSteer-TTS inherits the quality of strong pretrained TTS models and adds a direct continuous emotion-control interface on top.

5.2 Composite control

The paper also shows that the same steering machinery supports composite editing. In emotion replacement, the model first suppresses one emotion and then adds another; the result is verified by changes in predicted emotion probabilities. In multi-emotion steering, the method can synthesize blends such as fear plus happiness or sadness plus surprise, with the emotion predictor reflecting both components.

The authors report that these composite experiments show the intended diagonal effect: the target emotion increases and the source emotion decreases after replacement, while multi-emotion prompts lift the relevant emotion probabilities together. This is used to argue that EmoSteer-TTS is not just a one-emotion converter but a general control interface over an emotion subspace.

6. Steering-Dynamics Analysis and Ablations

6.1 Effect of top-$k$ token selection

The authors study the impact of the number of selected emotion-relevant tokens. Increasing $k$ generally raises emotion probability, especially for anger and happiness, and the gains plateau after roughly $k=200$. This is why all main experiments use $k=200$: it balances expressiveness and efficiency.

6.2 Which layers matter?

The paper divides layers into shallow, middle, deep, and spaced selections. Steering shallow layers yields moderate emotion influence; middle layers tend to work slightly better; deep layers alone become less effective, likely because those layers concentrate on acoustic refinement rather than high-level affect. Steering spaced layers such as $1,6,11,16,21$ performs best, consistent with the observation figure shown earlier.

6.3 Which flow-matching steps matter?

Because F5-TTS generates speech through 32 conditional flow-matching steps, the authors test early, middle, late, and all-step steering. Early steering has limited effect, while middle and late steps matter more; steering across all steps gives the strongest emotion shift. This is why the main setup steers all steps in F5-TTS and E2-TTS, and all 10 steps in CosyVoice2.

6.4 Prosody and F0 visualization

The appendix visualizes fundamental frequency contours to provide a qualitative check on the interpolation and erasure controls. The reported trends are intuitive: anger, happiness, and surprise tend to show higher pitch, sadness lower pitch, and erasure makes some contours flatter and calmer. The authors note that fear and sadness are less monotonic because these emotions can manifest in acoustically diverse ways, so pitch alone does not fully capture emotional perception.

7. Additional Observations and Practical Takeaways

The method is explicitly training-free: it does not require parameter updates to the TTS backbone, only steering vectors and runtime hooks. This makes it easy to attach to multiple pretrained models. The paper reports that the same approach works across F5-TTS, E2-TTS, and CosyVoice2, which supports the claim that the method is model-agnostic within the family of flow matching TTS systems.

The paper also highlights interpretability as a practical advantage. Instead of treating emotion as a black-box prompt effect, EmoSteer-TTS isolates a steerable activation direction and shows that modifying a small subset of tokens and layers changes emotion in a smooth and measurable way. That makes the control mechanism more transparent for debugging and for future emotion representation studies.

The appendix further reports that the authors recomputed open-source baseline results under a controlled protocol, but they treat those numbers only as reference because exact reproduction quality is not guaranteed. The main conclusions remain unchanged: the steered models preserve quality better than label-based baselines and provide more direct fine-grained control than prompt-based methods.

8. Limitations

The authors identify two main limitations. First, the method still depends on high-quality emotional speech samples to construct good steering vectors, even though the quantity required is modest compared with training a full controllable TTS system. Second, strong steering can introduce artifacts, especially when the steering strength becomes large; the paper notes that overly large $\alpha$ values may lead to unintelligible speech. The proposed future direction is to combine activation steering with learning-based approaches to improve robustness while keeping the control interface flexible.

9. Conclusion

EmoSteer-TTS demonstrates that emotional control in TTS can be reframed as an activation-steering problem. By extracting emotion directions from neutral-versus-emotional speech pairs and injecting them into selected internal activations of pretrained flow matching TTS models, the method achieves continuous conversion, interpolation, erasure, replacement, and multi-emotion blending without fine-tuning. Across both in-distribution and OOD evaluation, the method improves emotion controllability while preserving speech quality reasonably well, and the analysis suggests clear steering dynamics across token positions, layers, and diffusion steps.