Emotion Geometry

A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

This work compares emotion steering sites in hybrid TTS through representation geometry, revealing that the language model has clean, speaker-invariant emotion subspaces ideal for composable control, while the acoustic decoder suffers speaker entanglement limiting cross-speaker generalization.

• tts
• emotion
• prosody
• speech-driven
• voice-conversion

Authors: Siyi Wang, James Bailey, Ting Dang

Lab: The University of Melbourne

Categories: cs.SD, cs.LG

Published 2026-07-01 · Updated 2026-07-01

Links

Paper & demos

• arXiv

Impact

• Semantic Scholar0 citations

Abstract

While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.

---

Introduction & Motivation

Generating emotionally controllable speech is a core requirement for conversational agents, audiobook narration, and assistive communication. Human emotional expression is inherently nuanced: multiple emotions frequently co-exist within a single utterance, forming mixed affective states that current TTS systems struggle to reproduce with fine-grained quantitative control.

Existing emotion control strategies operate through the model's external interface. Label-based approaches enable explicit emotion conditioning but require costly annotated data and full retraining. Prompt-based methods describe target emotions in natural language but lack precise proportional control. Activation steering sidesteps both limitations by directly injecting learned direction vectors into intermediate activations at inference time — no retraining required. This paradigm has proven effective in large language models and text-to-image diffusion models, and is now being applied to speech.

State-of-the-art TTS systems increasingly adopt hybrid architectures that couple a speech language model (SLM) with a conditional flow-matching (CFM) decoder. The SLM autoregressively governs high-level prosodic and semantic structure, while the CFM renders fine-grained acoustic details. Each module is a candidate site for steering emotional expression. Prior work has demonstrated composable mixed-emotion steering via the SLM (Wang et al., CoCoEmo) and continuous single-emotion intensity control via the CFM (Xie et al., EmoSteer), but no study has systematically compared the two sites, related their geometric properties to steering effectiveness, or investigated whether jointly steering both modules yields complementary or interfering effects.

This paper presents the first comparative study of SLM and CFM modules as activation steering sites for mixed-emotion speech synthesis. The central thesis is that the geometry of emotion representations in each module — their dimensionality, linear separability, and speaker entanglement — directly predicts and explains practical steering outcomes.

Background: Hybrid TTS Architecture

The backbone system studied is CosyVoice2, a representative hybrid TTS architecture. Speech generation proceeds in two stages:

1. The SLM — a 24-layer Qwen2.5-based transformer with hidden dimension 896 — autoregressively generates discrete speech tokens $\mathbf{z} = f_{\text{SLM}}(\mathbf{x}, \mathbf{c}_{\text{ref}})$ from text input $\mathbf{x}$ and reference audio $\mathbf{c}_{\text{ref}}$, encoding high-level prosodic and semantic structure.
2. The CFM — a 56-layer Diffusion Transformer (DiT) with hidden dimension 256, operating over 10 denoising steps — transforms these tokens into a mel-spectrogram $\mathbf{m} = f_{\text{CFM}}(\mathbf{z}, \mathbf{c}_{\text{ref}}, \mathbf{v})$, rendering fine-grained acoustic details conditioned on speaker embeddings and reference speech.

A crucial architectural asymmetry: the SLM is not conditioned on explicit speaker embeddings, whereas the CFM is explicitly conditioned on both speaker embeddings and reference speech. This distinction has profound implications for speaker–emotion entanglement at each steering site.

Method

Geometry Analysis

The geometry analysis aims to characterize how emotions are organized in the representation spaces of the SLM and CFM, and in particular whether they form structures that support compositional control. Two complementary tools are used.

Linear Discriminability via Linear Probing

A linear classifier is trained at each layer of both the SLM and the CFM to assess the linear separability of emotion representations. Evaluation is performed under two conditions: within-speaker (training and test speakers overlap) and cross-speaker (held-out speakers). Higher linear probe accuracy indicates more separable emotion representations and, by extension, more reliable steering vectors for compositional steering. A large gap between within-speaker and cross-speaker accuracy signals speaker–emotion entanglement: the emotion representations are not generalizable across identities.

Local Intrinsic Dimensionality (LID)

Linear probing only reveals whether emotions are separable, not the geometric structure of the emotion manifold. Local Intrinsic Dimensionality (LID) addresses this by measuring how quickly the local volume around each sample expands as the neighborhood radius grows.

For each sample's activation representation, the $K$ nearest neighbors (in Euclidean distance) are identified, giving sorted distances $r_1, r_2, \ldots, r_K$. LID is estimated using the Levina–Bickel maximum likelihood estimator:

Higher LID indicates a more complex, higher-dimensional local geometry — emotion information is spread across many directions rather than concentrated in a compact subspace.

LID is computed in two settings:

• Per-emotion: LID estimated within samples of each individual emotion category, capturing each emotion-specific subspace's geometry.
• Pooled: LID estimated over all speech samples jointly, capturing the overall emotion space geometry.

The key diagnostic quantity is:

When $\Delta\text{LID} > 0$, pooling emotions increases estimated manifold dimensionality, meaning different emotions contribute additional independent directions of variation beyond those within individual emotion subspaces. This is favorable for mixed-emotion steering, because it signals that emotion-specific directions exist in representation space and can potentially be composed. When $\Delta\text{LID} < 0$, emotion categories largely lie on a shared acoustic manifold, making clean emotion-specific intervention difficult.

Activation Steering

Building on the geometric analyses, emotion steering vectors are extracted from activations and applied at inference time.

Steering Vector Extraction

For each layer $l$ at either the SLM or CFM, the emotion direction vector is computed as the mean activation difference between emotion-$e$ samples and neutral samples:

where $\mathbf{h}_{e,j}^{(l)}$ and $\mathbf{h}_{0,i}^{(l)}$ denote activations from emotion-$e$ and neutral samples respectively.

SLM steering vector: $\mathbf{v}_e^{(l)} = \mathbf{u}_e^{(l)}$, extracted from attention output activations at the last-token position of complete utterances. Based on geometric findings, steering is applied at SLM layers 14 and 17.

CFM steering vector: $\mathbf{u}_e^{(l)}$ is extracted from residual stream activations, then $L_2$-normalized, masked to the top-$k$ emotion-relevant frames (identified via an emotion classifier), and aggregated. Since emotion discriminability is uniformly distributed across CFM layers, steering vectors are applied at every 5th layer (12 layers total) across all 10 denoising steps.

Mixed-Emotion Composition

For mixed-emotion synthesis, single-emotion vectors are composed via weighted summation:

where $p_e$ denotes the proportion of emotion $e$, with $\sum_e p_e = 1$.

Inference-Time Steering

At inference, the activation at layer $l$ is modified as:

where $\alpha$ controls steering strength and $f_r$ renormalizes the modified activation to preserve the original scale. This is applied without any retraining of the model.

Experimental Setup

Datasets

Three emotion speech corpora spanning five emotions (angry, happy, neutral, sad, surprise) are used:

• ESD — Emotional Speech Dataset
• CREMA-D — Crowd-sourced Emotional Multimodal Actors Dataset
• RAVDESS — Ryerson Audio-Visual Database of Emotional Speech and Song

For linear probing: 30% of speakers are reserved for cross-speaker evaluation (4,530 utterances); 11,311 utterances from remaining speakers are used for probe training; 4,850 utterances for within-speaker evaluation.

For LID: 4,000 utterances are sampled for both per-emotion and pooled estimates, with $k=50$ neighbors, averaged over 10 resampling trials.

Steering vectors are extracted from 50% of speakers. Evaluation is performed on:

• CREMA-D — in-distribution test set
• IEMOCAP — out-of-distribution test set

Evaluation Metrics

Emotion control metrics:

• E-SIM: Cosine similarity between Emotion2Vec embeddings of synthesized and ground-truth speech — measures overall emotional similarity.
• TEP (Target Emotion Probability): Mean probability assigned to target emotions by the Emotion2Vec classifier — measures overall emotion intensity.
• $\rho$ (Spearman correlation): Correlation between the ranking of emotion probability increases and the ground-truth emotion ranking — measures proportional mixed-emotion control.
• H-Rt (Hit Rate): Fraction of samples where the ground-truth dominant emotion shows the largest probability increase — measures fidelity of dominant emotion prediction.

Speech quality metrics:

• S-SIM: Cosine similarity between WavLM speaker embeddings of synthesized and reference speech — measures speaker identity preservation.
• WER: Word error rate via Whisper-Large-v3 — measures intelligibility.

Results: Geometry Comparison

Linear Discriminability

In the SLM: within-speaker accuracy reaches 0.80 and cross-speaker accuracy 0.71, yielding a small mean gap of only 0.08. Discriminability peaks in mid-to-late layers (10–17), indicating that emotion information is concentrated in a localized, speaker-invariant subspace.

In the CFM: within-speaker accuracy is even higher at 0.89, but cross-speaker accuracy is substantially lower at 0.62, resulting in a mean gap of 0.32 — four times larger than the SLM gap. Moreover, discriminability is broadly uniform across all 56 layers and all 10 denoising steps, with no clear peak layer.

These results establish that SLM representations are more separable and generalizable across speakers, while CFM representations are strongly entangled with speaker identity and diffusely distributed — making the SLM a more suitable site for extracting robust, speaker-invariant emotion steering directions.

LID Trends

In the SLM, pooled LID follows a compression–expansion pattern: it first decreases (compression), then increases, and finally stabilizes in later layers. This is consistent with geometric dynamics previously observed in transformer representations more broadly.

In the CFM, LID consistently increases then decreases across layers at every denoising step — intermediate layers construct richer representations with more complex local geometry before compressing toward the prediction target. Additionally, LID progressively decreases across denoising steps, indicating that the representation manifold becomes increasingly structured and lower-dimensional as the CFM iteratively refines toward its final mel-spectrogram output, consistent with flow-matching theory.

Emotion Subspace Structure ($\Delta$LID)

$\Delta\text{LID}$ reveals a fundamental contrast:

• SLM: $\Delta\text{LID}$ is near zero in early layers (0–5), then becomes consistently positive from layer 6 onward (mean: +0.84). Combining emotion categories increases manifold dimensionality — emotions occupy distinct directions and contribute additional geometric structure beyond that of individual emotion manifolds. This is the favorable geometric signature for compositional steering.
• CFM: $\Delta\text{LID}$ is negative across all 56 layers and 10 denoising steps (mean: −1.48). Pooling emotions does not increase dimensionality — emotion categories largely reside on a shared acoustic manifold. This indicates that emotion variations are not encoded as independent directions, making clean compositional emotion steering difficult.

Geometry Summary Table

An important observation: despite the CFM having a lower LID (~13 vs. ~28 for SLM), it shows negative $\Delta$LID. This means the CFM's lower-dimensional manifold is shared across emotions rather than partitioned into distinct emotion subspaces. The SLM's higher-dimensional manifold (~28), in contrast, is structured such that each emotion occupies its own distinct sub-region — a property that directly enables compositional steering.

Results: Steering Comparisons

Three steering configurations are compared: SLM-only, CFM-only, and Joint (both modules simultaneously). For each configuration, two steering strengths $\alpha$ are reported. A no-steering baseline is also included. Performance is evaluated on both CREMA-D (in-distribution) and IEMOCAP (out-of-distribution).

Emotion Control

Overall emotion embedding similarity (E-SIM) and intensity (TEP) improve over the baseline for both SLM and CFM steering, with comparable performance between the two single-site methods. Both sites effectively align generated speech with target emotional embeddings. Joint steering yields the highest TEP across datasets, as combined perturbations reinforce overall emotion intensity.

Proportional control ($\rho$, H-Rt) tells a more nuanced story. SLM steering consistently outperforms CFM on both CREMA-D and IEMOCAP for fine-grained proportional control of each emotion. On CREMA-D: SLM achieves $\rho = 0.209$ vs. CFM's $\rho = 0.193$. On IEMOCAP: SLM achieves $\rho = 0.215$ vs. CFM's $\rho = 0.117$. This is directly consistent with the geometric analysis: the SLM's positive $\Delta$LID and low-dimensional emotion-specific subspaces support cleaner compositional steering, enabling precise control over emotion mixing ratios.

Critically, joint steering degrades proportional control on in-distribution data (CREMA-D joint $\rho = 0.176$ at $\alpha = 2.0$, below SLM-only's 0.209), suggesting that steering both modules simultaneously complicates fine-grained control over individual emotion ratios.

Speech Quality

Speaker identity (S-SIM) degrades noticeably under CFM steering (down to 0.807 on CREMA-D at $\alpha=2.0$), while SLM steering preserves speaker identity nearly perfectly (0.870–0.872 vs. baseline 0.871). This is architecturally expected: the SLM is not conditioned on speaker embeddings, so perturbing its activations does not directly interfere with speaker identity. The CFM, however, is explicitly conditioned on speaker embeddings and reference speech, so perturbing its activations directly disrupts speaker-dependent representations — consistent with the large within–cross gap and speaker–emotion entanglement revealed by geometry analysis.

WER remains stable for single-site methods (SLM slightly elevates WER at high strength; CFM actually reduces WER slightly). Joint steering causes a modest WER increase on CREMA-D (1.06 at joint $\alpha=2.0$ vs. baseline 1.07, minimal change) but is stable on IEMOCAP.

Overall conclusion: SLM steering provides the best balance between controllability (high $\rho$, H-Rt) and preservation (high S-SIM), making it the more suitable single site for emotion steering. CFM steering delivers stronger raw emotion intensity but at significant cost to speaker fidelity. Joint steering boosts intensity but introduces interference that undermines proportional control.

Discussion

Why Joint Steering Fails to Provide Additive Gains

Three compounding factors explain why joint steering does not deliver complementary improvements:

1. Distribution shift: SLM steering moves activations away from the neutral manifold before they reach the CFM module. This causes a mismatch between the distribution on which CFM steering vectors were extracted (neutral baseline) and the actual inference-time activations entering the CFM, especially in mixed-emotion settings where the perturbation is complex.
2. Speaker entanglement: CFM steering additionally perturbs speaker-dependent acoustics due to speaker–emotion entanglement in the flow-matching space. This introduces unwanted speaker variation that is amplified under joint steering.
3. Uncoordinated perturbation: Independent interventions at both sites accumulate noise rather than composing cleanly, leading to interference that reduces proportional control despite increasing overall emotion intensity.

Future Directions

The authors identify several promising directions to address identified limitations:

• Coordinated vector extraction: CFM steering vectors could be extracted conditioned on SLM-steered output, so that the extraction distribution matches actual inference-time conditions.
• Speaker direction orthogonalization: CFM vectors could be orthogonalized against speaker directions (e.g., via null-space projection) to mitigate speaker–emotion entanglement in the flow-matching space.
• Adaptive per-site tuning: Independent per-site $\alpha$ tuning, or frame-level adaptive steering, may allow the two modules to complement each other more effectively by matching their respective contribution magnitudes.
• Generalization to other architectures: Extending this analysis to architecturally distinct hybrid TTS systems (e.g., IndexTTS2) would test whether the geometric findings generalize beyond CosyVoice2.
• Per-layer, per-step analysis: The current geometry–steering relationship is characterized at the module level. Finer-grained analysis per-layer and per-denoising-step could pinpoint the most steerable directions within each module.

Key Contributions and Novelty

This work makes the following contributions to the TTS and controllable speech generation communities:

• First comparative geometry study of SLM vs. CFM as steering sites in hybrid TTS, using linear probing and LID as diagnostic tools that predict and explain practical steering outcomes.
• Introduction of $\Delta$LID as a diagnostic metric for compositional steerability: positive $\Delta$LID in the SLM (mean +0.84) correlates with effective mixed-emotion control; negative $\Delta$LID in the CFM (mean −1.48) correlates with poor proportional control and speaker entanglement.
• First joint steering study across both SLM and CFM modules, revealing that independent multi-site perturbation can interfere rather than complement, providing practical guidance for future multi-site strategies.
• Architectural interpretability insight: the SLM's lack of speaker conditioning explains its speaker-invariant emotion subspaces, while the CFM's explicit speaker conditioning explains its speaker–emotion entanglement — connecting architecture to geometry to steering behavior in a unified framework.
• Practical recommendation: SLM steering is identified as the superior single site for mixed-emotion control due to its combination of high proportional control ($\rho$, H-Rt) and excellent speaker identity preservation (S-SIM), evaluated across both in-distribution (CREMA-D) and out-of-distribution (IEMOCAP) settings.

Limitations

The study's scope and acknowledged limitations include:

• All experiments are conducted on a single backbone architecture (CosyVoice2). Generalizability to other hybrid TTS architectures remains to be established.
• Geometry analysis is performed at the module level (aggregated across layers); per-layer and per-denoising-step granularity could reveal additional structure.
• Joint steering uses the same $\alpha$ for both modules; adaptive per-site strength tuning may alter the conclusions about joint steering interference.
• CFM steering vectors are extracted from the neutral distribution baseline, not from SLM-steered outputs, which may explain part of the distribution mismatch observed during joint steering.
• Evaluation is limited to five discrete emotion categories; the framework's applicability to a wider continuous valence-arousal emotion space has not been tested.