Fine-Grained NV-TTS

Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control

This paper adds controllable non-verbal vocalizations (laughter, cries, sighs) to emotional TTS using fine-grained annotations specifying type, frequency, and duration. Unlike coarse-tag baselines, this enables precise emotional sound synthesis with significantly higher expressiveness and emotion recognition accuracy.

• tts
• emotion
• prosody
• audio-driven

Authors: Wangzixi Zhou, Bagus Tris Atmaja, Sakriani Sakti

Categories: eess.AS

Comment: 2025 28th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA)

Published 2026-05-25 · Updated 2026-05-25

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub0 stars0 watchers0 forksnull

Abstract

While current emotional Text-to-Speech (TTS) models have successfully controlled verbal prosody, they often ignore non-verbal vocalizations (NVs), which are essential for authentic human emotion. Although some non-verbal datasets have recently emerged, they often lack high-quality, fine-grained annotations, which restricts a model's ability to precisely control NV generation. To address this limitation, we propose a novel approach for fine-grained non-verbal expression synthesis. We curate and reprocess female NV utterances from the EARS corpus, develop a new annotation scheme using tags to encode NV types, frequencies, and durations, and build an emotional TTS benchmark to demonstrate its effectiveness. Our evaluation shows that while our NV approach leads to minor trade-offs in perceived naturalness, it significantly improves expressiveness (eMOS 4.20) and emotional recognition accuracy (78.8%). Emotion-specific analysis further reveals that NV cues are highly effective for high-arousal emotions like happy (82.5%) and fear (82.7%), and almost perfectly convey sadness (98.3%).

---

Introduction and Motivation

Emotional Text-to-Speech (TTS) synthesis has become a critical research area as conversational AI systems are increasingly expected to deliver emotionally rich, empathetic, and human-like responses. Existing emotional TTS models have made considerable progress in manipulating verbal prosody — controlling pitch, duration, and energy through emotional embeddings derived from linguistic content. However, a significant and often overlooked component of expressive human communication is non-verbal vocalizations (NVs): vocal expressions that carry no linguistic content, such as laughter, screams, sighs, cries, and cheering sounds.

As highlighted by Mehrabian's work on nonverbal communication, NVs are pervasive in spoken interaction and play a central role in conveying affective states. They represent a layer of spontaneous, authentic emotional expression that verbal prosody alone cannot replicate. Achieving truly expressive and human-like emotional TTS therefore requires not only manipulating verbal prosody but also accurately synthesizing appropriate non-verbal vocalizations.

A key obstacle is the state of existing NV datasets and annotation schemes. Current corpora typically use coarse-grained tags such as <laugh> or <cry> to represent entire non-verbal events. While functional, such annotations offer no mechanism for controlling finer properties like the frequency (how many discrete laughs), the duration (how long a cry lasts), or the sub-style (open vs. closed laughter). This lack of fine-grained control severely restricts the expressiveness achievable by trained TTS models.

This paper addresses these limitations through two primary contributions: (1) a curated fine-grained NV expression dataset derived and reprocessed from the EARS corpus with a novel annotation scheme encoding NV type, frequency, and duration; and (2) an emotional TTS system built on Grad-TTS with a specialized NV processing pipeline that exploits these fine-grained annotations to achieve more expressive and emotionally accurate speech synthesis.

Related Work

Non-Verbal Vocalization Datasets

Several NV datasets have been proposed, each with notable limitations for fine-grained controllable emotional TTS:

• NVTTS: A 17-hour open-access dataset derived from VoxCeleb and Expresso, featuring 10 NV types. However, acoustic quality is poor — only 1,525 of 3,642 utterances were deemed noise-free — making it difficult for smaller TTS models to learn without reproducing background noise artifacts.
• AMI Meeting Corpus: Offers 100 hours of meeting recordings with word-level annotations, but covers only a narrow NV range (predominantly laughter and coughs) in a domain-specific context with predominantly non-native English speakers.
• JNV Dataset: A Japanese emotional speech corpus where NVs were generated via ChatGPT prompts. Though high-quality, it is extremely small (only 407 seconds), limited to a tiny speaker set, and uses culturally specific expressions (e.g., the angry NV "onore") that cannot transfer to English TTS.

These limitations — poor acoustic quality, narrow NV scope, and cultural specificity — motivate the construction of a new high-fidelity dataset with precise, fine-grained annotations.

Emotional TTS with Non-Verbal Vocalizations

Recent emotional TTS systems have begun integrating NVs, typically leveraging large language models and proprietary large-scale data. For example, CosyVoice2 is a streaming TTS system that supports breathing, laughing, coughing, and sighing across multiple languages. NVTTS fine-tunes CosyVoice2 on their NV corpus to enable a wider variety of non-verbal sounds. However, both approaches either rely on proprietary instruction datasets or assume access to massive pre-trained models, offering little to conventional TTS frameworks that lack such resources.

Constructing the Fine-Grained Non-Verbal Expression Dataset

Data Source: The EARS Corpus

The authors utilize the EARS (Expressive Anechoic Recordings of Speech) corpus, a high-quality speech dataset featuring 107 speakers from diverse backgrounds, covering emotional speech, reading styles, non-verbal sounds, and conversational speech. For this study, NV utterances from 60 female speakers were selected, covering six NV categories:

• Laughter-open
• Laughter-closed
• Cheering
• Yelling
• Crying
• Screaming

Original audio files were approximately 10–12 seconds long, each containing multiple continuous instances of a given non-verbal sound.

Audio Segmentation

Preprocessing was performed using the pydub library. Each audio file was segmented based on periods of silence with the following parameters:

• Silence threshold: −40 dBFS
• Minimum silence duration: 200 ms
• Silence buffer kept around clips: 100 ms

This segmentation extracted 739 individual utterances (each approximately 2–6 seconds long) from the original 360 audio recordings.

Transcription and Fine-Grained Annotation Scheme

Initial transcriptions were generated using the Whisper speech recognition model, followed by manual verification. The central innovation of this dataset is the fine-grained annotation scheme, which contrasts sharply with the coarse single-tag approach used in prior work:

The annotation scheme encodes three properties:

1. Style: The overall NV category is specified in parentheses, e.g., (Laughter-open), (crying), (cheering).
2. Frequency control (discrete vocalizations): For sounds composed of discrete units (e.g., "ha", "whep"), the number of repetitions directly encodes frequency. For example, <(Laughter-open) ha ha ha> vs. <(Laughter-open) ha ha> specifies three vs. two laughs.
3. Duration control (continuous vocalizations): For sustained sounds (e.g., "wuu", "ah"), the duration is encoded by repeating the final character. For instance, wuu lasts approximately 1 second, with each additional "u" adding roughly 0.2 seconds — so wuuuuu indicates a longer cry than wuu.

The specific transcription tokens used per category are summarized below:

Non-Verbal Emotional TTS Architecture

Backbone: Grad-TTS with Emotion Encoder

The backbone model chosen is Grad-TTS, a diffusion-based TTS model recognized for its high-quality synthesis of reading-style speech. To support emotional synthesis, the model is augmented with an emotion encoder that incorporates emotional embeddings into the generation process.

Following Russell's circumplex model of affect, the authors represent emotion as continuous arousal and valence values rather than discrete emotion labels. This two-dimensional representation provides the model with more fine-grained, continuous control over emotional expression compared to a fixed set of categorical labels.

Specialized Non-Verbal Processing Pipeline

To enable the model to correctly parse and exploit the fine-grained NV annotations, a custom NV processor is integrated as the first stage of the text cleaning pipeline. For a transcription such as <(crying) wuuuuu whep> why you do this to me, the pipeline operates as follows:

1. NV Segment Identification: The processor first identifies the non-verbal segment enclosed in angle brackets: <(crying) wuuuuu whep>.
2. Style Parser: Extracts the overall NV style from the parenthetical tag — in this case, crying.
3. Discrete Unit Parser: Counts occurrences of discrete vocalizations within the segment — here, it counts one instance of whep.
4. Duration Parser: Calculates the duration implied by continuous vocalizations based on character repetition — here, wuuuuu (with four "u"s beyond the base) implies a cry of approximately $1 + 4 \times 0.2 = 1.8$ seconds.

The three parsers collectively produce structured NV tokens that encode style, frequency, and duration information. These tokens are then fed into the emotional Grad-TTS model, allowing the diffusion decoder to synthesize not only verbal speech but also emotionally congruent non-verbal sounds with precise control over their properties.

Experimental Setup

Training Data

A comprehensive 9-hour mixed dataset of English female speech (sampled at 22.05 kHz) was compiled for training the verbal component, drawn from:

• EXPRESSO: An expressive English speech dataset.
• SEMAINE: A multimodal affective interaction corpus.
• ESD (Emotional Speech Dataset): A multi-speaker emotional speech dataset.

For EXPRESSO and ESD, which lack continuous arousal/valence labels, a pre-trained Speech Emotion Recognition (SER) model was used to predict these values automatically.

For the non-verbal component, two distinct NV datasets were used to enable direct comparison:

• The authors' curated fine-grained NV expression data (from EARS).
• The NVTTS corpus (coarse-grained NV annotations) as the comparison baseline.

Acoustic Features and Vocoder

80-dimensional mel-spectrograms were extracted from the processed audio. The vocoder used to convert mel-spectrograms back to audio waveforms is HiFi-GAN. Training ran for 400,000 iterations on a single NVIDIA RTX A6000 GPU (48 GB VRAM).

Evaluation Protocol

Evaluation was subjective, conducted with 15 participants using headphones in quiet environments. Three system designs were compared:

1. Only Verbal: No NV cues; pure emotional prosody via verbal synthesis.
2. Verbal + Coarse-grained NV: NV from the NVTTS corpus, covering only 'happy' and 'sad' (due to the absence of fear/anger NV cues in that corpus).
3. Verbal + Fine-Grained NV: NV from the authors' proposed dataset, covering all four emotions.

To isolate vocal expression from textual content, 20 verbally ambiguous sentences (e.g., "what did you do") generated by ChatGPT were used as input text, preventing emotion inference from the words themselves. Each participant evaluated 60 samples total.

Metrics collected:

• nMOS (Naturalness Mean Opinion Score): Five-point Likert scale (1 = Bad, 5 = Excellent).
• eMOS (Emotion Expressiveness Mean Opinion Score): Five-point Likert scale.
• Emotional Recognition Accuracy: Four-choice task — happy, sad, fear, or anger.

A separate Preference Test was conducted for happy and sad emotions, each presenting four NV expression variants combined with the same verbal utterance. Participants ranked the four variants by preference.

Results and Discussion

Overall Perceptual Evaluation

The overall results reveal a clear trade-off structure:

• Naturalness (nMOS): The "Only Verbal" design achieves the highest naturalness score of 3.54, indicating that adding non-verbal cues incurs a minor perceptual penalty in naturalness. The fine-grained NV system scores somewhat lower, which the authors characterize as a "minor trade-off."
• Expressiveness (eMOS): The fine-grained NV system achieves the highest expressiveness score of 4.20, significantly outperforming both the verbal-only and coarse-grained NV systems.
• Emotion Recognition Accuracy: The fine-grained NV system achieves 78.8% overall accuracy — a 13.3 percentage point improvement over the verbal-only baseline. The coarse-grained NV system performed poorly overall, suggesting its NV cues were ambiguous or even counterproductive for certain emotions.

Emotion-Specific Analysis

Detailed per-emotion nMOS, eMOS, and recognition accuracy are presented below, along with confusion matrices:

Happy

The fine-grained NV system achieves 82.5% recognition accuracy for happy emotions — the highest among the three systems. Notably, the coarse-grained NV system performed especially poorly on happy emotions: its laughter samples were perceived as quiet and subdued, causing many listeners to misclassify them as "sad" rather than "happy." This illustrates how an imprecisely specified or low-energy NV can actively mislead emotion perception. The fine-grained system's cheering sounds (energetic, high-arousal vocalizations) were far more effective at communicating joy.

Sad

Sadness was the most accurately recognized emotion across all systems, with the fine-grained NV achieving near-perfect 98.3% accuracy. Sad emotions appear relatively easy to convey through both verbal prosody and NV cues. The fine-grained system's variety of crying sounds (whep, sneeze, prolonged wuu) provide a distinct emotional signature that listeners consistently and accurately identify.

Angry

Recognition accuracy for anger was 64.3% — the lowest among the four emotions tested with the fine-grained system, and only a modest improvement over the verbal-only baseline. The authors hypothesize this is because the primary NV used for anger was "yelling," which is a general high-arousal expression not uniquely associated with anger (unlike crying, which is uniquely sad). As a result, yelling may be attributed to fear or excitement by listeners. The eMOS for anger (4.06) is also lower than for other emotions (which score above 4.2), consistent with this interpretation.

Fear

The fine-grained system achieves 82.7% recognition accuracy for fear — an approximately 36 percentage point improvement over the verbal-only baseline. This dramatically demonstrates the value of non-verbal cues (screaming) in expressing fear. Fear is a high-arousal emotion that, like happiness, benefits strongly from salient, characteristic NVs.

Non-Verbal Expression Preference Evaluation

A focused preference test was run for happy and sad emotions, each with four NV expression variants.

Happy Emotion Preferences

Four happy NV variants were tested:

• <(cheering) Wo ho>
• <(cheering) Yo>
• <(Laughter-open) ha ha>
• <(Laughter-closed) ha ha>

Both cheering variants were strongly preferred. Laughter expressions were consistently ranked lower — <(Laughter-closed) ha ha> was ranked third by 62% of participants. This indicates that for conveying joy and excitement, energetic cheering vocalizations are more effective and natural-sounding than closed-mouth laughter.

Sad Emotion Preferences

Four sad NV variants were tested:

• <(crying) whep>
• <(crying) sneeze>
• <(crying) wuuuuuuu whep>
• <(crying) wuuuuuuu>

The complex multi-part expression <(crying) wuuuuuuu whep> — combining a prolonged cry with a discrete sob — was ranked first by 56% of participants, making it the clear favorite. Simple, single-element expressions received lower preference ratings: <(crying) wuuuuuuu> (prolonged cry only) was ranked fourth by 56% of participants. This finding validates the importance of designing multi-component NV expressions that more closely mimic the complexity of authentic human emotional vocalizations.

Key Design Insights and Limitations

The results collectively yield several important insights for NV-enhanced emotional TTS design:

• Fine-grained annotation is essential: Coarse-grained single-word tags (e.g., <laugh>) are insufficient and can even degrade emotion recognition if the synthesized NV does not adequately match the target emotion's arousal/valence profile.
• High-arousal emotions benefit most from NVs: Happy (82.5%) and fear (82.7%) showed the largest gains from NV inclusion, while sadness was already relatively well-recognized verbally but further improved by NVs (98.3%). Anger showed only modest gains due to the ambiguity of "yelling" as an anger-specific cue.
• Multi-component NV expressions are preferred for sadness: A structured, multi-part crying expression is perceived as more authentic than simple single-sound variants.
• Cheering outperforms laughter for happiness: Energetic, unambiguous vocalizations such as cheering are more effective than laughter for signaling joy, which can otherwise be misinterpreted as a more neutral or even negative emotion if not synthesized energetically enough.
• Naturalness trade-off is minor but present: The inclusion of NVs causes a slight decrease in naturalness MOS relative to verbal-only speech, which is to be expected when non-verbal segments are appended to otherwise fluent utterances.
• Limitation — anger NV design: The study acknowledges that the current NV cue for anger (yelling) is insufficiently distinctive, as yelling is shared across multiple high-arousal emotions. Future work could explore more anger-specific vocalizations (e.g., growls or harsh exhalations) to improve accuracy.
• Scope: The dataset focuses on a single gender (female speakers) from the EARS corpus, which may limit generalization. The emotional TTS evaluation covers only four emotion categories.

Conclusion

This paper presents a comprehensive approach to integrating fine-grained non-verbal vocalizations into emotional TTS synthesis. The authors curate and reprocess 739 NV utterances across six categories from the EARS corpus, developing a novel annotation scheme that encodes NV style, frequency, and duration through structured transcription tags. This dataset is used to train a Grad-TTS-based emotional TTS system augmented with a specialized NV processing pipeline (style parser, discrete unit parser, duration parser).

Subjective evaluation against verbal-only and coarse-grained NV baselines demonstrates that the fine-grained approach achieves:

• An eMOS of 4.20 (significantly above verbal-only and coarse-grained alternatives).
• Overall emotional recognition accuracy of 78.8% — a 13.3 percentage point improvement over verbal-only synthesis.
• Near-perfect sadness recognition at 98.3%.
• Strong high-arousal emotion recognition: happy at 82.5% and fear at 82.7%.

Preference testing further reveals that listeners prefer complex, multi-component NV expressions for sadness and energetic cheering sounds over laughter for happiness — insights that can directly guide the design of next-generation expressive and human-like emotional TTS systems. The work demonstrates that not merely the presence of NVs, but their precise design and fine-grained control, is crucial for achieving authentic, emotionally congruent speech synthesis.