STEB

STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity

STEB is a speech-to-speech translation benchmark that evaluates both translation fidelity and expressive aspects like emotion, scenario style, and nonverbal vocalizations. It uses a reference-free LLM-based method comparing structured expressive attributes, revealing challenges in preserving expressiveness.

• speech-to-speech
• llm
• multimodal
• emotion
• dubbing
• prosody
• voice-cloning

Demos

These demos showcase evaluation of speech-to-speech translation systems on STEB, focusing on translation fidelity and preservation of expressiveness like emotion and nonverbal vocalizations. Listen for how well systems maintain emotional tone, scenario style, NVs, and timing alignment essential for applications such as video dubbing. The radar chart visually summarizes system performance on these key aspects.

Authors: Sitong Cheng, Weizhen Bian, Songjun Cao, Jin Li, Bei Liu, Chunyang Jiang, Yike Zhang, Weihao Wu, Yiming Li, Chi-Min Chan, Long Ma, Wei Xue

Categories: cs.SD, cs.AI

Published 2026-06-24 · Updated 2026-06-24

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub13 stars0 watchers0 forks
• Hugging Face27 downloads0 likes

Abstract

Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.

---

Introduction

STEB is a benchmark for expressive speech-to-speech translation (S2ST), not just translation fidelity. The paper argues that a translated utterance should preserve three kinds of expressive information in addition to lexical meaning: emotion, scenario style (for example, news reporting, dramatic dialogue, audiobook narration), and nonverbal vocalizations (NVs) such as laughter, breathing, coughing, or crying. This matters for downstream uses like dubbing and conversational speech interfaces, where a semantically correct output can still feel wrong if it loses the source’s affect, scene, or vocal events.

The central challenge is evaluation. For expressive S2ST, there is usually no scalable way to collect a target-speech reference that is both translation-faithful and expressively matched to the source. Because of that, the paper proposes a reference-free evaluation framework that compares structured expressive descriptions of source and hypothesis speech instead of requiring a parallel target recording.

The benchmark covers Chinese$leftarrow$English translation in both directions, spans 32.6 hours of unique data, and is built from six real-world scenarios: drama, audiobooks, advertisements, interviews, news broadcasts, and commentary. The authors evaluate six S2ST systems covering cascaded pipelines, end-to-end models, and speech large language models. The core finding is that many systems do well on translation metrics, but expressiveness preservation remains weak, especially for emotion and NVs.

What STEB Contributes

• A new benchmark for evaluating beyond-fidelity S2ST, with explicit dimensions for emotion, scenario style, and NV preservation.
• A scalable curation pipeline that turns real-world audio into structured annotations using ASR, translation, NV tagging, audio captioning, summarization, and strict quality filtering.
• A reference-free LLM-as-a-judge evaluation method that scores expressive consistency on a 1--5 scale.
• A human validation study showing that the automatic expressiveness scorer correlates significantly with human judgments across all expressive dimensions.
• An empirical study of six S2ST systems demonstrating that semantic transfer is often much better than expressive transfer.

Benchmark Construction Pipeline

Data Collection

The data are collected from publicly accessible web sources and are initially drawn from approximately 80 hours of candidate audio. The final benchmark keeps fewer than 40 hours after filtering, with a selection rate below 50%. The paper emphasizes that the raw material is intentionally diverse and messy: multi-minute or multi-hour recordings, background music, environmental noise, and multiple speakers. That makes utterance-level annotation impossible without preprocessing.

Preprocessing and Cleaning

The preprocessing pipeline first converts all audio to 16 kHz mono WAV. It then applies:

• Source separation using BS-Roformer to isolate foreground speech from music and environmental noise.
• Speaker segmentation using a joint diarization setup: Silero VAD for speech activity, pyannote for speaker change boundaries, and CAM++ embeddings for same-speaker clustering. Adjacent same-speaker segments are merged.
• Duration filtering to retain only utterances between 3 and 30 seconds.
• Quality filtering that removes narrowband/telephone-quality audio, segments with insufficient SNR or clarity, and segments with DNSMOS below 3.0.
• Language identification using Whisper; only Chinese and English segments are kept.

The appendix adds more detail: the narrowband check rejects audio with effective bandwidth below 8 kHz, the SNR and clarity filters use per-frame statistics with thresholds on the 25th percentile and mean, and the filtering pipeline is intentionally strict to prioritize annotation reliability over quantity.

Annotation Strategy

Each retained utterance is annotated for multiple dimensions. The paper’s annotation pipeline combines task-specific models and multimodal LLMs:

• ASR with Qwen3-ASR to obtain the source transcript.
• Force alignment with Qwen3-ForceAlign to produce word-level timestamps.
• Translation with Qwen3-30B-A3B for Chinese$rightleftarrows$English text translation.
• NV detection with BEATs in the PretrainedSED framework, with event markers inserted according to overlap with word intervals.
• Audio captioning with Qwen3-Omni-Captioner to generate rich descriptions of emotion, style, pacing, energy, formality, and timbre.
• Summarization with Qwen3-30B-A3B to compress the caption into structured emotion and scenario style fields.

The authors explicitly note that the caption-then-summarize decomposition is designed to make expressive attributes easier to judge than if one relied on a monolithic label or raw audio comparison. Emotion is treated as a concise vocal state with category, intensity, and nuance. Scenario style is treated as a scene or genre description, such as news or drama. NVs are tracked at the event level rather than as a simple binary attribute.

Quality Assurance

Quality assurance combines automatic multimodal scoring with human validation. A strong multimodal LLM listens to each sample and scores the transcript, translation, emotion, style, and NV annotation accuracy on a 1--5 scale. The benchmark uses strict acceptance thresholds: for the normal subset, transcription, translation, emotion, and style must all be perfect or near-perfect, and for the NV subset the criteria focus on transcription, translation, and NV tagging.

The paper reports that among annotated candidates, 88.3% receive emotion score 5 and 91.4% receive style score 5 from the automatic quality gate. On a random subset of 170 samples, when the automatic score is above 4, the corresponding human score is also above 4 in 100% of transcript cases, 91.3% of translation cases, 86.7% of emotion cases, 88.8% of style cases, and 75.6% of NV cases.

Human annotators are bilingual, experienced in speech annotation, and review both speech quality and annotation accuracy. The appendix also notes that annotators consented to the use of their annotations for research and were compensated.

Benchmark Statistics

• Normal subset: 20,370 utterances, 32.27 hours.
• NV subset: 901 utterances, 1.26 hours.
• Total unique duration: 32.6 hours.

The NV subset overlaps with the normal subset, so the unique duration is not the sum of the two durations. Drama is the largest scenario source because it provides dense emotional variation, while the other scenarios increase coverage of scene style.

Reference-Free Expressiveness Evaluation

Because matched expressive target speech is hard to obtain at scale, the benchmark does not evaluate expressiveness against a ground-truth target audio reference. Instead, it evaluates whether source and hypothesis speech are consistent in the expressive attributes that matter.

Judge Pipeline

The evaluation pipeline first applies the same caption-then-summarize process to the translated speech hypothesis. In parallel, NVs are detected in the hypothesis with BEATs and aligned to the output transcription. The resulting source and hypothesis structured descriptions are then compared by an LLM judge using dimension-specific rubrics.

For each sample, the judge scores:

• Emotion: whether the core emotion category, intensity, and subtle attitude match.
• Scenario style: whether the scene or genre matches, with formality and delivery manner as supporting cues.
• NV preservation: whether event type, count, order, and approximate position are preserved, and whether new salient events were added.

The paper explicitly argues against simple label-matching classifiers because expressiveness is not just a closed-set classification problem. For example, two clips may share the same emotion label but differ substantially in intensity; style depends on scene and delivery, not just text topic; and NVs require event-level comparison. The benchmark therefore frames expressiveness as a contextual, comparative, graded scoring problem.

Scoring and Aggregation

The judge uses a 1--5 rubric for each dimension. Emotion and style are judged as a similarity score between structured descriptions. NVs are judged by event preservation, with special penalties for omissions and newly inserted salient events. To reduce variance, each sample is scored three times. The final score is obtained by an outlier-aware rule: identical scores are kept, large outliers can be removed, and all-different triples are aggregated by the median.

In implementation, the text judge is Qwen3-30B-A3B served via vLLM with default decoding and a maximum generation length of 2,048 tokens.

Human Validation of the Judge

The human correlation study uses 160 source-hypothesis pairs scored by multiple annotators. The authors compare three automatic judge designs:

• Summary-based scoring: caption-then-summarize annotations plus an LLM judge.
• Caption-based scoring: the judge uses the full captions directly.
• Audio-based direct scoring: a multimodal LLM listens to both clips and scores them directly.

The summary-based design is the only one that achieves statistically significant correlation with human judgments on all three dimensions. Direct audio-based and caption-based designs do not reach significance in this study.

These results support the paper’s design choice: decomposing speech into structured attributes before judging is more reliable than directly comparing raw audio or long captions. NV is the most stable and least subjective expressive dimension, while scenario style is the most subjective and therefore the weakest of the three for the judge.

Experimental Setup

The benchmark evaluates six S2ST systems:

• Three-Stage: ASR + NV detection, text translation with style instruction generation, then voice-cloning TTS.
• Two-Stage: direct audio-to-text translation with a multimodal LLM, then voice-cloning TTS.
• UniSS: a unified expressive S2ST model.
• SeamlessExpressive: a multilingual end-to-end S2ST model with prosody-aware design.
• Seed LiveInterpret 2.0: a commercial simultaneous speech translation system.
• Step-Audio 2: an end-to-end speech large language model.

Metrics fall into three groups:

• Translation fidelity: BLEU, COMET, and XCOMET. For speech outputs, the generated audio is first transcribed by ASR and then scored against the reference translation.
• Voice and timing: speaker similarity (SIM) from WavLM/ECAPA-TDNN embeddings, and speech length compliance $\mathrm{SLC}_{\epsilon}$ with $\epsilon \in \{0.2, 0.4\}$.
• Expressiveness: emotion, style, and NV scores on a 1--5 scale.

For duration alignment, the compliance metric is the fraction of utterances whose output-to-input duration ratio lies in $[1-\epsilon, 1+\epsilon]$. In other words, it measures how often generated speech stays within a specified timing tolerance around the source.

Main Results

The results are reported separately for Chinese$rightarrow$English and English$rightarrow$Chinese. Across both directions, the same broad pattern appears: translation quality is reasonably strong, but expressive transfer is much weaker.

Chinese$rightarrow$English

In this direction, the two cascaded systems achieve the strongest translation scores, with Two-Stage reaching 41.59 BLEU. But their emotion scores are only around 1.67--1.73, showing a large gap between semantic correctness and expressiveness. Style scores are higher, near 3.9 for cascaded models and above 4.0 for several end-to-end models, but still not consistently high. NV preservation is modest overall, with the best score only 2.31/5 from Two-Stage. Duration alignment is best for UniSS.

English$rightarrow$Chinese

For English$rightarrow$Chinese, translation fidelity is again strong: Two-Stage reaches 59.66 BLEU. Emotion preservation remains weak for cascaded systems, again around 1.68--1.70, while end-to-end systems rise to the 3.27--3.82 range. The best style score is 4.46 from Step-Audio 2. The best NV score is 2.21 from Three-Stage. UniSS again dominates duration alignment, with SLC$_{0.2}$ of 0.980 and SLC$_{0.4}$ of 0.990.

Intermediate Text Translation Analysis

The paper also evaluates the intermediate text translations before speech synthesis. Those scores are close to the final speech-output scores, which supports the authors’ claim that the main bottleneck is not semantic transfer alone, but expressive preservation across the full S2ST pipeline.

The gap between text translation and speech translation is small for most systems, again suggesting that the main failure modes are not just lexical transfer errors. Instead, expressive information is lost or altered during synthesis or across the speech pipeline.

Interpretation of the Findings

• Translation fidelity is not the bottleneck. Many systems, especially cascaded ones, reach strong BLEU/COMET/XCOMET values.
• Emotion is the hardest expressive dimension. Cascaded systems perform especially poorly, and even the best systems remain far from perfect.
• Scenario style is easier than emotion but still imperfect. Scores are generally higher than emotion and often above 4 for the better systems, but the paper still treats style as a meaningful open challenge.
• NVs benefit from explicit representation. Systems that explicitly carry NV tags into translation do better than end-to-end systems, but the best score is still only 2.31/5.
• Duration alignment requires dedicated control. UniSS is the clear winner here, which matches the model’s design.
• Adding style instructions to TTS does little. The w/o Instruct variants are almost unchanged from the full cascades, suggesting that simply giving TTS a style instruction is not enough to recover source expressiveness.

Limitations and Scope

• The benchmark currently covers only Chinese$leftrightarrow$English. The authors argue the pipeline is language-agnostic and can be extended, but that is future work.
• Expressiveness evaluation depends on a captioning model, so judge quality is bounded by the perceptual abilities of the audio-language stack.
• Scenario style is acknowledged as the most subjective expressive dimension and remains the hardest for the judge.
• The benchmark is built from publicly accessible web audio, and the paper notes that audio without redistribution permission is not redistributed; in those cases, only metadata and annotations are released.
• Samples with private or sensitive content are removed during filtering and validation.

Bottom Line

STEB reframes S2ST evaluation around a more realistic question: not just did the model translate the words correctly, but did it preserve how the source sounded? The benchmark’s results show a consistent gap between semantic transfer and expressive transfer. In the authors’ evaluation, current systems can already produce competent translations, but emotion and NV preservation remain weak, and even the best systems still fall short of faithful expressive translation. The paper’s main technical contribution is therefore both the benchmark itself and the validated reference-free method for scoring expressive consistency.