SpeechRole

SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents

SpeechRole presents a large dataset and benchmark to evaluate speech role-playing agents by measuring their ability to generate character-consistent voice, prosody, and dialogue coherence in speech-to-speech interactions. It uniquely focuses on paralinguistic cues and multidimensional speech evaluation beyond text.

• speech-to-speech
• prosody
• voice-cloning
• tts
• dialogue
• agent
• emotion

Demos

The demos showcase SpeechRole's ability to synthesize speech role-playing agents with coherent and contextually appropriate dialogue across various characters and languages. Watch for natural conversational flows, convincing character voice roles, and the system's capacity to handle both emotional nuance and diverse linguistic contexts as highlighted in the dataset examples.

Authors: Changhao Jiang, Jiajun Sun, Yifei Cao, Jiabao Zhuang, Xinmeng Che, Hui Li, Xiaoran Fan, Ming Zhang, Junjie Ye, Shihan Dou, Zhiheng Xi, Jingqi Tong, Yilong Wu, Baoyu Fan, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang

Categories: cs.CL

Published 2025-08-04 · Updated 2026-03-26

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub10 stars0 watchers1 forks
• Hugging Face4 downloads0 likes

Impact

• Semantic Scholar4 citations1 influential citation

Abstract

Speech is essential for realistic role-playing, yet existing work on role-playing agents largely centers on text, leaving Speech Role-Playing Agents (SRPAs) underexplored and without systematic evaluation. We introduce SpeechRole, a unified framework for developing and assessing SRPAs. SpeechRole-Data contains 98 roles and 111k speech-to-speech conversations with rich timbre and prosodic variation, providing large-scale resources for training SRPAs. SpeechRole-Eval offers a multidimensional benchmark that directly evaluates generated speech, preserving paralinguistic cues and measuring interaction ability, speech expressiveness, and role-playing fidelity. Experiments show that end-to-end SRPAs such as GPT-4o Audio achieve strong fluency and naturalness, but remain limited in prosody consistency and emotion appropriateness. In contrast, current open-source end-to-end models exhibit substantial performance gaps across multiple evaluation dimensions. Cascaded and end-to-end systems achieve comparable results in interaction ability and role-playing fidelity, suggesting that these aspects are still largely influenced by the underlying text-based language models. We release all data, code, and evaluation tools at https://github.com/yuhui1038/SpeechRole.

---

Introduction

SpeechRole addresses a gap in role-playing research: most prior work on role-playing agents is text-centric, while real spoken role-playing depends on voice, prosody, emotion, and character-specific vocal identity. The paper defines Speech Role-Playing Agents (SRPAs) as speech-to-speech systems that must do more than generate linguistically coherent answers; they must also preserve paralinguistic cues such as timbre, pitch, rhythm, and intonation, and they must do so consistently across interactions.

The framework has two parts: SpeechRole-Data, a large synthetic speech dialogue corpus for training, and SpeechRole-Eval, a benchmark for direct speech-level evaluation. The authors argue that speech-to-text conversion is insufficient for SRPAs because it discards the vocal information that is central to character portrayal. Their evaluation therefore operates on generated speech directly and scores both dialogue competence and expressive speech behavior.

The main contributions claimed in the paper are: (1) a 111k-sample, 98-role speech-to-speech dataset; (2) a multidimensional benchmark that evaluates speech directly; and (3) a systematic comparison of cascaded and end-to-end SRPA architectures, including a task-specific fine-tuned model.

SpeechRole-Data: Dataset Construction

SpeechRole-Data is designed to fill the lack of scalable, persona-consistent speech dialogue data. Because authentic character dialogue audio is hard to collect at scale and often constrained by copyright, the authors build the dataset through a controlled synthetic pipeline. The pipeline combines LLM-generated role-consistent dialogue text with TTS-based speech synthesis and voice cloning.

Roles and metadata

The dataset covers 98 roles drawn from television, film, animation, and games. The selection includes 18 roles from ChatHaruhi and 80 roles from RoleLLM. For each role, the authors collect structured persona grounding in three parts: role profile (temperament and preferences), background (identity, relationships, and setting), and character lines (dialogue and monologue excerpts). This metadata is used to steer dialogue generation toward consistent characterization.

The paper reports that the English portion is larger overall, with film characters especially prominent, while the Chinese subset is more balanced across source types. The dataset aims to expose SRPAs to diverse voices, contexts, genders, and character archetypes.

Text dialogue generation

The authors use gpt-4.1-2025-04-14 to generate both single-turn and multi-turn dialogues for each role. For every role, they generate approximately 800 single-turn and 800 multi-turn dialogues before filtering. The generation prompts explicitly specify the role, the source script, the available context, and the requirement that the dialogue remain natural, in-character, and grounded in the supplied persona material.

To reduce repetitiveness, they perform semantic deduplication with embedding similarity. English dialogues are compared using all-MiniLM-L6-v2, with a near-duplicate threshold of 0.9. Chinese dialogues use text2vec-bge-large-chinese, with a threshold of 0.85. Only one item from each near-duplicate cluster is retained.

The prompt templates share three core design principles stated by the authors: they explicitly ground the target role and context, impose a clear structured output format, and define the dialogue constraints so that generated samples are easier to evaluate and more reliable for later training.

Voice collection and speech synthesis

To preserve character voice identity, the authors collect reference audio from source material and carefully preprocess it. They note that some audio comes from sources such as Genshin Impact, whose assets permit non-commercial academic use. Other audiovisual works are used only for short speech fragments and are not redistributed. All audio is anonymized, segmented into short utterances, and converted to mono 16 kHz WAV.

Cleaning is performed with the Emilia framework, which does source separation, speaker diarization, and voice activity detection. The resulting clips are constrained to 3–10 seconds and are filtered using DNSMOS P.835 OVRL; clips with overall quality below 3 are discarded. For speaker identification, the pipeline transcribes clips with Whisper-large-v3-turbo, then uses gpt-4.1-2025-04-14 and DeepSeek-V3-0324 to infer the most likely character identity from transcript and context, followed by manual verification.

For each role, the authors choose a representative reference voice by ranking clips with CAM++ speaker embeddings and selecting the clip with the highest average cosine similarity to the others for that role. This clip becomes the timbral and prosodic reference for voice cloning.

Role utterances are synthesized with three TTS backends—CosyVoice2, F5-TTS, and E2 TTS—each conditioned on the selected reference clip. Generating multiple synthesized versions helps reduce model-specific artifacts. User utterances are synthesized with VolcEngine using a fixed voice, since the user does not need persona-specific vocal variation.

The duration analysis shown in the paper indicates that role utterances are typically longer and more expressive than user utterances. Role clips are generally concentrated around 10–20 seconds, while user clips are mostly around 3–6 seconds.

Dataset scale and splits

SpeechRole-Data contains 111k speech-to-speech conversations across the 98 roles. The dataset is split into training, out-of-domain, and test partitions. The training split uses 78 roles, while the out-of-domain split contains 20 unseen roles for generalization analysis. The test split covers all 98 roles with four curated dialogue sessions per role, yielding 392 dialogue-level evaluation instances, 1,448 role utterances, and about 4.2 hours of speech.

The paper emphasizes that this scale is especially useful for SRPA benchmarking because the test set is dialogue-dense and includes multiple role utterances per evaluation instance, not just single prompts.

Task Structure in SpeechRole

The prompts in SpeechRole are organized around three major task categories, which the authors further divide into eight subtask types. This structure is intended to stress different aspects of in-character behavior, ranging from internal reasoning to experiential narration and social communication.

According to the appendix, internal reasoning covers questions about inner states and motivations, experiential narration focuses on actions, skills, and background descriptions, and social communication probes interpersonal judgment and life advice. The task mix is intended to capture a broad range of role-playing behaviors rather than a single narrow interaction style.

SpeechRole-Eval: Direct Speech-Level Benchmarking

SpeechRole-Eval is designed to assess SRPAs in three dimensions: interaction ability, speech quality and expressiveness, and role-playing fidelity. These are operationalized through eight metrics:

• Instruction Adherence (IA) and Conversational Coherence (CC) for interaction ability.
• Speech Fluency (SF), Speech Naturalness (SN), Prosodic Consistency (PC), and Emotion Appropriateness (EA) for speech quality and expressiveness.
• Personality Consistency (PeC) and Knowledge Consistency (KC) for role-playing fidelity.

The appendix defines these metrics in plain terms. IA asks whether the response follows the instruction and remains fully in character; CC checks logical consistency with the dialogue history; SF and SN judge articulation quality and synthetic-artifact freedom; PC and EA measure whether pitch, stress, intonation, and emotional cues match the role and context; and PeC and KC capture whether personality traits and background knowledge are preserved.

LLM-based pairwise evaluation

Instead of absolute scalar grading, the benchmark uses pairwise comparison to reduce scale ambiguity. For each test case, the judge compares a candidate system response against a high-quality reference response. The judge is prompted to produce a brief rationale before a final paired score, which is meant to improve interpretability and consistency.

The judge model is gemini-2.5-pro, and the scoring range is 1 to 10. Each evaluation instance uses three reference variants synthesized with different TTS backends. For reference $j$, the metric uses the ratio between the test score and the reference score, and the final score averages across the three references and then across all $N$ instances:

$$ \text{Score} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{M} \sum_{j=1}^{M} \frac{s_i^{\text{test}}}{s_{i,j}^{\text{ref}}} \right), \quad M=3. $$

The authors justify the ratio-based formulation as a way to stabilize scores across prompts and reduce bias from any single synthesis backend. They also note that the LLM produces system-level rankings rather than relying on a single absolute truth answer.

The prompt explicitly asks the judge to evaluate the response as direct role-play of the target character, compare two model outputs, and produce both qualitative reasoning and paired quantitative scores. This matters because the benchmark is sensitive to prosody and other speech cues that would be lost in text-only evaluation.

Models and Training Setup

The benchmark covers eight SRPA systems spanning cascaded and end-to-end paradigms, plus an out-of-distribution evaluation setting for the fine-tuned model. The experiments are conducted in both English and Chinese, though some systems are English-only.

Cascaded systems

Cascaded SRPAs follow the classic ASR $$ text LLM $$ TTS pipeline. The paper evaluates open-source variants using Whisper-large-v3-turbo for transcription, one of Qwen3-8B, Llama-3.1-8B, or Mistral-7B for role-conditioned text generation, and F5-TTS for speech synthesis with reference-based voice cloning. The paper also includes a proprietary cascaded baseline built from Alibaba Cloud API, combining Paraformer Realtime ASR v2, Qwen-Plus-Character, and CosyVoice-v2.

End-to-end systems

End-to-end systems generate speech directly from speech input, which can reduce latency and error accumulation. The evaluated models are GPT-4o Audio, Qwen2.5-Omni-7B, and LLaMA-Omni. The latter is English-only and therefore excluded from Chinese evaluation.

SpeechRole-Agent fine-tuning

To test the value of task-specific speech-role training, the authors fine-tune Qwen2.5-Omni-7B on SpeechRole-Data, producing SpeechRole-Agent. They also evaluate an OOD variant on the 20 unseen roles. The appendix states that the fine-tuning script uses bfloat16 mixed precision, learning rate $10^{-4}$, gradient accumulation, periodic evaluation every 500 steps, and checkpoint management. The full run takes about 15 hours on 8×H100 GPUs.

Experimental Results

The paper reports results on all eight metrics plus an overall average for both English and Chinese. A key finding is that the strongest systems on speech naturalness and fluency are not always the strongest on role fidelity, and that the gains from end-to-end speech generation do not automatically translate into better character consistency.

English results

In English, Llama-3.1-8B has the best overall score in the reported table, while GPT-4o Audio stands out for speech fluency and naturalness. The open-source end-to-end models trail the stronger cascaded systems on most dimensions, especially on interaction ability and role fidelity. Fine-tuning improves SpeechRole-Agent substantially over base Qwen2.5-Omni, but it still does not close the gap to the best cascaded or proprietary models in English.

Chinese results

In Chinese, Alibaba Cloud API has the highest overall score. The fine-tuned SpeechRole-Agent is nevertheless strong: it is competitive with the best models across the expressive speech metrics and clearly improves over the base Qwen2.5-Omni. The OOD variant remains close to the in-domain variant, suggesting that the model learns transferable role-playing behavior rather than mere memorization of training characters.

The paper’s qualitative interpretation of the table is that GPT-4o Audio is especially strong in fluency and naturalness, while open-source end-to-end systems still show large gaps across multiple metrics. At the same time, cascaded and end-to-end systems can be closer on interaction ability and role-playing fidelity, which the authors interpret as evidence that these dimensions are still heavily shaped by the underlying text language model.

A notable pattern is that prosody consistency and emotion appropriateness remain harder than basic fluency. This matches the paper’s claim that the best end-to-end systems can sound natural yet still struggle to produce emotionally and prosodically appropriate role speech.

Human–LLM Agreement Study

To validate the automatic benchmark, the authors run a human study on a randomly sampled subset of SpeechRole-Eval. Three expert annotators independently rate model outputs on the same eight dimensions, using a 1–5 Likert scale. The annotators are fluent in the relevant evaluation language and have experience with dialogue or speech quality evaluation.

The paper reports strong alignment between human judgment and the Gemini-based automatic evaluation. The table below shows Spearman’s $\rho$, Kendall’s $\tau$, and a pairwise Human–LLM Agreement score for English and Chinese.

The authors summarize these results as evidence that the automatic evaluation produces system-level rankings that are broadly consistent with expert human judgments. In the main text they also report that Spearman’s $\rho$ ranges from 0.67 to 0.98 in English and from 0.64 to 0.96 in Chinese, while Kendall’s $\tau$ ranges from 0.50 to 0.93 and 0.43 to 0.90, respectively. The agreement scores exceed 0.75 for most dimensions.

Interpretation and Takeaways

The paper’s central takeaway is that SRPAs are evaluated by a combination of linguistic, acoustic, and persona-preservation requirements, and that these requirements do not improve uniformly across architectures. End-to-end systems, especially proprietary ones, can sound more fluent and natural, but role fidelity and emotional/prosodic control remain difficult. Cascaded systems can match or exceed end-to-end models on interaction and role consistency when they are backed by strong text LLMs, which supports the authors’ claim that some core role-playing abilities still come primarily from the language model rather than the speech pipeline itself.

SpeechRole-Agent shows that task-specific training on role-conditioned speech data helps close part of the gap. In English it improves clearly over the base Qwen2.5-Omni, and in Chinese it becomes competitive with the strongest reported systems on several metrics. The OOD result indicates that the benefit is not limited to memorizing seen roles.

The benchmark also illustrates why a direct speech-level evaluation is useful: the paper can separately diagnose fluency, naturalness, prosody, emotion, personality, and knowledge consistency instead of compressing them into a single text-based score.

Limitations

• The dataset and experiments are primarily limited to English and Chinese, so the benchmark does not yet cover broader multilingual or cultural settings.
• The evaluation speech is synthesized with TTS rather than collected from natural human recordings, so the benchmark may not capture the full variability and acoustic complexity of real-world speech.
• The automatic evaluator depends on a strong proprietary judge model, so it may still inherit biases or blind spots even though the human agreement is high.
• The reported results are mostly system-level averages; the benchmark does not directly isolate fine-grained controllability, long-term character drift, or adversarial/ambiguous role instructions.
• SpeechRole-Agent is built by fine-tuning an existing end-to-end model, so future work could explore alternative architectures, training objectives, and tighter language–speech integration.

Ethics and Data Handling

The paper states that no newly annotated human-labeled dataset is released. Human annotation is used only for the agreement study, and those annotations are not included in the released resources. The evaluated material consists of fictional role-playing dialogues, no personal data is collected, and the authors argue that the work raises no significant ethical concerns. They also note that they use AI assistants only for text refinement, not for study design, data collection, analysis, or scientific conclusions.

Conclusion

SpeechRole is presented as a unified foundation for SRPA research: a large synthetic dataset for training and a speech-native benchmark for evaluation. The paper’s evidence suggests that today’s best systems can already produce fluent and natural speech, but reliable prosody control, emotionally appropriate delivery, and stable persona expression remain open problems. By releasing data, prompts, evaluation code, and a validated LLM-based benchmark, the authors aim to make SRPA development more systematic and comparable across future models.