Sympatheia

where $\mathbf{x}$ are the input speech tokens, $z=(v,a)$ is the affect condition, and $\mathbf{y}$ are the target response tokens.

The affect supervision is not a single-point classification problem. The paper uses 12 emotion anchors in the VA plane:

The authors treat these as fixed design anchors rather than universal psychological coordinates. The point is to provide a consistent operating space that covers the major affective regions and allows interpolation between them. At inference time, the model can accept explicit continuous VA values, discrete emotions mapped to anchors, or estimates from external sensors.

To make the model robust to absent or noisy affect cues, the VA condition is randomly dropped for one third of the non-neutral training samples. This encourages the model to remain a capable speech dialogue system even when no external emotion estimate is provided, and to learn to use speech-internal cues when they are the only signal.

Training Data: Sympatheia-18k

The paper’s dataset is a synthetic emotion-conditioned spoken dialogue corpus with two complementary splits:

• Emotional split: roughly 12,000 examples across the 12 target emotions, about 1,000 per emotion.
• Neutral split: 500 neutral user queries, each paired with 12 emotion-conditioned assistant responses, for 6,000 examples total.

Together these yield about 18,000 query–response pairs. The emotional split teaches the model how emotionally expressive user speech should influence response style. The neutral split isolates explicit control: the input speech is neutral, so the desired response emotion must be supplied primarily through the VA prompt.

Construction is fully synthetic and proceeds in stages. First, Qwen3-32B-Instruct generates natural spoken-style user queries and empathetic response texts. The response prompts are designed so that the assistant acknowledges the user’s emotional state and still answers the actual request. Second, Qwen3-TTS synthesizes speech in emotion-consistent styles, providing prosody and pacing that match the intended affect. Finally, each example is paired with the corresponding VA metadata used during training.

The paper emphasizes that the response strategy is not simple emotion mirroring. For example, a frustrated user should receive calm and useful help, an anxious user should get reassurance and concrete guidance, and a tired user should receive low-energy, gentle support. The dataset thus teaches both emotional alignment and task completion.

Important data-generation details include:

• query generation with thinking mode disabled;
• response generation with thinking mode enabled;
• temperature 0.85 for query generation and 0.7 for response generation;
• a 70/30 train–evaluation split at the unique-query level;
• deduplication using all-MiniLM-L6-v2 embeddings with cosine similarity threshold 0.85.

The neutral split is especially important for evaluation: because the query itself contains little affective evidence, the model must rely on the explicit VA control signal. This is the main testbed for controllable emotional response generation.

Multimodal Emotion Sensing Interface

A major contribution of the paper is a shared mechanism for turning heterogeneous emotion recognizers into the same VA representation. For a modality $m$, the classifier outputs a distribution over its native labels, and the system computes a probability-weighted expected anchor coordinate:

where $p_m(y_k \mid x_m)$ is the class probability, and $\mu(y_k)$ is the VA coordinate associated with that class. This has two benefits: it preserves uncertainty, and it allows continuous interpolation between emotions instead of forcing a hard class decision.

The paper groups external affect sources into three families:

• Facial expression: AffectNet+ validation images, using an EfficientNet-style HSEmotion classifier.
• Biosignals: EEG and eye tracking from SEED-VII; ECG and GSR from YAAD.
• Textual affect descriptions: ISEAR self-reports, classified with a DistilRoBERTa-based model.

The system also maps some label sets to nearby anchors when the dataset’s taxonomies do not align exactly. For example, joy/happiness is mapped to happy, fear to anxious, and contempt to disgusted.

Standalone sensing accuracies reported in the appendix are:

• facial expression: 62.9%
• ECG: 41.7%
• GSR: 44.1%
• EEG: 48.4%
• eye tracking: 46.1%
• textual affect description: 64.7%

These are not the main focus of the paper, but they help explain why some sensing pathways improve response quality more than others.

Training Setup

Sympatheia is initialized from GLM-4-Voice-9B and fine-tuned with LoRA. The reported optimization details are:

• LoRA rank 32, $\alpha=32$, dropout 0.1;
• adapters inserted into fused Q/K/V projections, attention output projection, and both feed-forward projections;
• AdamW with $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$;
• maximum sequence length 2048 tokens;
• learning rate $10^{-4}$;
• weight decay 0.01;
• batch size 1 per device, 4 gradient accumulation steps across 4 GPUs, for an effective batch size of 16;
• 50 warmup steps and 5 epochs;
• checkpoints and evaluation every 200 steps;
• final checkpoint selected at step 2800 based on evaluation loss and manual inspection.

Training uses DeepSpeed ZeRO Stage 3, bfloat16 precision, and gradient clipping at 1.0. In the reported experiments, fine-tuning ran on 4 NVIDIA L40 GPUs and took about 8 hours. Dataset generation was more expensive: about 8 hours for query generation, 2 days for response generation, and 12 hours for TTS synthesis.

Evaluation Protocol

The paper evaluates the system from several angles: empathy, emotional appropriateness, prosody, multimodal conditioning, and preservation of general conversational ability. The main automated judge is Qwen3-Omni 30B, which listens to the generated audio so that prosody and pacing are evaluated alongside textual content. The paper also includes a human Emotion MOS study.

The core evaluation settings are:

• Sympatheia-Neutral: neutral user speech, with an explicit target emotion in the system prompt;
• Sympatheia-Emotional: expressive user speech, with no external emotion prompt;
• VoiceBench-CommonEval: real neutral spoken queries from VoiceBench.

For neutral-query settings, Sympatheia uses continuous VA prompts while baselines typically receive a discrete emotion label such as “the user is angry.” For the emotional setting, the model must infer affect from the spoken input itself. The evaluation also compares whether models actually vary their answers across target emotions by measuring pairwise semantic and lexical similarity between responses under different prompts.

The human study uses 20 native English speakers recruited through Prolific. Each participant rates one six-query batch; the full study contains 120 query–emotion trials per system and 840 response-level ratings in total. The study focuses on the neutral-query case, where the user’s speech is emotionally neutral and the target emotion is externally specified. Participants rate how well the response fits and supports the target emotion on a 1–5 scale.

Main Results

The headline result is that Sympatheia outperforms the spoken-dialogue baselines across all three automated empathy settings and in the human Emotion MOS evaluation. The largest gains appear when explicit affect conditioning matters most, but the model also does well when it has to infer emotion from expressive speech alone.

Two patterns stand out. First, Sympatheia’s advantage is especially large in the neutral-query regime, where explicit affect conditioning is needed. Second, the lower semantic and lexical similarity values mean that the model is changing its response content more across target emotions, rather than recycling a near-template answer for every emotion.

Two additional baseline comparisons reinforce the same conclusion:

• A cascaded Qwen3-Omni → Qwen3-TTS system improves over direct Qwen3-Omni in the neutral setting, but still remains far below Sympatheia.
• Providing Qwen3-Omni with a face image or text self-report helps, but Sympatheia still performs better when the same cues are mapped into the VA interface.

In other words, the gain is not just from having a stronger general-purpose model or better prompting. The learned continuous affect conditioning seems to matter.

Prosody Analysis

The paper checks whether the model’s emotional control is reflected in speech acoustics, not only in text. For each generated response, it measures mean and standard deviation of $F_0$, the central 80% range of $F_0$, mean and standard deviation of RMS energy, speaking rate, and spectral centroid. It then computes Spearman correlations with the target valence and arousal values.

The main qualitative pattern is that higher arousal targets produce more dynamic pitch, stronger pitch range, faster rate, and brighter spectral characteristics, while valence also influences energy-related features. Sympatheia shows the strongest and most consistent affect–prosody relationships among the compared systems.

These results suggest that VA conditioning is not only changing lexical choices but also influencing the acoustic realization of the response. In particular, arousal appears to be reflected strongly in pitch and energy dynamics, which is exactly what one would want if the goal is emotionally adaptive spoken interaction.

Multimodal Emotion Sensing Results

The sensing experiments test the system in the complementary scenario where the user’s speech is neutral or ambiguous, but another modality provides affect information. The evaluation is conducted as a three-stage pipeline: estimate a VA cue from the sensing modality, feed that cue to Sympatheia along with a neutral query, and score the response using the audio judge.

Every sensing modality improves response quality relative to omitting the cue. Facial expression and textual self-report produce the largest gains, while physiological signals still help despite being noisier and more person-dependent.

The live facial-expression study is particularly important because it tests the end-to-end pipeline on recordings from 10 human subjects who acted prompted emotions while speaking neutral queries. The live system also improves over the no-cue control, validating that the VA interface can operate in a more realistic interactive setting rather than only on offline datasets.

Ablation Studies and Additional Analyses

The paper includes two targeted ablations. The first asks whether emotion fine-tuning harms ordinary conversational ability. The second tests whether the learned VA space is smooth and robust to noise.

This result argues that affect fine-tuning does not damage the backbone’s general question-answering behavior. In fact, the fine-tuned model slightly improves naturalness and text/audio consistency.

The VA-noise analysis shows a smooth degradation rather than a brittle collapse. As the injected Gaussian noise on valence and arousal increases, the judge score decreases gradually. That suggests the model can tolerate imperfect upstream affect estimates and can use off-anchor conditions meaningfully.

Qualitatively, the paper also reports that the model’s responses vary across target emotions rather than staying nearly invariant. This supports the claim that the model is learning a genuine affect-control mapping, not just a few generic response templates.

Interpretation and Broader Takeaways

The paper’s main technical message is that continuous affect conditioning is a useful intermediate design point between rigid emotion classification and unconstrained dialogue generation. Compared with discrete labels, VA is easier to share across sensors and easier to interpolate across ambiguous states. Compared with unconditioned speech generation, it gives the model a direct handle on emotional style.

Sympatheia’s improvements appear in both semantic and acoustic dimensions. The model does not merely say “I understand you are upset”; it also changes the character of the response, including pitch, energy, and speaking style. The multimodal sensing experiments further show that the same VA interface can integrate very different sources of emotional evidence without changing the dialogue backbone.

The authors also make a careful distinction between emotion recognition and emotion-conditioned response generation. The sensing modules may be imperfect, but the dialogue model is trained to use their outputs as soft, continuous cues rather than hard decisions. This is important because in real systems, affect estimation will often be uncertain.

Limitations and Responsible Deployment

The paper is explicit that the work is still limited by the synthetic nature of the training data and by the compactness of the VA representation. Synthetic dialogue may not capture the full diversity, disfluency, and long-horizon dynamics of real conversation. A single VA coordinate also cannot fully represent blended emotions, fast shifts within a turn, or culturally variable interpretations of emotion labels.

Evaluation is another limitation. Much of the scoring relies on an audio-capable LLM judge, and the paper notes that the data, TTS, and judge are all Qwen3-based, which can introduce family-specific style bias. The authors partially mitigate this by including Qwen3-Omni as a baseline and by reporting human Emotion MOS, but they still call for independent audio judges and broader comparisons.

Deployment raises privacy and safety concerns. Facial expressions, speech, EEG, ECG, eye tracking, and self-reported emotion are all sensitive signals. The paper recommends opt-in sensing, disclosure of what is being collected, minimal retention of raw sensor data, and user override of affect conditioning. It explicitly warns against covert emotion surveillance, diagnosis, manipulation, protected-attribute inference, and high-stakes decisions without separate validation and governance.

Overall, the paper’s conclusion is that continuous affect conditioning is a practical step toward more emotionally adaptive voice assistants, but that robust deployment will require personalized calibration, better real-world human studies, and more careful handling of sensor uncertainty and consent.