VoxParadox

Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

VoxParadox exposes how Audio LLMs prioritize text over acoustic cues through an adversarial benchmark that mismatches transcripts with speech paralinguistics. Layer-wise probing identifies representation degradation, while PCLM and DPO jointly recover acoustic grounding that standard benchmarks fail to measure.

• llm
• multimodal
• prosody
• emotion
• asr
• audio-driven
• reasoning

Demos

VoxParadox reveals a critical failure mode in audio LLMs: they frequently ignore acoustic cues (prosody, emotion, speaker identity) when transcript semantics conflict, prioritizing language-implied answers instead. The interactive demo invites you to listen to audio clips where the true paralinguistic label (e.g., "elderly speaker," "angry tone") contradicts what the transcript asserts, then observe how unmodified models fail this acoustic ground truth test while improved models with PCLM+DPO achieve substantial gains.

Authors: Jiacheng Pang, Ashutosh Chaubey, Mohammad Soleymani

Categories: cs.SD, cs.LG

Comment: Accepted as a conference paper at ICML 2026. Project page: https://voxparadox.github.io/

Published 2026-05-26 · Updated 2026-05-26

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3 stars0 watchers0 forksUSC Research License
• Hugging Face286 downloads3 likes

Impact

• Semantic Scholar1 citation1 influential citation

Abstract

Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.

---

Introduction and Motivation

Speech is a rich communicative signal carrying not just what is said (lexical/semantic content) but also how it is said — paralinguistic attributes such as emotion, age, gender, pitch, volume, speaking rate, intonation, and speaker identity. These non-verbal cues are central to human communication and have been studied for decades in computational paralinguistics. Modern Audio Large Language Models (Audio LLMs) couple powerful speech encoders with large language model backbones, enabling conversational audio understanding and instruction following. However, their ability to reliably exploit paralinguistic information — especially when it conflicts with the lexical content of speech — has remained poorly characterized.

Existing audio benchmarks such as MMAU, MMAU-Pro, MMSU, and MMAR emphasize broad audio understanding but do not explicitly decouple lexical evidence from acoustic ground truth. This means a model can score well by relying on words alone, without ever truly "listening" to how something is said. Spoken-language benchmarks like MMSU do include paralinguistic categories, but they do not enforce a strict mismatch between what is said and how it sounds.

This paper introduces VoxParadox, a carefully constructed adversarial benchmark that forces a controlled contradiction between transcript content and vocal delivery, enabling a precise measurement of whether Audio LLMs rely on acoustic cues or default to transcript-implied shortcuts. Alongside the benchmark, the authors conduct rigorous layer-wise probing to understand where paralinguistic information is lost or ignored, and then propose Prompt-Conditioned Layer Mixer (PCLM) combined with Direct Preference Optimization (DPO) to substantially improve paralinguistic understanding.

The VoxParadox Benchmark

Design Philosophy

VoxParadox contains 2,000 verified multiple-choice examples spanning 10 paralinguistic tasks, with 200 examples per task. Every example is built around a deliberate, controlled contradiction between two labels:

• True label ($y_{\text{true}}$): the ground-truth paralinguistic attribute conveyed by the speech clip (the correct answer, derivable only from acoustics).
• Adversarial label ($y_{\text{adv}}$): the attribute explicitly asserted by the transcript content, intentionally set to conflict with $y_{\text{true}}$.

For instance, an elderly-sounding voice may say "I am a young person," or multiple synthesized speakers may all insist "there is only one person speaking." By decoupling what is said from how it sounds, VoxParadox exposes overreliance on language — correct answers must be inferred from non-verbal acoustic content. This adversarial-by-design approach is analogous to contradiction benchmarks in the vision-language community (e.g., GVQA) that expose modality shortcuts.

The Ten Paralinguistic Tasks

The benchmark covers:

• Age prediction — speaker claiming wrong age group (young vs. old)
• Gender prediction — speaker claiming wrong gender (male vs. female)
• Emotion recognition — vocal delivery contradicts stated emotion (happy/sad, angry/neutral)
• Pitch comparison — three concatenated segments with deterministic pitch shifts; transcript claims wrong ordering
• Volume comparison — three segments with gain-adjusted volume; transcript mismatches ordering
• Speed comparison — three segments with time-stretching; transcript mismatches ordering
• Vocal range comparison — three segments with range scaling; transcript mismatches ordering
• Intonation perception — SSML-controlled pitch contour (rising vs. falling); transcript asserts opposite
• Speaker identity recognition — multi-speaker conversation; transcript identity statements mislead toward wrong target
• Speaker counting — multi-turn TTS conversation; all speakers falsely claim a different number of speakers

Data Creation Pipeline

The pipeline uses GPT-4o to generate adversarial transcripts that explicitly assert $y_{\text{adv}}$ while excluding $y_{\text{true}}$. Acoustic attributes are enforced deterministically through three TTS engines, each suited to different tasks:

• ElevenLabs: used for age and gender tasks, where fixed speaker metadata guarantees the correct vocal profile.
• GPT-4o TTS: used for speaker counting, speaker identity, emotion, and signal-comparison tasks.
• Microsoft Azure SSML: used for intonation tasks, where pitch contours are specified via SSML markup.

For signal-comparison tasks (pitch, volume, speed, range), a seed utterance is synthesized and then two additional variants are created via deterministic signal processing (time-stretching, gain adjustment, pitch shifting, range scaling). Three resulting segments are concatenated in a random permutation, and the question asks which pattern best describes the ordering.

Verification and Quality Control

Quality assurance is multi-layered:

1. Transcript fidelity: Whisper large-v3 is run on every generated clip, and only clips with exact transcript match (word error rate $= 0$) are retained. For all non-emotion tasks, this is sufficient because acoustic attributes are enforced by construction.
2. Emotion filtering: a SpeechBrain Wav2Vec2-based speech emotion recognition (SER) model is applied to the reversed audio (to reduce reliance on lexical content) of each emotion clip; only clips where the top-1 prediction matches $y_{\text{true}}$ are kept.
3. Human validation: 200 randomly sampled examples (20 per task) were independently labeled by 60 annotators on Prolific under two framings — identifying the attribute asserted by the transcript ($y_{\text{adv}}$) and identifying the attribute conveyed by vocal delivery ($y_{\text{true}}$). Results show 88.7% / 94.4% (response/majority-vote) adversarial accuracy and 80.9% / 82.1% ground-truth accuracy, with Fleiss' $\kappa$ of 0.837 and 0.782 respectively, validating both sides of the contradiction. The gap between human GT accuracy (80.9%) and the best Audio LLM (30.85%) reflects genuine model limitations rather than dataset ambiguity.

Evaluation Metrics

GT Accuracy measures standard task performance — fraction of samples where the model prediction matches the acoustic ground truth:

Adversarial-Label Agreement (ALA) quantifies susceptibility to transcript-following — fraction of samples where the model prediction matches the transcript-implied adversarial label:

Since $y_{\text{adv}} \neq y_{\text{true}}$ by design, a high ALA with low GT accuracy directly indicates lexical shortcutting. $\Delta = \text{ALA} - \text{GT}$ summarizes this gap; larger positive values indicate stronger transcript-following behavior.

Benchmarking Audio LLMs on VoxParadox

Models Evaluated

The authors evaluate 12 models spanning three groups:

• Open-source Audio LLMs: Audio Flamingo 2 (AF2), Audio Flamingo 3 (AF3), Qwen2-Audio-7B-Instruct, SALMONN-7B, Kimi-Audio-7B-Instruct, VITA-Audio, MiMo-Audio-7B-Instruct, Step-Audio-R1
• Open-source Omni LLMs: Qwen2.5-Omni-7B, Qwen3-Omni
• Closed-source models: GPT-4o Audio, Gemini 2.5 Flash

Benchmarking Results: Consistently Low GT Accuracy

Across all 12 models, GT accuracy on VoxParadox is consistently low. The macro-average ranges from just 6.10% (SALMONN) to 30.85% (AF2), with no model exceeding 31%. This stands in stark contrast to performance on naturalistic benchmarks. For example:

• Step-Audio-R1 achieves 54.51% on the MMSU paralinguistic subset but only 17.45% on VoxParadox.
• Gemini 2.5 Flash drops from 51.05% on MMSU to 24.70% on VoxParadox.
• GPT-4o Audio collapses from 36.55% to 8.60%.

A notable exception is AF2, which achieves the highest VoxParadox GT accuracy (30.85%) despite a modest MMSU score (27.44%). The authors attribute this to AF2's CLAP-based audio encoder, which is trained via audio-text contrastive alignment rather than an ASR objective, reducing transcript-centric bias.

Adversarial-Label Agreement: Systemic Transcript Shortcutting

The ALA analysis reveals a near-universal pattern: most Audio LLMs exhibit high ALA alongside low GT accuracy. GPT-4o Audio matches $y_{\text{adv}}$ on 81.55% of examples while achieving only 8.60% GT accuracy ($\Delta = +72.95$). Qwen3-Omni shows 80.65% ALA against 10.60% GT ($\Delta = +70.05$). Across all 12 models, ALA averages 64.34% while GT accuracy averages only 15.33%, with most models showing $\Delta$ values exceeding 50 percentage points.

AF2 is the sole exception, with GT accuracy of 30.85% and ALA of 29.80%, yielding a small negative $\Delta$ of $-1.05$ — the only model where acoustic evidence marginally wins over transcript shortcuts.

Reversed-Audio Diagnostic

To further isolate the lexical dominance effect, the authors evaluate models on VoxParadox with all audio clips reversed — a transformation that removes intelligible linguistic content while preserving low-level acoustic properties (pitch, volume, timbre, tone). For temporally dependent tasks (signal comparison, intonation, speaker identification), answer choices and labels are also reversed accordingly.

This intervention produces a consistent behavioral shift: GT accuracy increases and ALA decreases. AF3 improves from 17.40% to 38.80% GT accuracy as ALA drops from 68.50% to 28.90%. This demonstrates that intelligible lexical content actively suppresses effective use of acoustic evidence in most Audio LLMs.

Layer-wise Probing Analysis

Probing Setup

Inspired by analogous work in vision-language models (Fu et al., 2025), the authors perform layer-wise probing on AF3 to understand why Audio LLMs fail on VoxParadox. The audio encoder and LLM are both frozen; lightweight 3-layer MLP probes with ReLU activations are trained via 10-fold cross-validation to predict task labels from mean-pooled hidden states extracted every two encoder/LLM layers. High probe accuracy indicates task-relevant information is readily retrievable at that layer, while low probe accuracy indicates it is absent or entangled.

Finding 1: A Large Utilization Gap

Across all tasks, probe accuracy is consistently and substantially higher than AF3's end-to-end accuracy. This reveals a utilization gap: task-relevant paralinguistic information is readily retrievable from the model's internal representations, yet the LLM frequently fails to use it when producing predictions. For speaker identity recognition in particular, probe accuracy increases progressively into deeper LLM layers while model output accuracy remains low, pinpointing the bottleneck in the LLM's decision policy rather than in the audio representations themselves. This mirrors findings previously observed in vision-language models.

Finding 2: Representation Degradation at Intermediate and Interface Layers

For low-level signal tasks (pitch, volume, speed, vocal range comparisons), earlier and middle encoder layers yield higher probe accuracy than the final encoder layer. This is consistent with the well-known phenomenon that ASR-pretrained encoders (like Whisper) progressively suppress acoustic and paralinguistic information in deeper layers as representations become more lexically aligned. Additionally, probe performance frequently exhibits a noticeable drop at the encoder–LLM boundary — an interface bottleneck where the cross-modal projection weakens acoustic information. Projecting only the final encoder layer into the LLM thus discards paralinguistic information that is stronger at intermediate depths.

Probing Robustness: Architectures and Encoders

The probing setup is replicated with three probe depths (linear, 3-layer MLP, 5-layer MLP) and four encoder families (AF3/AF-Whisper, Qwen2-Audio/Whisper, HuBERT, CLAP). Key findings:

• Probe depth invariance: layer-wise accuracy curves are nearly identical across all three probe capacities, confirming that trends reflect the underlying representations and not probe expressivity.
• Qwen2-Audio: reproduces both the utilization gap and the intermediate-layer advantage, demonstrating these are not artifacts of AF3's specific architecture.
• HuBERT: shows the same depth-wise degradation as Whisper-based encoders.
• CLAP: uniquely stable or improving toward the final layer, attributable to its audio-text contrastive training objective which does not enforce lexical alignment. This provides a representation-level explanation for AF2's superior VoxParadox performance.

Probing on VoxCeleb2-Derived Tasks

To verify that findings are not unique to synthetic adversarial speech, the authors repeat the probing analysis on a VoxCeleb2-derived task suite (2,000 examples per task, same MCQ format). Both the utilization gap and intermediate-layer advantage/interface degradation patterns persist on natural speech, reinforcing that these are structural properties of ASR-pretrained Audio LLMs.

Pilot: Intermediate-Layer Concatenation

As a simple baseline intervention, the authors concatenate projected tokens from intermediate AF-Whisper layers (layers 5 and 15) after the standard final-layer tokens at the audio–LLM interface. Formally, letting $H^{(l)}$ denote encoder hidden states at layer $l$ and $P^{(l)}$ a learnable projector, the conditioning sequence is:

Only intermediate projectors are trained (the LLM is kept frozen). This simple static augmentation improves VoxParadox GT accuracy from 17.40% to 19.75%. Increasing attention weight on layer-5 tokens by $10\times$ yields the best variant at 20.80%, with gains concentrated in signal-level comparisons and speaker identity. However, concatenation is a static strategy — it cannot adapt which layers to emphasize per task/prompt — motivating the prompt-conditioned approach described next.

Improving Paralinguistic Understanding: PCLM + DPO

Prompt-Conditioned Layer Mixer (PCLM)

Architecture

PCLM addresses both identified bottlenecks — representation degradation and the LLM utilization gap — through prompt-adaptive layer mixing. Let $\{H^{(l)}\}_{l \in \mathcal{L}}$ denote encoder hidden states from a selected set of layers $\mathcal{L}$. Given a text prompt $p$ (the question string), a lightweight BERT-small encoder produces a prompt embedding $e_p$. A small MLP outputs a softmax-normalized weight vector $\alpha \in \mathbb{R}^{|\mathcal{L}|}$:

Each selected encoder layer is projected into the LLM hidden space via a learnable projector $P^{(l)}$, and the final audio token representation is a prompt-weighted sum:

The mixed representation $\tilde{Z}$ replaces the standard final-layer audio tokens fed to the LLM. Intuitively, prompts asking about signal-level attributes can upweight earlier encoder layers rich in acoustic features, while prompts requiring semantic reasoning can favor later layers. The set of exposed layers is $\mathcal{L}_{\text{mid}} = \{5, 15, 25, 30\}$ plus the final layer — five candidate layers total. This choice balances representational diversity with parameter efficiency, adding less than 1% of original model parameters.

Training: Two-Stage SFT

PCLM training keeps the audio encoder frozen throughout and proceeds in two stages:

1. Stage 1 — Projector alignment and PCLM training: Only the intermediate-layer projectors $\{P^{(l)}\}$ and the PCLM weighting network (BERT + MLP) are trained, with the LLM frozen. Intermediate projectors are initialized by cloning the pretrained final-layer projector. This stage improves embedding quality by aligning intermediate-layer representations to the LLM embedding space and enabling prompt-conditioned layer selection.
2. Stage 2 — End-to-end SFT with PCLM enabled: The full LLM is fine-tuned alongside PCLM, with the audio encoder remaining frozen. This stage targets the utilization bottleneck, encouraging the LLM to leverage mixed audio tokens $\tilde{Z}$ for paralinguistic question answering. Learning rate is $5 \times 10^{-5}$ for both stages.

Training data is a mixture of paralinguistic and general audio QA sources totaling approximately 1.27 million samples across three blocks:

• AudioSkills-XL (~67K samples, ~914 hours): broad audio-text coverage from VoxCeleb2, MusicBench, MusicCaps, BBCSoundEffects, WavText5K — included to preserve general audio understanding.
• SpeechCraftSampled (~1,023K samples, ~1,287 hours): fine-grained expressive speech with paralinguistic attribute labels across pitch, speed, age, gender, and emotion (≈200K samples each).
• ParalinguisticDatasets (~177K samples, ~1,074 hours): VoxCeleb2-derived paralinguistic tasks (gender, pitch/volume/speed/range comparisons, intonation, speaker identity, speaker counting; 20K each) plus emotion recognition corpora (MSP-Podcast, Emov-DB, IEMOCAP, TESS, OMGEmotionChallenge).

VoxParadox is never used for training; all gains on VoxParadox reflect genuine generalization.

Direct Preference Optimization (DPO)

Even after SFT with PCLM, the utilization gap identified by probing suggests the LLM's decision policy may still favor transcript-implied shortcuts. DPO is applied as a final optimization stage to explicitly prefer acoustically grounded option selection over language-driven alternatives.

DPO Objective

The standard pairwise DPO loss is minimized over a dataset of triples $(x, y^+, y^-)$ where $x$ is the input (audio, question, candidate options), $y^+$ is the correct (acoustically grounded) answer, and $y^-$ is an incorrect option sampled from the remaining choices:

where $\sigma(\cdot)$ is the sigmoid function, $\beta = 0.1$ controls preference strength, and the preference margin is:

Both the trainable policy $\pi_\theta$ and the reference policy $\pi_{\text{ref}}$ are initialized from the PCLM Stage-2 SFT checkpoint ($\pi_{\text{ref}}$ is entirely frozen). During DPO, the audio encoder, all layer projectors, and the PCLM weighting network are also frozen — only the LLM parameters are updated. This isolates DPO to refining the output decision behavior given an unchanged audio-token interface. Learning rate is $5 \times 10^{-7}$.

DPO Training Data

A dedicated Paralinguistic DPO Dataset of ~20K preference triples is constructed from VoxCeleb2-derived tasks (2K pairs each for age, gender, pitch, volume, speed, vocal range, intonation, speaker identity, speaker counting) and emotion datasets (2K pairs from MSP-Podcast, IEMOCAP, Emov-DB, TESS, OMGEmotionChallenge). VoxParadox is excluded from DPO training.

Results

Main Results: PCLM and DPO on VoxParadox and MMSU

The table below summarizes class-wise GT accuracy on VoxParadox and the MMSU paralinguistic subset for AF3 and Qwen2-Audio with each proposed component:

Key observations from these results:

• PCLM alone provides the largest gains: AF3 jumps from 17.40% to 60.00% (+42.60 pp) and Qwen2-Audio from 14.85% to 68.65% (+53.80 pp) on VoxParadox.
• DPO adds further incremental gains: AF3 improves by an additional +5.20 pp to 65.20%; Qwen2-Audio by +3.65 pp to 72.30%.
• Signal-level tasks see the most dramatic improvements: Qwen2-Audio pitch goes from 26.50% to 100.00%; vocal range from 19.00% to 100.00%; volume from 22.00% to 98.00%.
• Higher-level tasks (intonation, speaker identity, speaker counting) see smaller but still substantial gains, reflecting the greater difficulty of these tasks for all components.
• SFT without PCLM is significantly weaker, especially for tasks like intonation (AF3 SFT: 7.50% vs. PCLM: 49.50%), confirming that the intermediate-layer access provided by PCLM is the critical ingredient rather than the fine-tuning data alone.

Effect on Adversarial-Label Agreement

DPO directly penalizes transcript-implied shortcutting. ALA drops dramatically:

• AF3: ALA falls from 68.50% to 22.60% with PCLM + DPO (a reduction of ~46 pp).
• Qwen2-Audio: ALA falls from 70.25% to 15.95% with PCLM + DPO (a reduction of ~54 pp).

Generalization to MMSU and General Speech Understanding

To assess whether paralinguistic gains come at the cost of general audio understanding, the authors report MMSU (All) results:

• AF3: MMSU All changes from 51.43% to 50.62% with PCLM + DPO — a negligible decrease of 0.81 pp, while MMSU paralinguistic improves by +17.04 pp.
• Qwen2-Audio: MMSU All actually improves from 50.82% to 55.43% with PCLM + DPO, alongside a +28.89 pp gain on MMSU paralinguistic.

These results confirm that PCLM + DPO produces genuine improvements in paralinguistic perception that generalize beyond VoxParadox's adversarial format, with minimal or no cost to broader speech understanding capabilities.

Ablations and Key Comparisons

PCLM vs. SFT without PCLM

For AF3, SFT without PCLM achieves 34.80% on VoxParadox — an improvement over the 17.40% baseline, but far below the 60.00% of PCLM. The contrast is most stark on intonation (7.50% SFT vs. 49.50% PCLM) and several signal tasks. This confirms that access to intermediate encoder layers — not merely additional supervised training — is the critical factor.

DPO without PCLM

AF3 + SFT + DPO without PCLM reaches 40.30% on VoxParadox, compared to 65.20% with PCLM + DPO. DPO alone cannot compensate for the degraded audio representations that result from relying solely on the final encoder layer.

Pilot Concatenation as a Baseline

Static concatenation of intermediate layers (without prompt conditioning) yields only 19.75% on VoxParadox, compared to 60.00% for PCLM. This confirms the value of prompt-adaptive mixing over a fixed static combination.

Related Work and Positioning

Paralinguistics Benchmarks

VoxParadox complements prior work in several ways. Classical challenge-style evaluations (INTERSPEECH ComParE series, IEMOCAP, SUPERB, SD-Eval) and modern audio LLM benchmarks (MMAU, MMAU-Pro, MMAR, MMSU) do not explicitly decouple lexical from acoustic evidence. CP-Bench evaluates contextual paralinguistic understanding in the wild, and LISTEN probes emotion recognition using decorrelated lexical/acoustic pairs. MULTIVOX probes paralinguistic behavior in naturalistic multi-speaker settings. VoxParadox occupies the niche of controlled adversarial counterfactuals that enforce explicit linguistic-acoustic contradiction, providing a targeted stress test for acoustic-only decision making.

Layer Mixing in Speech Models

Prior work shows ASR-pretrained encoders progressively suppress acoustic information in deeper layers (Pasad et al., 2022; Gong et al., 2023), and the optimal layer for different tasks varies with pretraining objective. Attentive merging has been applied for anti-spoofing (Pan et al., 2024). VARAN learns input-dependent layer aggregation. PaM applies a prompt-aware mixture over multiple audio encoders. PCLM differs by mixing intermediate layers within a single encoder, targeted specifically at paralinguistic cues that probing shows are strongest at intermediate depths.

DPO in Audio Settings

DPO (Rafailov et al., 2023) has been applied to speech generation tasks (expressiveness in TTS, intelligibility under challenging phenomena). In the current paper, preference pairs contrast language-implied answers with acoustically grounded responses, applying DPO to reward use of paralinguistic evidence under linguistic-acoustic contradiction — a novel application of preference optimization to audio understanding rather than audio generation.

Limitations

The authors identify two main limitations:

1. Post-hoc rather than integrated design: PCLM is applied as a post-hoc fix to a pretrained Audio LLM. Because paralinguistic information can already degrade inside the encoder and at the encoder–LLM interface, recovering it after the fact has natural ceiling effects. Larger and more reliable gains likely require incorporating multi-layer access and acoustic-grounding incentives during pretraining and instruction tuning, not only in downstream corrections.
2. Adversarial-by-design evaluation: VoxParadox enforces a controlled stress test that does not reflect all naturalistic communication scenarios. The benchmark should be viewed as a complement to naturalistic or crowdsourced paralinguistic benchmarks rather than a substitute. More broadly, common speech–language corpora intrinsically couple lexical content with paralinguistic cues; large-scale datasets that explicitly break or control these correlations would enable deeper study of paralinguistic reasoning independent of lexical signals.

Conclusions and Broader Impact

VoxParadox reveals a systematic and near-universal failure mode in current Audio LLMs: when lexical content contradicts acoustic evidence, most models default to transcript-implied answers even when explicitly asked about paralinguistic attributes. Layer-wise probing identifies two complementary bottlenecks: (1) paralinguistic cues degrade through deeper encoder layers and the encoder–LLM projection, and (2) even when cues are present in audio tokens, the LLM frequently fails to use them — a decision-policy bias beyond representation quality.

The proposed PCLM + DPO combination addresses both bottlenecks and yields large, consistent gains:

• AF3: VoxParadox $17.40\% \to 65.20\%$; MMSU Paralinguistic $37.74\% \to 54.78\%$; ALA $68.50\% \to 22.60\%$.
• Qwen2-Audio: VoxParadox $14.85\% \to 72.30\%$; MMSU Paralinguistic $34.37\% \to 63.26\%$; ALA $70.25\% \to 15.95\%$.

These gains transfer from VoxParadox to MMSU, confirming they reflect genuine improvements in paralinguistic perception rather than overfitting to the adversarial benchmark format. The fact that improvements are achieved without using VoxParadox during training underscores the generality of the approach.

The authors note that stronger paralinguistic inference has beneficial applications (more reliable speech interfaces, accessibility technologies) but also carries risks of intrusive profiling, surveillance, or discriminatory decision-making, and may amplify concerns when combined with high-fidelity speech synthesis. They encourage responsible use with appropriate consent, privacy protections, bias evaluation, and safeguards before high-stakes deployment.