Qwen2-Audio

Training Pipeline

Qwen2-Audio follows a three-stage training pipeline: (1) multi-task pre-training, (2) supervised fine-tuning (SFT), and (3) Direct Preference Optimization (DPO). This mirrors best practices from modern LLM alignment pipelines and extends them to the multimodal audio-language domain.

Stage 1: Pre-training

The pre-training stage is a large-scale multi-task learning phase spanning a broad range of audio understanding tasks. A key innovation compared to the original Qwen-Audio is the replacement of hierarchical task tags with natural language prompts. Where Qwen-Audio used structured symbolic tags to distinguish tasks, Qwen2-Audio simply phrases each training example as a natural language instruction-response pair. The authors find this leads to better generalization and improved instruction-following capabilities.

Pre-training data is further expanded significantly over Qwen-Audio. The dataset spans three high-level audio categories — Speech, Sound, and Music/Song — and covers a wide variety of tasks:

• Speech tasks: Automatic Speech Recognition (ASR, ~30k hours, multiple languages), Speech-to-Text Translation (S2TT, ~3.7k hours), Overlapped Speech Recognition (~<1k hours), Dialect ASR (~2k hours), Speech Recognition with Word-level Timestamps (English ~10k hours, Mandarin ~11k hours), Dialect Identification (~2k hours), Spoken Language Identification (~11.7k hours), Speaker Gender Classification (~4.8k hours), Emotion Recognition (~<1k hours), Speaker Verification (~1.2k hours), Speaker Diarization (~<1k hours), Speech Entity Recognition (~<1k hours), Keyword Spotting (~<1k hours), Intent Classification (~<1k hours), Slot Filling (~<1k hours), Speaker Age Prediction (~4.8k hours), Vocal Sound Classification (~<1k hours).
• Sound tasks: Automatic Audio Captioning (~8.4k hours), Sound Event Classification (~5.4k hours), Acoustic Scene Classification (~<1k hours), Sound Event Detection with timestamps (~<1k hours), Audio Question Answering (~<1k hours).
• Music & Song tasks: Singer Identification (~<1k hours), Singer and Music Emotion Recognition (~<1k hours), Music Caption (~25k hours), Music Instrument Classification (~<1k hours), Music Note Analysis (~<1k hours), Music Genre Recognition (~9.5k hours), Music Recognition (~<1k hours), Music Question Answering (~<1k hours).

Stage 2: Supervised Fine-tuning (SFT)

Pre-training equips the model with broad audio understanding, but SFT adapts the model to follow human instructions in realistic interaction scenarios. The SFT stage introduces two distinct interaction modes that are jointly trained so that no explicit mode-switching is required during use:

• Audio Analysis Mode: The user provides one or more audio clips (which may be speech, environmental sounds, music, or mixed audio) along with instructions — given either as text or as spoken audio. Qwen2-Audio processes the audio signal holistically and responds to the instructions. This mode is well-suited to offline or analytical workflows.
• Voice Chat Mode: The user engages in open-ended conversational interaction using voice. Qwen2-Audio treats the spoken content as natural dialogue turns. Text interaction is also available in this mode. This mode targets real-time, online dialogue use cases.

The authors emphasize that no system prompt is needed to switch between modes. Qwen2-Audio autonomously detects the nature of the input and responds accordingly. For example, given an audio clip where the first portion contains keyboard typing sounds followed by a spoken question "What is this sound?", the model correctly identifies the sound event and responds, "This is the sound of a keyboard."

A key finding from preliminary studies is that data quality and complexity in the SFT stage significantly affects final model performance. Accordingly, the SFT dataset was meticulously curated with rigorous quality control to ensure diversity, complexity, and accuracy of the instruction-response pairs.

Stage 3: Direct Preference Optimization (DPO)

To further align Qwen2-Audio's responses with human preferences — especially regarding factuality and adherence to desired behaviors — the model undergoes Direct Preference Optimization (DPO). DPO uses a dataset $\mathcal{D}$ of triplets $(\mathbf{x}, \mathbf{y}_w, \mathbf{y}_l)$, where $\mathbf{x}$ is an input (including audio), $\mathbf{y}_w$ is the human-annotated preferred response, and $\mathbf{y}_l$ is the dispreferred response. The DPO objective is:

where $\mathcal{P}_{\text{ref}}$ is the reference model (initialized from the SFT checkpoint), $\sigma$ is the sigmoid function, and $\beta$ is a hyperparameter controlling the deviation from the reference policy. DPO removes the need for a separate reward model, directly optimizing the policy using preference pairs.

Evaluation Benchmarks

Qwen2-Audio is evaluated on a comprehensive suite of 13 datasets covering four task categories, as summarized below:

The paper emphasizes that many older SLU and SER benchmarks are highly constrained and fail to reflect real-world performance, which is why AIR-Bench is treated as the primary evaluation for instruction-following and chat capabilities. All evaluation datasets are strictly excluded from training to prevent data leakage.

Main Results

Automatic Speech Recognition (ASR)

On Librispeech, Qwen2-Audio achieves WER of 1.3 / 3.4 / 1.6 / 3.6 on dev-clean, dev-other, test-clean, and test-other respectively, outperforming all prior systems including SpeechT5 (2.1/5.5/2.4/5.8), SLM-FT (–/–/2.6/5.0), SALMONN (–/–/2.1/4.9), SpeechVerse (–/–/2.1/4.4), and Qwen-Audio (1.8/4.0/2.0/4.2).

On Common Voice 15 (English, Chinese, Yue, French), Qwen2-Audio achieves WER of 8.6 / 6.9 / 5.9 / 9.6 versus Whisper-large-v3's 9.3 / 12.8 / 10.9 / 10.8. Note that Qwen2-Audio is not evaluated in a zero-shot manner on Common Voice 15, while Whisper is.

On Fleurs (zh), evaluated in zero-shot fashion for both models, Qwen2-Audio achieves WER 7.5 vs. Whisper-large-v3's 7.7.

On Aishell2 (Mic / iOS / Android), Qwen2-Audio achieves 3.0 / 3.0 / 2.9 WER, outperforming Qwen-Audio (3.3/3.1/3.3) and competitive with Paraformer-large (–/2.9/–).

Speech-to-Text Translation (S2TT)

On CoVoST2, Qwen2-Audio achieves state-of-the-art BLEU scores across all seven translation directions:

Speech Emotion Recognition (SER)

On the Meld dataset, Qwen2-Audio achieves accuracy of 0.553, slightly below Qwen-Audio (0.557) and above WavLM-large (0.542). This is one of the few tasks where Qwen2-Audio does not surpass its predecessor, illustrating that certain tightly constrained SLU benchmarks may not fully capture generative model strengths.

Vocal Sound Classification (VSC)

On VocalSound, Qwen2-Audio achieves 0.9392 accuracy, outperforming Qwen-Audio (0.9289), Pengi (0.6035), and CLAP (0.4945).

AIR-Bench Chat Benchmark

The AIR-Bench chat benchmark evaluates instruction-following across four audio modalities using GPT-4-based scoring (0–10 scale). Qwen2-Audio achieves state-of-the-art results across all four dimensions:

A notable caveat is that Gemini-1.5-pro's results are based on a reduced sample set — approximately one fifth of test samples were excluded due to SAFETY filter rejections during API evaluation. Qwen2-Audio's improvements over Gemini-1.5-pro are especially pronounced on sound (+1.50), music (+1.73), and mixed-audio (+1.50) dimensions.

Qualitative Case Studies

The paper includes several illustrated examples demonstrating Qwen2-Audio's diverse interaction capabilities across both modes. These examples showcase the model's ability to handle natural, free-flowing conversations and structured analytical queries.

Key Design Decisions and Contributions

Natural Language Prompts vs. Hierarchical Tags

One of the most impactful changes from Qwen-Audio to Qwen2-Audio is replacing the hierarchical symbolic tag system with natural language task prompts. The prior approach required careful construction and maintenance of a tag hierarchy to differentiate tasks during pre-training. By using plain language descriptions (e.g., "Transcribe the following speech into text." or "What emotions are being expressed by the speaker?"), the model learns a more uniform representation that generalizes better and supports zero-shot instruction following without needing to memorize special tags.

Seamless Dual-Mode Interaction Without System Prompts

The jointly-trained Audio Analysis and Voice Chat modes represent a significant usability advancement. In practical LALMs, mode switching usually requires explicit system prompts or different model checkpoints. Qwen2-Audio trains both modes simultaneously, allowing the model to autonomously determine how to interpret an audio input — whether as an analysis target or as a direct voice command — based on the context of the conversation.

DPO for Audio-Language Alignment

Applying DPO to a multimodal audio-language model is a notable contribution. The DPO stage specifically addresses factuality (ensuring responses are grounded in the actual audio content) and behavioral alignment (avoiding harmful, irrelevant, or low-quality responses). The preference annotation process produces triplets of audio+prompt inputs with annotated good and bad completions, enabling contrastive optimization without a separate reward model.

Scale and Breadth of Pre-training Data

Qwen2-Audio's pre-training dataset covers a remarkable breadth of tasks — over 30 distinct audio understanding tasks spanning speech (recognition, translation, speaker attributes, language ID, emotion, keyword spotting), sound (captioning, classification, detection), and music (captioning, genre, instruments, notes). The total data volume extends to tens of thousands of hours across key tasks like multilingual ASR, music captioning, and speech translation.

Limitations and Caveats

While the paper reports strong results across most benchmarks, a few limitations and caveats are worth noting:

• Inconsistent zero-shot evaluation: On Common Voice 15, Qwen2-Audio is evaluated after training on Common Voice data, whereas Whisper-large-v3 is evaluated in a true zero-shot manner. This makes direct comparison on that benchmark non-apples-to-apples. The Fleurs evaluation is zero-shot for both models.
• SER performance plateau: On the Meld speech emotion recognition benchmark, Qwen2-Audio (0.553) marginally underperforms Qwen-Audio (0.557). The paper itself notes that many SLU and SER benchmarks are limited in their ability to reflect real-world performance, motivating the heavier emphasis on AIR-Bench.
• Gemini-1.5-pro evaluation fairness: During AIR-Bench evaluation, approximately 20% of Gemini-1.5-pro's test samples were dropped due to safety filter refusals. The resulting reduced sample count may affect the representativeness of Gemini's scores, though the direction of bias is unclear.
• No ablation studies reported: The paper does not include formal ablation experiments (e.g., comparing natural language prompts vs. hierarchical tags, with vs. without DPO, or different data scales), making it difficult to quantitatively attribute improvements to specific design choices.
• Single LLM backbone: The system relies on Qwen-7B as its language backbone. The impact of scaling or changing the LLM component is not explored in this report.

Conclusion

Qwen2-Audio represents a substantial advance over Qwen-Audio in both audio understanding performance and interactive capabilities. The three-stage training pipeline — natural language prompt-driven pre-training at scale, joint SFT across Audio Analysis and Voice Chat modes, and DPO alignment — produces a model that achieves state-of-the-art results on a wide range of audio benchmarks without task-specific fine-tuning. Most notably, it surpasses Gemini-1.5-pro on the AIR-Bench chat benchmark across all four audio modalities (speech, sound, music, and mixed audio), demonstrating strong generalization to real-world audio instruction-following scenarios. The model is open-sourced to support continued progress in the multimodal language community.