Qwen-Audio
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
A unified audio-language model that handles speech, natural sounds, and music across 30+ tasks with a single encoder. It uses hierarchical task tags to resolve multi-task interference, enabling strong zero-shot transfer without task-specific fine-tuning—outperforming prior specialist audio systems.
Demos
Qwen-Audio demonstrates universal audio understanding across 30+ diverse tasks—speech recognition, audio captioning, scene classification, emotion recognition, and music analysis—without task-specific fine-tuning. Evaluate these demos for: (1) multi-modal comprehension across speech, environmental sounds, and music; (2) strong baseline performance relative to specialized models on Aishell1, Cochlscene, and ClothoAQA benchmarks; (3) multi-turn dialogue capabilities and reasoning over multiple audio inputs in Qwen-Audio-Chat.
Links
Abstract
Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
Introduction and Motivation
Large language models (LLMs) have demonstrated remarkable capabilities in knowledge retention, reasoning, and problem solving, but they inherently lack the ability to perceive non-textual modalities such as audio. Audio carries information far richer than plain text: prosody, emotion, tone, and intent in human speech; environmental cues such as train whistles, clock chimes, and thunder in natural sounds; melody and harmony in music. Enabling LLMs to understand these rich signals is therefore an important frontier in multimodal AI.
Prior instruction-following audio-language models have been constrained by a fundamental gap: the absence of pre-trained audio models capable of handling diverse audio types and tasks. Representative systems such as SpeechNet, SpeechT5, VIOLA, Whisper, and Pengi each cover only a subset of audio types — focusing on either human speech or natural sounds, but rarely both, and almost never music. This fragmentation forces practitioners to maintain separate specialist models and prevents cross-task knowledge transfer.
Qwen-Audio is introduced to fill this gap. It is a large-scale audio-language model that extends the Qwen-7B language model with a single shared audio encoder, trained jointly on more than 30 tasks spanning eight languages and all major audio types (human speech, natural sounds, music, songs). Building on top of Qwen-Audio, the authors also release Qwen-Audio-Chat, an instruction-tuned variant that supports flexible multi-turn dialogues involving mixed audio and text inputs.
Related Work
Multi-task Audio-Text Learning
Earlier multi-task audio systems such as SpeechNet and SpeechT5 formulate audio processing as a shared encoder-decoder problem but restrict themselves to human speech inputs and outputs. A line of work (VIOLA, VoxTLM, AudioPALM, SpeechX) converts continuous speech into discrete codec tokens and treats all tasks as conditional language-model generation, typically using decoder-only Transformers. Whisper introduced a systematic multi-task training format using special tokens — for language identification, task type, and sentence-level timestamps — but remained confined to speech recognition and translation. Pengi extends this idea to natural-sound understanding tasks by using text templates to unify data formats. None of these systems jointly handle speech, sound, and music in a single model.
Interacting with LLMs through Audio
Tool-use approaches (AudioGPT, HuggingGPT) leverage external audio specialist models as tools controlled by an LLM, but lose prosodic and emotional information that cannot be represented as text. End-to-end approaches train directly on audio-text pairs: SpeechGPT discretizes speech with HuBERT tokens and uses a three-stage pipeline; BLSP aligns audio representations with text by requiring the LLM to produce the same continuation for audio and its transcript; LLaSM synthesizes speech instruction data with a TTS API; LTU creates a 5 M audio-QA dataset and fine-tunes LLaMA with LoRA; SALMONN combines a text encoder and a speech encoder fused to a frozen LLM via Q-Former-style attention. Qwen-Audio differentiates itself by using a single encoder for all audio types, training end-to-end at scale across more than 30 heterogeneous tasks, and achieving strong zero-shot transfer to held-out benchmarks.
Model Architecture
The Qwen-Audio architecture consists of two components: an audio encoder and a large language model decoder. Given paired data $(\mathbf{a}, \mathbf{x})$ where $\mathbf{a}$ is an audio sequence and $\mathbf{x}$ is a text sequence, the training objective is to maximize the autoregressive next-token probability:
$$\mathcal{P}_{\theta}(x_t \mid \mathbf{x}_{
where $\theta$ and $\phi$ are the trainable parameters of the LLM and audio encoder respectively.
Qwen-Audio uses a single audio encoder for all audio types. The encoder is initialized from Whisper-large-v2, a 32-layer Transformer model with two convolution downsampling layers as a stem, totalling 640 M parameters. Although Whisper was trained only for speech recognition and translation, its learned representations encode rich information beyond speech content — including background noise and speaker characteristics — as demonstrated by subsequent research. Audio is preprocessed by resampling to 16 kHz and converting to an 80-channel mel-spectrogram with a 25 ms window and 10 ms hop. An additional pooling layer with stride 2 further compresses the sequence so that each encoder output frame corresponds to approximately 40 ms of audio. SpecAugment (LibriSpeech Basic policy) is applied during training as data augmentation.
The decoder is initialized from Qwen-7B, a 32-layer Transformer decoder with hidden size 4096 and 7.7 B total parameters. During multi-task pretraining, the LLM weights are frozen and only the audio encoder is optimized. During supervised fine-tuning (SFT), the audio encoder is frozen and only the LLM is optimized.
A core challenge of large-scale multi-task co-training in audio is that different datasets produce widely varying text output formats. The variation arises from differences in: (1) task focus (recognition vs. captioning vs. classification vs. QA); (2) language (English, Mandarin, German, French, Spanish, Italian, and others); (3) annotation granularity (word-level, sentence-level, or free-form); and (4) text structure (structured labels vs. unstructured natural language). Naively mixing all datasets creates a one-to-many mapping problem for the decoder — the same audio input may legitimately require different text responses depending on the implied task.
Simple remedies used in prior work — grouping similar tasks or appending a dataset ID — provide limited relief. Qwen-Audio proposes a more principled solution: a hierarchical tag sequence prepended to every decoder input that maximally shares information across similar tasks while unambiguously separating incompatible output formats.
The tag sequence appended to every input is defined by the following ordered slots:
Tags that are shared across many tasks (e.g., the audio language tag) encourage knowledge sharing; tags that differ between tasks (e.g., the transcription vs. analysis start tag) disambiguate the output format and prevent interference. The overall design principle is to make shared structure explicit while making task-specific divergences equally explicit.
The pretraining corpus spans over 30 distinct tasks, eight languages, and hundreds of thousands of hours of audio. The table below summarises the task categories and their approximate sizes.
Evaluation datasets are rigorously excluded from training to prevent data leakage. In total, the pretraining corpus represents tens of thousands of hours of labelled audio, making Qwen-Audio one of the largest-scale multi-type audio pretraining efforts to date.
A distinctive and novel aspect of Qwen-Audio's training is the inclusion of the SRWT task. Previous multi-task audio systems either omit timestamps entirely or predict only coarse sentence-level timestamps (as in Whisper). Qwen-Audio trains the model to interleave word-level start and end timestamps with the transcription tokens: for each word $w_i$, the output sequence contains $\langle t^{\text{start}}_i \rangle$, $w_i$, $\langle t^{\text{end}}_i \rangle$ in that order. This requires the model to maintain precise temporal alignment between audio frames and linguistic units.
The motivation is twofold. First, fine-grained temporal grounding inherently forces the model to learn better audio-text alignment, which is beneficial even when timestamps are not needed in the final output. Second, SRWT enables downstream grounding-based QA scenarios in Qwen-Audio-Chat — for example, identifying the time interval during which a person's name is spoken, or verifying whether a particular sound event occurs in an audio clip (audio event detection is a special case of event-timestamp prediction, since absence of a timestamp implies absence of the event).
Critically, the SRWT task shares the same audio data as the standard ASR task; incorporating it therefore does not require any additional audio collection. Only the label format changes.
Building on the pretrained Qwen-Audio backbone, Qwen-Audio-Chat is produced via supervised fine-tuning (SFT) on approximately 20k instruction-following examples. During SFT, the audio encoder weights are frozen and only the LLM weights are updated.
The SFT data is constructed from three sources: (1) manually written demonstrations converting raw dataset labels into question-answer format; (2) GPT-3.5-generated additional questions and answers conditioned on the raw text labels; and (3) a multi-audio dialogue dataset created through manual annotation, model generation, and strategy concatenation, incorporating reasoning, story generation, and multi-audio comprehension examples.
To handle multiple simultaneous audio inputs, the convention "Audio <id>:" labels each audio in order of appearance in the dialogue. The dialogue format follows the ChatML convention, marking each turn with
Both training stages use AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-6}$, weight decay 0.05, and gradient clipping at 1.0. Training is performed in bfloat16 with optimizer sharding. Key hyperparameters are summarised below:
Qwen-Audio is evaluated on 12 datasets spanning 8 task types — all held out from training:
On Librispeech, Qwen-Audio achieves WERs of 1.79 / 4.00 / 2.04 / 4.19 on dev-clean, dev-other, test-clean, and test-other, respectively — outperforming SpeechT5 (2.1/5.5/2.4/5.8), SLM-FT (−/−/2.6/5.0), and SALMONN (−/−/2.1/4.9) without any task-specific fine-tuning.
On Aishell1, Qwen-Audio achieves 1.22% / 1.29% WER on dev/test — state-of-the-art, surpassing MMSpeech-large (1.6/1.9) and Paraformer-large (−/2.0). On Aishell2, Qwen-Audio achieves 3.3/3.1/3.3 WER across the three microphone conditions, competitive with Paraformer-large (−/2.9/−).
On CoVoST2, Qwen-Audio substantially outperforms all baselines across all seven directions. Selected BLEU scores: en→de: 25.1 (vs. SALMONN 18.6, BLSP 14.1); de→en: 33.9 (vs. SpeechLLaMA 27.1); en→zh: 41.5 (vs. SALMONN 33.1); zh→en: 15.7 (vs. SpeechLLaMA 12.3); es→en: 39.7, fr→en: 38.5, it→en: 36.0 (vs. SpeechLLaMA 27.9/25.2/25.9).
On the Clotho test set, Qwen-Audio outperforms Pengi: CIDEr 0.441 vs. 0.416; SPICE 0.136 vs. 0.126; SPIDEr 0.288 vs. 0.271.
On the industrial test set evaluated by Average Absolute Score (AAS, lower is better), Qwen-Audio achieves 51.5 ms — better than Force-aligner (60.3 ms, which has access to ground-truth transcripts) and Paraformer-large-TP (65.3 ms, which like Qwen-Audio must jointly predict transcription and timestamps).
Qwen-Audio achieves 0.795 accuracy on CochlScene (vs. the dataset-specific model at 0.669 — a +12.6 point improvement) and 0.649 accuracy on TUT2017 (vs. Pengi 0.353 — nearly doubling the score).
On MELD, Qwen-Audio achieves 0.557 accuracy, edging out WavLM-large (0.542).
On ClothoAQA, Qwen-Audio achieves state-of-the-art results: overall accuracy 0.579 (vs. ClothoAQA baseline 0.542) and binary accuracy 0.749 (vs. Pengi 0.645).
On VocalSound, Qwen-Audio achieves 0.9289 accuracy — dramatically outperforming CLAP (0.4945) and Pengi (0.6035), with an improvement of more than 32 percentage points over the previous best.
On NSynth Qualities (MAP), Qwen-Audio achieves 0.4742 vs. Pengi 0.3860. On NSynth Instrument classification (accuracy), Qwen-Audio achieves 0.7882 vs. Pengi 0.5007.
Removing the SRWT task from pretraining while holding all other tasks constant degrades ASR performance across all Librispeech and Aishell1 splits:
The improvement is especially pronounced on Aishell1 (1.54→1.22 dev; 1.71→1.29 test), suggesting that fine-grained temporal alignment learning particularly benefits tonal language recognition.
SRWT also benefits QA tasks beyond speech — including natural-sound QA and music QA — demonstrating that temporal grounding generalises across audio modalities:
The authors interpret this as SRWT providing a cross-modal grounding signal that helps the model identify when and where specific audio events occur, which is essential for answering temporal questions about natural sounds and music.
Qwen-Audio-Chat extends the pretrained model with instruction-following capabilities via 20k SFT examples. Key capabilities include:
The paper does not explicitly enumerate limitations in a dedicated section, but several implicit constraints can be identified from the methodology:
Qwen-Audio presents a compelling demonstration that a single large-scale audio-language model can achieve state-of-the-art or highly competitive performance across a remarkably diverse set of audio understanding tasks — spanning automatic speech recognition in multiple languages, speech translation, audio captioning, acoustic scene and emotion classification, question answering, vocal sound classification, and music note analysis — all without task-specific fine-tuning. The key technical contributions are: (a) the hierarchical multi-task tag framework that enables clean co-training across 30+ heterogeneous tasks; (b) the identification of SRWT as a broadly beneficial temporal grounding auxiliary task; and (c) the demonstration that Whisper's encoder representations, despite being trained only for ASR/translation, carry sufficient information to support music, natural-sound, and emotion tasks when connected to a powerful LLM decoder. The resulting Qwen-Audio-Chat model further shows that 20k instruction examples suffice to unlock flexible, multi-turn, multi-audio conversational interaction grounded in these pretrained capabilities.
The Qwen-Audio repository provides PyTorch implementations of the multi-task audio-language model and its chat variant.
Pre-trained model checkpoints are released on ModelScope and Hugging Face (announced Nov. 2023); the repo itself contains
the core model architecture, audio processing pipeline, and evaluation scripts.
The paper's core contribution—a multi-task training framework with hierarchical tag conditioning to avoid
one-to-many interference across 30+ audio understanding tasks—is realized via the unified model architecture in
Audio Encoder
Large Language Model
Multitask Pretraining
The One-to-Many Interference Problem
Multi-task Training Format Framework
<|startoftranscripts|> for tasks requiring accurate linguistic transcription (ASR, speech translation). <|startofanalysis|> for all other tasks (captioning, classification, QA, etc.).<|unknown|> token.<|transcribe|>, <|translate|>, <|caption|>, <|analysis|>, and <|question-answer|>. For QA tasks, the corresponding question is appended immediately after the tag.<|timestamps|> or <|notimestamps|>. When set to <|timestamps|>, the model performs fine-grained word-level timestamp prediction (SRWT — Speech Recognition with Word-level Timestamps), interleaving start-time tokens before each word token and end-time tokens after. This differs from Whisper's sentence-level timestamps.Training Dataset
Audio Type Task Description Hours Speech ASR Automatic speech recognition (multiple languages) 30k S2TT Speech-to-text translation 3.7k OSR Overlapped speech recognition <1k Dialect ASR Automatic dialect speech recognition 2k SRWT (EN) English speech recognition with word-level timestamps 10k SRWT (ZH) Mandarin speech recognition with word-level timestamps 11k DID Dialect identification 2k LID Spoken language identification 11.7k SGC Speaker gender classification 4.8k ER Emotion recognition <1k SV Speaker verification 1.2k SD Speaker diarization <1k SER Speech entity recognition <1k KS Keyword spotting <1k IC Intent classification <1k SF Slot filling <1k SAP Speaker age prediction 4.8k VSC Vocal sound classification <1k Sound AAC Automatic audio captioning 8.4k SEC Sound event classification 5.4k ASC Acoustic scene classification <1k SED Sound event detection with timestamps <1k AQA Audio question answering <1k Music & Song SID Singer identification <1k SMER Singer and music emotion recognition <1k MC Music captioning 25k MIC Music instrument classification <1k MNA Music note analysis (pitch, velocity) <1k MGR Music genre recognition 9.5k MR Music recognition <1k MQA Music question answering <1k Speech Recognition with Word-Level Timestamps (SRWT)
Supervised Fine-Tuning: Qwen-Audio-Chat
<im_start> and <im_end> special tokens. Training supervises only the assistant response tokens (answers and special tokens) and not the role names or user prompts, ensuring the model learns to generate accurate responses without memorising the prompt template. An example of the SFT data format is shown below:
<im_start>user
Audio 1: <audio>emov-db/141-168-0155.wav</audio>what does the speaker say?<im_end>
<im_start>assistant
The speaker says in English, "Won't you draw up, gentlemen."<im_end>
<im_start>user
What's the mood of the person?<im_end>
<im_start>assistant
Based on the voice, the mood of the person is disgusted.<im_end>
Training Configuration
Configuration Multi-task Pre-training Supervised Fine-tuning Audio encoder init. Whisper-large-v2 Qwen-Audio (stage 1) LLM init. Qwen-7B (frozen) Qwen-7B (trainable) Peak learning rate $5 \times 10^{-5}$ $1 \times 10^{-5}$ Minimum learning rate $1 \times 10^{-5}$ $1 \times 10^{-6}$ LR schedule Cosine decay Cosine decay Training steps 500k 8k Warm-up steps 2000 3k Global batch size 120 128 Gradient accumulation 1 8 Audio encoder LR decay 0.95 0 (frozen) Model parallelism None 2-way Experiments and Results
Evaluation Benchmarks
Automatic Speech Recognition
Speech-to-Text Translation
Automatic Audio Captioning
Speech Recognition with Word-Level Timestamps
Acoustic Scene Classification
Speech Emotion Recognition
Audio Question Answering
Vocal Sound Classification
Music Note Analysis
Ablation Studies
Effect of SRWT on ASR
Method LS dev-clean LS dev-other LS test-clean LS test-other AISHELL1 dev AISHELL1 test w/o SRWT 1.93 4.18 2.22 4.21 1.54 1.71 Qwen-Audio (full) 1.79 4.00 2.04 4.19 1.22 1.29 Effect of SRWT on Audio Question Answering
Method ClothoAQA test ClothoAQA test-binary MusicAVQA audio-question w/o SRWT 0.5648 0.7418 0.7027 Qwen-Audio (full) 0.5795 0.7491 0.7211 Qwen-Audio-Chat: Interactive Dialogue Capabilities
Stated Contributions and Novelty
Limitations and Discussion
Conclusion
Code & Implementation
Repository Overview
Key Components
audio.py handles mel-spectrogram computation
(16 kHz, 80 Mel filters, 30-second chunks) using librosa-compatible filtering, and provides audio loading via FFmpeg.
It implements log_mel_spectrogram(), pad_or_trim(), and the multi-head attention mechanisms
used by the audio encoder.
modeling_qwen.py defines the main Qwen-Audio model
(inheriting from Qwen-7B LLM backbone) with support for flash-attention optimization, rotary embeddings, and
quantization-aware caching. Includes generation utilities in qwen_generation_utils.py for multi-turn chat.
tokenization_qwen.py, configuration_qwen.py,
and base_generation_config.json / chat_generation_config.json configure the tokenizer
and model hyperparameters.
eval_audio/ folder contains benchmark scripts for 12 tasks
(ASR, audio captioning, emotion recognition, scene classification, etc.) using standard metrics (WER, BLEU, etc.).
Paper–Code Mapping
modeling_qwen.py, which accepts tag-conditioned prompts during pre-training and inference.
The audio encoder (Whisper-large-v2 initialization) processes diverse audio types through audio.py's
spectrogram pipeline. Qwen-Audio-Chat instruction fine-tuning is implemented via generation configuration and
chat utilities, enabling multi-turn dialogue without task-specific fine-tuning.