Qwen2.5-Omni

Qwen2.5-Omni Technical Report

An end-to-end multimodal model that perceives text, images, audio, and video while generating text and streaming speech simultaneously. Its Thinker-Talker architecture decouples text and speech generation to enable real-time, interference-free concurrent outputs.

• llm
• multimodal
• dialogue
• tts
• streaming
• realtime
• autoregressive

Authors: Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, Junyang Lin

Categories: cs.CL, cs.CV, cs.SD, eess.AS

Published 2025-03-26 · Updated 2025-03-26

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub4,034 stars30 watchers322 forksApache-2.0
• Hugging Face629,250 downloads1,913 likes

Impact

• Semantic Scholar669 citations125 influential citations

Abstract

In this report, we present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. To synchronize the timestamps of video inputs with audio, we organize the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose \textbf{Thinker-Talker} architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness.

---

Introduction and Motivation

Humans naturally perceive and integrate visual and auditory signals simultaneously, then express responses through writing, speech, or physical action. While large language models (LLMs) have achieved remarkable progress on text-centric tasks, and specialist multimodal systems have extended these capabilities to audio and vision individually, building a single unified model that seamlessly handles all modalities end-to-end — and that can also speak its responses in real time — remains a hard open problem.

Qwen2.5-Omni addresses this gap. It is an end-to-end multimodal model that jointly perceives text, images, audio, and video, while simultaneously generating text and natural streaming speech in a single forward pass. The authors identify three core engineering challenges that must be solved together:

• Joint multimodal training: How to align and mutually reinforce text, image, video, and audio understanding, especially ensuring that audio and visual streams remain temporally synchronized in video.
• Interference-free multi-output generation: How to generate text tokens and speech tokens concurrently without the two output modalities disrupting each other during training or inference.
• Real-time streaming: How to minimise initial-packet latency for both multimodal inputs (chunked pre-filling) and audio output (streaming codec).

Architecture Overview: Thinker-Talker

Qwen2.5-Omni adopts a novel Thinker-Talker architecture that draws an analogy to the human brain-and-mouth system:

• The Thinker is a Transformer decoder (LLM backbone) initialised from Qwen2.5. It ingests multimodal inputs — text, audio, images, and video — via dedicated encoders, and produces both hidden-state representations and sampled text tokens autoregressively.
• The Talker is a dual-track autoregressive Transformer decoder that runs in parallel with the Thinker. It receives the full hidden representations and text embeddings emitted by the Thinker at each step, and generates discrete audio tokens. It does not operate as an independent model but shares the Thinker's entire historical context and is jointly trained end-to-end.

The dual-track design is critical: Talker simultaneously generates a text track (mirroring the Thinker's output) and a speech-token track. This way, speech generation does not require word-level or timestamp-level alignment with text, greatly simplifying training data requirements and the inference pipeline.

The motivation for passing high-level hidden representations rather than only discrete tokens to the Talker is that streaming voice generation must anticipate the overall tone and emotional intent of an utterance before the Thinker has completed generating all text. The continuous representations carry this latent prosodic and semantic context implicitly. Discrete sampled tokens are also passed to resolve phonetic ambiguity: words with very similar semantic embeddings can differ substantially in pronunciation.

Perceiving Multimodal Inputs

Text

Text is tokenised using Qwen's tokeniser, which applies byte-level byte-pair encoding (BPE) with a vocabulary of 151,643 regular tokens.

Audio

All audio (stand-alone or extracted from video) is resampled to 16 kHz and converted to a 128-channel mel-spectrogram with a 25 ms window and 10 ms hop. The audio encoder is carried over from Qwen2-Audio (itself based on Whisper-large-v3) and is adapted so that each output frame corresponds to approximately 40 ms of the original waveform. The audio encoder is modified to support block-wise attention (2-second blocks) rather than full-sequence attention, enabling chunked pre-filling for streaming input.

Images and Video (without audio)

The vision encoder is inherited from Qwen2.5-VL and is a Vision Transformer (ViT) with approximately 675 million parameters. It uses a patch size of 14 and a simple MLP that merges adjacent 2×2 patch tokens, supporting images at variable resolutions by packing tokens into sequences. Flash Attention is used for efficient training. Images are represented as two identical frames for consistency with video. Video is sampled at a dynamic frame rate to preserve information while matching the audio sampling cadence.

TMRoPE: Time-aligned Multimodal Rotary Position Embedding

Synchronising the temporal positions of audio and video frames within a shared context window is one of the central technical contributions of Qwen2.5-Omni. The authors propose TMRoPE (Time-aligned Multimodal RoPE), which extends M-RoPE (from Qwen-VL) with explicit, absolute temporal position IDs.

The rotary positional embedding is decomposed into three independent components:

• Temporal ID — one ID corresponds to 40 ms of real time.
• Height ID — spatial row index within a visual frame.
• Width ID — spatial column index within a visual frame.

For text: all three components use the same scalar position ID, making TMRoPE degenerate to standard 1D-RoPE. For audio: all three components share a single temporal ID that increments by one per 40 ms. For images: the temporal ID is constant across all tokens of a single image; height and width IDs vary per patch position. For video with audio: video frames carry incrementing temporal IDs reflecting real elapsed time (dynamically adjusted because frame rate is not fixed), while audio tokens interleaved within the same chunk also carry absolute temporal IDs at 40 ms resolution.

When multiple modalities appear in one context, position numbering for each modality is initialised by incrementing the maximum position ID of the previous modality by one, ensuring no positional collision.

Time-interleaving for Video with Audio

To enable the model to simultaneously process visual and auditory streams from video, the authors adopt a time-interleaving method: the video+audio representation is segmented into 2-second chunks. Within each chunk, the visual tokens are placed first and the audio tokens follow, producing an interleaved layout. This guarantees temporal coherence: audio and visual events that co-occur within the same 2-second window are always adjacent in the sequence.

Text and Speech Generation

Text Generation (Thinker)

Text is generated by the Thinker via standard autoregressive next-token prediction over the LLM vocabulary, with optional techniques such as repetition penalty and top-p sampling.

Speech Generation (Talker)

The Talker autoregressively generates discrete audio tokens, conditioned on both the Thinker's hidden representations and the sampled text-token embeddings. The authors design a bespoke speech codec, named qwen-tts-tokenizer, that efficiently encodes speech into discrete tokens and supports causal (streaming) decoding.

Audio tokens are decoded to waveforms via a two-stage pipeline:

1. A Flow-Matching DiT model transforms the discrete tokens into a mel-spectrogram.
2. A modified BigVGAN vocoder converts the mel-spectrogram back into a raw waveform.

Streaming Designs

The authors identify four contributors to initial-packet latency in a streaming multimodal system: (1) multimodal input processing delay, (2) delay from first text token to first speech token, (3) delay from first speech token to first audio waveform chunk, and (4) model-size-dependent compute latency. Qwen2.5-Omni targets reductions in all four dimensions.

Block-wise Multimodal Encoding (Chunked Pre-filling)

Both the audio encoder and the vision encoder are adapted to support block-wise attention along the temporal dimension. The audio encoder performs self-attention in non-overlapping 2-second blocks rather than globally over the full utterance. The vision encoder uses Flash Attention with a 2×2 MLP merge, enabling variable-resolution images to be processed efficiently in packed form. This approach decouples long-sequence multimodal perception from the LLM, assigning the perceptual role to compact encoders and the long-range modeling role to the Transformer decoder, while also enabling chunked pre-fill during streaming input.

Sliding-Window Block Attention in the DiT Codec Decoder

To enable streaming audio output with low latency, the DiT model uses a sliding window block attention mechanism. Adjacent discrete audio tokens are grouped into blocks. The DiT's attention mask limits each token to attending to at most 4 blocks: a lookback of 2 blocks and a lookahead of 1 block (plus the current block). Flow-Matching then generates the mel-spectrogram chunk-by-chunk. The BigVGAN vocoder similarly uses a fixed receptive field to facilitate streaming waveform production.

Pre-training

Pre-training proceeds in three stages:

1. Stage 1 — Encoder Training: The LLM parameters are frozen. Only the vision encoder and audio encoder (together with their adapter layers) are trained on large-scale audio-text and image-text paired data. This grounds the encoders' representations in the LLM's semantic space.
2. Stage 2 — Full Multimodal Training: All parameters are unfrozen. Training uses a broad mixture of image-text, video-text, video-audio, audio-text, and pure-text corpora. The authors report approximately 800 billion tokens of image/video data, 300 billion tokens of audio data, and 100 billion tokens of video-with-audio data in this phase. Maximum sequence length is capped at 8,192 tokens. Natural-language prompts replace hierarchical structural tags (following Qwen2-Audio), improving generalisation and instruction following.
3. Stage 3 — Long-Sequence Training: Long audio and video data are incorporated and the maximum context length is extended to 32,768 tokens, substantially improving the model's ability to process and reason over long multimodal sequences.

The LLM backbone is initialised from Qwen2.5; the vision encoder from Qwen2.5-VL; and the audio encoder from Whisper-large-v3.

Post-training

Data Format and Thinker Instruction Tuning

Post-training uses instruction-following data in ChatML format. The dataset encompasses pure text dialogue, visual-modality conversation, audio-modality conversation, and mixed-modality conversation. A representative example involves a video with two speakers, where the model is expected to transcribe and contextually describe what each person says, followed by a description of a visual scene.

Talker Three-Stage Training

The Talker undergoes a dedicated three-stage post-training process to achieve robust, natural, controllable speech generation:

1. Stage 1 — In-Context Learning (ICL) / Speech Continuation: The Talker is trained on next-token prediction over multimodal dialogue datasets that include spoken responses. It learns a monotonic mapping from semantic representations to speech tokens, and acquires contextually appropriate prosody, emotion, and accent. Timbre disentanglement techniques are applied to prevent the model from spuriously associating specific voices with rare textual patterns. Both text and audio token losses are applied simultaneously.
2. Stage 2 — DPO Stability Reinforcement: To combat hallucinations (pronunciation errors, attention misalignment, inappropriate pauses) arising from label noise in broad pretraining data, a reinforcement learning phase using Direct Preference Optimisation (DPO) is applied. A dataset of triplets $(\mathbf{x}, \mathbf{y}_w, \mathbf{y}_l)$ is constructed, where $\mathbf{x}$ is the input, $\mathbf{y}_w$ is the preferred speech output, and $\mathbf{y}_l$ is the dispreferred output. Samples are ranked by reward scores derived from Word Error Rate (WER) and punctuation-pause error rate. The DPO objective is: $$\mathcal{L}_{\text{DPO}}(\mathcal{P}_\theta; \mathcal{P}_{\text{ref}}) = -\mathbb{E}_{(\mathbf{x}, \mathbf{y}_w, \mathbf{y}_l) \sim \mathcal{D}} \left[ \log \sigma \!\left( \beta \log \frac{\mathcal{P}_\theta(\mathbf{y}_w \mid \mathbf{x})}{\mathcal{P}_{\text{ref}}(\mathbf{y}_w \mid \mathbf{x})} - \beta \log \frac{\mathcal{P}_\theta(\mathbf{y}_l \mid \mathbf{x})}{\mathcal{P}_{\text{ref}}(\mathbf{y}_l \mid \mathbf{x})} \right) \right]$$ where $\sigma$ is the sigmoid function and $\beta$ is a temperature parameter.
3. Stage 3 — Multi-Speaker Instruction Fine-tuning: Speaker-specific fine-tuning is performed to improve naturalness and controllability. Unique speaker-index tokens are introduced, replacing the beginning-of-speech token to condition the Talker on a target speaker identity.

Evaluation: Understanding (X → Text)

Text → Text

Qwen2.5-Omni-7B is benchmarked against Gemma2-9B, Llama3.1-8B, Qwen2-7B, and Qwen2.5-7B on a comprehensive suite covering general knowledge, mathematics, and coding. Results show that Qwen2.5-Omni-7B generally falls between Qwen2-7B and Qwen2.5-7B in pure-text capability, outperforming Qwen2-7B on most benchmarks. Selected results (higher is better):

The gap relative to Qwen2.5-7B is expected since the omni model must allocate capacity across all modalities, yet it surpasses the pure-text Qwen2-7B on nearly all dimensions.

Audio → Text

Qwen2.5-Omni-7B is evaluated on ASR, Speech-to-Text Translation (S2TT), Speech Entity Recognition (SER), Vocal Sound Classification (VSC), music understanding, and audio reasoning (MMAU), as well as voice-chatting benchmarks (VoiceBench and an in-house benchmark).

ASR highlights:

• CommonVoice-15 (en): WER 7.6% (vs. Whisper-large-v3: 9.3%, Qwen2-Audio: 8.6%)
• CommonVoice-15 (zh): WER 5.2% (vs. Qwen2-Audio: 6.9%)
• Fleurs (zh): WER 3.0%, matching MinMo — best reported result
• Librispeech test-other: WER 3.4% (matching Llama-3-70B; Qwen2-Audio: 3.6%)

S2TT (CoVoST2): Achieves best en→de BLEU (30.2), best zh→en BLEU (29.4), and competitive performance on de→en and en→zh.

Audio reasoning (MMAU): Qwen2.5-Omni-7B achieves 65.60% average (Sound: 67.87%, Music: 69.16%, Speech: 59.76%), substantially outperforming Gemini-Pro-V1.5 (54.90%) and Qwen2-Audio (49.20%).

VoiceBench: Average score of 74.12, surpassing all compared audio language models and omni models of similar size, including MiniCPM-o (71.69) and Baichuan-Omni-1.5 (71.14).

In-house voice-chat benchmark: Compared to Qwen2-7B (text input) and Qwen2-Audio (speech input), Qwen2.5-Omni-7B substantially narrows the gap to text-based intelligence when using speech instructions:

* Approximately 90% of text instructions suitable for speech conversion are used.

These results demonstrate that end-to-end speech instruction following in Qwen2.5-Omni is nearly on par with text input in the same model — a key contribution of the paper.

Image → Text

Qwen2.5-Omni-7B is compared to GPT-4o-mini, Qwen2.5-VL-7B, and the best-performing open-source omni models across college-level problems, math, general VQA, and OCR tasks. Selected results:

Qwen2.5-Omni-7B also demonstrates strong visual grounding performance on RefCOCO/RefCOCO+/RefCOCOg benchmarks, surpassing Qwen2.5-VL-7B on most splits, and achieves 42.2 mAP on open-vocabulary object detection (ODinW), compared to 37.3 for Qwen2.5-VL-7B (though below Grounding DINO's 55.0).

Video → Text

Qwen2.5-Omni-7B outperforms all compared open-source omni models and GPT-4o-mini on video understanding, achieving competitive or superior performance compared to the specialist Qwen2.5-VL-7B — notable given that Qwen2.5-Omni must share capacity across all modalities.

Multimodality → Text: OmniBench

OmniBench evaluates models on mixed-modality prompts combining image, audio (speech, sound events, music), and text. Qwen2.5-Omni-7B achieves state-of-the-art performance with an average of 56.13%, substantially outperforming all competitors:

Evaluation: Speech Generation (X → Speech)

Zero-Shot Speech Generation

Zero-shot speech generation is evaluated on the SEED TTS benchmark (test-zh, test-en, test-hard) measuring content consistency (WER, lower is better) and speaker similarity (SIM, higher is better). Two Qwen2.5-Omni configurations are compared: ICL (in-context learning, base model) and RL (after DPO reinforcement learning).

After RL, Qwen2.5-Omni-7B achieves highly competitive WER (1.42% zh, 2.33% en, 6.54% hard), outperforming MaskGCT and CosyVoice 2 on most splits. The RL stage specifically reduces attention misalignment, pronunciation errors, and inappropriate pauses on the challenging test-hard set. Speaker similarity remains somewhat below dedicated TTS systems (Seed-TTS), reflecting the trade-off inherent in a general-purpose omni model.

Single-Speaker Speech Generation

After speaker fine-tuning on specific target speakers (Speakers A–D), the model achieves near-human naturalness (NMOS) in both Chinese and English:

Speaker B surpasses the human recording baseline in naturalness for English (4.62 vs. unrecorded), and matches it in Chinese. Content consistency (WER) for speaker-fine-tuned variants is also preserved at near-RL-base levels (e.g., Speaker A: 1.29% zh, 1.86% en).

Key Contributions and Novelty

• TMRoPE: A principled positional encoding that decomposes rotary embeddings into temporal, height, and width components, enabling true time-alignment between audio and video tokens across an interleaved multimodal sequence. This is the first such explicit temporal-alignment encoding in a production omni model.
• Thinker-Talker Architecture: An end-to-end dual-decoder design that enables simultaneous text and speech generation without inter-modality interference. The Talker consumes continuous hidden representations from the Thinker, allowing prosodic context to flow before text generation is complete — a significant advantage for streaming naturalness.
• Block-wise streaming encoders: Audio encoder adapted to 2-second attention blocks; vision encoder using Flash Attention with MLP merge — both enabling chunked pre-fill for real-time multimodal input.
• Sliding-window DiT for streaming codec: Restricts the DiT's receptive field to 4 blocks (2 lookback + 1 lookahead), dramatically reducing initial-packet delay in audio decoding.
• Three-stage Talker training: ICL continuation → DPO stability → speaker fine-tuning, providing a systematic recipe for training a high-quality streaming speech decoder.
• Performance parity between speech and text inputs: Demonstrated on MMLU and GSM8K, showing that end-to-end speech instruction following nearly matches text-input intelligence — a milestone for conversational AI.

Limitations and Future Work

The authors note several open problems and limitations:

• Under-explored evaluation dimensions: Areas such as video OCR and audio-video collaborative understanding lack comprehensive benchmarks in the research community, requiring joint academic-industrial effort to address.
• Output modality scope: The current model generates text and speech only. Future work targets expanding generation capabilities to images, video, and music.
• Speed and robustness: The authors aim to develop faster and more robust variants of the model.
• Speaker similarity in zero-shot TTS: Qwen2.5-Omni's speaker similarity lags behind dedicated TTS systems (Seed-TTS RL), reflecting the inherent difficulty of maintaining voice identity in a general-purpose omni model without speaker-specific fine-tuning.
• Pure-text performance gap: As an omni model, Qwen2.5-Omni-7B underperforms the dedicated Qwen2.5-7B on several pure-text benchmarks (e.g., MMLU-Pro: 47.0 vs. 56.3), indicating that modality breadth still comes at some cost to text-only capability.

Conclusion

Qwen2.5-Omni represents a comprehensive and principled approach to building a unified, streaming, end-to-end multimodal model. Its three major technical contributions — TMRoPE for temporal alignment, the Thinker-Talker architecture for concurrent text-and-speech generation, and the suite of streaming engineering optimisations — together enable a model that is competitive with specialist systems on audio and vision benchmarks, approaches text-input performance when operating purely on speech, achieves state-of-the-art on multi-modal integration benchmarks such as OmniBench, and produces near-human-quality speech with low streaming latency. The work lays a strong foundation for next-generation conversational AI systems that can perceive and respond across all human communication modalities simultaneously.