VITA-1.5

VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

VITA-1.5 is a multimodal large language model for real-time vision and speech interaction in one system. Its progressive training preserves vision-language skills while adding speech understanding and generation without separate ASR or TTS, enabling fluent multimodal dialogue.

• llm
• multimodal
• dialogue
• asr
• tts
• speech-to-speech
• realtime

Demos

These demos showcase VITA-1.5's leaps in real-time multimodal interaction combining vision and speech, highlighting its ultra-low latency (1.5 seconds) and robust speech recognition with improved WER, alongside enhanced image and video understanding. Evaluate how fluently the system integrates speech input with visual context, the speed and accuracy of its responses, and its ability to maintain high-quality multimodal comprehension without latency compromise.

Authors: Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, Ran He

Categories: cs.CV, cs.SD, eess.AS

Comment: NeurIPS 2025 Spotlight, Code 2.4K Stars: https://github.com/VITA-MLLM/VITA

Published 2025-01-03 · Updated 2025-10-24

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub2,522 stars46 watchers182 forksNOASSERTION
• Hugging Face133 downloads50 likes

Abstract

Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction. Code has been released at https://github.com/VITA-MLLM/VITA.

---

1. Problem Setting and Core Idea

VITA-1.5 is a multimodal large language model (MLLM) designed to support vision, language, and speech in a single end-to-end system. The paper frames the main challenge as a modality conflict problem: vision data are spatial and image/video-centric, while speech data are temporal and acoustically structured, so naively adding speech capability can damage visual-language performance and vice versa. The authors target a more ambitious interaction setting than standard image-text MLLMs: they want the model to support real-time multimodal dialogue and speech-to-speech interaction without relying on a separate ASR pipeline followed by a separate TTS pipeline.

The central claim is that a progressive three-stage training strategy can introduce speech capability while preserving the model’s existing visual-language competence. The result is an "omni" assistant that can process images, video, text, and audio, and can generate spoken responses directly through an internal speech generation stack.

2. Model Architecture

Architecturally, VITA-1.5 follows a multimodal encoder-adapter-LLM design on the input side, and replaces the original VITA-1.0 style external speech synthesis with an end-to-end speech output module on the output side. The system therefore has two broad parts: an input stack that converts vision and audio into LLM-consumable tokens, and an output stack that turns LLM outputs into speech tokens and then waveform audio.

2.1 Visual pathway

For vision, the model uses InternViT-300M as the visual encoder with a fixed input size of $448 \times 448$ pixels, producing 256 visual tokens per image. For high-resolution images, the paper applies dynamic patching to preserve local details. Videos are handled as a special case of multiple-image input: clips shorter than 4 seconds use 4 uniformly sampled frames; clips from 4 to 16 seconds use 1 frame per second; and clips longer than 16 seconds use 16 uniformly sampled frames. The authors explicitly avoid dynamic patching on video frames to prevent the token count from becoming too large and slowing down processing.

A simple two-layer MLP adapter maps visual features into visual tokens that the LLM can consume.

2.2 Audio pathway

The audio input branch follows a speech-recognition-style design. The speech encoder uses multiple downsampling convolutional layers with $4\times$ downsampling followed by 24 Transformer blocks of hidden size 1024. The encoder has about 350M parameters and produces output at 12.5 Hz. Input features are mel-filter bank features with a 25 ms window and 10 ms shift.

A separate speech adapter uses multiple convolutional layers with $2\times$ downsampling. The paper also introduces a lightweight mechanism to help the model disambiguate modality: during audio input tuning, it adds special trainable input tokens to guide the speech understanding process, and later adds a classification head to distinguish whether the input comes from speech or text.

2.3 Speech output pathway

Unlike systems that require external TTS, VITA-1.5 generates speech with an internal codec-based output stack. The paper uses TiCodec with a single codebook of size 1024. The codec encodes speech into discrete tokens at 40 Hz and reconstructs waveform audio at 24,000 Hz.

Because the base LLM only emits text tokens, the model adds two speech decoders after the text tokens, following the approach cited by the paper. The first is a Non-Autoregressive (NAR) speech decoder that processes text tokens globally and models semantic speech features to generate an initial distribution over speech tokens. The second is an Autoregressive (AR) speech decoder that refines and generates higher-quality speech tokens step by step. Both decoders use 4 LLaMA decoder layers, have hidden size 896, and together add about 120M parameters. The final speech token sequence is decoded to waveform by the codec decoder.

3. Training Data

The multimodal instruction-tuning data span image, OCR/diagram, video, and pure text scenarios, and include both Chinese and English samples. The paper gives a detailed dataset breakdown with a total of 22,133.16K question instances across the multimodal instruction-tuning mixture. In addition, it uses 110,000 hours of internal speech-transcription paired ASR data for audio encoder training and alignment, and 3,000 hours of text-speech paired data generated by a TTS system for speech decoder training.

3.1 Sampling and language mix

The tabled instruction-tuning mixture includes both English and Chinese data. The authors also report that synthetic image data are derived from open-source image corpora such as Wukong, LAION, and CC12M. This is relevant because the model’s speech data and its generated speech responses are intended to support bilingual use, not only English-centered interaction.

4. Three-Stage Training Strategy

The most important methodological contribution is the staged curriculum. The authors argue that directly mixing speech with vision-language training can cause destructive interference, so they progressively add modalities in a way that preserves earlier capabilities. This is the paper’s main training novelty and the central mechanism behind the reported retention of vision-language performance after adding speech.

4.1 Stage 1: Vision-language training

Stage 1 has three substeps. In Stage 1.1, vision alignment, the model uses 20% of the descriptive caption data, and only the visual adapter is trainable while the other modules are frozen. This aligns visual features with the language space without disturbing the base model.

In Stage 1.2, vision understanding, the model uses all caption data, and the visual encoder, visual adapter, and LLM are trainable. The objective is to teach the model to transcribe image content into natural language descriptions.

In Stage 1.3, vision SFT, the model uses all QA data plus 20% of the caption data to improve instruction following for visual question answering. Again, the visual encoder, adapter, and LLM are trainable. The result is a strong image-and-video understanding foundation before audio is introduced.

4.2 Stage 2: Audio input tuning

Stage 2 introduces speech understanding while trying to avoid degradation of the already-trained vision-language behavior. In Stage 2.1, audio alignment, the model uses 11,000 hours of speech-transcription pairs. Training is two-step: first, the speech encoder is trained with a CTC loss so that it predicts transcription text from speech; second, the speech encoder is integrated with the LLM through the audio adapter so that the LLM can output the transcription text. The paper also notes the use of special trainable input tokens at this point to guide ASR-style speech understanding.

In Stage 2.2, audio SFT, the model trains on speech questions with text answers, using 4% of the caption data and 20% of the QA data. Approximately half of the text-based questions are randomly replaced by speech versions generated through a TTS system. The visual encoder and adapter, audio encoder and adapter, and LLM are all trainable, and the model additionally learns a classification head that distinguishes whether the input is speech or text.

4.3 Stage 3: Audio output tuning

Stage 3 adds speech generation capability while freezing the LLM so that multimodal understanding is not harmed. In Stage 3.1, codec training, the codec model is trained on 3,000 hours of text-speech data so that it can encode and decode discrete speech tokens.

In Stage 3.2, NAR + AR decoder training, text is tokenized and mapped into embeddings, speech is encoded into codec tokens, the embeddings are passed through the NAR decoder to obtain global semantic features, and the AR decoder predicts the final speech tokens. Because the LLM is frozen, this stage is explicitly designed to add speech output without changing the model’s core multimodal reasoning ability.

5. Evaluation Setup

The paper evaluates three main capabilities: image understanding, video understanding, and speech recognition. The benchmark suite is broad and mixes general perception, reasoning, OCR, mathematical reasoning, hallucination detection, and temporal understanding. For visual evaluation, the reported comparisons include both open-source and proprietary systems. For speech, the paper compares against speech-specialized baselines and earlier VITA versions.

5.1 Image benchmarks

The image benchmark set includes MMBench, MMStar, MMMU, MathVista, HallusionBench, AI2D, OCRBench, MMVet, and MME. These cover general visual reasoning, math, hallucination robustness, OCR, and chart/document-style understanding.

The main takeaway from the image table is that VITA-1.5 Stage 1 is already competitive with strong open-source models and even surpasses some closed-source entries on the reported average score, while VITA-1.5-Audio keeps the image-language average close to Stage 1. This supports the paper’s claim that audio can be added with only a small impact on visual-language performance.

5.2 Video benchmarks

For video understanding, the paper reports Video-MME with and without subtitles, MVBench, and TempCompass. VITA-1.5 Stage 1 and Stage 3 both remain in the range expected of strong open-source video-capable models, but the paper is explicit that the system still trails leading proprietary models such as GPT-4o and Gemini-1.5-Pro.

The video results show that VITA-1.5 is competitive with many open-source baselines, but the authors also stress that there remains a noticeable gap to proprietary systems. This is the paper’s clearest limitation on the visual side.

5.3 Speech / ASR benchmarks

Speech evaluation uses three Mandarin sets: aishell-1, test net, and test meeting, measured by character error rate (CER), and four English sets: dev-clean, dev-other, test-clean, and test-other, measured by word error rate (WER). The paper compares against Wav2vec2-base, Mini-Omni2, Freeze-Omni, and VITA-1.0.

VITA-1.5 is reported to outperform the compared speech-focused baselines on all listed ASR metrics. The gains are especially notable relative to VITA-1.0, indicating that the staged audio input and audio output design substantially improves speech capability.

6. What the Reported Results Suggest

The reported numbers support three specific conclusions. First, the staged training approach successfully preserves vision-language performance even after speech is added. Second, the model acquires strong ASR ability in both Mandarin and English. Third, the internal codec-based speech output stack enables speech-to-speech interaction without a separate ASR/TTS cascade, which is the basis for the paper’s real-time interaction claim.

The authors also note a practical latency improvement relative to VITA-1.0, stating that the end-to-end interaction latency in the actual demo drops from about 4 seconds to about 1.5 seconds, while the model’s direct end-to-end latency on NVIDIA A800 GPUs is about 700 ms. These numbers are mentioned in the paper text as evidence that the new architecture is more suitable for live conversational use.

7. Main Contributions and Novelty

• Progressive multimodal curriculum. The paper’s main technical idea is the three-stage progression from vision-language training to audio input tuning to audio output tuning, used specifically to mitigate modality interference.
• End-to-end speech interaction. The system removes the dependency on a separate ASR + TTS loop for speech conversation, replacing it with a codec plus NAR/AR speech decoder stack.
• Strong retention of visual ability. The paper reports that adding audio has only a small effect on image-language scores, which is a key engineering result for an omni assistant.
• Bilingual speech capability. The ASR training and evaluation are both Mandarin-English oriented, and the model is intended to support both languages in interaction.
• Real-time orientation. The architecture and reported latency are explicitly designed around conversational responsiveness rather than offline batch generation.

8. Limitations and Future Work

The paper is relatively explicit about its remaining limitations. On the vision side, the video benchmark table shows that VITA-1.5 still lags behind proprietary leaders such as GPT-4o and Gemini-1.5-Pro. On the product side, the authors state that the system is still a generic assistant and does not yet incorporate personalized preferences. They also identify long-term memory as an important open problem for interactions that last many minutes or hours.

More broadly, the paper does not claim that the speech-output path fully closes the gap to proprietary omni models; rather, it shows that open-source multimodal interaction can be pushed much further by carefully staged training and a direct speech-generation stack.

9. Practical Takeaway for Conversational-AI Teams

For a talking-head or conversational-AI system, VITA-1.5 is interesting less as a single benchmark winner and more as a design pattern. The paper argues that if you want a single model to do visual grounding, speech understanding, and speech generation, you should not simply "bolt on" audio. Instead, you should train vision first, align audio second, and only then add the speech output stack while freezing the base LLM. That recipe is the main reason the system can keep its visual-language abilities while becoming much better at speech interaction.