Baichuan-Omni-1.5

Baichuan-Omni-1.5 Technical Report

Baichuan-Omni-1.5 is a 7B omni-modal model unifying text, image, video, and audio understanding with end-to-end audio and text generation. It features a novel audio tokenizer and multi-stage training to maintain strong language ability and enable seamless cross-modal interactions for real-time conversational AI.

• llm
• multimodal
• asr
• tts
• speech-to-speech
• realtime

Demos

The demo showcase highlights Baichuan-Omni-1.5's omni-modal capabilities to understand and generate across text, image, video, and audio inputs, producing both text and speech outputs. Evaluate the architecture for its unified multi-modal processing design, observe the multi-stage training pipeline ensuring alignment of modalities, and assess performance charts demonstrating state-of-the-art benchmarks across vision-language, audio, video, and medical tasks. Watch for comprehensive integration and controllable bilingual real-time interaction potential.

Authors: Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, Chong Li, Yuanbo Fang, Dongdong Kuang, Mingrui Wang, Chenglin Zhu, Youwei Zhang, Hongyu Guo, Fengyu Zhang, Yuran Wang, Bowen Ding, Wei Song, Xu Li, Yuqi Huo, Zheng Liang, Shusen Zhang, Xin Wu, Shuai Zhao, Linchu Xiong, Yozhen Wu, Jiahui Ye, Wenhao Lu, Bowen Li, Yan Zhang, Yaqi Zhou, Xin Chen, Lei Su, Hongda Zhang, Fuzhong Chen, Xuezhen Dong, Na Nie, Zhiying Wu, Bin Xiao, Ting Li, Shunya Dang, Ping Zhang, Yijia Sun, Jincheng Wu, Jinjie Yang, Xionghai Lin, Zhi Ma, Kegeng Wu, Jia li, Aiyuan Yang, Hui Liu, Jianqiang Zhang, Xiaoxi Chen, Guangwei Ai, Wentao Zhang, Yicong Chen, Xiaoqin Huang, Kun Li, Wenjing Luo, Yifei Duan, Lingling Zhu, Ran Xiao, Zhe Su, Jiani Pu, Dian Wang, Xu Jia, Tianyu Zhang, Mengyu Ai, Mang Wang, Yujing Qiao, Lei Zhang, Yanjun Shen, Fan Yang, Miao Zhen, Yijie Zhou, Mingyang Chen, Fei Li, Chenzheng Zhu, Keer Lu, Yaqi Zhao, Hao Liang, Youquan Li, Yanzhao Qin, Linzhuang Sun, Jianhua Xu, Haoze Sun, Mingan Lin, Zenan Zhou, Weipeng Chen

Categories: cs.CL, cs.SD, eess.AS

Published 2025-01-26 · Updated 2025-01-26

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub192 stars7 watchers9 forksApache-2.0
• Hugging Face258 downloads48 likes

Impact

• Semantic Scholar110 citations9 influential citations

Abstract

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.

---

Introduction

Baichuan-Omni-1.5 is a 7B omni-modal model designed to unify text, image, audio, and video understanding with end-to-end text and audio generation. The central problem addressed by the paper is the same one that constrains many multimodal systems: adding new modalities can improve coverage while degrading core language ability, increase latency if the system relies on cascaded ASR/TTS modules, and create modality conflicts during training. The report’s answer is to combine three ingredients: a large cleaned and synthesized multimodal corpus, an audio tokenizer that bridges speech and language spaces, and a multi-stage training curriculum that gradually aligns modalities instead of mixing everything from the start.

The model is positioned as an omni-modal alternative to systems that either support only vision-language tasks or require separate speech recognition and synthesis modules. A key design choice is non-cascading audio generation: the model directly alternates between text tokens and audio tokens, and the generated audio tokens are decoded into waveforms by a dedicated audio decoder. This is intended to reduce latency and complexity while keeping speech understanding and speech generation within one model. The paper also emphasizes that the final system retains strong pure-language performance, which is an important signal that multimodal training has not catastrophically erased language competence.

Core Contributions and Design Goals

The paper’s stated contributions can be summarized as follows. First, it constructs a large-scale multimodal data pipeline that yields about $500$B high-quality data spanning text, audio, and vision, with careful cleaning, synthesis, and balancing to preserve text capability. Second, it introduces the Baichuan-Audio-Tokenizer, a residual-vector-quantization-based audio tokenizer that captures both semantic and acoustic information at a frame rate of $12.5$ Hz, enabling tighter integration between audio and an MLLM backbone. Third, it proposes a multi-stage training strategy that incrementally brings in vision, then audio, then cross-modal interaction, then supervised fine-tuning. In the reported experiments, the model is competitive on pure language, strong on image and video understanding, strong on speech understanding and generation, and especially notable in medical image settings.

The report also states that the project open-sources the code, the model, and constructed audio and medical benchmarks. In addition to being a general omni-modal model, it is presented as a baseline for real-time interaction scenarios and as a step toward medical conversational assistance.

Overall System Overview

The system has three main components: a visual branch, an audio branch, and a pretrained large language model backbone. The visual branch converts image and video inputs into visual tokens that can be consumed by the language model. The audio branch converts speech to discrete tokens and reconstructs speech from generated tokens. The language backbone performs cross-modal reasoning and generation. The architecture is built so that image, video, text, and audio can all be handled in a unified token space without splitting the system into separate expert models for each modality.

Visual branch

The visual branch uses NaViT from Qwen2-VL as the visual encoder, which supports images and videos of arbitrary resolution and aspect ratio. A two-layer MLP projector then compresses the visual feature map by a $2 \times 2$ factor. This is a classic efficiency-versus-capacity tradeoff: the encoder preserves detailed visual information, while the projector reduces the token burden before visual tokens are injected into the language model.

Audio branch

The audio branch is the paper’s main novelty beyond standard multimodal design. The Baichuan-Audio-Tokenizer is based on residual vector quantization (RVQ) and multi-objective training. Audio is first converted to Mel spectrogram features, and a Whisper Large encoder extracts high-level representations. A residual convolutional network downsamples these features to a low-frame-rate sequence, and an 8-layer RVQ quantizes the sequence into discrete audio tokens at $12.5$ Hz. These tokens are used both for reconstruction and for transcript prediction, making the tokenizer useful for both semantic and acoustic alignment.

The paper describes two downstream paths from the audio tokens. One path feeds tokens into an audio decoder for Mel spectrogram reconstruction, and the other feeds tokens into the pretrained language model so that the model can learn speech semantics jointly with text. The audio decoder is structurally symmetric to the Whisper encoder and is trained with a multi-scale Mel loss. The final waveform is produced with a HiFi-GAN vocoder after the decoder’s flow-matching module predicts target Mel spectrograms. The flow-matching decoder uses a U-Net with one down-sampling block, one up-sampling block, and 12 intermediate blocks, and it is trained on $24$ kHz audio.

For speech generation, the model uses an independent audio head consisting of 3 depth-transformer layers and 8 classification heads. This head is responsible for generating audio tokens, while the language model alternates between text-token prediction and audio-token prediction via a special modality-switching token. This is the mechanism that enables the non-cascading text-plus-audio output path.

Data Construction

The report gives unusual emphasis to data quality. Rather than simply aggregating heterogeneous public datasets, the authors build a cleaning and synthesis pipeline that produces balanced corpora for text, image-text, audio-text, video-text, and mixed image-audio-text / video-audio-text interaction. The motivation is explicit: adding audio and video naively can damage text performance and produce modality conflicts. The solution is to maintain pure text in the mixture, synthesize missing forms of supervision, and use a staged curriculum to avoid overfitting any single modality.

Pretraining data by modality

Image data

Image pretraining is organized into caption data, interleaved image-text data, OCR data, and question-answer data. The corpus is intentionally broad, spanning general web images, documents, charts, tables, and specialized scientific content. The paper lists many public sources such as DenseFusion-1M, Synthdog, DreamLIP, InternVL caption data, LAION-5B, OBELIC, Cauldron, ArxivQA, TGDoc, MMTab, AnyWord-3M, TinyChartData, and DocStruct4M, alongside large in-house collections. A notable synthesis strategy is to parse books and papers into interleaved image-text, OCR, and chart data, and to generate additional Chinese captions and interleaved examples so that the model does not become English-dominant.

For image SFT, the data is grouped into several capability buckets: GeneralQA, OCR, graphical reasoning, mathematics, spatiotemporal reasoning, captioning, and medical understanding. The authors explicitly state that OCR proportions had a large impact on overall performance and were tuned carefully; the final OCR share is 26.51% of image data in the image SFT mix.

Video data

Video pretraining and video SFT use a mix of captioning and QA sources. The paper mentions public datasets for general video understanding, action recognition, and temporal localization, plus synthetic captions and GPT-4o-generated captions for YouTube clips. For video SFT, the dataset is curated and rebalanced by scene type, task difficulty, answer quality, and overall video quality. GPT-4o is used to categorize the video data fine-grainedly before rebalancing.

For omni-modal pretraining, video is sampled at $1$ frame per second with a maximum of $32$ frames per video, and each frame is resized to at most $560 \times 1120$ pixels. The paper also notes an extension to a $64$k sequence length to support long audio and video streams.

Audio data

Audio data is split into understanding and generation sets. Understanding includes ASR, audio question answering, speech-to-text translation, and audio-text interleaving. Generation includes TTS, interleaved TTS, and pure audio. A particularly important synthesis trick is the construction of interleaved audio-text examples, where sentences are segmented and a random quarter of the text is converted into audio to form mixed-modality sequences. The paper reports 44 distinct voice types in the cross-modal interaction data and 10,000 distinct voice tones for synthesized audio instructions in the SFT stage. An ASR-based quality filter is used to compare generated audio with source text, and mismatches are fed back into either the TTS or ASR pools, which helps turn failure cases into training signal.

Cross-modal interaction data

The cross-modal interaction dataset is especially relevant for real-world conversational systems. For image-text interaction, a portion of the textual description is replaced by generated audio and the model is asked to combine image and audio to recover or extend the full description. For video-text interaction, the audio is taken directly from the original video. This design is meant to teach the model to coordinate multiple input streams instead of treating speech as a simple transcription channel.

Training Strategy

The training strategy is staged to preserve existing ability while adding new modalities. The paper’s overall logic is: first align images, then add audio, then introduce mixed cross-modal pretraining, and finally perform multimodal instruction tuning. The authors repeatedly freeze some modules while training others to reduce representational drift and avoid catastrophic forgetting.

Stage 1: image-text pretraining

Image-text pretraining is itself split into two phases. In the first phase, the visual projector is trained while the large language model and visual encoder remain frozen. The learning rate is $10^{-3}$. This phase is described as establishing the initial alignment between image representations and text. In the second phase, the visual encoder and the language model are unfrozen. The language model and projector are trained with learning rate $10^{-5}$, while the visual encoder uses the lower learning rate $10^{-6}$. The data mix includes public and in-house image-text data, plus OCR and chart data, and pure text makes up 40% of the total to keep language ability intact.

Stage 2: image-audio-text pretraining

Audio is introduced after the visual alignment stage. Here the model starts from the visual model and is extended to end-to-end speech understanding. In the first audio stage, the large language model, visual modules, and audio tokenizer are frozen; only the audio embedding layer and audio head are trained, with learning rate $10^{-4}$. The training mix includes ASR, TTS, interleaved audio-text, and interleaved TTS data. In the second audio stage, training is expanded to all parameters except the visual encoder and the audio tokenizer, with learning rate $10^{-5}$. The data ratio is audio $0.2$, image $0.4$, and pure text $0.4$, explicitly balancing the modalities to maintain both speech and visual-language ability.

This is also where the paper’s audio-tokenizer design matters most. By freezing the pretrained language model in part of the stage, the authors try to make the audio tokens semantically align to the text space instead of allowing the tokenizer and language model to drift apart.

Stage 3: omni-modal pretraining

After the image and audio branches are aligned, the paper trains all parameters on high-quality cross-modal interaction data spanning image-audio-text and video-audio-text. The learning rate is $4 \times 10^{-6}$. This stage extends the maximum sequence length to $64$k to support long audio and video streams. The stated goal is to consolidate cross-modal reasoning and interaction after the individual branches already know how to speak the same representational language.

Stage 4: multimodal supervised fine-tuning

The SFT stage uses around 17M data pairs across text, image, video, audio, and image-audio tasks. The paper describes a two-step SFT schedule. First, it freezes the audio head and trains using omni-modal understanding data, with the aim of improving modality interactivity and multitask comprehension. Second, it activates only the audio head and audio embedding layer and trains on audio generation data to refine speech output. The report also notes a packing technique using cuseq_len from flash-attention2 to concatenate multiple samples while preserving sample boundaries, improving throughput and memory efficiency.

Experimental Setup

The paper evaluates the 7B model across pure language, image understanding, video understanding, audio understanding, omni-modal interaction, and medical benchmarks. Unless otherwise stated, experiments are zero-shot. Many results are reproduced by the authors using the same settings as the baselines, and the paper emphasizes fairness in evaluation. For multimodal benchmarks, it uses VLMEvalKit where applicable; for audio tasks and omni tasks, GPT-4o-based judging is used in several places.

A useful diagnostic detail is that the model is evaluated in two audio directions: speech-to-text ($s \to t$) and speech-to-speech ($s \to s$), where the latter is a non-cascaded generation path that outputs interleaved text and audio. This separation makes it possible to see whether the model is better at understanding speech than generating speech, and whether direct end-to-end audio output introduces quality loss.

Pure language benchmarks

On pure language tasks, Baichuan-Omni-1.5 remains competitive with strong text models and clearly outperforms earlier omni-modal baselines. The paper’s main point here is not that it beats proprietary frontier models like GPT-4o, but that it avoids the common multimodal failure mode where language skill falls apart after audio and vision are introduced. Its gains over MiniCPM-o 2.6 on CMMLU, C-Eval, MMLU, and AGIEval support that claim.

Image understanding

On image tasks, the model is strongest on benchmark-style multimodal reasoning and OCR-heavy evaluation. It leads the reported open-source omni-modal systems on MMBench-EN and MMBench-CN, and it is also strong on MMMU and OCRBench. Compared with Qwen2-VL-7B, its image performance is broadly comparable, though Qwen2-VL remains stronger on some pure visual-text retrieval-style metrics such as TextVQA and OCRBench.

Video understanding

The video results show a consistent strength on general video VQA and a strong result on open-ended video QA, especially on MSVD-QA. One nuance worth preserving from the table is that the model does not dominate every single video metric; for example, MiniCPM-o 2.6 slightly edges it on ActivityNet-QA accuracy. The broader takeaway is that Baichuan-Omni-1.5 is competitive across both perception-heavy and open-ended video reasoning tasks while using a relatively small $7$B backbone.

Audio understanding and generation

The audio benchmark suite, OpenAudioBench, includes Reasoning QA, Spoken Llama Questions, Web Questions, TriviaQA, and AlpacaEval. The paper evaluates both $s \to t$ and $s \to s$ settings. The reported results are:

These results suggest a nuanced picture. In pure speech-to-text evaluation, GPT-4o-Audio remains stronger on several knowledge-style metrics. Among open-source systems, however, Baichuan-Omni-1.5 is very competitive and clearly strong in the direct speech-to-speech setting, where it surpasses GLM-4-Voice by a wide margin on Reasoning QA and AlpacaEval. The gap between raw audio and transcript-based evaluation also shows that current omni-modal audio recognition is still a bottleneck: transcript inputs consistently outperform raw audio inputs across models.

Omni-modal interaction

OmniBench is valuable because it directly tests mixed input streams. The paper’s main diagnostic conclusion here is that using audio transcripts is easier than using raw audio, which reflects the current limitations of speech understanding rather than a weakness of the image branch. Baichuan-Omni-1.5 is strongest on the raw multimodal settings that combine images with audio, but MiniCPM-o 2.6 retains an edge when the audio has already been transcribed. This is a useful indicator of where future work should focus: tighter speech recognition and richer spoken-state reasoning.

Medical understanding

Medical evaluation is one of the strongest parts of the report. The paper introduces OpenMM-Medical, built from 42 public medical image datasets and totaling 88,996 images, each paired with multiple-choice VQA. GMAI-MMBench is also used, with 284 clinical datasets, 38 modalities, 18 clinical VQA tasks, and 18 clinical departments. Baichuan-Omni-1.5 achieves the highest score on OpenMM-Medical and is very competitive on GMAI-MMB-VAL, though it does not surpass Qwen2-VL-72B on that specific validation split. The broader message is that the model is unusually strong in medical reasoning for a 7B omni-modal system.

Interpretation and What the Results Suggest

Three empirical patterns stand out. First, maintaining a substantial pure-text share during multimodal training appears to protect language competence. The language table shows that the model does not collapse on text benchmarks even after adding audio and video. Second, direct end-to-end speech modeling is feasible without reducing the system to a cascade of ASR plus TTS modules. The audio results, especially in the $s \to s$ setting, show that the unified audio head can produce usable speech and often beats prior open-source omni-modal systems. Third, the largest gains are seen when modality mixing is hardest: mixed image-audio tasks and medical visual reasoning. These are exactly the kinds of scenarios where a single unified representation can be more useful than separate per-modality systems.

At the same time, the results reveal the remaining bottlenecks. Audio understanding is still weaker on raw audio than on transcripts; some image metrics remain below the best specialized vision-language models; and video performance, while strong, is not uniformly dominant over every baseline. In other words, the model is strong as a balanced omni-modal system, but the report does not claim that every subskill is best-in-class.

Ablations and Diagnostic Observations

The provided LaTeX does not include a dedicated ablation table with controlled numerical sweeps. Instead, the paper justifies the design choices through the staged curriculum and by showing the resulting benchmark behavior. The closest thing to an ablation-style observation is the repeated comparison between raw audio and transcript-based settings in OmniBench and the contrast between $s \to t$ and $s \to s$ audio evaluation. These comparisons indicate that speech recognition remains harder than text generation and that the direct speech output path is still more challenging than text-only decoding.

There are also several implicit design lessons encoded in the training setup: using a large amount of pure text prevents language degradation, OCR and chart data improve visual-text alignment, interleaved audio-text data helps contextual speech modeling, and freezing the language backbone during early audio alignment protects the original textual knowledge. These are not presented as separate ablations, but they are the paper’s strongest methodological claims.

Limitations and Future Work

The conclusion section is explicit about remaining limitations. The authors say there is still substantial room to improve (1) pure text understanding, (2) longer video-frame understanding, and (3) audio understanding and generation beyond human voices. They specifically mention natural environmental sounds such as flowing water, bird songs, and collision noises as examples of audio phenomena that the system should eventually model. This is an important limitation because it means the current audio branch is optimized mainly for speech-centric interaction rather than general acoustic scene understanding.

More broadly, the paper suggests that better omni-modal systems will require continued work on scaling data quality, supporting longer temporal context, and improving the audio tokenizer and decoder so that the model can handle richer acoustic diversity without losing text or visual reasoning strength.

Conclusion

Baichuan-Omni-1.5 is a carefully engineered omni-modal system whose main value lies in the combination of broad modality coverage and strong preservation of language competence. Its most distinctive technical pieces are the cleaned and synthesized multimodal data pipeline, the Baichuan-Audio-Tokenizer, and the multi-stage training curriculum that gradually aligns image, audio, and video with the language model backbone. Experimentally, the model is competitive across language, image, video, audio, omni-modal, and medical benchmarks, with particularly strong results in image-language tasks, speech-to-speech interaction, and medical image understanding. The paper’s final message is that effective omni-modal modeling is less about simply adding modalities and more about carefully managing data balance, tokenization, and stagewise training so that new capabilities do not erase old ones.