Mini-Omni

Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

Mini-Omni is the first fully end-to-end, open-source conversational model enabling real-time speech interaction by generating speech and text together. Its decoding methods support natural, low-latency voice conversations while keeping the language model's reasoning intact.

• llm
• multimodal
• dialogue
• asr
• tts
• streaming
• realtime
• speech-to-speech

Demos

These demos illustrate Mini-Omni's core capability as a multimodal large language model that simultaneously processes speech input and generates streaming audio output in real time. Watch for smooth, uninterrupted voice conversation showcasing its unique ability to "hear, talk while thinking," without separate ASR or TTS modules. The included framework diagram clarifies its integrated architecture supporting continuous speech-to-speech interaction and streaming audio synthesis.

Authors: Zhifei Xie, Changqiao Wu

Categories: cs.AI, cs.CL, cs.HC, cs.LG, cs.SD, eess.AS

Comment: Technical report, work in progress. Demo and code: https://github.com/gpt-omni/mini-omni

Published 2024-08-29 · Updated 2024-11-05

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3,561 stars79 watchers311 forksMIT
• Hugging Face1 downloads442 likes

Abstract

Recent advances in language models have achieved significant progress. GPT-4o, as a new milestone, has enabled real-time conversations with humans, demonstrating near-human natural fluency. Such human-computer interaction necessitates models with the capability to perform reasoning directly with the audio modality and generate output in streaming. However, this remains beyond the reach of current academic models, as they typically depend on extra TTS systems for speech synthesis, resulting in undesirable latency. This paper introduces the Mini-Omni, an audio-based end-to-end conversational model, capable of real-time speech interaction. To achieve this capability, we propose a text-instructed speech generation method, along with batch-parallel strategies during inference to further boost the performance. Our method also helps to retain the original model's language capabilities with minimal degradation, enabling other works to establish real-time interaction capabilities. We call this training method "Any Model Can Talk". We also introduce the VoiceAssistant-400K dataset to fine-tune models optimized for speech output. To our best knowledge, Mini-Omni is the first fully end-to-end, open-source model for real-time speech interaction, offering valuable potential for future research.

---

Introduction and Problem Setting

The paper targets a specific gap in multimodal LLMs: while contemporary systems can understand text, images, and sometimes audio, most academic speech-capable models still depend on a cascade pipeline in which the language model first produces text and a separate text-to-speech system converts that text into speech. The authors argue that this design introduces undesirable latency and prevents truly real-time voice interaction. They position Mini-Omni as an end-to-end conversational model that can listen and speak in streaming, while also retaining useful text capabilities.

The central claim is that direct speech interaction can be approximated by leveraging a strong text model, but only if speech generation is trained and decoded in a way that preserves reasoning and supports streaming. To that end, the paper proposes a simple architecture around a small base LLM, audio tokenization, parallel text-audio decoding, and a staged training recipe that minimizes changes to the original language model.

• Model goal: real-time speech interaction with speech input and streaming speech output.
• Main challenge: reasoning directly in audio is difficult, and naive audio-only training can produce incoherent outputs.
• Design philosophy: keep the architecture simple, use discrete speech tokens, and transfer text capability into the audio path with minimal extra data.
• Author claim: this is the first fully end-to-end open-source model for real-time speech interaction.

Model Architecture Overview

The architecture keeps the backbone intentionally lightweight. The base language model is Qwen2-0.5B, a transformer with 24 blocks and hidden size 896. Speech input is encoded with Whisper-small, and the model uses adapters to align audio features with the language model. For output speech, the system uses a multi-codebook audio tokenization scheme based on SNAC, chosen because the authors prefer higher audio quality over a lower-bitrate codec. The paper explicitly frames this as a trade-off: a richer codec increases token count, but the authors compensate with parallel and delayed decoding.

Conceptually, the model treats text tokens and discrete speech tokens as members of a shared token space. Let $Y=(y_i \in \mathcal{V}_{\text{txt}})$ denote a text utterance and $D=(d_i \in \mathcal{V}_{\text{dst}})$ denote discrete speech tokens produced by tokenizing audio. The combined vocabulary is written as

$$\mathcal{V}_{\text{voxt}}=\mathcal{V}_{\text{txt}}\cup\mathcal{V}_{\text{dst}}.$$

In this formulation, the model can generate text, speech tokens, or mixed text-speech sequences in a single autoregressive framework. The paper describes the training objective over paired text-audio outputs $(T,A)$ conditioned on input $X_j$ as

$$\mathcal{L}(T,A\mid C)=\sum_{j=1}^{m}\sum_{i=1}^{n_j}\log P(T_{i,j},A_{i,j}\mid T_{<i,j},A_{<i,j};X_j).$$

In words, the model maximizes the joint likelihood of text and audio tokens over the training corpus, rather than treating speech generation as a separate post-processing step. The authors emphasize that this lets the model reuse the text model’s reasoning skill while producing streamed audio.

Streaming Decoding Strategy

The most distinctive technical contribution is the decoding strategy. Instead of generating text fully and then invoking an external TTS system, the model produces text and audio tokens in parallel. The intuition is that text has higher information density, so a textual stream can carry the core reasoning content while the speech stream is generated as an online synthesis process. The paper notes that text can be generated first, with audio tokens then conditioned on those text tokens, which reduces first-token delay and makes the system suitable for conversation.

Text-instructed parallel generation

The authors refer to this as text-instructed speech generation. Their implementation pads the sequence with $N$ tokens so that the text side is produced before the corresponding audio tokens. This is described as a hyperparameter-controlled delay mechanism. The decoder therefore emits a text token first, followed by SNAC code tokens, so the speech stream is grounded in a textual plan rather than being generated blindly from audio tokens alone.

The paper also states that speaker and style embeddings can be used, allowing the system to control who is speaking and how the speech should sound. This is important because the model is not just doing recognition plus synthesis; it is aiming for conversational output that behaves like a voice assistant.

Text-delay parallel decoding

The paper adapts ideas from parallel generation methods used in music modeling to speed up audio inference. Because codec-based speech representations often contain multiple layers or codebooks, several tokens can be predicted in one step. Mini-Omni uses SNAC and a set of eight sub-language-model heads so that, in one decoding step, the system can generate a text token and multiple speech-code tokens with a one-step delay between adjacent layers. The authors describe this as text-first delayed parallel decoding.

The motivation is practical: real-time speech output requires hundreds of tokens per second on standard devices. The delayed parallel scheme is the mechanism that makes this feasible without discarding the richer audio representation.

Batch parallel decoding

The second decoding trick is batch parallel decoding. The authors observe that even after parallel generation, speech-side reasoning is still weaker than text-side reasoning, and audio answers tend to be simpler. Their solution is to expand inference for a single user input into a batch of two samples. One sample must produce both text and audio, while the other produces only text. The text output from the text-only sample is then embedded into the text token positions of the audio-producing sample, and the audio stream is rendered using the text content from the text-only branch.

In effect, the method tries to borrow the stronger reasoning path of the text-only branch and inject it into the audio branch at inference time. The paper claims that this almost entirely transfers text capability into audio generation with very little extra overhead, and that this is a key reason a 0.5B model can produce comparatively strong conversational results.

Any Model Can Talk: Training Recipe

The training strategy is designed to preserve the original language model as much as possible. The authors present this as a recipe that can be applied to other strong text models that do not yet have robust speech interaction. They call the method Any Model Can Talk, emphasizing that the architecture changes are minimal and most of the work is done by staged training and adapter design.

Adapters and modality alignment

The audio front end uses a speech encoder and adapter layers to map speech features into the language model’s internal space. The paper says the ASR adapter is a two-layer MLP, while the TTS adapter extends the original model by adding six transformer blocks. During the first stage, the core language model is frozen and only the adapters learn. This stage uses speech recognition and speech synthesis data to align the audio modality with the text model.

The design goal is to avoid retraining the whole language model just to learn basic speech alignment. Instead, the model learns how to interpret and emit audio through a narrow set of trainable modules, while preserving the base model’s text competence.

Three-stage training pipeline

1. Modality alignment: freeze the base model and train only the adapters on speech recognition and speech synthesis data.
2. Adaptation training: freeze the adapters and train the model to answer textually when prompted with audio inputs, using speech recognition, spoken question answering, and text response data.
3. Multimodal fine-tuning: unfreeze all weights and train the full system on broader multimodal data so the model can handle voice interaction end to end.

The paper emphasizes that the third stage is where full multimodal interaction is learned, but only after the audio-text alignment has been established. This staged approach is intended to minimize catastrophic degradation of the original text model.

Input formatting for parallel generation

Because the decoder operates over multiple parallel streams, the input construction is more complex than in standard LLMs. The paper states that the model can accept either text or audio inputs, and the corresponding tokens are placed into modality-specific sequences. For audio input, the input tokens and Whisper features are mapped by adapters into the same dimensionality and concatenated. The special <answer> token is placed at different positions depending on the task, so the model knows which stream should begin generation.

The authors also note that the multiple sequences are summed and averaged before entering the model. This implementation detail is part of how the parallel streams are integrated into a single transformer backbone.

Datasets and Training Setup

To teach the model speech input, speech output, and voice-assistant behavior, the paper combines several dataset families. A key motivation for introducing a new dataset is that many open-source QA corpora contain code, symbolic expressions, or long-form text that is not well suited for speech output. The authors therefore create VoiceAssistant-400K, a GPT-4o-synthesized dataset intended to look more like a voice assistant’s responses.

The paper says this dataset contains over 400,000 entries and is designed to avoid code-heavy or overly verbose outputs. It is used in the final stage of training to improve voice-assistant style responses.

The modality notation in the table indicates whether the input or output is audio or text. For example, $A_1|T_1$ denotes audio input and text output, while $A_1|T_1|A_2|T_2$ denotes a more complex interaction involving both audio and text on the input and output sides.

The paper reports the following training configuration: training on 8 A100 GPUs, cosine annealing learning-rate scheduling with a minimum learning rate of $4\times10^{-6}$ and a maximum learning rate of $4\times10^{-4}$, batch size 192, and 40,000 steps per epoch. Fine-tuning uses a learning rate range from $4\times10^{-6}$ to $5\times10^{-5}$. The paper also states that the model uses Whisper-small as the speech encoder and Qwen2-0.5B as the base model.

Reported Experiments and Results

The experimental section is explicitly framed as a foundational capability test rather than a complete benchmark suite. The paper says that additional experiments will be included in a later version. The reported quantitative evaluation is concentrated on ASR, while the rest of the evidence is mainly qualitative case studies showing real streaming output.

ASR evaluation on LibriSpeech

The paper evaluates recognition on the LibriSpeech test-clean, test-other, dev-clean, and dev-other splits. The comparison includes wav2vec2-base, VITA, Whisper-small, and Mini-Omni. The authors conclude that Mini-Omni slightly trails Whisper-small but still achieves strong speech comprehension.

Relative to the baselines reported in the paper, Mini-Omni is better than wav2vec2-base and VITA on the shown splits, but it does not surpass Whisper-small on ASR. The authors interpret this as evidence that the speech pathway is functional and reasonably accurate even though the system is optimized for real-time speech interaction rather than pure recognition.

Qualitative case studies

The case-study section provides streaming examples rather than a broad quantitative speech-to-speech benchmark. The authors say that these examples show speech understanding and reasoning, but also make clear that speech-side reasoning is weaker than text-side reasoning. That observation is one of the main motivations for batch parallel decoding. The paper presents the examples as evidence that the model can produce live, conversational audio output in real time.

Technical Contributions and Novelty

The paper’s novelty is concentrated in a small number of pragmatic ideas rather than in a large architectural overhaul. First, it treats speech as a discrete token stream that can be modeled by the same language-model machinery used for text. Second, it adds a text-instructed streaming decoding scheme so speech output can start immediately rather than waiting for an entire text response. Third, it uses a batch trick to import stronger text reasoning into the audio path during inference. Fourth, it introduces a staged adapter-based training recipe that is intentionally designed to preserve the base model’s text ability.

• Unified token modeling: text and discrete speech tokens are modeled jointly in a single autoregressive framework.
• Streaming output: text-first delayed parallel decoding makes real-time speech generation feasible.
• Inference-time reasoning boost: batch parallel decoding improves audio-side reasoning with minimal overhead.
• Low-modification adaptation: Any Model Can Talk is meant to retrofit speech interaction onto existing text models.
• Data contribution: VoiceAssistant-400K addresses the lack of speech-assistant-style supervision data.

Limitations and Scope of the Report

The paper is a technical report and repeatedly signals that it is still a work in progress. It does not present a full benchmark suite for speech-to-speech generation, nor does it report extensive ablations on the decoding or training choices. The authors also state that more experiments will be added in a future version.

Two limitations stand out from the text. First, speech reasoning is still described as weaker than text reasoning, which is why batch parallel decoding is needed. Second, the authors acknowledge that direct audio reasoning is hard and may be limited by either model capacity or insufficient audio data. The work therefore should be read as an early but concrete step toward real-time end-to-end voice interaction, not as a fully solved system.

Another practical limitation is that the paper favors audio quality and reusability over maximal simplicity: the SNAC-based output path is token-heavy, and the system depends on multiple adapters, parallel heads, and staged training. This complexity is still far smaller than building a fully new speech model from scratch, but it is not yet a minimal inference stack.

Conclusion

Mini-Omni is presented as an end-to-end speech interaction model built by combining a small but capable text backbone, discrete audio tokens, text-instructed streaming decoding, and a three-stage adapter-based training recipe. The paper’s strongest concrete claims are that real-time speech output is possible with a 0.5B model, that language capability can be largely preserved, and that a specialized data recipe such as VoiceAssistant-400K helps produce more natural voice-assistant behavior. The overall contribution is a practical recipe for giving existing language models speech output while keeping them streamable and conversational.