LLaMA-Omni

LLaMA-Omni: Seamless Speech Interaction with Large Language Models

LLaMA-Omni enables real-time, low-latency speech interaction with open-source LLMs by generating text and speech responses simultaneously from speech input, bypassing typical ASR and TTS steps. It delivers natural spoken dialogue with minimal delay and better quality than prior models.

• llm
• multimodal
• dialogue
• speech-to-speech
• tts
• asr
• streaming
• low-latency

Authors: Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng

Categories: cs.CL, cs.AI, cs.SD, eess.AS

Comment: ICLR 2025

Published 2024-09-10 · Updated 2025-03-01

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3,141 stars34 watchers223 forksApache-2.0
• Hugging Face113 downloads418 likes

Abstract

Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

---

Overview

LLaMA-Omni is an end-to-end speech-interaction architecture for open-source LLMs that aims to match the low-latency conversational behavior of systems such as GPT-4o while preserving strong response quality. The central idea is to let an LLM consume a speech instruction directly, generate a text response autoregressively, and simultaneously generate the corresponding speech units with a streaming decoder, rather than using a conventional ASR-to-LLM-to-TTS cascade.

The paper argues that cascaded ASR/TTS systems incur avoidable latency because they must first transcribe the user speech, then generate text, and only then synthesize speech. Prior speech-language models can remove the transcription step, but many either rely on sequential generation of text and speech or require much larger-scale training. LLaMA-Omni is designed to address both issues: it uses a pretrained speech encoder, a lightweight speech adaptor, a strong open-source LLM, and a non-autoregressive streaming speech decoder trained with CTC.

The paper’s main claimed contributions are:

• an end-to-end architecture for simultaneous text and speech response generation from speech input;
• a streaming speech decoder trained with CTC so the model does not need explicit speech-text pre-alignment;
• a speech-instruction dataset, InstructS2S-200K, tailored to speech interaction style rather than text-chat style;
• strong latency/quality trade-offs on a new benchmark, including a reported minimum latency of 236 ms in streaming mode.

Model architecture

LLaMA-Omni decomposes the problem into four modules:

• Speech encoder $mathcal{E}$: the encoder from Whisper-large-v3, frozen during training;
• Speech adaptor $mathcal{A}$: a trainable adapter that downsamples the speech representation and maps it to the LLM embedding space;
• LLM $mathcal{M}$: Llama-3.1-8B-Instruct, used to generate the text response;
• Streaming speech decoder $mathcal{D}$: a causal, non-autoregressive Transformer stack that predicts discrete speech units from the LLM hidden states using CTC.

Note: the strange symbol here is just a rendering safeguard in the draft; the actual math below is written in standard KaTeX-safe notation.

Let the user’s speech instruction be $X^S$, the text response be $Y^T$, and the speech response be $Y^S$. The speech encoder produces a sequence of hidden states

$$ \mathbf{H} = \mathcal{E}(X^S) = [\mathbf{h}_1, \ldots, \mathbf{h}_N]. $$

The adaptor first downsamples by concatenating every $k$ consecutive frames, and then applies a 2-layer MLP with ReLU:

$$ \mathbf{S} = \mathcal{A}(\mathbf{H}) = \operatorname{Linear}(\operatorname{ReLU}(\operatorname{Linear}(\operatorname{DownSample}(\mathbf{H})))). $$

The adapted speech representation is inserted into the prompt at the speech placeholder and passed to the LLM, which is trained with a standard autoregressive cross-entropy objective:

$$ \mathcal{L}_{\text{text}} = -\sum_{i=1}^{M} \log p\bigl(y_i^T \mid \mathcal{P}(\mathbf{S}), Y^T_{

For speech generation, the paper first converts the target waveform to discrete units using HuBERT features and K-means clustering. Consecutive identical unit indices are merged, yielding $Y^U = [y_1^U, \ldots, y_L^U]$ with $y_i^U \in \{0, \ldots, K-1\}$. A unit-based HiFi-GAN vocoder with a duration predictor then synthesizes waveforms from those units.

The speech decoder takes the LLM’s text-side hidden states $\mathbf{C} = [\mathbf{c}_1, \ldots, \mathbf{c}_M]$, upsamples each state by a factor $\lambda$ into $\widehat{\mathbf{C}}$, and predicts a longer sequence of unit emissions. The model uses CTC with a blank symbol $\epsilon$ to align the decoder output to the unit sequence. If $A$ is an alignment and $\beta(A)$ collapses repeated symbols and blanks, then training minimizes

$$ \mathcal{L}_{\text{CTC}} = -\log \sum_{A \in \beta^{-1}(Y^U)} \prod_{i=1}^{\lambda M} p(a_i \mid \mathbf{O}), $$

where $\mathbf{O}$ is the decoder output and $\beta^{-1}(Y^U)$ denotes all alignments that collapse to the target unit sequence. This is the key mechanism that lets the model generate speech units in a non-autoregressive, streaming fashion while the LLM is still producing text.

The concrete configuration reported in the paper is:

• Whisper-large-v3 encoder, frozen;
• speech adaptor with $5\times$ downsampling;
• Llama-3.1-8B-Instruct as the LLM;
• speech decoder with 2 Transformer layers, hidden size 4096, 32 attention heads, FFN size 11008, and about 425M parameters;
• upsample factor $\lambda = 25$.

Two-stage training

The paper uses a strict two-stage training strategy:

1. Stage 1: train the speech adaptor and the LLM on speech-instruction-to-text-response pairs using $\mathcal{L}_{\text{text}}$. The speech encoder is frozen, and the speech decoder is not used.
2. Stage 2: freeze the speech encoder, adaptor, and LLM; train only the speech decoder using $\mathcal{L}_{\text{CTC}}$ on discrete speech units.

Optimization details reported by the authors:

• batch size 32 in both stages;
• 3 epochs;
• cosine learning-rate schedule with 3% warmup;
• peak learning rate $2\times 10^{-5}$ for stage 1;
• peak learning rate $2\times 10^{-4}$ for stage 2;
• total training time about 65 hours on 4 NVIDIA L40 GPUs.

Inference and streaming synthesis

At inference time, the LLM generates text tokens autoregressively. As each token is produced, the corresponding hidden state is upsampled and fed to the causal speech decoder. Because the decoder is non-autoregressive, the discrete-unit alignment for a chunk can be produced in parallel. Once the number of new units reaches a chunk threshold $\Omega$, the model immediately sends that unit segment to the vocoder and begins playing the speech output.

This design has two important consequences:

• the user can start hearing speech before the entire response is complete;
• the latency is controlled by the chunk size $\Omega$, allowing a trade-off between response speed and speech continuity.

The paper emphasizes that the decoding speed for text+speech generation is close to text-only decoding because the speech side is generated in parallel inside the non-autoregressive chunk.

Construction of InstructS2S-200K

The paper’s dataset construction is important because the authors argue that ordinary text instruction data does not match speech interaction well. The resulting dataset, InstructS2S-200K, contains 200K triplets of speech instruction, text response, speech response.

The pipeline has three steps:

1. Instruction rewriting. Existing text instructions are rewritten to sound more natural when spoken. The rewriting adds moderate fillers such as “hey”, “so”, “uh”, or “um”, converts numbers and symbols into spoken forms, and keeps the utterances relatively brief. The paper uses Llama-3-70B-Instruct for rewriting.
2. Response generation. The corresponding answers are also rewritten to be suitable for speech interaction: concise, direct, and free of elements that are awkward for TTS such as ordered lists or parentheses. Again, Llama-3-70B-Instruct is used.
3. Speech synthesis. Instructions are synthesized with CosyVoice-300M-SFT, randomly choosing male or female voices, while responses are synthesized with a VITS model trained on LJSpeech.

The dataset is built from about 50K Alpaca instructions and about 150K UltraChat instructions. For UltraChat, the paper keeps only the first 150K entries and only the first-round instruction from each conversation. The resulting dataset statistics are:

So the dataset comprises about 1,476 hours of synthesized speech in total, with responses substantially longer than instructions, which is consistent with the speech-response setting.

Experimental setup

The authors evaluate the model on a new benchmark, InstructS2S-Eval, constructed from the helpful_base and vicuna subsets of Alpaca-Eval. They remove math and code questions and end up with 199 instructions, which are then synthesized into speech using CosyVoice-300M-SFT.

The paper defines two tasks:

• S2TIF: speech-to-text instruction following;
• S2SIF: speech-to-speech instruction following.

For S2SIF, the authors consider both:

• offline: generate the full text response first, then synthesize the whole speech response;
• streaming: generate speech units and synthesized audio while the text response is still being produced.

Decoding is done with greedy search for reproducibility.

The evaluation metrics are:

• ChatGPT Score: GPT-4o assigns a score from 1 to 5 based on helpfulness, relevance, fluency, and suitability for speech interaction;
• ASR-WER: Whisper-large-v3 transcribes the generated speech, and word error rate is computed between the transcription and the text response;
• UTMOS: a predicted MOS score for speech naturalness;
• Latency: time from input speech to the start of speech output;
• WPS: words per second, used to quantify speech rate.

The baselines are:

• SpeechGPT: a speech-language model that sequentially outputs text instruction, text response, and speech response using chain-of-modality prompting;
• SALMONN + Orca: a speech-understanding LLM followed by the Orca TTS model;
• Qwen2-Audio + Orca: another speech-understanding model followed by Orca.

For streaming TTS baselines, the paper controls latency by triggering synthesis after a word chunk size $\Theta$ new words arrives. For LLaMA-Omni, latency is controlled by the unit chunk size $\Omega$.

Offline results

The paper reports that LLaMA-Omni is best on the main quality metrics in the offline setting, especially on ChatGPT Score and UTMOS. It is also the most balanced model in the sense that the gap between S2TIF and S2SIF is not as severe as for SpeechGPT.

Offline results on InstructS2S-Eval

The paper’s interpretation of these results is:

• LLaMA-Omni achieves the highest ChatGPT Score on both S2TIF and S2SIF, which the authors attribute to using Llama-3.1-8B-Instruct plus speech-aligned training data.
• SpeechGPT performs worst overall, especially on S2SIF, because it generates text and speech sequentially and its speech-text alignment is weak.
• Cascade systems minimize ASR-WER, especially SALMONN + Orca, because the TTS backend is industrial-strength and the text-to-speech interface is explicit.
• LLaMA-Omni has the best UTMOS, indicating high speech naturalness, but its ASR-WER is somewhat higher than the cascade systems because its speech decoder is trained on much less data than industrial TTS.

The paper explicitly notes that the S2SIF scores are generally lower than the S2TIF scores because speech synthesis errors and ASR-based evaluation both make the task harder. Even so, LLaMA-Omni is reported to outperform the prior speech-language models in both content and style.

Streaming results and latency-quality trade-offs

The streaming experiments are one of the paper’s most important contributions because they show that LLaMA-Omni can begin speaking before the full response is complete. The model keeps speech generation tightly coupled to the LLM token stream, and latency is controlled by the unit chunk size $\Omega$.

LLaMA-Omni streaming table

The key pattern is a clean latency-quality trade-off:

• smaller $\Omega$ yields lower latency, with the minimum reported latency of 236.18 ms at $\Omega = 10$;
• larger $\Omega$ slightly improves UTMOS and slightly increases ASR-WER stability;
• speech rate stays almost constant across settings, which the paper treats as evidence that the prosody is relatively stable even under aggressive streaming.

The authors also observe that at low chunk sizes, speech is split into more segments, which can reduce continuity and naturalness. This explains why UTMOS is lower at smaller $\Omega$, even though the latency is best.

Baseline streaming tables

SpeechGPT

SALMONN + Orca

Qwen2-Audio + Orca

These tables make the paper’s main streaming conclusion very clear:

• SpeechGPT is far too slow because it still serializes text instruction, text response, and speech response generation; its quality is also weak.
• SALMONN + Orca can be very low latency, but the speech rate drops sharply at low-latency settings and the text/speech alignment depends on the TTS stage.
• Qwen2-Audio + Orca also achieves low latency, but its ChatGPT Scores and ASR-WER are worse, especially at the lowest latency setting.
• LLaMA-Omni is the only method reported here that combines very low latency with simultaneous end-to-end generation of both text and speech, while keeping the speech rate and prosody comparatively stable.

The paper highlights that the end-to-end design preserves rhythm and prosody better than a streaming cascade because only the vocoder is cascaded; the speech-unit generation itself remains inside the model.

Human evaluation

To complement the automatic metrics, the authors run a side-by-side human evaluation. They compare LLaMA-Omni at $\Omega = 40$ and latency 347 ms against SALMONN + Orca at $\Theta = 3$ and latency 352 ms, and against Qwen2-Audio + Orca at $\Theta = 3$ and latency 420 ms. They sample 20 speech instructions, collect outputs from the compared systems, and ask 5 participants to vote on helpfulness and naturalness with win/tie/lose judgments.

The paper reports that LLaMA-Omni achieves a higher win rate than the cascade baselines on both dimensions, supporting the claim that its responses are better aligned with human preferences in real-time speech interaction.

Qualitative case study

The appendix includes a case study on the instruction “How do I wrap a present neatly?”. The qualitative comparison reinforces the numerical findings:

• Qwen2-Audio produces a very long response with line breaks, parentheses, and explicit step numbering, which is awkward for speech synthesis;
• SALMONN is shorter but still somewhat verbose;
• SpeechGPT is concise and speech-like, but it contains less information;
• LLaMA-Omni is concise and informative, making it more useful for conversational speech interaction.

Stated limitations and future work

The paper does not include a formal limitations section, but it does acknowledge a few concrete constraints and future directions:

• the speech decoder is trained on only about 1K hours of speech data, which is far less than what industrial TTS systems typically use; this likely explains why cascade systems still have lower ASR-WER;
• smaller chunk sizes improve latency but can hurt speech continuity and naturalness;
• the authors plan to improve the expressiveness of generated speech responses and further enhance real-time interaction.

In other words, the paper’s own evidence suggests that LLaMA-Omni already reaches a strong quality/latency compromise, but better speech training data and more expressive vocoding remain open directions.

Takeaway

LLaMA-Omni shows that a relatively efficient training recipe can turn a modern open-source LLM into a real-time speech-interaction model without relying on ASR transcription. The key design choices are: a frozen Whisper encoder for speech understanding, a lightweight adaptor, a strong Llama-3.1-8B-Instruct backbone, and a CTC-based streaming decoder that predicts speech units in parallel with text generation. On the paper’s benchmark, this combination yields the best reported ChatGPT Scores, the best UTMOS, and a very low streaming latency of 236 ms, while keeping the training cost low enough to be practical on 4 GPUs.