Freeze-Omni

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

Freeze-Omni is a speech-to-speech dialogue model that keeps a large language model frozen to preserve reasoning while enabling low-latency spoken interaction. It connects speech input and output with the text LLM for natural duplex dialogue using limited data.

• llm
• multimodal
• dialogue
• speech-to-speech
• streaming
• low-latency
• asr
• tts

Demos

The demos highlight Freeze-Omni's low-latency, streaming speech-to-speech dialogue capabilities with both Mandarin and English question answering, as well as prompt following and role play. Watch for natural, coherent speech synthesis and the model's ability to handle interruptions and follow conversational context in real time. These showcase the system's robust acoustic understanding and AR-based speech output from a frozen LLM without fine-tuning degradation.

Authors: Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, Long Ma

Categories: cs.SD, cs.AI, cs.CL, eess.AS

Comment: Project Page: https://freeze-omni.github.io/

Published 2024-11-01 · Updated 2024-12-08

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub386 stars7 watchers27 forksNOASSERTION
• Hugging Face0 downloads20 likes

Abstract

Rapidly developing large language models (LLMs) have brought tremendous intelligent applications. Especially, the GPT-4o's excellent duplex speech interaction ability has brought impressive experience to users. Researchers have recently proposed several multi-modal LLMs in this direction that can achieve user-agent speech-to-speech conversations. This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni. Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We design a three-stage training strategy for modeling both the speech input and output, enabling Freeze-Omni to obtain speech-to-speech conversation ability using text-speech paired data (such as ASR and TTS data) and only 60,000 multi-round text Q&A data on 8 GPUs. Moreover, we can effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level compared with that in the text modality of its backbone LLM, while achieving low latency end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multi-task training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.

---

Introduction

Freeze-Omni is a speech-to-speech dialogue system built around a frozen text LLM. The paper’s central idea is to attach streaming speech input and streaming speech output modules to an existing language model without updating the LLM’s parameters during multimodal training. The authors argue that this avoids catastrophic forgetting, preserves the backbone model’s original reasoning and instruction-following ability, and reduces data and compute requirements relative to approaches that fine-tune the LLM itself.

The target setting is low-latency spoken interaction similar in spirit to GPT-4o-style duplex dialogue, but implemented with a modular architecture: a chunk-wise speech encoder maps incoming speech to the LLM embedding space, and a token-based speech decoder turns the LLM’s output into speech tokens and then waveform. The paper emphasizes three practical goals:

• keep the LLM parameters frozen throughout multimodal training,
• support speech input and speech output using text-speech paired data plus a relatively small amount of multi-round question answering data, and
• enable natural duplex interaction through chunk-level state prediction and streaming inference.

The reported backbone for the experiments is Qwen2-7B-Instruct. The paper’s broader claim is not that Freeze-Omni replaces the LLM, but that it provides a practical speech interface that preserves the LLM’s text-side intelligence while making speech-to-speech dialogue feasible under limited training resources.

Main architectural idea

Freeze-Omni decomposes the problem into two alignments: speech input to text and text to speech output. These are trained separately, then connected end-to-end. The resulting system operates in a streaming manner: speech is processed in chunks, the LLM produces text tokens and hidden states, and the speech decoder consumes these signals in chunked form to produce low-latency audio output.

The main novelty is the training strategy. Rather than updating the backbone LLM, the paper uses three-stage procedures for both the input side and the output side, plus an additional duplex state prediction task. This is meant to make the speech modules adapt to the language model while leaving the language model itself intact.

System overview

At a high level, the architecture has four functional parts:

• Streaming speech encoder: converts mel features into a higher-level representation and down-samples the frame rate before handing features to the LLM adapter.
• Adapter: maps speech representations into the LLM embedding space.
• Frozen LLM: performs the actual language reasoning, generation, and hidden-state production.
• Speech decoder: turns LLM outputs into speech tokens and then waveform, using a non-autoregressive prefix path and an autoregressive token generation path.

The system is explicitly designed for streaming: both the input and output sides use chunking to reduce delay and support incremental processing.

Speech input modeling

The speech-input side aims to make the LLM understand spoken language while preserving the backbone’s original behavior. The paper uses a chunk-wise streaming speech encoder composed of down-sampling convolution layers and Transformer blocks. The adapter is also convolutional and further reduces the frame rate. The motivation for aggressive down-sampling is to reduce the number of speech frames the LLM must process during prefill, which in turn lowers latency.

The encoder input is mel-filterbank features with a 25 ms window and 10 ms shift. The encoder has approximately 350M parameters, uses four-times down-sampling in the convolution stack, includes 24 Transformer layers with hidden size 1024, and produces an output frame rate of 12.5 Hz. The adapter uses multi-convolution layers with two-times down-sampling.

Three-stage training for speech input

The paper uses a three-stage training schedule:

1. Stage 1: ASR-style pretraining. The model is trained like a standard speech recognition system. Speech features are mapped to transcripts, and the loss is Connectionist Temporal Classification (CTC).
2. Stage 2: Speech-to-text alignment with a frozen LLM. The stage-1 encoder is initialized and connected to the LLM through the adapter. The output target is still the transcript. Trainable special tokens are inserted to guide the LLM during training. In this stage, the LLM is frozen, while the speech encoder, adapter, and other active components are trainable.
3. Stage 3: Multi-round speech question answering. The authors construct a multi-turn Q&A dataset whose answers are generated by the backbone LLM so that the data is compatible with that backbone. They then synthesize the questions into speech using a multi-speaker TTS system, add trainable prompt embeddings before each question, freeze the speech encoder to preserve acoustic robustness, and keep the LLM frozen. Only the prompt embedding parameters remain trainable in this stage.

The key design point is that stage 3 removes the special tokens used in stage 2 and replaces them with a shared prompt embedding for all questions. This is intended to make the model robust in a conversational setting while preventing drift in the backbone language model.

In the paper’s interpretation, stage 2 establishes speech understanding, while stage 3 adapts the model to conversational speech input without modifying the LLM.

Speech output modeling

The speech-output side generates waveform from the LLM’s text-side outputs. The decoder is inspired by VALL-E-style token-based synthesis and is split into a non-autoregressive (NAR) prefix path, a NAR speech decoder, an autoregressive (AR) speech decoder, and a codec decoder that converts speech tokens into waveform.

The paper uses TiCodec as the codec model and configures it as a single-codebook system with codebook size 1024 and speech-token frequency 40 Hz. This single-codebook choice is justified as a simpler and lower-latency option for the limited speaker diversity considered in the paper. The speech decoder itself uses 4-layer Llama-style decoder blocks with hidden size 896, and the speech decoder parameters are about 120M. The codec output sample rate is 24 kHz.

Three-stage training for speech output

The speech decoder is also trained in three stages:

1. Stage 1: Codec training from speech only. A single-codebook codec model is trained using speech data only.
2. Stage 2: Text-to-speech alignment with a frozen LLM embedding layer. The authors build text-speech paired data, tokenize text with the backbone LLM tokenizer, convert tokens to embeddings, and feed them to the NAR speech decoder. The AR decoder predicts the speech tokens with teacher forcing. Labels come from the stage-1 codec. The NAR and AR decoders share parameters in this stage, and the LLM embedding layer is frozen.
3. Stage 3: Prefix fine-tuning to match the LLM’s hidden states. Using the same multi-round Q&A data as in speech-input stage 3, the authors feed both text tokens and the backbone LLM hidden-state sequence into an additional NAR prefix speech decoder. This prefix module produces a kv-cache that conditions the speech decoder more tightly on the LLM’s internal representation. The text-token labels are converted to speech via a TTS system and then to speech tokens by the stage-1 codec. In this stage, only the NAR prefix decoder is trainable; the rest of the network is frozen.

The paper explicitly motivates stage 3 as a way to reduce bad cases caused by the mismatch between generic text-speech paired data and the actual output distribution of the backbone LLM. In other words, the prefix module adapts the speech decoder to the LLM’s own hidden-state space without altering the backbone model.

The authors also test an additional pre-network between the LLM embedding layer and the NAR speech decoder to better model the mismatch between semantic embeddings and speech-token generation space. This improves synthesis quality substantially in their reported CER results.

Duplex dialogue and streaming interaction

After connecting the speech encoder, frozen LLM, and speech decoder, Freeze-Omni can perform speech input to speech output. To better match a natural duplex conversation style, the paper adds chunk-level state prediction. This is used to decide whether the user is interrupting the agent or whether the current speech chunk ends without interruption.

Chunk-level state prediction

The state classifier is attached to the output hidden state corresponding to the last frame of each chunk. The paper defines three states:

• State 0: the LLM should continue to receive speech,
• State 1: the user interrupts the dialogue and the LLM should switch to a new generation turn,
• State 2: there is no need to interrupt the dialogue.

According to the paper, state labels are only valid on the final frame of each chunk. The state classifier and the LLM are optimized jointly with a multi-task cross-entropy objective in stage 3 of the speech-input training procedure.

The authors also describe a model-as-a-server inference strategy. Multiple model instances can be launched as a server, and chunks from active users are scheduled to idle instances. Because the kv-cache and CNN cache are separated per user, the server only needs to preserve per-user inference state, allowing any idle model to handle any user chunk.

Data and training configuration

The paper’s data strategy is deliberately modest compared with large-scale speech-dialogue pretraining. For conversational fine-tuning, it randomly selects 60,000 multi-round Q&A examples from moss-003-sft-data and uses the backbone LLM to generate new answers in place of the original ones. These questions are synthesized into speech with a zero-shot TTS system.

For speech-input modeling, the authors use 110,000 hours of internal bilingual Chinese-English ASR data in stages 1 and 2. For stage 3, they use the speech version of the multi-round Q&A data.

For speech-output modeling, they use about 3,000 hours of text-speech paired data generated by a zero-shot TTS system in stages 1 and 2, then use the multi-round Q&A data again in stage 3.

Training uses AdamW with warmup. The reported learning rates are:

• Speech input stages 1/2/3: $2\times 10^{-4}$, $1\times 10^{-4}$, and $6\times 10^{-4}$, respectively.
• Speech output stages 2/3: $5\times 10^{-5}$ for both.
• Speech output stage 1: same hyperparameters as TiCodec.

All experiments are reported on 8 GPUs.

Experimental results

The experimental section evaluates three aspects: speech-input understanding, speech-output quality, and spoken question answering. The paper also reports end-to-end latency.

Speech input: ASR-style understanding

To measure how well the speech encoder and adapter capture speech semantics, the paper evaluates ASR accuracy on Mandarin and English sets using the model from stage 2 of speech-input training. The authors argue that because the stage-3 speech-input encoder and adapter are unchanged, these results also represent the input-side understanding ability of the final Freeze-Omni system.

They test on aishell-1, test_net, and test_meeting for Mandarin, measured with CER, and on dev-clean, dev-other, test-clean, and test-other for English, measured with WER.

The main takeaway is that dynamic chunk training improves robustness to different chunk sizes, while a fixed chunk-4 setup without dynamic chunk training gives the best reported recognition numbers in the table. The paper nevertheless notes that this fixed setup reduces flexibility for stage 3, where the chunk size can no longer be changed. The authors choose the last-row configuration for the remainder of the experiments.

Speech output: synthesis quality and prefix alignment

For speech generation, the paper evaluates 1,000 utterances by comparing synthesized speech against label text using CER computed by paraformer-zh. The input to the speech decoder is either text tokens and hidden states output by the LLM, and the experiment is run under several AR decoding settings.

The paper’s interpretation is that adding the LLM hidden-state prefix makes the speech decoder more tightly aligned with the backbone model, reducing bad cases and lowering CER. The additional pre-network improves quality further by giving the NAR path extra capacity to bridge the semantic space of LLM embeddings and the speech-token generation space. Among the reported rows, the best CER is 1.69% at top-k = 2 with both the prefix decoder and the pre-network.

Spoken question answering

To test whether the frozen-LMM design preserves intelligence in speech mode, the paper evaluates spoken question answering on three datasets: Web Questions, LlaMA-Questions, and TriviaQA. Because Web Questions and TriviaQA are text-only, the authors synthesize them into speech using edge-tts with the voice en-US-BrianNeural. The reported metric is accuracy.

The key observation is that Freeze-Omni substantially outperforms earlier audio-text systems in all three speech QA benchmarks and narrows the gap to the text-only backbone. Relative to the backbone LLM, the paper reports that the speech-side performance is much closer than in Moshi, supporting the claim that freezing the LLM can preserve much of its intelligence while still enabling speech interaction.

End-to-end latency

The paper distinguishes statistical latency from non-statistical latency. Statistical latency is measured from the moment the LLM is interrupted to the first PCM chunk produced. Non-statistical latency refers to the time from the real end of speech to the moment the LLM emits the interrupt state; the paper reports this as about one to two speech-encoder chunks, or roughly 160 ms to 320 ms in their configuration. They also include network delay estimates of about 200 ms to 300 ms when discussing real deployment.

Interpreting the reported components, the dominant delay comes from the LLM-to-text-generation stage and the speech-decoder prefill/generation stages, while token-to-PCM conversion is comparatively small. The authors summarize the overall real-world latency, including network cost, at roughly 1.2 seconds on average.

What the paper claims as contributions

• Frozen-LLM multimodal integration. Freeze-Omni connects speech input and output to a text LLM without updating any LLM weights.
• Low-data speech-to-speech learning. The system is trained with text-speech paired data and only 60,000 multi-round Q&A examples, using limited compute compared with large-scale spoken-dialogue training.
• Comparable intelligence across modalities. The paper reports that spoken QA performance is close to the text-only backbone relative to prior audio-text systems.
• Low-latency duplex dialogue. Streaming chunking, prefix conditioning, and chunk-level state prediction enable interactive speech-to-speech use.
• Backbone flexibility. Because the LLM remains frozen, the method is presented as compatible with any text-capable LLM, and style changes can be handled by text-only fine-tuning of the backbone if needed.

Discussion and stated future work

The paper’s conclusion frames Freeze-Omni as a practical route to speech-to-speech dialogue under the constraint of a frozen LLM backbone. The main tradeoff is that the speech modules must learn to match the backbone model’s semantics and generation style without direct adaptation of the backbone itself. The results suggest this is feasible with the proposed staged training and chunk-based design.

The authors explicitly outline three directions for future work:

• upgrade the speech encoder into a more general audio encoder for tasks such as emotion understanding and audio captioning,
• add more speech-dialogue tasks while keeping the LLM frozen, and
• support multiple voices and stronger instruction-following behavior in the speech decoder so that richer speaking styles can be generated from the LLM hidden state.

Overall, the paper’s technical message is that a carefully designed streaming speech interface can be attached to a frozen LLM and still yield strong speech understanding, speech generation, and duplex conversational behavior, provided the speech components are trained in stages and explicitly aligned to the backbone model’s representations.