GLM-4-Voice

GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

GLM-4-Voice is an intelligent bilingual spoken chatbot enabling real-time voice conversations with human-like expressiveness. It combines a low-bitrate speech tokenizer with a pretrained large language model, offering more natural and interactive dialogue than traditional text-based pipelines.

• llm
• dialogue
• asr
• tts
• prosody
• speech-to-speech
• realtime
• low-latency

Demos

These demos illustrate GLM-4-Voice's advanced end-to-end spoken chatbot capabilities, showcasing its ability to understand and generate fluent bilingual speech in real time. Key aspects to evaluate include its naturalness in voice modulation, emotion control, varied dialects, and low-latency audio response, highlighting the effectiveness of integrated discrete tokenization and flow-based decoding.

Authors: Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang

Categories: cs.CL, cs.SD, eess.AS

Published 2024-12-03 · Updated 2024-12-03

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub3,198 stars29 watchers280 forksApache-2.0
• Hugging Face15,307 downloads119 likes

Abstract

We introduce GLM-4-Voice, an intelligent and human-like end-to-end spoken chatbot. It supports both Chinese and English, engages in real-time voice conversations, and varies vocal nuances such as emotion, intonation, speech rate, and dialect according to user instructions. GLM-4-Voice uses an ultra-low bitrate (175bps), single-codebook speech tokenizer with 12.5Hz frame rate derived from an automatic speech recognition (ASR) model by incorporating a vector-quantized bottleneck into the encoder. To efficiently transfer knowledge from text to speech modalities, we synthesize speech-text interleaved data from existing text pre-training corpora using a text-to-token model. We continue pre-training from the pre-trained text language model GLM-4-9B with a combination of unsupervised speech data, interleaved speech-text data, and supervised speech-text data, scaling up to 1 trillion tokens, achieving state-of-the-art performance in both speech language modeling and spoken question answering. We then fine-tune the pre-trained model with high-quality conversational speech data, achieving superior performance compared to existing baselines in both conversational ability and speech quality. The open models can be accessed through https://github.com/THUDM/GLM-4-Voice and https://huggingface.co/THUDM/glm-4-voice-9b.

---

Overview

GLM-4-Voice is an end-to-end spoken chatbot designed to be both intelligent and human-like. The paper’s core claim is that a strong spoken chatbot should not merely transcribe speech and then synthesize a reply, but should instead learn speech as a first-class modality through large-scale pre-training and then be adapted for low-latency conversational use.

The model supports Chinese and English, can conduct real-time voice conversations, and is intended to vary vocal properties such as emotion, intonation, speech rate, and dialect according to user instructions. The system is built on a pre-trained text language model, GLM-4-9B-Base, which is expanded to handle speech tokens and further trained on a mixture of speech-only, speech-text, and text-only corpora at scale.

A central design choice is the use of a single-codebook, low-bitrate speech tokenizer at 12.5 Hz and 175 bps, derived from an ASR model by inserting a vector-quantized bottleneck into the encoder. This tokenizer is paired with a flow-matching speech decoder that can stream audio generation. The paper emphasizes that the combination of a text-model warm start, speech pre-training on up to 1 trillion tokens, and a streaming conversational fine-tuning stage is what enables both strong linguistic capability and improved speech quality.

Problem Setting and Motivation

The paper starts from the observation that most mainstream conversational AI systems are text-centered. While these systems are effective at semantic reasoning and information retrieval, they do not naturally capture the paralinguistic dimensions of human speech. Conventional spoken chatbots usually decompose the problem into automatic speech recognition (ASR), text-only LLM reasoning, and text-to-speech (TTS). That modular pipeline is functional, but the authors argue it has several limitations:

• Latency: the pipeline adds serial delays between ASR, LLM, and TTS.
• Error accumulation: mistakes in ASR can propagate into the language model and synthesis stages.
• Weak expressiveness: a text-only intermediary tends to lose emotion, prosody, and other speech-specific cues.

Previous speech language models and end-to-end spoken chatbots are presented as promising alternatives, but the paper argues they still face two structural problems: speech corpora are far smaller than text corpora, and many prior spoken chatbot methods fine-tune from a text LLM with spoken-dialogue data without enough dedicated speech pre-training. According to the authors, this leaves them short on both intelligence and natural speech generation.

Method Overview

GLM-4-Voice is designed with minimal modifications to the autoregressive transformer. The system has three major components:

1. A supervised speech tokenizer that converts waveforms to discrete tokens at low bitrate.
2. A large language model backbone initialized from GLM-4-9B-Base and extended to handle speech tokens.
3. A speech decoder that turns generated speech tokens back into audio, with streaming support.

The paper’s training recipe is split into two stages:

• Stage 1: joint speech-text pre-training on 1 trillion tokens using text, speech-only, and speech-text data.
• Stage 2: supervised fine-tuning on high-quality conversational speech data, including style-controlled dialogues.

The result is a model that can both answer questions and render spoken responses with better conversational quality than several baseline spoken chatbot systems.

Speech Tokenization

The tokenizer is one of the paper’s main technical contributions. The authors explicitly argue that an ideal tokenizer for speech-language modeling should have three properties:

• Low sampling rate with a single codebook, so autoregressive generation remains feasible.
• Alignment with text, enabling transfer from large text LLMs.
• High-quality reconstruction, so the produced speech tokens can be decoded into natural audio.

To satisfy these constraints, the paper adopts a 12.5 Hz speech tokenizer variant described as supervised semantic tokenization. The tokenizer is built by fine-tuning a pretrained ASR model, specifically Whisper large-v3, with an additional pooling layer and a vector quantization layer inserted into the encoder. The encoder output is discretized through the VQ bottleneck, and the resulting codebook indices become the speech tokens.

The codebook is trained with exponential moving average (EMA) updates, and codebook collapse is addressed by resetting rarely used vectors with randomly selected continuous representations before quantization. The authors also note that the codebook size is increased as the sampling rate decreases to offset information loss from pooling.

Streaming-capable tokenizer

To support live voice interaction, the tokenizer is modified for causal streaming inference. The architecture replaces the convolution before the encoder transformer with causal convolution and swaps bidirectional self-attention for block causal attention. This preserves local context while ensuring the encoding process can proceed incrementally on incoming speech chunks.

Tokenizer training data and procedure

The tokenizer is fine-tuned from Whisper large-v3 for 2 epochs with batch size 4096 and learning rate $10^{-5}$. The training mixture includes both supervised and pseudo-labeled speech:

• Supervised ASR datasets: LibriSpeech, GigaSpeech, MLS-Eng, Wenet, CommonVoice, AISHELL-1, and a proprietary Chinese ASR dataset of 10k hours.
• Unsupervised speech data: 700k hours, with pseudo labels produced by Whisper large-v3 for English and Paraformer-large for Chinese.

The supervised-to-pseudo-labeled sample ratio is 1:3. The EMA decay coefficient is 0.99, and the commitment loss coefficient is 10.0.

Tokenizer evaluation

Tokenizer quality is evaluated in two ways: semantic retention via ASR performance after tokenization/decoding, and reconstruction quality via WER, VisQOL, and MOSNet. The paper compares the proposed tokenizer with several baselines, including SpeechTokenizer, Moshi’s Mimi tokenizer, and direct ASR models.

The authors select the 12.5 Hz variant for the final model because it balances compression and quality. At that setting, the tokenizer maintains strong semantic retention while keeping the bitrate at 175 bps. The table also suggests a trade-off across frame rates: the higher-rate variants improve reconstruction, while the very low-rate 6.25 Hz variant suffers substantial degradation in ASR/reconstruction quality.

Speech Decoder

The speech decoder converts discrete speech tokens back into waveforms and is crucial for naturalness and expressiveness. The paper adopts the decoder architecture from CosyVoice, composed of:

• a speech token encoder,
• a conditional flow matching model, and
• a HiFi-GAN vocoder.

The decoder is trained in two stages. In pre-training, it sees all speech samples in the unsupervised speech data, covering diverse speakers and quality levels. In fine-tuning, it uses high-quality speech from a single speaker. This is meant to exploit large-scale diversity first and then sharpen audio quality on cleaner data.

For streaming inference, the decoder is trained on truncated audio samples of length $n \cdot b$ seconds, where $b$ is the block size and $n=1,2,3,\dots$. At inference time, it uses the first $(n-1)b$ seconds as prompt and predicts the next chunk. The paper reports a block size of $b=0.8$ seconds, implying a minimum delay of about 10 speech tokens before the first audio output.

The decoder evaluation table, reproduced from the paper’s referenced reconstruction results, shows that the proposed low-rate tokenizer/decoder pair can still produce reasonable speech quality at 12.5 Hz, with MOSNet 3.39 and WER 8.43 on the reconstruction benchmark. Higher rates improve WER and sometimes quality, but the selected 12.5 Hz setting provides the intended efficiency–quality trade-off.

Decoder reconstruction results

Inference and Conversation Design

The paper makes an important conceptual move: instead of treating spoken dialogue as a direct speech-to-speech generation problem, it decouples the task into two subtasks:

1. Speech-to-text: generate a textual answer $A_t$ from the user speech input $Q_s$.
2. Speech-and-text-to-speech: generate spoken output $A_s$ conditioned on both $Q_s$ and $A_t$.

This decomposition is motivated by the idea that text provides strong semantic guidance, while speech generation needs extra cues for prosody and naturalness. The authors state that this setup improves response quality, although it introduces a latency issue because the system would otherwise have to wait for the full text answer before starting the speech response.

To reduce this delay, they introduce a streaming thoughts template. The model alternates between text tokens and speech tokens during generation, so that the textual reasoning and spoken realization can proceed in a coordinated streaming fashion. The paper describes the schedule as alternating 13 text tokens and 26 speech tokens under the 12.5 Hz tokenizer, using a 1:2 ratio to ensure text remains ahead of speech. Elsewhere in the latency derivation, the first chunk is described as 13 text tokens plus 10 speech tokens, corresponding to the first audio waveform segment.

The latency breakdown is given as:

$$ T_{\text{total}} = T_{\text{speech\_tokenize}} + T_{\text{llm\_prefill}} + T_{\text{llm\_decode}} + T_{\text{speech\_decode}}. $$

Here, streaming tokenization processes only the current audio block, prefill cost scales with the number of speech tokens produced by the tokenizer, and decoding cost is measured over the initial text-and-speech token chunk needed to generate the first audio response.

Training Procedure

Stage 1: Joint speech-text pre-training

The first stage aims to transfer knowledge from the text LLM into the speech domain and to teach the model to process interleaved speech and text. The training mixture contains four corpus types:

• Interleaved speech-text data: synthesized from text pre-training corpora using a text-to-token model.
• Unsupervised speech data: 700k hours of speech.
• Supervised speech-text data: ASR and TTS data.
• Text-only data: retained to preserve text LLM capability.

The paper provides token statistics for this training mix:

In total, the model is trained on 1 trillion tokens, with a fixed sampling ratio of 30% text data, one epoch each of unsupervised speech and supervised speech-text data, and the remainder coming from interleaved speech-text data. The model is initialized from GLM-4-9B-Base and its vocabulary is expanded to include speech tokens. Optimization uses AdamW with $\beta_1 = 0.9$ and $\beta_2 = 0.95$, a sequence length of 8192, and a learning rate that linearly decays from $6 \times 10^{-5}$ to $6 \times 10^{-6}$.

Stage 2: Supervised fine-tuning

The second stage aims to make the model behave like a human-like spoken assistant. The fine-tuning data are constructed from two categories:

• Multi-turn conversational spoken dialogues: derived primarily from text-based data after filtering. Code and math content are removed, long answers are shortened, and responses are adapted for verbal delivery. Speech outputs are synthesized to match the refined dialogue, and human annotators also record speech inputs to increase input diversity.
• Speech style-controlled spoken dialogues: high-quality multi-turn spoken dialogues designed to control speed, emotion, or dialect.

The training objective is split according to the two subtasks used at inference time. Each conversational sample consists of $Q_s$, the corresponding text input $Q_t$, a text output $A_t$, and a speech output $A_s$. Because the model learns text outputs faster than speech outputs, the authors split each sample into two variants: one masks the speech-output loss so the model focuses on text prediction, and the other masks the text-output loss so the model focuses on speech generation conditioned on the text output.

Fine-tuning runs for 20 epochs on speech output and 4 epochs on text output, with learning rate decayed from $10^{-5}$ to $10^{-6}$. To reduce overfitting, the model uses weight decay 0.1, dropout 0.5 on hidden layers, and gradient clipping at 1.0.

Evaluation Setup

The paper evaluates the base model and the final chat model on a mixture of speech-language, ASR/TTS, and conversational benchmarks. A consistent theme in the evaluation is comparing speech-to-speech ($S \rightarrow S$) against speech-to-text ($S \rightarrow T$) settings to understand how much the model depends on textual intermediate supervision.

For the speech-language and spoken QA tasks, the contexts and continuations are synthesized using the VolcEngine multi-speaker TTS API. For the chat model evaluation, the authors use GPT-4o as a judge for answer quality and correctness, and UTMOS to estimate speech naturalness. For text-speech alignment in chat, speech responses are transcribed with Whisper large-v3 and compared against the text outputs using WER.

Base Model Results

Speech language modeling

Speech language modeling is evaluated on Spoken StoryCloze and Spoken Topic-StoryCloze. The task measures whether the model can predict the correct continuation from a speech context. The paper reports the following accuracies:

The main takeaway is that GLM-4-Voice achieves the best reported $S \rightarrow T$ results and remains competitive in $S \rightarrow S$. The paper also notes that the $S \rightarrow T$ numbers are consistently better than $S \rightarrow S$, indicating that text guidance remains helpful for intelligent speech chatbots.

Spoken question answering

Spoken QA is evaluated on Web Questions, Llama Questions, and TriviaQA. This task probes broad factual knowledge without external retrieval. The results show a large jump over prior systems, especially in the $S \rightarrow T$ setting:

GLM-4-Voice is strongest in the $S \rightarrow T$ setting, where it reaches the best results on all three datasets. In $S \rightarrow S$, it still exceeds prior systems, but the gap to the text-guided setting remains substantial. The authors interpret this as evidence that direct speech-to-speech intelligence is improving, but textual guidance remains an important scaffold.

ASR and TTS abilities

The paper also evaluates whether the base model can do traditional speech tasks. Using the pre-training prompt format, it measures ASR on LibriSpeech and AISHELL-1, and TTS on LibriTTS and Seed-TTS. GLM-4-Voice is compared against Whisper large-v3 and CosyVoice.

The reported ASR/TTS results show that GLM-4-Voice is competitive with specialized models, although it does not outperform Whisper on English ASR and is weaker than CosyVoice on LibriTTS clean speech. The main strength is that it provides these competencies within a unified spoken-chatbot model, while achieving especially strong Chinese ASR and strong TTS on Seed-TTS.

Chat Model Results

The final conversational model is evaluated with GPT-4o on general QA and knowledge tasks, and with UTMOS and ASR-WER for speech quality and text-speech alignment. The baselines come from the paper’s earlier spoken-chatbot comparisons.

These results are the paper’s strongest evidence that the pre-training plus fine-tuning recipe improves practical spoken-chatbot behavior. GLM-4-Voice improves both response quality and speech naturalness while also reducing ASR-WER relative to prior systems. The authors highlight that it achieves the best scores among the reported baselines on all four measures.

What the Paper Claims as Novel

• Speech tokenizer from ASR: a low-bitrate, single-codebook tokenizer built by inserting a VQ bottleneck into a pretrained ASR encoder.
• Streaming-capable speech modeling: causal tokenization and a streaming decoder to support real-time interaction.
• Large-scale text-to-speech knowledge transfer: synthetic interleaved speech-text data created from text pre-training corpora and used for continued pre-training.
• Unified training recipe: joint speech-text pre-training followed by conversational fine-tuning on style-controlled dialogues.
• Streaming thoughts inference: an alternating text/speech generation template that reduces the delay of direct spoken responses.

Interpretation of the Results

The empirical picture is fairly consistent across the paper:

• Text guidance helps. In both speech language modeling and spoken QA, $S \rightarrow T$ is stronger than $S \rightarrow S$.
• Pre-training matters. Compared with methods that rely mainly on instruction tuning, the model benefits from large-scale speech pre-training and transfers more knowledge from the text backbone.
• Speech quality improves with dedicated training. The final chat model achieves better UTMOS and lower ASR-WER than previous spoken chatbots.
• Low-bitrate tokenization is viable. The 12.5 Hz tokenizer keeps enough semantic information to support both downstream modeling and reconstruction.

At the same time, the tables show that some specialized single-task systems still dominate specific subtasks. For example, Whisper remains stronger on English ASR, and CosyVoice is better on LibriTTS clean speech. The paper’s contribution is not to win every standalone benchmark, but to demonstrate a strong unified spoken assistant with broad competence.

Limitations and Open Questions

The paper does not include a standalone limitations section, but the reported experiments reveal several practical constraints and unresolved issues:

• Speech-to-speech remains harder than speech-to-text. The gap between $S \rightarrow S$ and $S \rightarrow T$ remains noticeable across tasks.
• Streaming is approximate, not free. The streaming-thought design and block-based decoding reduce latency, but the first audible chunk still depends on a nontrivial amount of text and speech generation.
• Specialized systems remain strong on narrow tasks. The model is competitive, but not always best, on pure ASR and some reconstruction settings.
• Training relies on large curated and synthetic data. The method depends on substantial data construction, including synthetic interleaved speech-text corpora and high-quality conversational speech.

These points suggest that the main challenge ahead is not merely scaling parameters, but improving the direct coupling between semantic planning, prosody, and low-latency speech realization.

Conclusion

GLM-4-Voice is presented as a step toward an end-to-end spoken chatbot that can be both intelligent and expressive. Its technical recipe is notable for combining a speech tokenizer derived from ASR, a streaming speech decoder, and large-scale speech-text pre-training initialized from a 9B text model. The paper’s results show strong gains on speech language modeling, spoken question answering, ASR/TTS, and conversational evaluation, with especially strong performance in the text-guided $S \rightarrow T$ setting.

For conversational-AI systems, the most important takeaway is that the authors treat spoken dialogue as a full model-training problem rather than a post-hoc TTS attachment problem. The resulting architecture and training recipe are explicitly designed to preserve the strengths of text LLMs while adding speech-native competence, low latency, and controllable vocal style.