SpeechGPT
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
SpeechGPT is an intrinsic cross-modal large language model that treats speech as discrete tokens alongside text, enabling natural multi-modal conversational abilities. Unlike traditional cascaded systems, it integrates speech perception and generation directly in one model for fluent speech-text interactions.
Links
Paper & demos
Code & resources
Abstract
Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-model content. With discrete speech representations, we first construct SpeechInstruct, a large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow multi-modal human instructions and highlight the potential of handling multiple modalities with one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
1. Problem Setting and Motivation
The paper targets a limitation in existing speech-language and multi-modal language models: they usually work in a cascade paradigm or only accept multi-modal inputs while still producing text-only outputs. In that setup, speech is handled by separate ASR and TTS modules or by task-specific experts, which creates several drawbacks: the language model cannot transfer knowledge directly into the speech domain, paralinguistic cues such as emotion and prosody are easily lost, and systems that can synthesize speech often cannot truly understand speech semantics. The authors argue that this prevents genuine cross-modal conversational ability.
SpeechGPT is proposed as an intrinsic cross-modal large language model that can both perceive and generate speech and text. The key design choice is to represent speech with discrete tokens so that speech can be incorporated into an LLM by expanding its vocabulary, rather than by attaching separate speech front-ends and back-ends.
2. High-Level Approach
The system combines three ingredients:
- a discrete unit extractor that maps continuous speech into a sequence of discrete units,
- a large language model that is extended with those unit tokens, and
- a unit vocoder that can decode discrete units back into speech.
Speech is first discretized, then treated as another token stream alongside text. This makes it possible to train the LLM with next-token prediction objectives over speech units, instruction data, and mixed modality prompts. The overall goal is to let the same model handle speech understanding, speech generation, text generation, and mixed speech-text dialogue in one framework.
3. Model Architecture
3.1 Discrete Unit Extractor
The discrete unit extractor uses HuBERT to convert continuous speech into discrete labels. The speech waveform is encoded into intermediate representations, those representations are clustered with $k$-means, and the resulting cluster indices are used as discrete units. Adjacent duplicate indices are removed to shorten the sequence. The output is a unit sequence $$U = (u_1, u_2, \ldots, u_T), \quad u_i \in \{0, 1, \ldots, K-1\},$$ where $K$ is the size of the unit inventory.
This choice is central to the paper: by turning speech into discrete symbols, SpeechGPT can use the same token-based interface that LLMs already expect.
3.2 Large Language Model Backbone
The backbone is LLaMA-13B. To integrate speech, the original vocabulary $V$ is expanded with a new set of unit tokens $V'$, giving the new vocabulary $$V'' = V \cup V'.$$ If the original embedding matrix is $E \in \mathbb{R}^{|V| \times d}$, the model creates a new embedding matrix $E' \in \mathbb{R}^{|V''| \times d}$, copies the original embeddings into the first $|V|$ rows, and randomly initializes the new rows for the speech unit tokens. The same expansion is applied to the LM head so the model can both consume and generate speech units.
The paper emphasizes that this vocabulary expansion is what enables intrinsic cross-modal operation: speech tokens live in the same autoregressive modeling space as text tokens, instead of being mediated by an external ASR/TTS pipeline.
3.3 Unit Vocoder
For waveform reconstruction, the authors train a multi-speaker unit HiFi-GAN. The generator embeds the discrete units using lookup tables, upsamples them through transposed-convolution blocks and residual blocks with dilated layers, and concatenates a speaker embedding to each upsampled frame. The discriminator uses a Multi-Period Discriminator and a Multi-Scale Discriminator in the style of HiFi-GAN.
The paper notes that the multi-speaker vocoder is used because a single-speaker unit vocoder is too limiting for the broader speech setting they study.
4. SpeechInstruct: The Instruction Dataset
A major contribution of the paper is SpeechInstruct, described as the first large-scale speech-text cross-modal instruction-following dataset in the work. It is split into two parts:
- Cross-modal Instruction, which covers speech-text pairs for ASR/TTS-style tasks, and
- Chain-of-Modality Instruction, which introduces speech-input / speech-output and mixed speech-text instruction formats.
4.1 Cross-modal Instruction
To build cross-modal instruction data, the authors collect large-scale English ASR datasets, specifically GigaSpeech, Common Voice, and LibriSpeech. They tokenize speech with mHuBERT and remove repeated adjacent units to obtain reduced discrete unit sequences. This yields 9 million unit-text data pairs.
The paper then uses GPT-4 in a zero-shot prompting setup to generate diverse task descriptions for ASR and TTS. The appendix shows prompts that ask GPT-4 to produce 100 diverse instructions per task, aiming to increase linguistic variety and avoid templated instruction language. For each speech-text pair, the system samples a task description, combines it with the speech unit sequence and the transcription, and formats it into an instruction-response example.
The instruction template is of the form: the human provides a task description and an input sequence, and SpeechGPT produces the corresponding text or speech output. The authors also concatenate examples into multi-turn conversations up to the model's maximum input length to support longer dialogue-style sequences.
4.2 Chain-of-Modality Instruction
The second part of SpeechInstruct is designed to teach the model to move across modalities inside a single instruction. Because there is not enough naturally available data with speech input and speech output, the authors train a separate text-to-unit generator with a Transformer encoder-decoder architecture. This generator is trained on the LibriSpeech unit-text pairs from the cross-modal instruction data.
Using that generator, the authors select 37,969 samples from the moss-002-sft-data dataset, restricted to responses shorter than 35 words. They convert both the instructions and responses into unit sequences, producing quadruplets $(\text{SpeechI}, \text{TextI}, \text{TextR}, \text{SpeechR})$.
From those quadruplets they build four chain-of-modality formats:
- speech instruction to speech response,
- speech instruction to text response,
- text instruction to speech response, and
- text instruction to text response.
The goal is to force the model to reason through an instruction, optionally transcribe speech input into text internally, and then produce the requested modality at the end.
5. Training Strategy
The training pipeline is divided into three stages. Each stage is designed to move the model from general speech-unit modeling to cross-modal alignment and then to chain-of-modality conversational behavior.
5.1 Stage 1: Modality-Adaptation Pre-training
In the first stage, SpeechGPT is further pretrained on unpaired speech data using a next-token prediction objective over discrete speech units. The paper uses LibriLight, a 60,000-hour unlabeled English audiobook corpus.
If the training corpus contains unit sequences $U_1, U_2, \ldots, U_m$, the objective is the negative log-likelihood:
$$\mathcal{L}(L \mid C) = -\sum_{j=1}^{m} \sum_{i=1}^{n_j} \log P(u_{i,j} \mid u_{
This stage is meant to make the LLM comfortable with the discrete speech-token space before instruction tuning begins.
In the second stage, the model is aligned on paired data. The authors mix the cross-modal part of SpeechInstruct with the
Each example is a prefix-response sequence, and the loss is computed only on the response portion, not the prefix. If $p_j$ denotes the number of prefix tokens in sample $T_j$, the objective is:
$$\mathcal{L}(L \mid I) = -\sum_{j=1}^{x} \sum_{i=p_j+1}^{y_j} \log P(t_{i,j} \mid t_{
This stage is where the model learns to map between speech units and text while also following natural-language task instructions.
The third stage uses LoRA to efficiently fine-tune the model on the chain-of-modality portion of SpeechInstruct. The paper adds LoRA adapters to the attention mechanisms and trains only the new low-rank parameters. The same loss as stage 2 is used.
This final stage is explicitly aimed at speech dialogue and more complex modality switching, while keeping the number of trainable parameters small compared with full-model fine-tuning.
The experiments use LLaMA-13B as the backbone model. The reported training setup is as follows:
For decoding, the paper sets the maximum sequence length to 2048, temperature to 0.8, top-$k$ sampling with $k=60$, and top-$p$ sampling with $p=0.8$.
The paper does not report standard benchmark scores. Instead, evaluation is based on human judgment and qualitative case studies of whether the model follows instructions correctly in both cross-modal and spoken dialogue settings.
The main results section presents case-based evidence rather than a quantitative leaderboard. For cross-modal instruction following, the examples show that SpeechGPT can:
The cited examples demonstrate ASR-style transcription and TTS-style generation, such as transcribing an utterance into written form and reading a sentence aloud as a speech clip. These examples are used to argue that the model can follow task instructions across modality boundaries rather than merely handling one fixed speech task.
For spoken dialogue, the paper shows examples in which a speech instruction leads to a speech response. The demonstrated dialogues include pronunciation help, poem generation, advice-seeking, answering factual questions, and a safety-related refusal. The authors state that the responses adhere to the HHH criteria: Harmless, Helpful, and Honest.
Because the paper relies on case studies, the evidence is qualitative: it supports the claim that the model can behave as a spoken conversational agent, but it does not provide task-level accuracy numbers, BLEU/WER scores, or comparative benchmark tables in the text provided here.
The paper's stated contributions are:
The main methodological novelty is not simply adding a speech front-end or speech back-end, but instead treating speech as a discrete token stream that can be jointly modeled with text in the same autoregressive framework.
The paper is explicit about several limitations:
These limitations are important for a talking-head or conversational-AI team because they indicate that the current system is stronger at instruction following and speech-text conversion than at rich, long-horizon, emotionally expressive interaction.
SpeechGPT shows a concrete path toward unifying speech and text in one language model by using discrete speech units, vocabulary expansion, and staged training. The paper's evidence suggests that this design can support both speech understanding and speech generation, along with mixed speech-text instruction following. Within the scope of the reported experiments, the strongest claim is qualitative but clear: discrete tokenization makes it possible to bring speech more directly into the LLM interface, opening a route toward more general cross-modal conversational agents.
The The training scripts in The inference implementation is located in Additional utilities related to speech tokenization and vocoder models are provided under Overall, the repository closely follows the methodology described in the paper, offering a complete pipeline from multi-modal training data through to end-to-end speech and text generation via a unified large language model.5.2 Stage 2: Cross-modal Instruction Fine-Tuning
moss-002-sft dataset to form the training set $I$.
5.3 Stage 3: Chain-of-Modality Instruction Fine-Tuning
6. Experimental Setup
Setting
Stage 1
Stage 2
Stage 3
Batch size
768
1536
128
Peak learning rate
2e-4
2e-4
2e-4
Max length
1024
512
1024
Training steps
900
4000 in the appendix table; the experimental setup text also mentions 2100
4200
LoRA rank
-
-
8
LoRA alpha
-
-
16
Trainable parameters
13B
13B
6M
Training device
96 A100 GPUs
96 A100 GPUs
8 A100 GPUs
7. Reported Results and Qualitative Behavior
8. Novelty and Contributions
9. Limitations
10. Takeaway
Code & Implementation
SpeechGPT repository provides the implementation of the SpeechGPT model that empowers large language models with intrinsic cross-modal conversational abilities, as presented in the paper. The codebase is organized primarily under the speechgpt/ directory, containing source code for both training and inference.speechgpt/src/train/, such as ma_pretrain.py, correspond to the paper's three-stage training strategy. This includes modality-adaptation pre-training and cross-modal instruction fine-tuning, enabling the model to learn representations that integrate speech and text modalities.speechgpt/src/infer/cli_infer.py, which handles input preprocessing, speech-to-unit conversion, multi-modal prompt construction, and text or speech generation. It interfaces with the underlying LLaMA-based causal language model and uses a HiFi-GAN vocoder for speech synthesis, enabling the model to produce spoken output directly.speechgpt/utils/. The README includes usage examples and links to demos demonstrating SpeechGPT's capabilities.