AudioPaLM

AudioPaLM: A Large Language Model That Can Speak and Listen

AudioPaLM is a multimodal large language model that combines speech and text processing to perform ASR, speech-to-text, and speech-to-speech translation. It preserves speaker voice and intonation while using text knowledge to enhance multilingual speech understanding and generation.

• llm
• multimodal
• asr
• speech-to-speech
• tts
• voice-cloning
• voice-conversion
• autoregressive

Authors: Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, Christian Frank

Categories: cs.CL, cs.AI, cs.SD, eess.AS, stat.ML

Comment: Technical report

Published 2023-06-22 · Updated 2023-06-22

Links

Paper & demos

• arXiv
• Project page

Abstract

We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples

---

Introduction

AudioPaLM is a decoder-only multimodal language model for speech understanding and generation. The central idea is to merge a pretrained text LLM and a pretrained speech language model into one shared token space so that the same Transformer can consume and emit both text and speech tokens. In the paper, the authors build on PaLM-2 for text modeling and AudioLM for audio token modeling, then unify them into a single architecture capable of automatic speech recognition (ASR), automatic speech translation (AST), text-to-speech (TTS), and speech-to-speech translation (S2ST).

The motivation is that audio systems benefit from the linguistic and translation competence already encoded in large text LLMs, while speech tokenization and audio generation preserve paralinguistic information such as speaker identity and intonation. The paper argues that mixing both modalities in one autoregressive model allows the model to learn bidirectional mappings between speech and text, rather than using separate encoder-decoder pipelines.

Two themes recur throughout the paper:

• Text pretraining transfers to speech tasks: initializing from a text-only checkpoint is substantially better than training from scratch.
• Audio token quality matters: the choice of speech tokenizer strongly affects downstream performance.

Method

AudioPaLM uses a decoder-only Transformer over a unified vocabulary containing both text tokens and discrete audio tokens. In this setup, audio is first tokenized into a discrete sequence; the model then treats text and audio as just token sequences. This makes multimodal modeling look almost identical to standard text-only next-token prediction, except that the vocabulary now spans both modalities.

Let the text vocabulary size be $t$, the audio vocabulary size be $a$, and the embedding dimension be $m$. A text-only model has an input embedding matrix $E $ of size $t \times m$ and, with tied input/output embeddings, a final projection of size $m \times t$. AudioPaLM expands this to $(t+a) \times m$ by adding new rows for audio tokens while keeping the rest of the decoder architecture unchanged. The first $t$ tokens remain SentencePiece text tokens, and the next $a$ tokens correspond to audio tokens.

The key implementation choice is to initialize from a pretrained text LLM checkpoint and then add randomly initialized audio-token embeddings. The paper reports that the entire model must be finetuned; freezing most weights was not found to work well. This is one of the main practical differences from some multimodal adapter-style systems.

Audio tokenization

Raw waveforms are converted into discrete tokens using a self-supervised speech encoder followed by vector quantization. The paper evaluates three tokenizers:

• w2v-BERT: multilingual w2v-BERT embeddings quantized with k-means. The authors use a multilingual model rather than the English-only setting of prior AudioLM work, and they do not normalize the embeddings before k-means because normalization degraded multilingual performance. This tokenizer yields tokens at $25$ Hz with a vocabulary of size $1024$.
• USM-v1: the same quantization recipe applied to the larger multilingual Universal Speech Model (USM) encoder, using the middle layer of the $2$B parameter encoder. It also yields $25$ Hz tokens with vocabulary size $1024$.
• USM-v2: a further improved quantizer trained with an auxiliary ASR loss and finetuned for multilingual performance. This is the best-performing tokenizer in the paper.

Task formulation and prompting

Training examples can contain any combination of the fields audio, transcript, translated audio, and translated transcript. The paper defines five component tasks:

• ASR: audio $7$ transcript
• AST: audio $7$ translated transcript
• S2ST: audio $7$ translated audio
• TTS: transcript $7$ audio
• MT: transcript $7$ translated transcript

Tasks are specified using ordinary text tags, not special control tokens. Example prompts include [ASR French], [TTS English], and [S2ST English French]. The language names are included in the prompt, which the authors found especially helpful for low-resource languages.

The paper also introduces combined tasks, in which the model is trained to emit intermediate outputs for a composite task in one autoregressive pass. For example, a combined S2ST prompt may ask the model to first produce English text, then French text, and finally French audio tokens. This is analogous to a chain-of-thought style decomposition, but applied to speech-text generation. The authors report that combined tasks often improve AST quality, though they can slightly hurt ASR.

Audio decoding back to waveforms

Audio tokens produced by AudioPaLM are converted back to waveform using one of two decoders:

• AudioLM stage 2 + stage 3: an autoregressive decoder that generates SoundStream tokens conditioned on audio tokens and a short voice-conditioning sample, followed by a higher-bit-rate residual quantizer reconstruction stage.
• SoundStorm: a non-autoregressive iterative decoder that generates audio tokens in parallel. The paper reports that SoundStorm produces comparable quality with much better consistency and is substantially faster.

For voice transfer, the model conditions on a 3-second voice sample, represented as both audio tokens and SoundStream tokens. This allows the translated speech to preserve the source speaker's voice across languages.

Training objective and optimization

Training is standard next-token prediction with loss masking on the input portion. This means the model is only penalized on the desired outputs, even though the inputs may themselves be audio-token sequences. The paper uses the same finetuning recipe as PaLM: Adafactor, constant learning rate $5 \times 10^{-5}$, and dropout $0.1$.

Datasets and evaluation

The paper trains on a mixture of speech-text and synthetic speech-text datasets. The main real data sources are:

• CoVoST2 / CVSS: speech translation data for $21$ source-language pairs to English, with CVSS providing speech targets.
• VoxPopuli ASR: parliamentary speech with transcripts for ASR.
• VoxPopuli S2ST: parliamentary speech with spoken translations for S2ST.
• Common Voice 11: multilingual read speech paired with text.
• Conversational EsEn: conversational Spanish-English data with speech, transcripts, and translations.
• YouTube ASR: multilingual unlabeled audio automatically transcribed using a USM-2B ASR model.
• WMT/TED TTS: text translation corpora augmented with synthetic TTS audio.
• PaLM MT TTS: synthetic S2ST data created by translating transcripts with PaLM-2 and synthesizing speech with a prior AudioPaLM S2ST model.

The training mixtures are constructed with SeqIO and are balanced by dataset size, with larger datasets downweighted. The paper trains two main families of models: an AST mixture that includes ASR and AST tasks, and an S2ST mixture that additionally includes TTS and S2ST tasks.

Evaluation uses the following metrics:

• BLEU for AST on CoVoST2 and FLEURS.
• WER for VoxPopuli ASR; for CoVoST2 ASR on Japanese and Chinese, the paper reports CER instead.
• ASR-BLEU for S2ST on CVSS, computed by transcribing generated speech with a fixed ASR system and scoring the transcript against reference text.

Main results

The headline result is that AudioPaLM substantially improves speech translation and remains competitive on ASR. The strongest AST model is AudioPaLM-2 8B AST, while the strongest S2ST system is AudioPaLM 8B S2ST decoded with SoundStorm.

Interpretation of the main table:

• AudioPaLM beats prior systems on CoVoST2 AST and CVSS S2ST.
• AudioPaLM-2 further improves AST, indicating that stronger text-only translation priors transfer to speech.
• ASR performance is competitive but not uniformly best-in-class; the strongest ASR numbers in the paper are still comparable to specialized ASR systems.
• The cascaded AudioPaLM-2 ASR+translation pipeline slightly outperforms the direct AST model on CoVoST2 AST, but AudioPaLM still provides a unified single-model approach.

Zero-shot AST on FLEURS

A major claim of the paper is that the model can do zero-shot speech-to-text translation for language pairs not directly seen as AST during training, as long as the model saw ASR for the source language. On FLEURS, the authors split languages into two groups:

• AST-observed: languages for which X $7$ English speech translation data was seen in training.
• ASR-observed only: languages for which the model saw transcription data but not translation data.

The paper emphasizes two takeaways from this experiment. First, AudioPaLM-2 clearly improves over AudioPaLM in both seen and unseen translation settings. Second, the model can translate speech into English even for language pairs where it never saw speech translation data, so long as it had ASR exposure for those languages. This supports the paper's thesis that translation competence is inherited from the text LLM and transferred into the speech domain.

Speech quality and voice preservation

For CVSS-T voice-transfer evaluation, the authors compare AudioPaLM against the ground-truth synthesized references from the dataset and against Translatotron 2. They report both objective and subjective scores, plus a measure of acoustic consistency.

The subjective study sampled $210$ source utterances, filtered to exclude extremely noisy examples, and collected $10$ ratings per generated sample. The result is that AudioPaLM is judged better than both the dataset's synthetic references and Translatotron 2 on both speech quality and voice similarity. This is a notable result because the model is not merely intelligible; it also preserves speaker identity and acoustic conditions more effectively than the baselines.

Ablations and design choices

The paper includes several ablations that explain why the final system works. The most important are summarized below.

1) Adding ASR helps AST

Adding ASR supervision improves AST even though ASR is not the target metric. The authors interpret this as helping the model connect audio inputs to its text knowledge.

2) Finetuning from a text LLM is much better than training from scratch

This is one of the strongest empirical messages of the paper. Simply adding audio tokens to a pretrained text checkpoint and finetuning it gives a very large gain over random initialization, even though the audio embeddings themselves are new and the input modality is completely different.

3) Audio tokenization quality strongly affects results

The move from w2v-BERT to USM-v1 to USM-v2 produces large improvements. The paper explicitly concludes that the audio tokenizer is a critical bottleneck and an important future research area.

4) Combined tasks help AST, with a small ASR trade-off

Decomposing a complex task into simpler intermediate outputs improves translation quality, consistent with chain-of-thought-like benefits. The downside is a mild degradation on ASR, likely because checkpoint selection is tuned to AST and because the task mix changes.

5) Adding S2ST introduces speech generation capability, but can slightly hurt text outputs

Outputting audio tokens is fundamentally different from outputting text tokens under the loss-masked setup, so enabling S2ST consumes model capacity. The trade-off is that the model gains a new generative capability at the cost of some text-task performance.

6) More training data helps, especially when it includes ASR and synthetic translation data

Scaling data consistently improves translation. Additional ASR data especially helps AST, and synthetic TTS/MT-generated speech gives the strongest S2ST results.

7) SoundStorm improves decoding quality for S2ST

SoundStorm is the better waveform decoder in this setup, matching the paper's claim that it improves consistency and speed while preserving or improving intelligibility.

8) PaLM-2 is better than PaLM for speech translation

The authors interpret this as evidence that PaLM-2's stronger translation competence transfers into the audio domain, especially for AST. ASR effects are mixed and seem more dependent on the data mixture.

9) Scaling the model helps

With PaLM-2 checkpoints of $128$M, $1$B, and $8$B parameters, performance improves monotonically with scale. The paper highlights that moving from $128$M to $1$B yields large gains, and moving from $1$B to $8$B adds more improvements. On the full Public + YT + WMT/TED mixture, the $8$B model is best across the reported ASR and AST metrics and substantially outperforms Whisper Large-v2 on VoxPopuli ASR and AST.

Discussion of what the paper shows

The paper's strongest contribution is not just that a speech-capable LLM can be built, but that it can be built by minimally extending a text LLM. The only architectural change is expanding the token vocabulary to include audio tokens; the rest of the Transformer is left intact. Despite this simplicity, the model learns to:

• transcribe speech to text,
• translate speech to text in many languages,
• generate speech from text,
• perform direct speech-to-speech translation, and
• transfer a voice across languages from a short prompt.

The results also show that the model inherits and exposes the properties of its components:

• Text LLM priors drive translation quality and zero-shot behavior.
• Speech tokenization determines how much of the audio signal is accessible to the model.
• Audio decoders such as SoundStorm affect the final speech realization and voice consistency.

Limitations and open questions

The paper is unusually explicit about its limitations. First, AudioPaLM depends heavily on the quality of the speech tokenizer; the authors show that tokenization choice changes results dramatically. Second, they found it necessary to finetune the entire model rather than freezing most parameters. This means the approach is less modular than adapter-based multimodal systems and does not preserve the original text-only behavior by construction. Third, the evaluation space for generative audio is still much less mature than for text, so a large part of the paper focuses on speech translation benchmarks where established metrics exist.

Open research questions highlighted by the paper include:

• what properties a good audio tokenizer should have,
• how to measure those properties reliably,
• how to design better benchmarks for generative audio, and
• how to scale these methods beyond the current speech translation and recognition tasks.

Conclusion

AudioPaLM shows that a large text LLM can be extended into a speech-text model by adding discrete audio tokens and finetuning on a carefully designed mixture of speech and translation tasks. The method is conceptually simple but empirically strong: it achieves state-of-the-art or near-state-of-the-art speech translation results, competitive ASR, zero-shot AST for many unseen language combinations, and high-quality voice-preserving S2ST. The paper's main technical lesson is that speech and text can be unified successfully in a single decoder-only Transformer, provided that the tokenizer, task formulation, and training mixture are designed well.