AudioLM
AudioLM: a Language Modeling Approach to Audio Generation
AudioLM generates high-quality, long-consistent audio by modeling audio as a language over discrete tokens. It combines semantic tokens for structure and acoustic tokens for detail, enabling coherent speech and music generation without transcripts or symbolic input.
Links
Paper & demos
Impact
Abstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
Introduction
AudioLM frames audio generation as language modeling over discrete audio tokens. The paper argues that audio signals live at multiple levels of abstraction: in speech, short-range acoustics interact with phonetics, prosody, syntax, and semantics; in music, local waveform detail coexists with melody, harmony, and rhythm. The central challenge is to generate audio that is both high fidelity and long-range coherent, especially when the model is trained only from raw waveforms and not from transcripts or symbolic annotations.
The paper’s main contribution is a hybrid tokenization scheme and a hierarchical autoregressive Transformer that first models coarse semantic structure and then refines the output using acoustic details. The authors combine two complementary representations: semantic tokens extracted from a self-supervised speech model, and acoustic tokens extracted from a neural audio codec. This enables the model to preserve the long-term structure of a sequence while still reconstructing audio at high perceptual quality.
The paper demonstrates the approach on two very different domains: speech continuation and piano continuation. For speech, AudioLM is trained without transcripts and can generate syntactically and semantically plausible continuations that preserve speaker identity, prosody, and recording conditions even for unseen speakers. For piano, the same framework generates coherent continuations in melody, harmony, tone, and rhythm, despite being trained without symbolic music representation.
Problem formulation and overall architecture
AudioLM operates on a single-channel waveform $x \in \mathbb{R}^{T}$ and uses three frozen components: a tokenizer that maps audio to discrete tokens, a decoder-only Transformer language model, and a detokenizer that maps predicted tokens back to waveform space. If $h$ denotes the token sequence, the model learns an autoregressive factorization
$$p(h) = \prod_{t=1}^{T'} p(h_t \mid h_{
The key design choice is that AudioLM does not use a single discrete representation. Instead, it separates audio into semantic tokens for global structure and acoustic tokens for signal-level detail. These are modeled in a hierarchy: semantic tokens are generated first, then coarse acoustic tokens are generated conditioned on the semantic sequence, and finally fine acoustic tokens are generated conditioned on the coarse acoustic tokens.
The paper first studies the strengths and weaknesses of the two token types in isolation. The conclusion is that neither representation alone is sufficient: semantic tokens are good for long-range structure but poor for waveform reconstruction, while acoustic tokens reconstruct audio well but do not encode enough high-level structure for coherent generation.
Acoustic tokens are produced by SoundStream, a neural audio codec. A convolutional encoder maps the waveform to embeddings at 50 Hz for 16 kHz audio, which corresponds to a 320-fold reduction in temporal rate. Each embedding is discretized by a residual vector quantizer with $Q$ codebook levels, each of size $N$; the paper uses $N = 1024$ and, for the speech setup, $Q = 12$. With $Q = 4$ and $N = 1024$, the bitrate is 2000 bps; with $Q = 12$ and $N = 1024$, the bitrate is 6000 bps. The SoundStream decoder reconstructs waveform samples from the discrete codes, and the codec is trained end to end with reconstruction and adversarial losses.
These acoustic tokens preserve perceptual quality, but a language model trained only on them tends to preserve speaker identity and recording conditions while losing linguistic coherence. In the paper’s qualitative analysis, such a model often produces speech that sounds like babbling despite retaining the prompt’s voice characteristics.
Semantic tokens are derived from w2v-BERT XL, a 0.6B-parameter self-supervised speech representation model trained with masked language modeling and contrastive objectives. AudioLM selects an intermediate layer of the MLM module, normalizes the embeddings to zero mean and unit variance per dimension, and clusters them with $k$-means into $K$ discrete symbols. The paper selects the 7th MLM layer and uses $K = 1024$ clusters after comparing multiple layers and cluster sizes.
These semantic tokens are sampled at 25 Hz, so for 16 kHz audio they correspond to 250 bps when $K = 1024$. They are much better at representing phonetic and lexical structure than acoustic tokens, but by themselves they are poorly invertible and yield weak audio reconstruction.
The paper evaluates the two token types along two axes: phonetic discriminability and reconstruction quality. Phonetic discriminability is measured by ABX error rate, with lower being better; reconstruction quality is measured by ViSQOL, with higher being better. The result is a clear trade-off: semantic tokens improve phonetic structure but reconstruct poorly, while acoustic tokens achieve much better reconstruction but are weak at discriminating phonemes.
The core argument for AudioLM follows from this trade-off: semantic tokens provide the right abstraction for long-range consistency, but need acoustic tokens to produce high-quality audio; acoustic tokens preserve signal detail, but need semantic conditioning to avoid incoherent generations.
AudioLM’s language model is not a single Transformer over one flattened sequence. Instead, it uses three stages of decoding that progressively refine the representation. This structure mirrors the paper’s assumption that semantic content should be generated first, and that lower-level acoustic details can then be generated conditioned on that semantic scaffold.
The first Transformer models the semantic token sequence autoregressively:
$$p(z_{1:T_S}) = \prod_{t=1}^{T_S} p(z_t \mid z_{
The second Transformer predicts the coarse acoustic codes from the first $Q'$ residual quantizers, conditioned on the entire semantic sequence. The paper uses $Q' = 4$ for speech. The acoustic codes are flattened in row-major order so that the model predicts tokens sequentially across time and quantizer index. In effect, the model estimates
$$p(y_t^q \mid z_{1:T_S}, y_{
The third Transformer predicts the remaining fine acoustic quantizers, conditioned on the already generated coarse acoustic tokens. Its factorization is
$$p(y_t^q \mid y_{1:T_A}^{\leq Q'}, y_{
The conditional-independence intuition behind the hierarchy is that semantic tokens should be approximately independent of past acoustic tokens once past semantic tokens are known. This motivates a cleaner factorization than simply interleaving semantic and acoustic symbols in one long sequence.
The paper describes three inference modes.
For continuation, the paper extracts the semantic prompt tokens and the coarse acoustic prompt tokens, generates the continuation of semantic tokens first, appends the semantic continuation to the prompt, and then generates coarse and fine acoustic tokens. The final waveform is reconstructed with the SoundStream decoder. The default sampling temperatures are 0.6, 0.8, and 0.6 for the three stages.
For speech, all components are trained on the Libri-Light unlab-60k split, which contains 60,000 hours of English speech. This is a notably more diverse and noisier training set than the 6,000-hour clean subset used in some prior work, and the paper highlights that AudioLM remains robust in this setting.
Semantic-token configuration is selected by inspecting ABX, sWUGGY, and sBLIMP scores across different w2v-BERT layers and cluster counts, plus a small subjective listening test. The 7th MLM layer and $K = 1024$ clusters are chosen as the best compromise.
For the speech acoustic tokenizer, the paper trains SoundStream with 12 residual vector-quantizer layers, each with a 1024-entry codebook, and a convolutional stack with strides $(2, 4, 5, 8)$. The resulting bitrate is 6000 bps. The hierarchical generation then uses the first four quantizers as the coarse stage and the remaining eight as the fine stage.
The language model in each stage is the same architecture: a decoder-only Transformer with 12 layers, 16 attention heads, hidden size 1024, feed-forward dimension 4096, dropout 0.1, and T5-style relative position embeddings. Each stage has about 0.3B parameters. Training uses random cropping to input lengths of 30 seconds, 10 seconds, and 3 seconds for stages 1, 2, and 3, respectively. Consecutive repetitions of semantic tokens are removed in the first two stages, following prior practice. Each stage is trained on 16 TPUv4s with batch size 256 for 1 million steps.
The speech experiments target three claims: semantic tokens carry linguistic content, acoustic tokens carry speaker and recording information, and the full hierarchy can generate realistic continuations from short prompts.
In this experiment, the model is given ground-truth semantic tokens from real speech and asked to generate acoustic tokens. The purpose is to test whether semantic tokens preserve the sentence content. The generated speech is then transcribed with a Conformer Transducer-L ASR system and compared against the original transcript.
The paper evaluates on LibriSpeech test-clean utterances between 4 and 10 seconds long, using three generated samples per utterance. As a comparison point, it also reports the unit-to-speech resynthesis results of GSLM using HuBERT-derived units.
The authors interpret these numbers as evidence that the semantic tokens preserve the transcript-level content fairly well: the resulting speech remains transcription-friendly even after the model resamples the acoustic representation. The remaining errors are attributed mainly to proper nouns, sentence-ending timing, and background noise differences.
To test whether acoustic tokens encode speaker identity and recording conditions, the paper trains a convolutional speaker classifier on log-mel spectrograms. The classifier uses 1-second crops and a six-block convolutional stack with channel sizes $[64, 128, 256, 256, 512, 512]$. It is trained on the union of LibriSpeech train-clean-100 and test-clean, containing 291 speakers, and reaches near-perfect accuracy on held-out real audio.
The table below shows that SoundStream reconstruction remains easy to classify as the original speaker, but acoustic generation from semantic tokens does not: speaker identity varies substantially when only the semantic content is fixed. In contrast, continuation from a prompt preserves the prompt speaker very well.
The paper notes that 3.2% is far above the $1/291 \approx 0.3\%$ random baseline, but still low enough to indicate that most speaker identity is controlled by the acoustic tokens rather than the semantic tokens. Subjective inspection suggests that prosody is mostly controlled by semantic tokens, with some contribution from acoustic tokens, while recording conditions are mainly represented in the acoustic stream.
To quantify the linguistic knowledge learned by the semantic-token language model, the paper evaluates on the ZeroSpeech 2021 development sets using sWUGGY and sBLIMP. These are zero-shot metrics that ask whether the model assigns higher likelihood to real words than to nonce words, and to grammatical sentences than to minimally perturbed ungrammatical variants. The model normalizes sequence log-likelihood by length for the sBLIMP comparison to avoid bias toward shorter sentences.
AudioLM is compared against both text-based toplines and audio-based baselines, including BERT and CPC-BERT variants, HuBERT-only models, GSLM, and an LSTM baseline. The paper emphasizes that several of these baselines are non-causal and therefore not directly usable for generation, whereas AudioLM remains causal.
AudioLM achieves the best non-text-supervised results in both tasks. The paper reports that it improves sBLIMP by about 8% relative over the previous state of the art among comparable audio models, and even surpasses a supervised phone-topline when log-likelihood is normalized by length. This is one of the paper’s strongest pieces of evidence that language modeling over semantic tokens does capture meaningful lexical and syntactic regularities.
The main generation setting is continuation from a short prompt. The model is given a 3-second prefix from an unseen speaker and asked to generate a 7-second continuation. The speaker classifier then checks whether the continuation matches the prompt speaker. The 92.6% accuracy indicates that AudioLM can preserve speaker identity surprisingly well even for speakers not seen during training.
The paper also runs a human evaluation where raters hear 10-second samples and judge whether each is original or synthetic. Half of the samples are genuine speech compressed with SoundStream to match the codec artifacts of model output, and the other half are AudioLM continuations. Across 1,000 ratings from 10 English-proficient raters, the correct-label rate is 51.2%, which is not statistically different from random guessing ($p = 0.23$). The authors interpret this as evidence that short continuations can be very hard for humans to distinguish from real speech.
Because a model that can continue speech convincingly can also enable spoofing or impersonation, the paper explicitly addresses misuse risk. It proposes a detector for AudioLM-generated speech using the same convolutional architecture as the speaker classifier, but trained for binary classification between original audio and generated continuations.
The detector is trained on original samples and prompts from LibriSpeech train-clean-100, and it compares generated continuations against SoundStream-compressed originals rather than against uncompressed originals, so that the task is not trivially solved by codec artifacts. On a balanced evaluation set, this detector reaches 98.6% accuracy. The authors present this as a practical mitigation, though not a complete solution, to the risks created by realistic speech synthesis.
To show that the method is not speech-specific, the paper retrains AudioLM on an internal dataset of 40,000 hours of piano music spanning different skill levels, pieces, and recording conditions. The hyperparameters remain the same as in speech, except for the acoustic stage: the authors found that a codec with only 3 quantization layers and a larger codebook size of $2^{14}$ already gives high reconstruction quality, so they omit the third acoustic stage and directly predict the 3 acoustic layers in the second stage.
At inference, they condition on a 4-second prompt from the Maestro dataset. The paper compares the full hierarchical model to a version trained on acoustic tokens alone. Both variants produce high audio quality, but only the full AudioLM hierarchy produces continuations that remain coherent in melody and temporal structure.
In a subjective preference study, 10 raters compared 15 prompt-matched pairs of 20-second continuations. The raters preferred the samples generated by the full AudioLM system in 83.3% of the pairs, indicating that hierarchical semantic-to-acoustic generation improves musical consistency as well as speech coherence.
The paper’s conceptual contribution is to show that audio generation becomes much easier when the problem is decomposed into content planning and signal realization. Semantic tokens act as a planning layer that captures long-range structure; acoustic tokens act as a realization layer that captures voice, timbre, and waveform detail. In this sense, AudioLM is not just a codec-plus-language-model pipeline, but a carefully structured factorization of audio generation that matches the multi-scale nature of the data.
The experiments support this decomposition. Semantic tokens are sufficient to recover transcript-like content, while acoustic tokens are necessary to preserve speaker identity and recording conditions. Prompted generation shows that the two together can maintain both linguistic continuity and acoustic continuity over long spans. The piano results show that the same strategy generalizes beyond speech, suggesting that the hierarchy is a more general principle for audio continuation.
The paper is explicit about the dual-use nature of the work. On the positive side, the method could help with assistive communication or music composition. On the negative side, it can be used for spoofing biometric systems, impersonation, or generating deceptive speech. The authors also note the possibility of bias and mismatched accent or dialect behavior for underrepresented groups, since the model inherits the distributional properties of its training data.
A technical limitation is that the strongest speech results come from a carefully engineered multi-stage system with frozen tokenizers, separate Transformers, and hand-chosen prompt lengths and temperatures. The hierarchy improves quality and coherence, but it also adds system complexity. Another limitation is that, although human listeners may be fooled by short continuations, a simple classifier can still detect them with high accuracy. This implies that perceptual realism and machine detectability are not the same thing.
The paper therefore positions detection as an important complementary capability, not as a complete safeguard. The broader takeaway is that high-quality generative audio systems should be developed together with mitigation mechanisms and careful downstream-use analysis.
AudioLM presents a strong case for hierarchical discrete modeling as a general strategy for audio generation. By combining semantic tokens from a self-supervised speech model with acoustic tokens from a neural codec, and by training separate causal Transformers over these levels, the system achieves both long-term consistency and high-fidelity synthesis. The paper’s speech experiments show that the method can generate plausible, speaker-consistent continuations without text supervision, and its piano experiments show that the same idea extends beyond speech to music.
The main empirical message is simple but important: semantic abstraction and acoustic detail should be modeled separately. AudioLM operationalizes that insight with a practical three-stage generation pipeline, and the reported results indicate that this decomposition substantially improves audio continuation quality over modeling acoustic codes alone.
Representation trade-offs: semantic versus acoustic tokens
Acoustic tokens from SoundStream
Semantic tokens from w2v-BERT
Quantitative trade-off study
Tokenization
Bitrate
ABX within / across
ViSQOL
Semantic (w2v-BERT)
250 bps
6.7 / 7.6
1.1
Semantic (w2v-BERT)
6000 bps
5.6 / 6.2
1.4
Acoustic (SoundStream)
2000 bps
22.4 / 28.7
3.3
Acoustic (SoundStream)
6000 bps
17.8 / 26.6
3.9
Hierarchical modeling of semantic and acoustic tokens
Stage 1: semantic modeling
Stage 2: coarse acoustic modeling
Stage 3: fine acoustic modeling
Inference and generation modes
Training setup and model selection
Speech experiments
Acoustic generation from ground-truth semantic tokens
Setting
CER
WER
Original audio
0.8
2.5
SoundStream reconstruction
0.9
2.6
AudioLM acoustic generation
3.4
6.0
GSLM unit-to-speech
2.9
6.6
What the acoustic tokens carry
Setting
Speaker classification accuracy
SoundStream reconstruction
100.0%
AudioLM acoustic generation
3.2%
AudioLM continuation
92.6%
Linguistic probing: sWUGGY and sBLIMP
Model
sWUGGY all
sWUGGY in-vocab
sBLIMP
Forced alignment topline
92.2
—
63.7
Phone topline
97.9
—
66.8
BERT baseline
67.7
75.6
56.1
HuBERT-only
70.9
79.8
59.5
Harwath et al.
67.6
75.4
56.7
CPC-BERT
—
80.0
59.9
van Niekerk et al.
64.3
72.3
54.0
GSLM
—
68.7
57.1
AudioLM
71.5
83.7
64.7
Prompted speech continuation
Safety and synthesized-speech detection
Piano continuation
Interpretation of the method
Limitations and broader impact
Conclusion