Whisper

Robust Speech Recognition via Large-Scale Weak Supervision

Whisper is a large-scale sequence-to-sequence model trained on 680,000 hours of weakly supervised web audio transcripts. It enables robust zero-shot speech recognition, translation, and language identification across many languages without fine-tuning, approaching human transcription accuracy and robustness.

• asr
• multimodal
• speech-to-speech

Demos

These demos showcase Whisper's capabilities as a robust, multitask speech recognition model trained via large-scale weak supervision. Viewers should watch for its multilingual transcription accuracy, speech translation quality, and language identification effectiveness across diverse audio inputs. The WER breakdown by language visualizes performance variation, while the approach diagram clarifies the model's sequence-to-sequence multitask training design that unifies ASR, translation, and identification tasks.

Authors: Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever

Categories: eess.AS, cs.CL, cs.LG, cs.SD

Published 2022-12-06 · Updated 2022-12-06

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub103,933 stars751 watchers12,649 forksMIT
• Hugging Face76,340 downloads420 likes

Impact

• Semantic Scholar7,560 citations1,041 influential citations

Abstract

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

---

1. Problem setting and core claim

This paper asks a simple but important question for speech systems: what happens if we train directly on a very large weakly supervised corpus of audio/transcript pairs and aim for zero-shot transfer, instead of relying on self-supervised pretraining plus task-specific fine-tuning? The authors argue that scaling supervised-but-noisy web speech data can yield a single model that is both more robust and more usable than pipelines that need dataset-specific adaptation. Their central claim is that a sequence-to-sequence Transformer trained on 680,000 hours of labeled audio can transfer well across English ASR, multilingual ASR, speech translation, language ID, noise robustness, and long-form transcription, often without any fine-tuning.

The paper emphasizes two related ideas. First, a strong decoder matters: if the model is trained end-to-end to emit transcripts and task tokens, it can perform multiple speech tasks with one interface rather than requiring separate components for ASR, VAD, translation, and language ID. Second, robustness should be evaluated out of distribution. A model that looks excellent on an in-domain benchmark can still fail badly elsewhere, so the paper repeatedly measures zero-shot performance on held-out datasets and compares it to humans and supervised baselines.

2. Data collection, filtering, and segmentation

The dataset is built from audio on the internet that already comes with transcripts. The authors intentionally take a minimalist preprocessing stance: instead of aggressively normalizing transcripts, they train the model to predict raw transcript text and rely on sequence-to-sequence capacity to learn the mapping from speech to naturalistic written output. This avoids a separate inverse-text-normalization stage at inference time.

Because web text is noisy, the paper devotes substantial effort to filtering. The training set is cleaned with automated heuristics to remove clearly machine-generated transcripts, detect mismatches between spoken language and transcript language, and reduce duplicate or low-quality content. The language mismatch check uses an audio language detector derived from a prototype Whisper model and VoxLingua107, with transcript language verified by CLD2. If the transcript language is English and the audio language does not match, the example is kept as X→en speech translation data rather than ASR data. They also apply fuzzy de-duplication and remove sources that manual inspection reveals to contain partially transcribed, poorly aligned, or caption-like machine-generated text.

Audio files are split into 30-second segments, and the training set includes segments with no speech as well, albeit with subsampled probability. Those speechless segments are useful for voice activity detection. To reduce contamination, the authors also deduplicate at the transcript level against at least TED-LIUM 3, which they identify as a higher-risk evaluation set for overlap.

The final pretraining corpus spans 680,000 hours. Of this, 117,000 hours cover 96 non-English languages, and 125,000 hours are X→en translation pairs. The paper’s core qualitative point is that the corpus is not merely larger than prior supervised speech data; it is also broader in domain, recording setup, speaker demographics, and task mix.

3. Model architecture and task formulation

Whisper uses an off-the-shelf encoder-decoder Transformer. The input is resampled to 16 kHz, converted into an 80-channel log-magnitude Mel spectrogram using 25 ms windows and 10 ms stride, and globally scaled to roughly zero mean and the range $[-1, 1]$. The encoder begins with two convolution layers of width 3 and GELU activations, with the second convolution using stride 2. Sinusoidal position embeddings are added before the Transformer blocks, and the encoder ends with layer normalization. The decoder uses learned position embeddings and tied input/output token embeddings.

The paper uses a single text generation interface for all speech tasks. Decoding begins with a start-of-transcript token, followed by a language token, an optional no-speech token, a task token such as transcription or translation, and a timestamp-mode token when needed. Timestamp tokens are quantized to 20 ms, which matches the model’s native temporal resolution, and timestamps are interleaved with text when timestamp prediction is enabled. The decoder is effectively an audio-conditional language model, and the model can also be conditioned on previous transcript text with some probability so that it can exploit longer-range linguistic context.

The training objective is standard autoregressive token prediction conditioned on audio and task tokens:

$$\mathcal{L}(\theta) = -\sum_{t \notin \text{previous context}} \log p_\theta(y_t \mid y_{<t}, x)$$

In other words, the model predicts all tokens except the prepended previous-context text, which is masked from the loss. This is a small detail but important: the model is not trained as a pure transcript memorizer; it is trained to operate as a general decoder over multiple speech-processing tasks.

The tokenizer is GPT-2-style byte-level BPE for English models; for multilingual models the vocabulary is refit to the same size to reduce fragmentation on non-English text. The paper explicitly notes that a shared tokenization interface is one reason the same model can cover ASR and translation across many languages.

4. Training procedure and optimization

Training uses data parallelism with FP16, dynamic loss scaling, and activation checkpointing. Optimization is done with AdamW, gradient clipping, and a linear learning-rate decay to zero after a warmup of 2048 updates. The standard training run uses a batch size of 256 segments and 2²⁰ updates, corresponding to roughly two to three passes through the full dataset. The paper deliberately avoids data augmentation and regularization in the main runs, arguing that the diversity and scale of the training set are sufficient to encourage robustness.

Hyperparameters reported in the appendix include $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-6}$, and weight decay 0.1. The model-specific peak learning rates are $1.5 \times 10^{-3}$ for Tiny, $1 \times 10^{-3}$ for Base, $5 \times 10^{-4}$ for Small, $2.5 \times 10^{-4}$ for Medium, and $1.75 \times 10^{-4}$ for Large.

The paper also reports an improved Large V2 model trained for 2.5× more epochs and regularized with SpecAugment, Stochastic Depth, and BPE Dropout; its training uses 655,360 updates and batch size 1024. Unless otherwise stated, the reported results are updated to use this improved model.

One practical issue the authors discovered is that the model sometimes learned to hallucinate speaker names. This happens because many training transcripts explicitly include speaker metadata, but the current 30-second acoustic window often does not contain enough evidence to infer the name. Their workaround is a brief fine-tuning step on transcripts that do not include speaker annotations, which suppresses this failure mode.

5. Evaluation protocol and text normalization

The paper evaluates in a strict zero-shot setting: for each benchmark, no training data from that benchmark is used. This is crucial to the authors’ argument, because they want to measure broad generalization rather than in-distribution fitting. They evaluate English ASR, multilingual ASR, speech translation, language identification, noise robustness, and long-form transcription.

A major metric issue in speech recognition is that word error rate (WER) can over-penalize harmless formatting differences. Since Whisper emits flexible UTF-8 text rather than a rigid symbol set, the authors use a custom normalization pipeline before computing WER. The English normalizer removes bracketed text, fillers such as “uh” and “um”, punctuation that is not semantically relevant, British spellings, and many formatting variations; it also standardizes numbers and currencies. For non-English text they use a more conservative lowercasing and punctuation-stripping process, and for languages without spaces such as Chinese and Japanese they insert spaces between every character so the score effectively behaves like character error rate.

The authors are explicit that this normalization is imperfect and may overfit to some of Whisper’s output peculiarities, but their comparison against FairSpeech’s normalizer suggests that the effect is broadly similar across systems and does not merely benefit Whisper. Still, it is an important methodological choice: on some datasets the WER reduction can be as large as roughly 50% when the reference transcript format differs in contractions, whitespace, or number formatting.

6. Zero-shot English ASR, robustness, and human comparison

On English ASR, the paper’s main message is that zero-shot Whisper is not just competitive on the benchmark it was trained closest to; it generalizes far better than conventional LibriSpeech-centered baselines. The authors compare a large zero-shot Whisper model against a strong wav2vec 2.0 baseline with no language model and show that, while both can be similar on LibriSpeech clean, Whisper is dramatically better on out-of-distribution test sets.

Using the paper’s effective-robustness framing, the zero-shot Whisper model makes 55.2% fewer errors on average than the matched wav2vec 2.0 baseline across the out-of-distribution English datasets. This is the paper’s strongest empirical evidence that large-scale weak supervision changes not just benchmark accuracy but the shape of generalization.

The paper also argues that the right comparison for human performance is not in-distribution fine-tuning but zero-shot transfer. On LibriSpeech, modern supervised models can reach or surpass human-level WER, yet still make substantially more mistakes than humans on other datasets. Whisper’s robustness frontier is much closer to the human profile. For one Kincaid46 subset, the authors report that a computer-assisted human transcription service is only 1.15 percentage points better than Whisper in aggregate WER, while fully human transcription is only a fraction of a point better. The paper therefore treats Whisper as approaching human-level robustness, though not perfect accuracy.

7. Multilingual ASR transfer

Multilingual behavior is one of the paper’s main differentiators from earlier English-only systems. The authors show that zero-shot Whisper can perform well on multilingual LibriSpeech (MLS), but that performance is more mixed on VoxPopuli. On MLS, the large V2 model reaches 7.3 WER, outperforming XLS-R (10.9), mSLAM-CTC (9.7), and the other reported baselines in their zero-shot setup. On VoxPopuli, however, Whisper is weaker: it obtains 13.6 WER, behind Maestro (8.1), mSLAM-CTC (9.1), and XLS-R (10.6), though still better than the VP-10K+FT baseline from the original paper.

The Fleurs analysis is especially informative because it spans many more languages than MLS or VoxPopuli. The paper finds a strong squared correlation coefficient of 0.83 between log training data per language and log WER, with a fitted slope implying that WER approximately halves for every 16× increase in training data. The main outliers are languages with unique scripts or larger linguistic distance from the dominant Indo-European training data, such as Hebrew, Telugu, Chinese, and Korean. The authors interpret this as evidence that multilingual scaling works, but also that data quality, script coverage, and tokenizer fit matter.

In short, the paper’s multilingual story is not “Whisper solves every language equally well.” It is that broader multilingual supervision produces useful transfer, but performance remains data-dependent and uneven across languages, especially for lower-resource or script-divergent ones.

8. Speech translation and language identification

For speech translation, the paper evaluates X→en translation on CoVoST2 and also repurposes Fleurs as a translation benchmark by treating the English transcript as the reference translation. On CoVoST2, zero-shot Whisper reaches 29.1 BLEU overall, with 36.2 on high-resource languages, 32.6 on mid-resource languages, and 25.2 on low-resource languages. This is enough to exceed the baselines they compare against overall and in the medium/low-resource bins, though not always on the highest-resource languages.

The paper attributes much of the translation ability to the pretraining corpus itself: it includes roughly 125,000 hours of X→en translation data, which is vastly more than the 861 hours available in CoVoST2. On Fleurs translation, the relationship between per-language translation supervision and BLEU is much weaker than for ASR, with an $R^2$ of only 0.24. The authors highlight a noisy example: Welsh has an unexpectedly large amount of supposed translation data, but inspection shows that much of it is actually English audio misclassified as Welsh by the language detector.

For language identification, Whisper is not competitive with the best supervised models on Fleurs. The paper reports 64.5% accuracy overall, versus 77.7% for mSLAM-CTC and 71.4% for w2v-bert-51. The authors note that Whisper has no training data for 20 of the 102 Fleurs languages, so the overall ceiling is only 80.4%; on the overlapping 82 languages, the best Whisper model reaches 80.3%.

9. Long-form transcription and decoding heuristics

Whisper is trained on 30-second windows, so long-form transcription requires an external strategy that stitches together successive windows. The authors transcribe a 30-second chunk, shift the window using predicted timestamps, and repeat. They found that long-form performance depends heavily on decoding heuristics, especially on avoiding repetition loops and timestamp drift.

Their reliable long-form decoding recipe is worth summarizing because it is one of the paper’s more practical contributions. They use beam search with 5 beams, start at temperature 0, and increase temperature in steps of 0.2 up to 1.0 when either the average log probability falls below $-1$ or the gzip compression ratio exceeds 2.4. They also condition on previous text when the temperature is below 0.5, use a combined no-speech threshold of 0.6 plus average log-probability threshold of $-1$ for VAD, and constrain the first timestamp to lie between 0.0 and 1.0 second to avoid dropping initial words.

The table shows that each heuristic helps in at least some settings, although the improvements are not uniform across datasets. The main practical takeaway is that Whisper’s base decoder is good enough to enable long-form transcription, but a production-like system still benefits from carefully designed search, timestamp, and VAD rules.

10. Scaling, multitask transfer, and data-size ablations

The scaling plots are one of the paper’s strongest pieces of evidence against saturation at small model sizes. Across multilingual ASR, speech translation, and language ID, performance continues to improve as the model grows. For English ASR, gains taper off earlier, which the authors interpret as consistent with approaching human-level performance on the easier in-domain benchmarks.

The multitask-transfer analysis is also nuanced. When training jointly on English ASR, multilingual ASR, translation, and other tasks, small models can suffer negative transfer relative to English-only models trained with the same effective compute. However, the joint setup scales better: for the largest experiments, the multitask/multilingual models outperform English-only models, even before adjusting for the fact that only about 65% of compute in the joint setup is spent on English ASR.

This ablation makes a very direct point: scaling the amount of weak supervision helps all tasks, but the marginal benefit is not constant. English ASR improves quickly up to roughly 54k hours and then shows diminishing returns. Multilingual ASR follows a similar pattern, and speech translation is especially data-hungry: performance is near zero at the smallest scales and improves only once the dataset becomes large enough to support cross-lingual transfer. The authors explicitly suggest that the full 680k-hour dataset may still be under-trained, but they also allow that the field may be approaching a saturation regime where additional data gives smaller gains.

11. Limitations and future work

The paper is candid about several limitations. First, decoding errors remain, especially in long-form settings: the model can repeat itself, omit the first or last few words of a segment, or hallucinate text unrelated to the audio. The decoding heuristics help, but they are described as a workaround rather than a complete solution. The authors speculate that supervised fine-tuning or reinforcement learning aimed directly at decoding quality could reduce these failures.

Second, the system is still uneven across languages. Performance is strongly correlated with per-language supervision, and many languages have less than 1000 hours in the training corpus because the dataset is English-centric. The authors see a clear path forward: intentionally increase coverage for lower-resource languages, which should improve average multilingual quality even without massively enlarging the whole corpus.

Third, the paper studies only zero-shot transfer. That is appropriate for the robustness story, but it leaves open how much better Whisper would become if fine-tuned on task-specific high-quality data. The authors explicitly list fine-tuning as an important future direction, both for direct performance gains and for fairer comparison with prior specialized systems.

Fourth, it is still unclear how much of Whisper’s strength comes from the encoder, the decoder, or their interaction. The authors suggest that ablations such as decoder-less CTC models or language-model comparisons on top of existing encoders could clarify whether the decoder is the main driver of robustness.

Finally, the paper notes that it does not use the self-training or self-supervision methods that dominate much recent speech recognition work. The authors do not claim those approaches are unnecessary in general; they only show that they are not required to obtain a highly competitive, broadly robust zero-shot speech system at this scale.

12. Bottom line

The paper’s main technical contribution is not a new loss or a new pretraining trick. It is the demonstration that large-scale weak supervision plus a carefully designed sequence-to-sequence interface can yield a single, robust, multi-task speech model that transfers surprisingly well without fine-tuning. In English ASR, Whisper is dramatically more robust out of distribution than standard supervised baselines. In multilingual ASR, the model benefits from scaling but remains strongly data-dependent. In speech translation, it is competitive or state of the art in zero-shot settings on some benchmarks. In long-form transcription and human comparison, it approaches the quality of professional transcribers while outperforming several commercial and open-source systems in the evaluated conditions.

The overall message for a conversational-AI or talking-head team is that Whisper should be viewed as a foundation model for speech processing: a single decoder-driven interface for multiple speech tasks, trained on massive weak supervision, with robustness that is better explained by data scale and diversity than by any specialized downstream adaptation. The paper’s strongest evidence is empirical and cross-domain: when you train on enough diverse speech, zero-shot behavior becomes much closer to the kind of generalization humans expect from speech systems in the wild.