SeamlessM4T

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

A unified multilingual model supporting speech-to-speech, speech-to-text, text-to-text, text-to-speech translation, and ASR across 100 languages in a single architecture. It replaces cascaded systems with a modular design trained on 470K hours of automatically mined multilingual speech data.

• speech-to-speech
• asr
• tts
• multimodal

Authors: Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang

Categories: cs.CL

Published 2023-08-22 · Updated 2023-10-25

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub11,808 stars152 watchers1,173 forksCC-BY-NC-4.0
• Hugging Face332,306 downloads989 likes

Impact

• Semantic Scholar195 citations27 influential citations

Abstract

What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication

---

Introduction & Motivation

SeamlessM4T is a unified, massively multilingual and multimodal machine translation system from Meta AI Research, conceived to realize the vision of a "Babel Fish"—a single model capable of translating speech or text between any two languages. The work is motivated by a persistent modality gap in modern MT: while text-to-text translation (T2TT) models such as NLLB cover more than 200 languages, speech translation systems remain fragmented, limited in language coverage, and heavily reliant on cascaded pipelines.

The three core deficiencies of prior speech-to-speech translation (S2ST) systems that SeamlessM4T addresses are:

• Language coverage: Existing systems focus predominantly on high-resource languages (English, Spanish, French), leaving low-resource languages behind.
• Directionality: Most S2ST systems only translate into English (X→eng), not from English into other languages (eng→X).
• Architecture fragility: Cascaded systems (ASR → MT → TTS) compound errors and cannot leverage shared representations across modalities.

SeamlessM4T is the first system to unify ASR, speech-to-text translation (S2TT), text-to-text translation (T2TT), text-to-speech translation (T2ST), and speech-to-speech translation (S2ST) in a single model, supporting up to 100 source speech languages and 35 target speech languages, with text coverage reaching 95 languages.

Task Definitions and Language Coverage

SeamlessM4T supports the following tasks:

• ASR – Automatic Speech Recognition for 96 languages.
• S2TT – Speech-to-Text Translation from 100 source speech languages into 95 target text languages.
• S2ST – Speech-to-Speech Translation from 100 source speech languages into 35 target speech languages.
• T2ST – Text-to-Speech Translation (zero-shot) from 95 text languages into 35 target speech languages.
• T2TT – Text-to-Text Translation from 95 languages into 95 languages.
• TTS – Text-to-speech synthesis for 36 languages.

Languages are classified by speech resource level: high (>1,000 hours of primary supervision), medium (500–1,000 hours), low (<500 hours), and zero-shot (no direct supervision). The full language table covers 100 languages spanning Indo-European, Afro-Asiatic, Sino-Tibetan, Austronesian, Dravidian, Uralic, Turkic, and many other families, written in Latin, Arabic, Devanagari, Cyrillic, CJK, and other scripts.

Two model sizes are released: SeamlessM4T-Large (2.3B parameters) and SeamlessM4T-Medium (1.2B parameters).

Data: SeamlessAlign — Mining 470,000 Hours of Aligned Speech

Speech-Language Identification (LID)

Mining aligned multilingual speech data begins with a robust language identification system. The authors built a new multilingual LID model aligned with the 100-language target coverage. This model was trained on a mixture of labeled and pseudo-labeled data and is used to classify and filter audio segments by language before mining.

SONAR: A Multimodal Sentence Embedding Space

Central to the data mining pipeline is SONAR (Sentence-level multimOdal and laNguage-Agnostic Representations), a new multimodal sentence embedding space that supports both speech and text encoders. SONAR text encoders cover 200 languages; speech encoders cover 37 languages. These embeddings lie in a shared representation space, enabling direct comparison of speech in one language against text in another—a prerequisite for large-scale cross-modal data mining.

SONAR is built on top of LASER3 encoders and extends them with speech modality. The similarity metric used for mining is xsim and its enhanced variant xsim++, which account for cross-lingual cosine similarity in the shared embedding space.

Oversegmentation and Audio Processing

Raw audio (totaling roughly 4 million hours of data across 95+ languages) is first segmented into sentence-length segments via an oversegmentation procedure that uses voice activity detection (VAD) and forced alignment heuristics. The resulting segments are then encoded with SONAR speech encoders, and the closest cross-lingual text matches are retrieved via SONAR text encoders to form aligned speech-translation pairs.

Scale and Statistics of SeamlessAlign

The resulting SeamlessAlign corpus contains more than 470,000 hours of automatically aligned speech translations. The mining covered speech in 37 languages paired with English text translations. Detailed statistics for individual languages are provided in the paper's data tables, covering total raw audio hours (ranging from ~1,400 hours for Maltese to ~490,000 hours for German), ASR training hours, BLEU quality scores compared to Whisper, and the volume of mined audio in three directions:

• Sen2Txx: speech in language X aligned with English text (202,796 hours total)
• Sxx2Ten: speech in X aligned against English text translations (239,767 hours total)
• Sxx2Sen: speech in X aligned with speech in English (29,161 hours total)

The average X→eng BLEU of our ASR system used during mining is 23.3, compared to 22.5 for Whisper-Large-v2 across 37 languages. SeamlessM4T notably outperforms Whisper on several low-resource and morphologically complex languages (e.g., Bengali: 18.9 vs. 13.2; Maltese: 23.2 vs. 13.5; Swahili: 22.6 vs. 7.2; Uzbek: 17.9 vs. 6.0; Kannada: 20.0 vs. 11.6).

Combined Training Data

The final training data combines three source types:

• Primary data: open-source supervised speech translation and pseudo-labeled ASR data.
• Mined data: a filtered subset of SeamlessAlign.
• Human-labeled data: curated human translations for select language pairs.

In total, the training corpus for X2T tasks amounts to approximately 406,000 hours when combining primary and mined subsets. S2ST training data statistics are listed per language, distinguishing primary and mined portions for both X→eng and eng→X directions.

Model Architecture

SeamlessM4T is built by fusing four main building blocks into a unified multitask architecture:

1. w2v-BERT 2.0 — Speech Encoder

The speech encoder is w2v-BERT 2.0, a self-supervised model pre-trained on 1 million hours of unlabeled open speech audio. It is an extension of the w2v-BERT model combining wav2vec 2.0-style contrastive pre-training with masked language modeling in the feature space. It covers 143 languages. After pre-training, the encoder is finetuned on labeled data as part of the SeamlessM4T pipeline. The speech encoder uses a Conformer architecture that processes raw audio waveforms and outputs frame-level speech representations.

2. SeamlessM4T-NLLB — Massively Multilingual Text Encoder-Decoder

For text, SeamlessM4T uses a Transformer encoder-decoder backbone initialized from a massively multilingual T2TT model derived from NLLB (No Language Left Behind). This text encoder-decoder supports 95+ languages in both source and target directions. The text encoder produces contextual representations of tokenized input, while the text decoder generates output tokens autoregressively.

3. T2U — Text-to-Unit Sequence-to-Sequence Model

A dedicated Transformer sequence-to-sequence model, T2U, translates output text tokens into discrete speech units. These discrete units (also called "speech tokens") are learned via k-means clustering of speech representations from the w2v-BERT 2.0 encoder on unlabeled multilingual audio. The T2U model thus bridges the gap between text translation output and the speech synthesis stage.

4. HiFi-GAN Vocoder — Speech Synthesis

A multilingual HiFi-GAN vocoder synthesizes speech waveforms from the discrete speech units produced by the T2U model. The vocoder is trained to support 36 target languages and produces natural-sounding audio output from unit sequences.

UnitY Architecture

The overall SeamlessM4T design is based on the UnitY framework, which introduces a two-pass decoding strategy for S2ST:

• First pass: The speech encoder (or text encoder) feeds into the text decoder, which generates a text translation autoregressively.
• Second pass: The T2U model takes the text translation and generates discrete speech units, conditioned on both the text output and encoder representations.

This two-pass design allows the model to leverage the text translation as an intermediate, interpretable representation while still producing high-quality speech output. The architecture is modular: depending on the task, different combinations of components are activated.

Task Routing and Multitask Training

SeamlessM4T uses a single set of shared weights for all tasks. Task conditioning is achieved through language tokens and modality-specific input pathways. During multitask training, batches from different tasks (ASR, S2TT, T2TT, S2ST, T2ST) are mixed according to a sampling schedule. Encoder representations are shared across speech and text inputs where possible, encouraging the model to learn a unified cross-lingual and cross-modal representation.

Training proceeds in stages:

1. Stage 0: Self-supervised pre-training of w2v-BERT 2.0 on 1 million hours of unlabeled speech.
2. Stage 1: Finetuning the X2T model (combining speech encoder + text encoder-decoder) on combined ASR, S2TT, and T2TT data.
3. Stage 2: Finetuning the full S2ST model including the T2U module and HiFi-GAN vocoder on speech-to-speech paired data.

Evaluation Framework and Metrics

SeamlessM4T introduces a comprehensive suite of automatic and human evaluation metrics:

BLASER 2.0

A key contribution of this work is BLASER 2.0, a language and modality-agnostic quality estimation metric. It builds on the original BLASER model but extends it to handle both speech and text as inputs or outputs, enabling unified evaluation across all five tasks. BLASER 2.0 uses SONAR embeddings to represent both source and hypothesis, and is trained to predict human quality judgments. It achieves similar accuracy to its predecessor on text quality estimation while generalizing to speech.

ASR-BLEU and ASR-ETOX

For S2ST evaluation, the model's speech output is first transcribed using Whisper (Whisper-Medium for English targets, Whisper-Large-v2 for non-English targets), and then standard BLEU is computed on the normalized transcriptions. This metric is called ASR-BLEU. Analogously, ASR-ETOX applies the ETOX toxicity word-list detector to ASR transcriptions of the model's speech output, enabling toxicity evaluation of speech outputs at scale.

XSTS — Human Evaluation Protocol

XSTS (Cross-lingual Semantic Textual Similarity) is a human evaluation protocol where annotators rate the semantic similarity of a translation to a reference on a 1–5 scale. XSTS scores are calibrated across annotators. The paper reports both raw XSTS scores and the proportion of outputs scoring ≥3 (acceptable) and ≥4 (good).

Robustness Benchmarks

New robustness benchmarks based on FLEURS are introduced to evaluate performance under:

• Background noise — audio corrupted with various types of environmental noise at different SNR levels.
• Speaker variation — speech from different speakers, accents, and recording conditions.

Results: Automatic Evaluation

Speech-to-Text Translation (S2TT)

On FLEURS, SeamlessM4T-Large achieves an improvement of 20% BLEU (4.2 BLEU points) over the previous SOTA end-to-end S2TT model (AudioPaLM-2-8B-AST) for X→eng translation. Compared to strong cascaded models (Whisper-Large-v2 + NLLB-3.3B), SeamlessM4T-Large improves translation quality by 1.3 BLEU points for into-English S2TT.

For eng→X translation on CoVoST 2, SeamlessM4T-Large outperforms the previous SOTA (XLS-R-2B-S2T) by 2.8 BLEU points. On FLEURS eng→X, performance is on par with cascaded systems.

Speech-to-Speech Translation (S2ST)

On FLEURS, SeamlessM4T-Large outperforms strong 3-stage cascaded models by 2.6 ASR-BLEU points for into-English S2ST. On CVSS, SeamlessM4T-Large outperforms a 2-stage cascaded model (Whisper-Large-v2 + YourTTS) by 8.5 ASR-BLEU points, a 58% relative improvement (reported as 50% in some parts of the paper).

ASR

SeamlessM4T-Large outperforms Whisper-Large-v2 on FLEURS ASR with an average WER reduction of 45% over 77 overlapping languages.

Text-to-Text Translation (T2TT)

On FLORES, SeamlessM4T matches the performance of NLLB-3.3B for into-English T2TT and improves by approximately 1 chrF++ point on average for from-English T2TT.

Performance vs. Input Duration

Results: Human Evaluation (XSTS)

Human evaluation was conducted using the XSTS protocol on FLEURS for both S2TT and S2ST tasks, covering 24 languages in both directions (eng→X and X→eng). Key findings:

• For S2TT eng→X: XSTS scores for all 24 languages are consistently above 4 (out of 5), indicating high translation fidelity. For example, Indonesian (4.81), Portuguese (4.79), French (4.76), and Catalan (4.73).
• For S2TT X→eng: SeamlessM4T shows significant improvement over Whisper-Large-v2 for 7 of 24 languages, particularly Bengali (4.03 vs. 2.84 for Whisper), Swahili (4.03 vs. 1.65), and Telugu (3.84 vs. 3.14).
• For S2ST: Scores are generally lower than S2TT due to the additional TTS step, but the model delivers substantially better results than Whisper + YourTTS cascades in several lower-resource languages.

MOS Naturalness of S2ST Output

Robustness Evaluation

SeamlessM4T-Large is evaluated on two custom robustness benchmarks derived from FLEURS:

• Noise robustness: Average improvement of 38% over Whisper-Large-v2 in S2TT quality under background noise conditions.
• Speaker variation robustness: Average improvement of 49% over Whisper-Large-v2 in S2TT quality across different speakers.

Responsible AI: Toxicity Evaluation

Added Toxicity Definition and Metrics

Added toxicity is defined as the introduction of toxic content in the translation output that was not present in the source input. The paper introduces ASR-ETOX as a new metric for detecting added toxicity in speech output: Whisper transcribes the speech output, and then ETOX (a word-list-based toxicity detector using Toxicity-200 word lists) is applied to the transcription. For text output, ETOX is applied directly.

Toxicity Results on FLEURS

Overall, added toxicity prevalence on FLEURS is low, averaging 0.15% across languages, tasks, and directions. Breakdown:

• S2TT X→eng: 0.11% added toxicity averaged across languages.
• S2ST X→eng: 0.12% averaged across languages.
• S2TT eng→X: 0.21% averaged across languages.
• S2ST eng→X: 0.16% averaged across languages.

Compared to Whisper-Large-v2 + YourTTS baselines, SeamlessM4T-Large reduces added toxicity by 63% in S2TT X→eng, 62% in S2ST X→eng, and 26% in S2TT eng→X.

Toxicity Results on HolisticBias

The HolisticBias dataset (472K+ English sentences with demographic descriptors across 13 axes) was extended to speech using TTS synthesis. Results show:

• S2TT eng→X: 0.19% added toxicity, 84 languages affected. SeamlessM4T reduces toxicity by 34% vs. Whisper + NLLB-3.3B.
• S2ST eng→X: 0.13% added toxicity, 34 languages affected.

Human validation of ETOX detections reveals that on HolisticBias, 97–98% of detections are true positives, while on FLEURS, 81–84% are false positives—a difference attributed to FLEURS having more neutral content. This confirms that real toxicity mitigation in SeamlessM4T is genuine.

The demographic axes with the highest concentration of added toxicity across both S2TT and S2ST are nonce (0.79% S2TT, 0.46% S2ST) and sexual orientation (0.62% S2TT, 0.35% S2ST). The least toxic axes are age for S2TT and political ideologies for S2ST.

Responsible AI: Gender Bias Evaluation

Methodology

Gender bias is evaluated using the Multilingual HolisticBias dataset, which contains translations of 325 short English sentences (average 6 words) in up to 50 languages, each provided in masculine and feminine reference forms. The paper extends this dataset to speech using MMS-TTS synthesis.

Two evaluation directions are considered:

• eng→X (overgeneralization): Translating gender-neutral English sentences and comparing against masculine vs. feminine references in the target language.
• X→eng (robustness): Translating masculine vs. feminine versions of the same sentence and measuring the quality gap in the English output.

The relative gender preference is quantified as: $$\Delta = \frac{\omega(M) - \omega(F)}{\omega(\min(M, F))}$$ where $\omega \in \{\text{chrF}, \text{ASR-chrF}, \text{BLASER 2.0}\}$, and $M$, $F$ denote masculine and feminine scores respectively.

Bias Results: eng→X (Overgeneralization)

In the absence of gender information in the English source, SeamlessM4T-Large exhibits an average preference of approximately 10% toward translating to the masculine form (measured as $\Delta$ chrF for S2TT: 10.9%; ASR-chrF for S2ST: 10.6%; BLASER: 0.6%). This preference is consistent across both S2TT and S2ST and is observed for all languages except Thai. The highest gender gaps are in Catalan (10.3 chrF points), Slovak (10.1), and Spanish (10.0).

Bias Results: X→eng (Robustness)

For X→eng, SeamlessM4T-Large shows a robustness gap of approximately 3% between masculine and feminine source inputs (chrF $\Delta$: 3.7% for S2TT; ASR-chrF $\Delta$: 3.5% for S2ST). Performance is generally better when translating from masculine source sentences (15 of 18 languages for S2TT). The largest gaps are for Tamil (6.4 chrF points) and Urdu (5.0 chrF points).

Demographic Axis Analysis

Analysis across HolisticBias demographic axes reveals that the axes most sensitive to quality degradation (lowest average chrF/ASR-chrF) are cultural, body type, socioeconomic class, and religion. The highest gender bias (largest masculine–feminine chrF difference) is observed for ability, body type, religion, and characteristics—consistent with findings from the Multilingual HolisticBias paper.

Gender Data Representation

Using word-list matching of gendered nouns and pronouns (11 masculine nouns, 4 masculine pronouns, 10 feminine nouns, 4 feminine pronouns from HolisticBias), the authors find that masculine representations dominate all analyzed datasets. FLEURS, FLORES, and EuroParl show very low overall gendered word density, reflecting domain-neutral content. The SeamlessAlign mined data also shows masculine dominance, which may contribute to the observed bias in model outputs.

FAIRSEQ2 Toolkit

SeamlessM4T is developed and released alongside FAIRSEQ2, a redesigned open-source sequence modeling library (MIT license). FAIRSEQ2 provides:

• State-of-the-art Transformer components (attention, layernorm, embeddings, etc.).
• A scalable streaming data pipeline API (fairseq2.data) for multi-terabyte datasets without explicit preprocessing.
• Efficient training components (optimizers, LR schedulers, loss functions).
• Optimized inference with incremental beam search.

FAIRSEQ2 emphasizes the dependency inversion principle, enabling component drop-ins without copy-pasting large code blocks. It cleanly separates core and experimental code, avoiding the "if-else accumulation" problem of its predecessor FAIRSEQ.

Ablations and Design Choices

The paper discusses several important design choices validated through ablations:

• Two-pass vs. direct S2ST: The UnitY two-pass decoding (text intermediate + T2U) consistently outperforms direct discrete unit prediction without a text intermediate, both in quality and in interpretability.
• Mined data contribution: Adding SeamlessAlign mined data significantly improves performance across low-resource languages, with some languages (e.g., Swahili, Uzbek) showing dramatic BLEU improvements over systems trained without mined data.
• Data filtering: Filtering of imbalanced or toxic content in training data is credited with improved safety metrics.
• w2v-BERT 2.0 pre-training scale: Pre-training on 1 million hours of audio (vs. smaller subsets) improves downstream ASR and S2TT performance, especially for low-resource languages.
• SONAR embedding quality: The use of SONAR for mining (vs. LASER or LASER3) yields better alignment quality as measured by downstream translation quality on mined pairs.

Limitations

The authors candidly discuss several limitations:

• Word-list-based toxicity detection: ETOX produces high false positive rates on some datasets (81–84% FP on FLEURS) and can miss toxicity due to ASR transcription errors (false negatives). It does not capture context-dependent toxicity.
• Gender bias detection via noun lists: Using a fixed vocabulary of 30 gendered nouns may not generalize to all occupational or other gendered nouns, and does not handle inclusive plural forms consistently across languages.
• Short-form model: SeamlessM4T was trained on short speech and text inputs; performance degrades on long-form inputs.
• Research model: Not intended for production deployment, medical/legal domain use, or certified translations.
• ASR performance variability: Like other speech systems, performance varies by gender, accent, race, and language.
• Prosody and expressivity: Speech output lacks preservation of prosodic, emotional, or stylistic properties of the source speech.
• Latency: The current system is not streaming/low-latency; real-time deployment would require additional engineering.
• Gender overgeneralization: The model systematically prefers masculine forms when gender is ambiguous, reflecting imbalances in training data.

Open-Source Contributions

All of the following are open-sourced at https://github.com/facebookresearch/seamless_communication:

• Model weights and inference code for SeamlessM4T-Large (2.3B parameters) and SeamlessM4T-Medium (1.2B parameters), with finetuning recipes backed by FAIRSEQ2.
• SONAR — speech encoders for 37 languages and text encoders for 200 languages.
• SeamlessAlign — metadata to recreate the unfiltered 470,000-hour multimodal aligned corpus.
• BLASER 2.0 — model weights and inference scripts for the cross-modal quality estimation metric.
• Stopes-based pipelines for reproducing the data mining procedure.

Social Impact and Future Directions

SeamlessM4T is positioned as a tool for augmenting "world-readiness"—the ability to communicate across language barriers. The authors highlight its potential value for:

• Accessibility for users with visual impairments or low literacy who rely on voice interfaces.
• Languages with multiple writing systems or no standardized script, where speech-first technology may be more inclusive.
• Low-resource language communities where text-based MT tools have limited reach.

Future research directions identified include:

• Low-latency and streaming S2ST for live conversation applications.
• Prosody and expressivity preservation in speech output.
• Bias and toxicity mitigation strategies (the current work focuses on evaluation, not mitigation beyond data filtering).
• Extension to non-English-centric translation directions (X→Y, not just X→eng or eng→X).
• Improved handling of slang, proper nouns, and domain-specific content.