Moshi

Moshi: a speech-text foundation model for real-time dialogue

A speech-text foundation model for real-time dialogue via end-to-end speech-to-speech generation instead of cascaded components. By jointly modeling overlapping audio streams with text-token prediction, it achieves 200ms latency while preserving emotion and handling natural conversational dynamics like interruptions.

• dialogue
• speech-to-speech
• streaming
• low-latency
• llm
• asr
• tts

Authors: Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour

Categories: eess.AS, cs.AI, cs.CL, cs.LG, cs.SD

Published 2024-09-17 · Updated 2024-10-02

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub10,506 stars98 watchers975 forksApache-2.0
• Hugging Face4,116 downloads64 likes

Impact

• Semantic Scholar614 citations117 influential citations

Abstract

We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.

---

Introduction and Motivation

Current spoken dialogue systems — from early voice assistants like Alexa and Siri to modern GPT-based pipelines — are built from a chain of independent components: voice activity detection (VAD), automatic speech recognition (ASR), a large language model (LLM) for text dialogue, and text-to-speech (TTS) synthesis. While such cascaded architectures have enabled question-answering over voice, they suffer from three fundamental limitations that prevent them from emulating the feel of real human conversation:

• High latency. Each stage in the pipeline adds delay. The compounded latency of a typical VAD → ASR → LLM → TTS system is several seconds, whereas natural human response times are on the order of 200–300 ms.
• Textual information bottleneck. Because language understanding and generation happen in the text domain, all non-verbal signal — emotion, accent, paralinguistic cues, surrounding audio events — is discarded. The model operates as if speech were simply read text.
• Turn-based segmentation. Pipeline systems assume that conversation is a neat sequence of non-overlapping single-speaker segments. This misses backchannelling ("uh-huh", "I see"), interruptions, and overlapping speech, which account for 10–20% of spoken time in natural conversation.

To address all three limitations simultaneously, Moshi casts spoken dialogue as a full-duplex, speech-to-speech generation problem. Rather than cascading specialist modules, a single model jointly encodes and generates audio for both participants of a conversation in real time. The paper introduces a complete system: the Helium text LLM backbone, the Mimi neural audio codec, the RQ-Transformer generative architecture, multi-stream modeling, and the Inner Monologue training procedure.

System Overview

Moshi is a multi-stream speech-to-speech Transformer model. At a high level:

• Helium is a 7B-parameter autoregressive text LLM pre-trained on 2.1 trillion tokens of public English text. It forms the Temporal Transformer backbone of Moshi.
• Mimi is a neural audio codec that converts raw audio into discrete tokens via residual vector quantization (RVQ), combining acoustic and semantic information into a single causal, streaming tokenizer.
• Moshi's generative model uses an RQ-Transformer — a large Temporal Transformer (Helium) coupled with a small Depth Transformer — to autoregressively predict all token sub-sequences at each time step.
• Multi-stream modeling allows the model to simultaneously process and generate two separate audio streams: one for Moshi's own speech and one for the user's speech, eliminating the concept of explicit speaker turns.
• Inner Monologue is a training and inference procedure where time-aligned text tokens are predicted as a prefix to audio tokens, providing a textual scaffold that dramatically improves the linguistic quality of generated speech.

The resulting system achieves a theoretical latency of 160 ms (200 ms in practice), below the 230 ms average human response time measured across 10 languages.

Helium: The Text Language Model Backbone

Architecture

Helium is a decoder-only autoregressive Transformer with the following design choices, in line with recent best practices:

• RMS Normalization at the input of attention blocks, feed-forward blocks, and the output linear layer.
• Rotary Positional Embeddings (RoPE) with a context length of 4,096 tokens.
• FlashAttention for memory-efficient training.
• Gated Linear Units (GLU) with SiLU activation in the feed-forward blocks.
• A SentencePiece unigram tokenizer with 32,000 tokens, all numbers split into single digits, and byte-fallback to prevent information loss.
• Model dimension 4096, MLP dimension 11264, 32 heads, 32 layers (approximately 7B parameters).

Training uses AdamW with weight decay 0.1, momentum 0.9, squared-gradient decay 0.95, cosine learning rate schedule starting at $3 \times 10^{-4}$ with linear warmup, for 500k steps with a batch size of 4.2M tokens on H100 GPUs using FSDP and activation checkpointing.

Pre-training Data and Filtering

The training corpus is 2.1 trillion tokens of English text. It is composed of:

• 12.5% curated sources: Wikipedia (five dumps from 2017–2022), Wikibooks, Wikisource, Wikinews, StackExchange, and the peS2o scientific article collection.
• 87.5% filtered CommonCrawl from ten crawls (2018–2023), processed through a multi-stage pipeline.

The CommonCrawl filtering pipeline consists of three stages:

1. Deduplication. Line-level deduplication using FNV-1a hashes and a Bloom filter to remove boilerplate. A fastText classifier then performs fuzzy deduplication of blocks of at least 3 consecutive similar lines.
2. Language identification. A fastText language classifier retains only documents classified as English with confidence above 0.85.
3. Quality filtering. A 9-category fastText classifier trained on high-quality sources (Wikipedia, Wikibooks, etc.) vs. random CommonCrawl pages assigns domain-specific scores aggregated at document level (length-weighted average of line scores). Only documents above a per-domain threshold are retained.

Text LM Evaluation Results

Helium is evaluated on a standard battery of benchmarks: ARC (easy/challenge), OpenBookQA, HellaSwag, WinoGrande, PIQA, SocialIQA, TriviaQA, Natural Questions, and MMLU. On most benchmarks, Helium is on-par with or outperforms models trained with similar compute (MPT, Falcon, Llama 2, OLMo). It reaches 54.3 on MMLU, competitive with Mistral and Gemma which use up to 3× more compute. This validates the quality of the text data pipeline.

Mimi: The Neural Audio Codec

Motivation

Audio language models historically require two separate tokenizers: semantic tokens from self-supervised speech models (e.g. HuBERT, WavLM) capture linguistic content, while acoustic tokens from neural codecs (e.g. Encodec) capture fine audio detail. Using both in sequence enables intelligible, high-quality synthesis, but the non-causal nature of semantic encoders is incompatible with streaming inference. Mimi addresses this by distilling semantic information directly into the first level of a causal RVQ codec.

Base Architecture

Mimi is a SeaNet autoencoder with a residual vector quantizer (RVQ) bottleneck:

• The encoder cascades residual convolutional blocks with dilated and strided convolutions, ELU non-linearities, and weight normalization. Four blocks with strides $(4, 5, 6, 8)$ plus a final stride-2 convolution project 24 kHz audio to a latent at 12.5 frames/second with dimension $D = 512$. All convolutions are causal.
• The decoder mirrors the encoder with transposed convolutions.
• The RVQ uses $Q = 8$ quantizers each with codebook size $N_A = 2048$, for a total bitrate of 1.1 kbps at 12.5 Hz.
• Embeddings are projected to 256 dimensions before RVQ and back to 512 before the decoder.

Transformer-Based Bottleneck

Two Transformer modules are inserted in the bottleneck: one immediately before quantization and one immediately after. Each has 8 layers, 8 heads, RoPE positional encodings, causal masking, a 250-frame (20 s) finite context, GELU activations, model dimension 512, and MLP dimension 2048. LayerScale (initial diagonal value 0.01) stabilizes training. The Transformer in the encoder is especially helpful for semantic distillation, as distilling a large non-causal Transformer into a purely convolutional encoder is challenging.

Split RVQ for Semantic–Acoustic Decoupling

Rather than placing all 8 levels in a single RVQ chain (where the semantic first level must pass its residual to acoustic levels), Mimi uses a split RVQ: a standalone VQ for the semantic token (distilled from WavLM) plus a 7-level RVQ in parallel for acoustic tokens. Their outputs are summed, so both contribute to reconstruction. This removes the constraint that acoustic residuals build on the semantic quantizer's output, improving the semantic–acoustic trade-off significantly.

WavLM Distillation

WavLM-Large produces 1024-dimensional embeddings at 50 Hz from 16 kHz audio. To align with Mimi's 12.5 Hz latent rate, WavLM embeddings are downsampled via average pooling (stride 4, kernel 8) applied non-causally (compatible with streaming since it is training-only). A linear projection maps the first VQ output to 1024 dimensions; cosine distance to the WavLM target is the distillation loss.

Adversarial-Only Training

Mimi is trained with two configurations tested against each other: a standard mix of multi-scale mel-spectrogram reconstruction loss + multi-scale STFT discriminator (following Encodec), versus adversarial losses only (feature matching + discriminator, no reconstruction loss). Subjective MUSHRA evaluation shows the adversarial-only model achieves a striking MUSHRA score of 81.0 vs. 58.8 for the mixed-loss variant, despite dramatically lower VisQOL objective scores. This highlights a severe mismatch between objective metrics and perceived quality.

Quantization Rate

Following prior work, Mimi applies quantizer dropout for bitrate scalability. Additionally, quantization is applied to the latent space only 50% of the time during training (passing raw embeddings to the decoder on the other 50%), which significantly improves objective quality metrics (especially VisQOL).

Training Details

AdamW optimizer, weight decay $5 \times 10^{-2}$ on Transformer parameters only, learning rate $8 \times 10^{-4}$, momentum 0.5, squared-gradient decay 0.9, EMA decay 0.99. Batch size 128, random 12-second windows, 4M steps. Transformer context during training is limited to 10 seconds.

Audio Tokenization Evaluation

Mimi is evaluated on two axes:

• Semantic quality via triphone-based ABX error rate on LibriSpeech dev-clean. Lower is better; Mimi achieves 8.1–8.7% ABX with distillation (vs. ~23–31% without). SpeechTokenizer achieves 3.3% but at much higher bitrate and lower audio quality.
• Acoustic quality via VisQOL, MOSNet (automatic), and MUSHRA (human). The adversarial-only Mimi achieves MUSHRA 81.0, far outperforming RVQGAN (31.3), SemantiCodec (64.8), and SpeechTokenizer at 1.5 kbps (45.1).

Crucially, Mimi is the only causal codec among compared systems, making it uniquely compatible with streaming inference. Its low frame rate (12.5 Hz vs. 50–75 Hz for alternatives) is essential to keeping the cost of each Temporal Transformer forward pass manageable.

Generative Audio Modeling Architecture

The RQ-Transformer

Modeling $Q = 8$ RVQ levels at 12.5 Hz would require 100 tokens per second if flattened — generating 30,000 tokens for 5 minutes of audio and running 100 tokens/s is incompatible with real-time streaming. The RQ-Transformer splits this burden across two models:

• The Temporal Transformer (Helium, 7B parameters) operates at $S$ time steps, one per audio frame. At step $s$ it consumes all previous time steps $V_0, \ldots, V_{s-1}$ and produces a temporal context vector $z_s \in \mathbb{R}^d$: $$z_s = \mathrm{Tr}_{\mathrm{Temp}}(V_0, \ldots, V_{s-1})$$
• The Depth Transformer (6 layers, dimension 1024, 16 heads) operates over the $K$ token levels within a single time step. Given $z_s$ and previously predicted tokens $V_{s,1}, \ldots, V_{s,k-1}$, it produces logits for level $k$: $$l_{s,k} = \mathrm{Tr}_{\mathrm{Depth}}(z_s, V_{s,1}, \ldots, V_{s,k-1}) \in \mathbb{R}^{N_k}$$ The first token uses a dedicated linear head: $l_{s,1} = \mathrm{Lin}(z_s)$.

The model is trained so that $\operatorname{softmax}(l_{s,k})$ approximates the conditional distribution: $$\operatorname{softmax}(l_{s,k}) \approx p(V_{s,k} \mid V_0, \ldots, V_{s-1}, V_{s,1}, \ldots, V_{s,k-1})$$

A key design choice is depthwise parametrization: unlike prior RQ-Transformer works that share weights across all levels in the Depth Transformer, Moshi uses separate parameters per level index $k$ for the linear projections and fully connected layers. This reflects that each RVQ level encodes qualitatively different information (semantic vs. fine acoustic detail). Ablations confirm this is beneficial at no additional cost given the Depth Transformer's small size.

Acoustic Delay

Simply setting the multi-sequence $V = A$ (audio tokens without any delay) leads to unstable generation. Moshi introduces an acoustic delay $\tau$ between the semantic first codebook and the acoustic higher codebooks. For delay $\tau$ and step $s$:

This delay allows the large Temporal Transformer to model the dependency between semantic and acoustic tokens, reducing the burden on the Depth Transformer. Ablations show that $\tau = 1$ (160 ms latency) or $\tau = 2$ (240 ms latency) both substantially outperform $\tau = 0$ (80 ms). Pre-training uses $\tau = 2$; fine-tuning and inference use $\tau = 1$ for the final 160 ms theoretical latency.

Multi-Stream Modeling

To model a two-speaker conversation without explicit turn segmentation, Moshi extends the single-stream formulation to two parallel audio streams: $(A_{t,q})$ for Moshi's speech and $(A'_{t,q})$ for the user's speech. Both are encoded separately by Mimi and interleaved into the joint sequence $V$, with the same acoustic delay applied to each. At inference, Moshi's tokens are sampled autoregressively, while the user's tokens are fed in from the actual microphone input (the model's prediction of the user stream is discarded during real interaction but used for offline evaluation of generated dialogues).

Inner Monologue

Pure audio-to-audio generation, even with the RQ-Transformer, tends to produce speech of limited linguistic coherence. The key insight of Inner Monologue is that having the model also predict the textual transcription of its own speech — at each audio frame, as a prefix to the semantic token — provides a powerful scaffold that dramatically improves linguistic quality.

Text–Audio Alignment

Given a transcript of Moshi's speech (obtained with Whisper and word-level timestamps), each word is mapped to SentencePiece tokens $w_{i,j}$ and a start frame index $t_i$ (timestamp divided by 12.5 Hz frame rate). Two special tokens, PAD and EPAD, are used to fill frames with no word onset. The aligned text sequence $W_t$ is built as:

• Initialized to PAD for all frames.
• For each word $i$: frame $t_i - 1$ is set to EPAD; frames $t_i + j$ are set to $w_{i,j}$.

In English conversational speech, padding tokens represent approximately 65% of all text stream positions. The EPAD token serves as a useful two-step signal: first decide to end padding, then choose the next word.

Final Joint Sequence

Combining multi-stream modeling and Inner Monologue, the full joint sequence to model has $K = 2Q + 1 = 17$ sub-sequences per time step:

Streaming ASR and TTS via Delay Inversion

A single delay hyperparameter controls which modality drives the other:

• Streaming ASR: delay the text stream by 2 seconds behind the audio. At inference, teacher-force the audio tokens from the actual input signal and sample text tokens freely. The output text stream is a streaming transcription with 80 ms word-level timestamp precision.
• Streaming TTS: delay the audio stream by 2 seconds behind the text. At inference, provide padded text tokens and sample audio tokens freely. The model learns to place PAD/EPAD tokens naturally between words, and a logit bonus on padding tokens can control speech rate. Voice is controlled via a speaker prefix.

Training Loss

Given ground-truth discrete tokens $V_{s,k}$ and estimated logits $l_{s,k}$, the training objective is:

where $\alpha_k = 100$ for the semantic audio token ($k$ corresponding to $A_{s,1}$) and $\alpha_k = 1$ for all acoustic tokens. The text token ($k=1$, Inner Monologue) and the combined audio tokens receive equal total weight. The semantic token receives 100× weight relative to acoustic tokens within the audio group, reflecting its greater importance for intelligible speech generation.

Datasets and Training Pipeline

Text Data

2.1 trillion tokens as described in the Helium section (Wikipedia, StackExchange, peS2o, filtered CommonCrawl).

Audio Data

• Unsupervised audio dataset: 7 million hours of freely available audio content, predominantly English speech. Transcribed with Whisper large-v3. Used for single-stream audio pre-training; all audio resampled to 24 kHz mono.
• Fisher dataset: 2000 hours of telephone conversations recorded with separate channels per speaker, providing ground-truth separated speaker streams for multi-stream training. Original 8 kHz audio upsampled to 24 kHz with AudioSR.
• Supervised multi-stream dataset: 170 hours of natural and scripted conversations between pairs of participants, recorded on separate channels. Used only to train the streaming multi-stream TTS system (not Moshi directly) and to fine-tune Helium on real conversation transcripts.

Speech–Text Instruct Data

20,000+ hours of synthetic speech generated by a multi-stream TTS system conditioned on a single actor's voice (covering 70+ speaking styles). Transcripts are generated by a fine-tuned Helium using prompts built from Wikipedia paragraphs, StackExchange posts, and role-play scenarios. Categories of generated conversations include:

• General knowledge Q&A (seeded from Wikipedia/StackExchange context).
• Voice instruction role-play (92 speaking styles; see table in appendix).
• Misspelling robustness (user mispronounces words, Moshi asks for clarification).
• False-fact correction.
• Basic math, grammar, and trivia single-turn pairs.
• Safety conversations (Moshi refuses unethical/NSFW requests).

The user stream's voice is randomly sampled per example for robustness. During instruction fine-tuning, extensive data augmentation is applied to the user stream: random gain (−24 to +15 dB, 50% of the time), additive noise from the DNS challenge (30% of the time, −30 to +6 dB relative), simulated echo from Moshi's stream (scaled by uniform $[0, 0.2]$, delay uniform $[100, 500]$ ms, 30% probability), and reverb augmentation.

Training Stages

Training proceeds through four stages, all using AdamW with weight decay 0.1 on H100 GPUs with FSDP:

During single-stream pre-training, text masking (30% probability) and randomized text–audio delay (±0.6 s) are applied. To prevent catastrophic forgetting, 50% of training steps in pre-training use pure text batches with a separate optimizer state. The text embedding and output layer learning rate is multiplied by 0.75 during text-only batches from audio training, and padding tokens receive 50% reduced weight in the cross-entropy loss.

Ablation Studies

RQ-Transformer Ablation

Compared to using separate independent classification heads for the 8 RVQ levels (following MusicGen), the RQ-Transformer provides only marginal improvement when using the staggered delay pattern $[0,1,2,3,4,5,6,7]$ (perplexity 40.3 vs. 42.2). However, this delay pattern induces 8 × 80 ms = 640 ms of theoretical latency — unacceptable for real-time dialogue.

Switching to the low-latency pattern $[0,2,2,2,2,2,2,2]$ (240 ms latency, with all acoustic levels delayed by 2 steps) dramatically changes the picture: without the RQ-Transformer, perplexity explodes to 135.4; with it, perplexity is 36.8. The RQ-Transformer is thus critical under strict latency constraints.

Delay Pattern, Semantic Token Weight, and Inner Monologue Ablation

The paper evaluates quality using transcript NLL (scored with a lightweight 460M-parameter text LM) and transcript length (a strong proxy for model quality — weak models collapse to silence). Key findings:

Inner Monologue provides the most dramatic single improvement: transcript length increases from 602 to 1920 characters and NLL drops from 3.65 to 2.77 — a step change in linguistic quality. The combination of semantic weight 100, depthwise parametrization, and Inner Monologue is used throughout all subsequent experiments.

Evaluation Results

Audio Language Modeling (Textless NLP Benchmarks)

Moshi is evaluated on sWUGGY (lexical), sBLIMP (syntactic), sTopic-StoryCloze and sStoryCloze (semantic) benchmarks. Key results:

• In the Audio-only, Cold Start setting (randomly initialized, audio-only data, no Inner Monologue), Moshi scores 74.8 sWUGGY / 59.9 sBLIMP / 80.9 sTopic-StoryCloze / 56.9 sStoryCloze, surpassing GSLM, AudioLM, and TWIST.
• With Helium warm start, Moshi matches or beats TWIST-13B and Spirit-LM on all audio metrics.
• After multi-stream instruction fine-tuning, Moshi achieves 63.0 sWUGGY / 55.2 sBLIMP / 83.6 sTopic-StoryCloze / 62.7 sStoryCloze in the multimodal category.
• MMLU on text (without audio tokens): Moshi scores 49.7–49.8 after audio training (vs. 54.3 for Helium), and 12 points higher than Spirit-LM (36.9).

The paper notes that sWUGGY degrades after instruction fine-tuning due to noisy/reverberant user stream conditions, not due to an actual loss of lexical knowledge.

Spoken Question Answering

Moshi with Inner Monologue achieves remarkable results on spoken QA benchmarks (0-shot):

Inner Monologue nearly triples Moshi's spoken QA accuracy on all benchmarks, at negligible inference cost (17 tokens per step instead of 16). Moshi significantly outperforms Spectron and SpeechGPT while being the only model compatible with streaming inference — Chain-of-Modality methods (Spectron, SpeechGPT) must complete the full text answer before generating any audio.

Generated Dialogue Quality and Turn-Taking

Moshi can generate full conversations autonomously (both sides), enabling offline evaluation of conversational dynamics. Evaluated on 1000 10-second prompts from Fisher with 32 continuations each:

At temperature 0.8, Moshi achieves perplexity 41.9 — matching the cascaded topline and vastly outperforming the non-cascaded dGSLM baseline (195.9). At temperature 1.0, turn-taking statistics (gap, overlap) closely match the ground truth. This demonstrates that Moshi generates coherent multi-party conversational dynamics in real time.

Streaming ASR and TTS

• Streaming TTS (2 s audio lookahead): 4.7% WER on LibriSpeech test-clean — better than Vall-E (5.9%) which requires the full input sequence, though worse than NaturalSpeech 3 (1.81%).
• Streaming ASR (2 s text lookahead): 5.7% WER on LibriSpeech test-clean — worse than Streaming FastConformer (3.6%) but also provides 80 ms word-level timestamps as a byproduct.

Model Compression and Quantization

Post-training quantization (PTQ) is studied to enable deployment on resource-constrained devices. Activations are dynamically quantized to 8 bits (AbsMax, symmetric) at the input of every linear layer; model weights are quantized asymmetrically (MinMax) at various bitwidths and block sizes. Embedding layers, RMSNorms, and Mimi are left unquantized.

Linguistic Quality under Compression

Helium (text-only) is substantially more robust than Moshi to quantization. With block size 32, W4A8 keeps Helium within 2 MMLU points of the BF16 baseline (52.97 vs. 54.3) at 4.37 GB model size. Moshi degrades more: W4A8 with block 32 yields MMLU ~42–46 (vs. 49.7–49.8 baseline). The online demo uses W8A8 as a reasonable compromise (−2 MMLU points, ~2× compression).

Audio Quality under Compression

Audio quality (as measured by MOSNet) is robust to quantization down to 4 bits. However, MOSNet is insensitive to certain severe artifacts at very low bitwidth. The paper introduces an entropy-spectrum artifact detection method: Shannon entropy $H$ is computed over sliding windows of $C=64$ tokens for the text and each audio codebook stream independently. Three artifact categories are defined:

• Repetitive text: text entropy is nearly flat (slope below $10^{-3}$) but non-zero.
• Background noise: text stream is silent ($H^0 = 0$) but audio entropy is high (median across codebooks $> 2$).
• Gibberish: text entropy $> 3.5$ during active speech (incoherent token stream).
• Noisy audio: standard deviation of entropy across audio codebooks $> 0.6$.

Artifact summary at 2-bit compression: W2A8 with block size 256 shows 83.1% gibberish and only 5.9% clean samples. At 4 bits (block 32), 95.7% of samples are clean — essentially matching the unquantized model.

Safety Analysis

Toxicity

Evaluated on the ALERT benchmark (hate, self-harm, weapon, crime, sex, substance). Moshi achieves an overall safety score of 83.05, placing it in the middle of a comparison including GPT-3.5 (96.95), GPT-4 (99.18), Llama 2 (99.98), Alpaca (62.13), Falcon (88.11), Mistral (75.45), and OLMo (85.90). Industry models benefit from extensive private annotation and red-teaming.

Regurgitation of Training Data

An audio fingerprinting system (constellation map over mel-spectrograms + hash encoding, inspired by Shazam-style matching) is used to identify the most frequent audio segment in the 7M-hour training set. Key findings from 100,000 generations:

• Without deduplication, the pre-trained model regurgitates the most frequent segment ~0.13–0.19% of the time unprompted, and up to 98.4% when prompted with the first 3 seconds.
• Dataset deduplication alone brings the regurgitation rate to 0%, even when prompted.
• The instruction fine-tuned multi-stream model also shows 0% regurgitation in all tested conditions.

Voice Consistency

Over 100 hours of generated conversations, WavLM speaker embeddings show that 98.7% of Moshi's speech segments are classified as closer to Moshi's reference voice than the user's voice. Speaker consistency remains stable over time (98.4–99.3% across 20–45 second windows). Simply training with a consistent voice during instruction fine-tuning is sufficient to lock in Moshi's identity at inference.

Watermarking

Signal-based watermarking (AudioSeal) is defeated by Mimi re-encoding: detection score drops to 0.0805 (indistinguishable from no watermark at 0.0855). Generative watermarking (biasing token sampling probabilities via a hash function) is explored but hampered by codec non-idempotence: re-encoding a decoded waveform produces substantially different tokens, especially for higher RVQ levels and under temporal shifts. This is identified as a negative result and several open directions are discussed (marking only the first RVQ level, training for idempotence, latent-space watermarking).

Audio Fingerprinting System (Deduplication)

The audio matching system is based on a constellation map approach:

From each keypoint $(t_k, f_k)$ in the constellation $\mathcal{C}$, a hash signature $s_k = (f_b, f_k, f_f, t_k - t_b, t_f - t_k)$ is computed using a forward and backward neighbor in the temporal window $[m, M) = [4, 20)$ frames. The hash key can take $64^3 \cdot (M - m)^2 = 2^{26} \approx 67$ million distinct values. Matching is done via inverted file with 1D Hough temporal voting for consistency. For deduplication, a "duplicate signature set" is precomputed from frequently co-occurring segments and used as a runtime filter.

Limitations

• Knowledge degradation from audio training. Moshi's MMLU drops from 54.3 (Helium) to ~49.7 after multi-stream instruction fine-tuning. The gap is especially large on TriviaQA (7.3 vs. 56.4 for Helium text), partly attributable to multi-sentence questions and formal syntactic structures absent in oral-style training data.
• sWUGGY regression after fine-tuning. Instruction fine-tuning on noisy/reverberant conditions for the user stream degrades lexical discrimination benchmarks, though the model retains practical intelligibility.
• Turn-taking at low temperature. At temperature 0.8, Moshi produces too-short IPUs (35.1 s) and too-long pauses (13.2 s) compared to ground truth, suggesting over-conservative speech activity.
• ASR quality gap. Streaming ASR (5.7% WER) is significantly worse than dedicated streaming ASR systems (3.6% for Streaming FastConformer with similar lookahead). The Inner Monologue ASR is presented as a demonstration of the framework's flexibility, not a competitive ASR system.
• Quantization sensitivity. Moshi is more sensitive to quantization than Helium alone. Below 4 bits, linguistic performance degrades severely; quantization-aware fine-tuning (QAT) is suggested but not implemented due to the complexity of the multi-stage training pipeline.
• Watermarking remains unsolved. Neither signal-based nor generative watermarking reliably survives Mimi re-encoding. The paper identifies codec non-idempotence as the primary obstacle.
• Safety score below best proprietary models. Moshi's ALERT score of 83.05 lags behind GPT-4 (99.18) and Llama 2 (99.98), reflecting the absence of large-scale human preference annotation and red-teaming.
• Objective audio metrics are unreliable proxies. VisQOL and MOSNet correlate poorly with human perception under distribution shifts (e.g., changing training objective from mixed to adversarial losses). MUSHRA human evaluation is required for reliable quality assessment.

Summary of Contributions

• Helium: A 7B-parameter text LLM competitive with models trained on similar compute budgets (MPT, Falcon, Llama 2, OLMo) and within striking range of models using 3× more compute (Mistral, Gemma).
• Mimi: A causal, streaming neural audio codec with split RVQ combining semantic distillation (from WavLM) and high-quality adversarial acoustic reconstruction. Achieves MUSHRA 81.0 at 1.1 kbps and 12.5 Hz frame rate — the lowest frame rate among compared codecs, enabling real-time generation.
• RQ-Transformer with acoustic delay: A hierarchical two-Transformer architecture that makes streaming audio generation of semantic + acoustic tokens tractable under 160 ms latency constraints. Depthwise parametrization in the Depth Transformer is shown to be beneficial.
• Multi-stream full-duplex modeling: The first model to jointly autoregressively model two speaker audio streams in parallel, removing speaker-turn segmentation and supporting natural conversational dynamics.
• Inner Monologue: A training method that generates time-aligned text tokens as a per-frame prefix to audio tokens. This single design choice nearly triples spoken QA accuracy and multiplies generated speech length by more than 3×. The same framework, by varying the delay between text and audio, yields zero-shot streaming ASR and streaming TTS.
• Complete training and safety analysis: A four-stage pipeline from text pre-training through instruction fine-tuning, with comprehensive evaluation of toxicity, regurgitation, voice consistency, watermarking, and model compression robustness.