PRIME-Speech
Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation
PRIME-Speech converts speech-to-text LLMs into speech-to-speech systems by freezing the backbone and attaching a trainable audio post-decoder synchronized to intermediate hidden states. This enables concurrent speech generation while preserving original S2T capabilities and avoiding serial TTS bottlenecks.
Links
Paper & demos
Impact
Abstract
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.
Introduction and Motivation
Speech-to-speech (S2S) interaction is a natural goal for conversational AI: a system should listen to speech, reason over the user's intent, and respond directly in speech. The dominant practical solution remains a cascade pipeline — automatic speech recognition (ASR) converts input to text, a text LLM produces an answer, and a text-to-speech (TTS) system renders the answer as audio. While modular and convenient, this approach has fundamental limitations: recognition errors are locked into text before the LLM sees them, non-lexical acoustic cues are discarded before reasoning, and speech generation cannot begin until enough text has been produced by the downstream TTS system.
This paper focuses on a specific, increasingly important formulation of the S2S problem: how to convert a strong speech-to-text (S2T) LLM into an S2S model without sacrificing the capabilities that made the backbone useful in the first place. Modern S2T LLMs already provide speech perception, text reasoning, and instruction-following. The challenge is that end-to-end S2S training is not merely rendering a completed textual response into speech — it requires generating speech tokens within the same autoregressive process that supports text-oriented reasoning. This can alter output patterns learned by the pretrained backbone and degrade speech understanding, reasoning, and instruction-following capabilities. Conversely, keeping the backbone intact and generating speech only after a text response is complete reduces the system to a serial TTS pipeline, reintroducing the very bottleneck that motivates the S2S approach.
Existing S2S architectures expose this tension in different ways. Unified token-interleaving models (e.g., GLM-4-Voice, LLaMA-Omni) place text and audio tokens in a single autoregressive stream, giving a direct synchronization mechanism but requiring the main decoder to balance heterogeneous text and audio objectives simultaneously. Decoupled Thinker–Talker systems (e.g., Qwen2.5-Omni, Qwen3-Omni, MiniMo) separate semantic reasoning from speech rendering, improving modularity, but the talker is typically driven by finalized text, fixed text chunks, or text-side hand-off states — meaning the speech branch follows an already-materialized representation rather than the evolving hidden states of the speech–text backbone. Multi-token prediction (MTP) can reduce the number of codec-token updates, but applying MTP only to a lightweight talker yields limited end-to-end speedup because the thinker path remains unchanged, while applying it to the main backbone risks degrading S2T capabilities.
PRIME-Speech — standing for Preserving Reasoning and Intelligence while enabling More Efficient Speech-to-Speech generation — proposes a frozen-backbone conversion framework that addresses all three failure modes simultaneously: it keeps the backbone frozen to preserve S2T capabilities, synchronizes a trainable audio branch to the backbone's evolving hidden states (avoiding the serial TTS bottleneck), and applies MTP to the synchronized audio branch to improve latency without modifying the reasoning path.
Method
Overview and Core Factorization
PRIME-Speech converts a frozen S2T LLM into an S2S model by adding a trainable speech-generation branch around the original text pathway. Given input $x = (x^\tau, x^a)$, where $x^\tau$ is an optional text prompt and $x^a$ is the input speech waveform, the frozen backbone produces a text response $y^\tau$, while the added audio branch produces a codec-token response $y^a$. The joint distribution is factored as:
$$P(y^\tau, y^a \mid x) = P_{\mathrm{bb}}(y^\tau \mid x) \cdot P_{\mathrm{aud}}(y^a \mid y^\tau, H^{\mathrm{mid}}; \theta_a)$$where $P_{\mathrm{bb}}$ is the frozen backbone distribution, $P_{\mathrm{aud}}$ is the trainable audio-branch distribution, $H^{\mathrm{mid}}$ is a sequence of intermediate backbone states used for speech conditioning, and $\theta_a$ denotes the audio-branch parameters. During streaming, the dependence on $y^\tau$ is prefix-restricted. At each decoding update $s$, the frozen backbone exposes a hidden state $h^{\mathrm{mid}}_s$; the text head and the audio branch consume this state in parallel. This design ensures the backbone remains solely responsible for speech perception and reasoning, with the audio branch acting as a synchronized speech readout of the preserved backbone trajectory.
Frozen Backbone and Codec Targets
The backbone maps speech and text inputs into a shared autoregressive context. For an input waveform $x^a$, a frozen speech encoder and projection module produce acoustic embeddings $e^{\mathrm{sp}}$; text tokens $x^\tau$ are mapped to embeddings $e^\tau$. The frozen transformer stack processes the concatenated sequence layer by layer:
$$h^{(\ell)} = \mathrm{Backbone}^{(\ell)}([e^\tau, e^{\mathrm{sp}}]), \quad \ell = 1, \ldots, L$$The speech-conditioning interface uses a fixed middle-layer stream $H^{\mathrm{mid}} = h^{(\ell_{\mathrm{mid}})}$, where $\ell_{\mathrm{mid}}$ is set to approximately two-thirds of the backbone depth. This choice is guided by a layer-wise centered kernel alignment (CKA) analysis, which identifies this depth as carrying the richest paralinguistic information while retaining semantic grounding. The final text logits, text-token embeddings, and text KV cache remain entirely those of the frozen backbone — the original S2T pathway is never updated by S2S training.
Target speech is represented as semantic codec tokens from the CosyVoice2 tokenizer at 25 Hz:
$$y^a = \{y^a_t\}_{t=1}^{T_a}, \quad y^a_t \in \{1, \ldots, V_a\}$$where $T_a$ is the number of codec frames and $V_a$ is the codec vocabulary size. The audio post-decoder predicts these tokens, and the paired CosyVoice2 codec decoder converts them back to waveform.
Hidden-State-Synchronized Audio Post-Decoder
The audio post-decoder is a causal transformer executed in the same streaming update loop as the frozen text path. Both branches are indexed by the same update step $s$. At update $s$, the frozen backbone state $h^{\mathrm{mid}}_s$ fans out to two concurrent branches:
- The text head predicts one text token $y^\tau_s$.
- The audio post-decoder predicts an audio block $\mathbf{y}^a_s = (y^a_{s,1}, \ldots, y^a_{s,B_s})$, where $B_s = 1$ without MTP and $B_s \leq k$ with MTP horizon $k$.
The audio branch is causal: it conditions on the current hidden state and the history committed before $s$, not on the text token being predicted in the same update. The post-decoder models:
$$P(\mathbf{y}^a_s \mid \mathbf{y}^a_{The conditioning state combines three signals via a mixed conditioning vector. Let $e^\tau_{s-1}$ be the embedding of the previously committed text token, and let
$$r^a_{s-1} = \frac{1}{B_{s-1}} \sum_{j=1}^{B_{s-1}} e^a_{s-1,j}$$be the mean embedding of the codec tokens committed by the previous audio update (beginning-of-sequence embeddings are used for $s=1$). The mixed conditioning vector is then:
$$h^{\mathrm{mix}}_s = w_h \, h^{\mathrm{mid}}_s + w_\tau \, e^\tau_{s-1} + w_a \, r^a_{s-1}$$with $w_h = w_\tau = w_a = 1.0$ in all experiments (selected as the best simple fixed-weight setting in held-out ablations). The three terms respectively provide semantic context from the backbone's internal state, lexical anchoring from the previous text token, and local acoustic continuity from recent audio history. Text and audio are generated by parallel branches with separate caches; hidden states serve as the synchronization interface.
Training follows the same causal graph with teacher forcing: the frozen backbone is evaluated on the reference text prefix and the audio post-decoder receives reference histories up to $s-1$; only the current audio block contributes to the speech-generation loss. At inference, generated text tokens and audio blocks replace the reference histories. In both cases, the audio branch never conditions on future text or on the text token being generated concurrently at update $s$.
Multi-Token Prediction as an Audio Efficiency Adapter
Codec tokens are generated at 25 Hz, so autoregressive audio decoding can dominate response latency. After the single-token audio branch has learned stable hidden-state alignment, PRIME-Speech attaches MTP heads to the audio post-decoder. In the timestamp formulation, MTP sets the audio block size to $k$. At update $s$, one post-decoder state predicts $k$ future codec-token distributions:
$$p_{s,i} = P(y^a_{s,i} \mid \mathbf{y}^a_{where positions beyond the utterance boundary are masked. At inference, each synchronized audio update commits up to $k$ codec tokens, reducing the effective codec prediction rate from 25 Hz to $25/k$ Hz.
The placement of MTP is a critical design choice. In a Thinker–Talker system, MTP applied only to the lightweight talker compresses the talker-side loop but leaves the thinker's text generation path unchanged — the total sequential cost still includes $U$ thinker updates for $U$ text tokens plus $\lceil T_a / k \rceil$ talker updates. In PRIME-Speech, each update already represents one shared-backbone synchronization step followed by the audio post-decoder, and MTP compresses that synchronized loop itself — so every update commits $k$ codec frames directly. This means MTP reduces the effective $1{:}k$ audio-per-text schedule at the synchronized decoding clock rather than compressing only a detached talker stage. The frozen reasoning path is never touched.
Training-Time Multi-Turn Packing and Cache Reset
Realistic multi-turn S2S data is costly to collect, and training with long cross-turn audio histories is unstable. PRIME-Speech does not require additional multi-turn S2S supervision. Instead, it reuses the frozen S2T backbone's ability to maintain dialogue-level text context and trains the audio branch with a turn-level cache policy. During training, unrelated single-turn examples are concatenated into packed pseudo-dialogues, exposing the audio branch to long text-side context and explicit turn boundaries without treating one turn's acoustic realization as useful context for the next turn.
The policy is applied consistently in training and inference. Let $n$ index a dialogue turn (or packed segment during training) and $m \in \{\tau, a\}$ denote text or audio modality. The cache update rule is:
$$\mathbf{C}^{(n)}_m = \begin{cases} \mathbf{C}^{(This training-time packing plus turn-local audio reset allows PRIME-Speech to exploit the backbone's text-side multi-turn capability without collecting new multi-turn S2S data, while preventing stale audio states from causing repetition or drift across turns.
Model Architecture and Configuration
PRIME-Speech is built on Phi-4-MM-7B, an advanced S2T LLM pre-trained on 2M hours of speech data and 5 trillion text tokens. This large-scale pre-training provides a controlled testbed for measuring whether S2S adaptation can preserve existing speech understanding and text reasoning capabilities. The choice of a well-characterized S2T backbone (as opposed to released S2S checkpoints where architecture, data, and fine-tuning effects are already entangled) is deliberate and methodologically important.
Trainable Modules
- Audio post-decoder: approximately 2 billion parameters organized in 10 causal transformer decoder layers. Each layer follows the hidden size and attention configuration of the backbone to minimize representational mismatch between $H^{\mathrm{mid}}$ and the speech branch.
- MTP module: a multi-head MLP with hidden dimension 2048 and approximately 100M parameters, attached to the audio post-decoder.
Frozen Modules
The speech encoder, projection module, transformer backbone, text LM head, and entire text decoding path remain frozen throughout training and evaluation. As a result, any change in S2S quality is attributable to the learned speech branch, while S2T performance should remain governed by the original backbone (up to decoding stochasticity).
Training Data and Curriculum
Training Data
Since the backbone is frozen, training does not teach the model to reason from speech — it teaches the audio branch to render the responses of a strong S2T LLM faithfully and intelligibly. The task-balanced mixture covers three needs: codec-level speech realization, semantic preservation when output is not a transcript of input, and assistant-style spoken responses. After resampling, the mixture contains approximately 100k weighted hours.
| Dataset | Type | Language | Hours | Stage |
|---|---|---|---|---|
| LibriHeavy | TTS | EN | 46k | S1 |
| In-house X2EN | S2ST | EN | 10k | S1, S2 |
| CoVoST-2 X2EN | S2ST | EN | 1k | S1, S2 |
| VoiceAssistant-400K | SQA | EN | 4k | S1, S2 |
| TriviaQA | SQA | EN | 2k | S1, S2 |
LibriHeavy (46k hours of transcribed English speech) serves as the main alignment source, providing dense audio-text correspondence to stabilize the randomly initialized audio post-decoder in early training. Multilingual speech translation data (CVSS, CoVoST-2 X2EN, and approximately 10k hours of synthesized x-to-English S2S pairs following an in-house synthesis and filtering procedure) forces the audio branch to render inferred content rather than merely repeat input transcripts. VoiceAssistant-400K, TriviaQA, and Natural Questions expose the post-decoder to assistant-style outputs including short factual answers and longer explanatory responses. For all components, target-side speech is synthesized using Microsoft Azure TTS with speakers sampled from several hundred identities, and encoded by the CosyVoice2 tokenizer at 25 Hz.
Training Curriculum
Training proceeds in two stages, designed to avoid asking a randomly initialized speech branch to solve long-horizon codec prediction before it can reliably follow frozen backbone states.
Stage 1 — Audio-branch training: Trains the audio post-decoder at the native 25 Hz codec rate with standard next-token prediction. Training runs for one epoch on the full task-balanced mixture using AdamW with a learning rate of $1 \times 10^{-4}$ and linear decay. LibriHeavy provides acoustic-token modeling stability; translation, spoken QA, and assistant-style examples teach the post-decoder to render content inferred by the frozen backbone rather than only verbatim transcripts.
Stage 2 — MTP training: Enables multi-token codec prediction and continues training for 20k steps with the same learning-rate schedule. Pure alignment examples are down-weighted while translation and spoken QA examples remain active. At this point the post-decoder has already learned stable hidden-state conditioning, so MTP is trained as an efficiency component that compresses codec-token prediction without modifying the frozen reasoning path.
Experiments
Baselines and Evaluation Suite
PRIME-Speech is compared against a comprehensive set of baselines: GLM-4-Voice, Kimi-Audio, Step-Audio-2-Mini, VocalNet, Qwen2.5-Omni, Qwen3-Omni, GPT-4o, and the frozen Backbone-LLM (Phi-4-MM-7B, reported only in S2T mode since it has no speech-generation branch). Public systems are evaluated with their recommended inference settings.
The evaluation suite covers:
- Speech-to-Speech Translation (S2ST): FLEURS and CoVoST-2 X2EN, using BLEU/ASR-BLEU metrics.
- Spoken QA (Speech Conversation): UltraEval-Audio (LLaMA-QA, TriviaQA, WebQ subtasks), plus WER of the S2S output against the text response.
- Multi-turn Dialogue: An in-house set of 28 human-validated question-answer conversations with 179 turns, designed to stress anaphora, ellipsis, and references to previous answers.
- Speech Understanding: VocalBench (Knowledge, Reasoning, Creativity, Fluency via UTMOS, Single-turn, Overall) and BigBench-Audio.
S2T and S2S scores are always reported separately. In S2S mode, generated waveforms are transcribed by Whisper Large-V3, and task metrics are computed on ASR transcripts. WER against the corresponding text response measures rendering consistency and intelligibility. Unless otherwise stated, PRIME-Speech uses MTP horizon $k = 4$.
Main Results
The results address two simultaneous requirements: preservation (adding speech output should not damage the S2T backbone) and realization (generated speech should preserve task correctness and remain easy to transcribe).
| Model | S2ST | UltraEval-Audio | Multi-turn | BigBench-Audio | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| FLEURS | CoVoST | LLaMA-QA | TriviaQA | WebQ | WER↓ | S2T | S2S | WER↓ | S2T | S2S | |
| Qwen3-Omni-30B | 33.25 / 32.72 | 41.25 / 37.62 | 83.00 / 71.33 | 61.43 / 57.52 | 55.95 / 52.51 | 14.92 | 79.89 | 70.39 | 11.28 | 83.7 | 72.0 |
| GPT-4o | 33.86 / — | 37.09 / — | 83.00 / — | 76.07 / — | 50.98 / — | — | — | — | — | 70.2 | 67.2 |
| GLM-4-Voice-9B | — | — | 64.70 / 50.70 | 39.10 / 26.50 | 32.20 / 15.90 | — | 74.86 | 70.95 | 7.83 | 44.8 | 42.7 |
| Kimi-Audio-7B | 7.68 / — | 7.40 / — | 76.67 / 62.33 | 46.78 / 37.99 | 41.98 / 35.37 | 14.85 | 73.18 | 65.36 | 10.9 | 59.4 | 51.0 |
| Step-Audio-2-Mini-7B | 29.03 / 24.85 | 33.25 / 27.21 | 61.00 / 60.33 | 33.40 / 32.23 | 33.02 / 31.69 | 8.56 | 70.39 | 69.27 | 6.15 | 50.9 | 47.5 |
| VocalNet-8B | — | — | 76.33 / 69.00 | 44.63 / 38.38 | 44.05 / 39.27 | 7.68 | 74.86 | 68.72 | 8.52 | 45.9 | 44.9 |
| Qwen2.5-Omni-7B | 34.59 / 5.94 | 39.72 / 10.52 | 76.33 / 71.00 | 47.66 / 45.60 | 42.18 / 39.42 | 21.5 | 69.83 | 67.04 | 4.23 | 54.2 | 53.6 |
| Backbone-LLM-7B | 31.41 / — | 40.65 / — | 78.67 / — | 47.07 / — | 42.18 / — | — | 79.33 | — | — | 66.5 | — |
| PRIME-Speech-9B | 31.40 / 33.24 | 41.29 / 40.98 | 79.00 / 74.42 | 46.98 / 44.54 | 42.04 / 40.18 | 3.33 | 80.45 | 79.33 | 3.34 | 66.2 | 63.4 |
Preserving the reasoning path: The controlled comparison with Backbone-LLM is the most direct evidence for preservation. Across translation, spoken QA, and BigBench-Audio, PRIME-Speech's S2T scores are nearly unchanged from the frozen backbone. This is expected but important: the speech-generation branch does not rewrite the text pathway, so S2S gains are not obtained by trading away the backbone's original speech understanding or reasoning capability.
From text correctness to spoken correctness: On translation and spoken QA, PRIME-Speech converts strong text responses into speech with small degradation and consistently low WER. Notably, on FLEURS translation, PRIME-Speech achieves S2S BLEU of 33.24 compared to its S2T BLEU of 31.40 — the S2S score actually exceeds the S2T score, indicating effective audio branch learning. On CoVoST-2, S2T/S2S scores are 41.29/40.98 — a near-perfect match. In multi-turn dialogue, PRIME-Speech achieves S2T 80.45, S2S 79.33, and WER 3.34% — by far the smallest S2T–S2S gap and lowest WER among all systems evaluated. On BigBench-Audio, PRIME-Speech achieves S2T 66.2 and S2S 63.4, closely tracking the frozen backbone's 66.5.
Speech naturalness (VocalBench): PRIME-Speech achieves the highest VocalBench Overall score (78.76) among all 7B-class systems, along with competitive scores on Knowledge (68.9), Reasoning (4.23), and the highest Single-turn score (4.29).
Ablation Studies
Frozen Backbone vs. LoRA Adaptation
The ablation compares PRIME-Speech against two variants that update the backbone via LoRA: LoRA + ESI and LoRA + Post LM. These variants use the same data but adapt the backbone weights.
| Variant | Frame Rate | FLEURS S2T/S2S | WER↓ | LLaMA-QA S2T/S2S | BigBench S2T | BigBench S2S |
|---|---|---|---|---|---|---|
| LoRA + ESI | 37.5 Hz | 29.37 / 31.11 | 2.07 | 70.00 / 66.33 | 53.75 | 53.25 |
| LoRA + Post LM | 25 Hz | 29.59 / 30.96 | 2.62 | 70.33 / 67.00 | 52.96 | 52.36 |
| PRIME-Speech S1 | 25 Hz | 31.39 / 33.57 | 1.51 | 79.00 / 73.33 | 66.30 | 59.10 |
| + MTP=1 | 25 Hz | 31.39 / 33.58 | 1.45 | 79.00 / 72.33 | 66.30 | 63.86 |
| + MTP=2 | 12.5 Hz | 31.39 / 33.56 | 1.52 | 79.00 / 74.67 | 66.40 | 64.16 |
| + MTP=4 | 6.25 Hz | 31.40 / 33.24 | 2.19 | 79.00 / 74.42 | 66.20 | 63.38 |
LoRA variants remain viable on in-domain S2S metrics but show substantially lower BigBench-Audio S2T scores (~53 vs. ~66 for PRIME-Speech). This demonstrates the cost of adapting the reasoning model and supports PRIME-Speech's decomposition: reasoning stays in the frozen backbone, while speech realization is learned in the post-decoder. PRIME-Speech's S1 model already substantially outperforms both LoRA variants in reasoning capability preservation (BigBench-Audio S2T: 66.30 vs. 53.75/52.96).
Effect of MTP Horizon
MTP is interpreted as an efficiency adapter, not as the source of semantic alignment. The stage-1 model already shows strong performance. Moving from $k=1$ to $k=2$ reduces the effective codec rate from 25 Hz to 12.5 Hz while actually improving multi-turn WER (from 3.61% to 2.07%) and BigBench-Audio S2S (from 63.86 to 64.16). Moving to $k=4$ further reduces the rate to 6.25 Hz with broadly stable S2S quality across all benchmarks. This confirms that MTP serves its intended role: compressing the codec-generation loop without changing the frozen reasoning path or causing significant task-quality loss.
Multi-Turn Audio Cache Reset
| Cache Policy | Turn 1 Acc/WER | Turn 2 Acc/WER | Turn 3 Acc/WER | Turn 4 Acc/WER | Turn ≥5 Acc/WER |
|---|---|---|---|---|---|
| Text accum. + audio reset | 92.86 / 2.44% | 82.14 / 1.94% | 71.43 / 1.97% | 85.71 / 3.65% | 73.13 / 1.48% |
| w/o audio reset | 92.86 / 2.77% | 78.57 / 5.62% | 39.29 / 65.57% | 10.71 / 129.63% | 0.00 / 143.27% |
The audio cache ablation provides a stark causal diagnostic. With text accumulation and turn-local audio reset, WER stays below 4% in every turn-position bucket and S2S accuracy remains stable as dialogue length grows. Without audio reset, the first two turns appear acceptable, but from turn 3 onward WER jumps sharply — exceeding 100% WER and reaching 0% accuracy at 5+ turns. Since the text cache is accumulated in both rows, the degradation is isolated to reused audio-side state rather than lost dialogue memory. This strongly validates the hybrid cache policy: audio KV and audio positions must reset at turn boundaries, while text context should be preserved.
Efficiency Analysis
| System | Frame Rate (Hz) | TTFT↓ (ms) | TTFA↓ (s) | Throughput↑ (tok/s) | RTF↓ |
|---|---|---|---|---|---|
| Qwen2.5-Omni-7B | 50.0 | 58 | 1.01 | 45.75 | 1.093 |
| VocalNet-8B (k=1) | 12.5 | 38 | 0.51 | 216.89 | 0.250 |
| VocalNet-8B (k=3) | 6.25 | 38 | 0.40 | 220.16 | 0.243 |
| VocalNet-8B (k=5) | 4.17 | 38 | 0.40 | 225.93 | 0.243 |
| PRIME-Speech (k=1) | 25.0 | 61 | 1.07 | 30.62 | 1.088 |
| PRIME-Speech (k=2) | 12.5 | 60 | 0.63 | 62.17 | 0.548 |
| PRIME-Speech (k=4) | 6.25 | 58 | 0.39 | 123.76 | 0.296 |
All experiments are conducted on a single NVIDIA H100 GPU. VocalNet's shadow and dense talker architecture gives high codec-generation throughput and low RTF, representing an efficiency-oriented point on the S2S design trade-off — but this comes with a larger S2T–S2S modality gap. PRIME-Speech operates in a different regime: a 2B post-decoder synchronized with hidden states from a 7B frozen reasoning backbone costs more computation than a shallow talker at $k=1$, but preserves the S2T pathway and yields stronger text–speech consistency.
Within the frozen-backbone regime, MTP provides a controllable efficiency knob. Increasing $k$ from 1 to 4: reduces the effective codec rate from 25 Hz to 6.25 Hz; reduces Time-to-First-Audio (TTFA) from 1.07 s to 0.39 s; improves throughput from 30.62 to 123.76 tok/s; and reduces Real-Time Factor (RTF) from 1.088 to 0.296 — while ablation results show broadly stable S2S task quality. At $k=4$, PRIME-Speech achieves faster first-audio response than Qwen2.5-Omni-7B (0.39 s vs. 1.01 s) and competitive RTF with VocalNet-class systems, while maintaining substantially better S2T–S2S alignment.
Key Contributions and Novelty
- Frozen-backbone conversion framework: PRIME-Speech formulates S2S adaptation as a frozen-backbone conversion problem, keeping the complete S2T pathway unchanged and attributing all S2S learning to the added audio branch. This is validated by the near-identity between PRIME-Speech and Backbone-LLM S2T scores across all benchmarks.
- Hidden-state synchronization: The audio post-decoder is conditioned on intermediate backbone states (at approximately two-thirds depth, selected via CKA analysis) as they are produced, rather than waiting for completed text responses or fixed text chunks. This creates timestamp-level synchronization — not chunk-level — enabling speech generation concurrent with text generation without force alignment.
- Mixed conditioning: The post-decoder combines backbone hidden states, text-token embeddings, and a summarized audio history embedding into a unified mixed conditioning vector, providing semantic context, lexical anchoring, and acoustic continuity simultaneously.
- Turn-level audio cache reset with text accumulation: A simple but critical multi-turn cache policy is identified: text KV cache accumulates across turns (preserving dialogue semantics), while audio KV cache and positional indices reset at each assistant turn (preventing acoustic drift). Training-time packing of single-turn examples into pseudo-dialogues teaches this behavior without requiring multi-turn S2S data collection.
- Audio-side MTP as a latency knob: MTP is applied to the synchronized audio branch — not a lightweight detached talker — so it compresses the actual bottleneck in the frozen-backbone S2S loop. At $k=4$, TTFA drops from 1.07 s to 0.39 s and RTF drops from 1.088 to 0.296 with minimal task quality degradation.
Limitations and Scope
The paper explicitly acknowledges several scope boundaries:
- Expressive prosody and speaker consistency: Target-side speech is synthesized from text using Microsoft Azure TTS with sampled speaker identities. Accordingly, the paper does not claim improvements in expressive prosody or speaker consistency, and evaluation focuses on transcript-level task correctness, S2T–S2S semantic gap, WER, and decoding efficiency (with UTMOS-based fluency from VocalBench as a naturalness indicator).
- Not universally best: PRIME-Speech is not the best system on every individual metric, especially compared with larger proprietary systems like Qwen3-Omni-30B or GPT-4o. The claim is a balanced profile across translation, spoken QA, multi-turn stability, and reasoning preservation — rather than task-specific supremacy.
- Single-codebook codec: The CosyVoice2 tokenizer operates as a single semantic codec stream at 25 Hz. The system does not address multi-stream or residual-quantizer codec architectures.
- Data synthesis for target speech: All target-side speech is synthesized, which provides clean supervision but may not capture the full distribution of natural spoken responses.
- Generative AI use disclosure: The authors note that generative AI tools were used only for language polishing (grammar, spelling, LaTeX formatting); all technical content, experiments, and conclusions were produced and verified by the authors.
Conclusion
PRIME-Speech presents a principled framework for adding speech-to-speech capability to a strong S2T LLM without altering its reasoning backbone. The central design — freezing the complete backbone and synchronizing a trainable audio post-decoder to intermediate backbone states — enables concurrent text and audio generation from the same evolving semantic trajectory, avoiding both the catastrophic-forgetting risk of backbone fine-tuning and the serial bottleneck of downstream TTS systems.
The empirical results demonstrate that across speech translation, spoken QA, multi-turn dialogue, and speech understanding, PRIME-Speech preserves the S2T behavior of the frozen Phi-4-MM-7B backbone while producing accurate spoken responses with consistently low rendering WER. The turn-level audio cache reset is shown to be critical for stable multi-turn operation — without it, WER exceeds 100% and accuracy collapses to zero by the fifth turn. Multi-token prediction provides a controllable efficiency adapter, reducing TTFA from 1.07 s to 0.39 s and RTF from 1.088 to 0.296 at horizon $k=4$ with broadly stable task quality, enabling real-time factor well below 0.3 while maintaining the full frozen-backbone S2T pathway.