JSTIP

Speech-text joint training (the naive baseline) adds text-only data by passing zero speech inputs through the encoder for text-only batches. While this allows text sequences to update LLM parameters and improve text-side metrics, the authors find it fails to close the modality gap: text-side language modeling improvements do not propagate to speech-conditioned entity recognition in practice.

JSTIP: Joint Speech–Text Interleaved Pre-Training

JSTIP departs from dataset-level scheduling. Instead, it constructs interleaved sequences inside each aligned speech-text pair. Given a speech-text pair $(A, T)$, word-level or segment-level alignment yields a sequence of segment pairs:

where each $A_i$ is the acoustic span aligned to text segment $T_i$. From this aligned pair, two complementary interleaved sequences are constructed by alternating speech and text segments:

• Speech-first variant: $(A_1, T_2, A_3, T_4, \dots, T_n)$
• Text-first variant: $(T_1, A_2, T_3, A_4, \dots, T_n)$

In both cases, the final segment is always a text segment, and all text tokens in the interleaved sequence serve as training targets. The model is thus trained to predict text conditioned on distributions of the form:

This design forces the decoder to predict text tokens after both speech and text contexts within the same sequence, preserving textual modeling behavior while simultaneously learning speech conditioning. Deterministic alternation is used in all experiments; adaptive or word-type-based selection is left for future work.

Loss Masking

Cross-entropy loss is applied only to text tokens regardless of training format. In ASR-only training, the loss covers transcript tokens. In text-only training, it covers text tokens. In interleaved training, it covers all text tokens in the interleaved sequence. Speech positions are always masked out. Special task/modality indicator tokens are excluded from the loss. When multiple examples are packed into the 8k context window, the implementation uses packed SFT with cumulative sequence lengths (cu_seqlens) in FlashAttention to prevent cross-example attention leakage.

Word-Level Interleaving

At word-level granularity, each $T_i$ is a single word (or token span) and $A_i$ is its aligned acoustic region. The sequence alternates at maximum frequency, and special tokens between speech and text segments are skipped to enforce tight cross-modal coupling at the finest available scale.

A key engineering challenge arises because continuous speech representations are used (not discrete tokens): processing many short acoustic spans independently creates large zero-padding overhead in the speech encoder, threatening GPU memory during large-batch training. The paper addresses this with a concatenate-then-reinsert strategy: all speech segments are concatenated into a single longer sequence at the encoder input level, and the resulting representations are placed back into their correct interleaved positions after the adapter forward pass. This makes word-level interleaving scalable for continuous speech inputs.

Segment-Level Interleaving

At segment granularity, consecutive words are merged into phrase- or sentence-level segments. The interleaved sequence takes the form:

where $\langle s \rangle$ and $\langle /s \rangle$ are start/end tokens and $\langle N \rangle$ is a modality boundary token. Segment boundaries can be defined by acoustic silence alone, or by both acoustic silence and punctuation signals.

Mixed Interleaving (JSTIP Full)

The best-performing configuration mixes word-level and segment-level interleaved sequences. Word-level interleaving contributes fine-grained cross-modal coupling and ASR-specific entity benefits, while segment-level interleaving provides longer-range semantic context and stronger modality-gap reduction. The combined strategy outperforms either alone.

Architecture and Training Setup

The model follows a decoder-only speech-LLM architecture:

• Speech encoder: 400M-parameter Conformer with temporal downsampling factor of 8. Input is 80-dimensional log Mel filterbanks extracted with 10 ms frame shift, yielding an effective 80 ms per speech token at the decoder.
• Decoder: An internal 7B-parameter LLM pretrained on 5 trillion text tokens.
• Adapter: A lightweight ~20M-parameter projection layer mapping encoder outputs into the LLM latent space.

Training is conducted in two stages:

1. Stage 1 (adapter warm-up): Only adapter parameters are updated using ASR data to stabilize cross-modal alignment. Learning rate: $1 \times 10^{-4}$ with linear decay (AdamW), covering 10% of total ASR tokens.
2. Stage 2 (full joint training): All parameters — encoder, adapter, and LLM — are jointly optimized. Learning rate: $4 \times 10^{-5}$ with linear decay (AdamW). ASR, interleaved, and text-only sequences are packed into 8k context length with packed SFT.

Training Data

ASR Data

38,000 hours of anonymized English in-house ASR data, corresponding to approximately 2.3B training tokens at a 12.5 Hz token rate. Evaluation speakers are disjoint from training speakers.

Interleaved Data

Word-level alignments are obtained on the 38k-hour ASR data using an HMM-based hybrid ASR system. Utterances with failed alignments are removed; no confidence-based filtering is applied. Two interleaving granularities are derived: word-level (each word as a unit) and segment-level (consecutive words merged by silence or silence+punctuation). 2.3B interleaved tokens are used per run to balance with the ASR data. The paper notes that segment-level interleaving is expected to be less sensitive to local boundary errors than word-level interleaving, and that robustness to CTC or timestamp-based forced alignment is left as future work.

Text Data

To preserve textual modeling capacity, the following text-only corpora are included:

• 2.3B text tokens from PubMed abstracts (medical domain).
• 0.1B text tokens from TTS-pair transcriptions.

All text data is cleaned by simple rule-based filtering and formatted for LLM pretraining (truncated at 8k context, EOS appended).

Synthetic Domain Data (TTS-Pairs)

For domain adaptation experiments, 9,000 hours of synthetic medical-domain ASR data (TTS-pairs) are created. Transcriptions are generated by prompting GPT for entity-rich utterances across medical topics; audio is synthesized using an in-house TTS model. Two uses of this data are compared: using full synthetic speech-text pairs vs. using only the transcriptions (TTS-transcription), which is cheaper to obtain.

Evaluation Protocol

General ASR (TER)

Two held-out in-house test sets are used: conversation (spontaneous short-form speech) and dictation (longer-form dictated speech with rich punctuation). Performance is measured by Token Error Rate (TER), which counts all tokens including capitalization and punctuation.

Entity Error Rate (EER)

Nine held-out in-house domain test sets are used: eight medical domain sets and one banking set. References and entity spans are human-annotated. EER is defined as $1 - \text{recall}$ over annotated entity instances, computed before text normalization (so capitalization, punctuation, and exact spelling matter; spelling variants are not accepted). EER is the primary metric because LLM knowledge transfer is concentrated on entities, while aggregate TER is dominated by common non-entity tokens.

• Medical aggregate: 261 utterances, 49.281 hours.
• Banking set: 36 utterances, 5.567 hours.

Modality Gap: Speech-MMLU

A speech version of MMLU is constructed and evaluated in 5-shot setting. Performance is measured under text-to-text (T2T) and speech-to-text (S2T) conditions. Answers are predicted via next-token selection over options {A, B, C, D} (random baseline: 25%). The gap $\Delta = \text{S2T} - \text{T2T}$ quantifies how much textual reasoning ability is lost after speech adaptation.

Zero-shot Speech Question Answering (SQA)

Three datasets from UltraEval-Audio are used: LLaMA-QA, TriviaQA, and WebQA. Average accuracy is reported. SQA requires open-ended answer generation and serves as a diagnostic for whether text-side generative behavior is preserved under speech input. The paper interprets it as supporting evidence for knowledge preservation rather than a standalone explanation for entity gains.

Main Results

Table 1 (Overall Results) shows the full set of results across configurations. Below is a summary of the key findings:

Key Observations from Main Results

Interleaving alone (no domain text): Adding interleaved sequences to ASR-only training reduces Medical-AVG EER from 7.97% to 7.32% and dramatically recovers zero-shot SQA capability: SQA-S2T jumps from 0.05% to 41.92%. This reveals that ASR-only training essentially destroys the LLM's generative prior for speech-conditioned generation, and interleaving restores it.

Naive joint training vs. interleaving: Adding PubMed text without interleaving (ASR+PubMed) substantially improves MMLU-T2T (43.01% → 64.1%) because text batches update LLM parameters, but MMLU-S2T improves much less (35.68% → 43.77%), and EER only drops modestly (7.97% → 7.49%). This exposes the modality gap: text-side improvements do not transfer to speech-conditioned inference without interleaving. With interleaving, MMLU-S2T reaches 58.98%, EER drops to 6.87%, and SQA-S2T reaches 41.03%.

Transcription vs. synthetic speech pairs for domain adaptation: Without interleaving, adding TTS-transcription yields negligible EER improvement (7.97% → 7.85%). With interleaving, TTS-transcription alone achieves Medical-AVG EER of 6.81%, matching the result from full synthetic TTS-pairs (6.86% without interleaving, 6.72% with interleaving on pairs). This means domain transcription text alone, when combined with interleaving, is as effective as expensive synthetic speech-text pairs for entity adaptation — a practically important finding for domain adaptation without TTS infrastructure.

JSTIP-Best-EER: Combining all resources (38k hours ASR + PubMed text + TTS-transcription + TTS-interleaved sequences + ASR-interleaved sequences) achieves the lowest Medical-AVG EER of 6.60%, representing a 17.2% relative improvement over the ASR-only baseline (7.97%).

Comparison with Open-Source Systems

The paper emphasizes that this comparison is an external reference rather than a fully controlled ranking, as the open-source models differ in architecture, training data volume (potentially millions of hours), model size, prompting, and decoding settings. Despite being trained on only 38k hours of ASR data, JSTIP-Best-EER outperforms Whisper-Large-V3, Qwen3-ASR-1.7B, Voxtral-Mini-3B, Gemma-3n-E4B, and Qwen2.5-Omni-7B on Medical-AVG EER. The benefit on medical domains is attributed to the LLM pretraining prior and medical domain text (PubMed + TTS-transcription). The model remains weaker on banking, where no comparable domain text is included in training. JSTIP is outperformed on medical domains by Qwen3-Omni-30BA3B and Voxtral-Small-24B, which the paper attributes to their larger LLM backbones.

Ablation Study: Interleaving Granularity and Segmentation Strategy

Key Findings from Ablation

Word-IL alone: Improves entity recognition (7.97% → 7.64%) but shows limited modality-gap reduction. The $\Delta$ metric moves only from −7.33 to −6.89. Word-level coupling helps ASR entity performance but is insufficient on its own for preserving broad text-side reasoning.

Segment-IL alone: Substantially reduces the modality gap (silence: $\Delta = -4.45$; silence+punctuation: $\Delta = -0.77$), significantly better than Word-IL. However, Segment-IL achieves similar or slightly worse EER compared to Word-IL, suggesting the two approaches are complementary.

Punctuation-based segmentation: Using both silence and punctuation for segment boundaries achieves near-zero modality gap ($\Delta = -0.77$) compared to silence-only ($\Delta = -4.45$). The reason is that acoustic silence alone can be sparse, producing very long segments; punctuation creates more balanced segment-length distributions, making the supervision signal more even.

Mixed-IL (Word + Segment): The combination with silence+punctuation segmentation achieves the best EER (7.32%) and a positive $\Delta$ (+0.61%), meaning MMLU-S2T exceeds MMLU-T2T — an indicator that speech conditioning and text-side reasoning are well-aligned. This demonstrates that word-level interleaving provides ASR-specific benefits beyond what segment-level interleaving alone (as used in prior systems such as Voxtral and Xie et al.) can offer.

Transcription vs. speech pairs under interleaving: Without interleaving, TTS-transcription gives EER of 7.85% while TTS-pairs give 6.86% — a large gap. With mixed interleaving, TTS-transcription reaches 6.81%, matching TTS-pairs performance. This confirms that interleaving is the mechanism that enables domain transcription text to transfer into speech-conditioned entity recognition.

Analysis: Why Does Interleaving Work?

The paper provides a coherent mechanistic interpretation. Under ASR-only training, the decoder specializes in speech-conditioned prediction and the text-side generative prior degrades rapidly: MMLU-S2T is 35.68% despite the 7B LLM backbone being pretrained on 5 trillion text tokens, and zero-shot SQA-S2T essentially collapses to 0.05%.

Naive joint training with separate text batches improves T2T metrics (MMLU-T2T: 43% → 64%) but fails to bridge the modality gap, as evidenced by persistently low MMLU-S2T (43.77%) and SQA-S2T (10.1%). Text-only batches update the LLM in a context-independent manner; the decoder never learns to predict text given mixed speech-and-text context.

JSTIP solves this by making text prediction occur after both speech and text contexts within the same sequence. The decoder therefore cannot ignore the textual prior when speech is present, because the training objective explicitly trains text prediction after speech spans. The near-zero or positive $\Delta$ values under Mixed-IL confirm that the modality gap is substantially closed, and the zero-shot SQA recovery (0.05% → ~42%) strongly supports knowledge preservation. The paper argues that this preserved generative prior is the likely mechanism by which domain entity recognition improves: the model can leverage the broader world knowledge and linguistic prior from LLM pretraining rather than relying solely on observed speech-text co-occurrences in the ASR training corpus.

Relation to Prior Work

The paper situates JSTIP relative to three bodies of work:

• Speech-LLM integration systems (Qwen-Audio, Phi-4-Multimodal, Moshi, Step-Audio, Kimi-Audio, Voxtral): These systems commonly report interleaved or mixed training as useful for general multimodal ability, but precise construction details and ASR-specific analysis are usually not disclosed. JSTIP isolates interleaving as an ASR-oriented mechanism.
• Spirit-LM (Nguyen et al.): Interleaves spoken and written language using discrete speech tokens in a shared decoder formulation. JSTIP differs by operating with continuous speech representations, targeting a decoder-only ASR architecture, and specifically analyzing the effect on entity recognition and the modality gap.
• Pre-LLM joint speech-text training (JOIST, MAESTRO, SpeechLM, SpeechT5, SLAM, etc.): These methods inject text-only data into AED or transducer-based ASR to exploit linguistic regularities. They required alignment heuristics (duration-based upsampling, repetition) to handle the text–speech input mismatch. JSTIP revisits text injection under the decoder-only LLM paradigm, with the specific goal of preserving the pretrained LLM prior rather than merely strengthening a learned language model decoder.

Contributions and Novelty

• Diagnosis: Identifies over-specialization to speech-conditioned decoding as the root cause of poor LLM prior transfer in the large supervised-data regime, even when text-only data is naively mixed.
• JSTIP method: An ASR-oriented interleaved pretraining strategy that constructs word-level and segment-level interleaved sequences within aligned speech-text pairs, ensuring text prediction occurs after both speech and text contexts.
• Scalable word-level interleaving: A concatenate-then-reinsert implementation that makes fine-grained word-level interleaving practical for continuous speech representations at training scale.
• Domain adaptation insight: Under JSTIP, domain transcription text alone achieves entity performance comparable to synthetic TTS speech-text pairs, enabling cost-efficient domain adaptation without requiring TTS synthesis for every text example.
• Empirical evidence: Consistent entity recognition improvement (up to 17.2% relative), near-zero modality gap, and recovery of zero-shot SQA capability — all without introducing task-specific supervision beyond ASR.

Limitations and Future Work

The paper explicitly acknowledges several limitations:

• The primary training data, domain evaluation sets, and backbone LLM are internal/proprietary. Results should be interpreted as controlled improvements within this system rather than a universal ranking over Speech-LLMs.
• Only HMM-based hybrid ASR alignments are used. Robustness to CTC alignments or timestamp-based forced alignment tools is identified as future work.
• JSTIP uses deterministic alternation for modality selection. Adaptive, random, or word-type-based modality selection strategies are left for future work.
• The analysis focuses on English medical and banking ASR. Generalization to other languages and domains remains to be demonstrated.
• The banking domain does not benefit from domain-specific text in the current setup, and the model lags behind larger open-source models (Qwen3-Omni-30BA3B, Voxtral-Small-24B) due to backbone LLM size differences.
• SQA accuracy can be affected by prompting, decoding, and answer normalization, and is interpreted as supporting evidence rather than a fully controlled diagnostic.