VIB-AVSR

Information Bottleneck Principle

The Information Bottleneck (IB) principle (Tishby et al.) provides a framework for learning representations that retain task-relevant information while discarding redundant variation. For an input $X$, a learned representation $Z$ parameterized by $\phi$, and a target $Y$, the IB objective is:

where $I(\cdot;\cdot)$ denotes mutual information and $\beta \geq 0$ controls the compression–prediction trade-off. Maximizing $I(Z; Y)$ preserves information predictive of the output, while minimizing $I(Z; X)$ discards redundancy — such as acoustic noise. In the AVSR setting, noise corrupts the audio input $X^a$, causing audio hidden states $\mathbf{H}_a$ to encode noise-specific features. The authors apply the IB objective selectively to $\mathbf{H}_a$, leaving video tokens $\mathbf{H}_v$ and text tokens $\mathbf{H}_t$ unaffected, since they are not directly corrupted by acoustic noise.

Variational Formulation

Direct optimization of the IB objective is intractable due to difficulty estimating mutual information in high-dimensional spaces. Following the variational formulation, the authors derive a tractable lower bound tailored to audio hidden states. The audio encoder and projector produce initial audio representations $\mathbf{H}^0_a$, which are passed into the LLM. Let $\mathbf{H}^l_a \in \mathbb{R}^{T \times d}$ denote the audio hidden states at the output of LLM layer $l$, where $T$ is the number of audio tokens and $d$ is the hidden dimension. A VIB module $g_\phi^l$ takes $\mathbf{H}^l_a$ as input and produces a compressed representation $Z^l_a$.

For the compression term $I(Z^l_a; \mathbf{H}^l_a)$, a factorizable variational prior $r(Z^l_a)$ is introduced and the non-negativity of KL divergence is applied:

For the predictive term $I(Z^l_a; Y)$, the true posterior $p(Y \mid Z^l_a)$ is approximated by a variational approximation $q_\phi(Y \mid Z^l_a, X^v, X^t)$, parameterized by the LLM decoder. Omitting the constant $H(Y)$ with respect to model parameters:

Combining both bounds gives the variational IB objective at layer $l$:

The first term is the autoregressive transcription likelihood and the second term regularizes the audio hidden states by penalizing deviation from the prior. $\beta$ controls the compression–prediction trade-off.

Practical Implementation

Using a Monte Carlo approximation and converting to a minimization objective over a mini-batch of $N$ samples:

Both the posterior and prior are modeled as diagonal Gaussians.

Prior distribution. The prior is a learnable diagonal Gaussian:

where $\mu^l_r \in \mathbb{R}^d$ and $\sigma^l_r \in \mathbb{R}^d_+$ are per-layer parameters shared across all samples, admitting a closed-form KL divergence.

Posterior distribution. The posterior is parameterized by a position-wise two-layer MLP $g_\phi: \mathbb{R}^d \rightarrow \mathbb{R}^{2d}$, applied independently to each audio token embedding — analogous to the feed-forward sublayer of a Transformer. The posterior is:

where $[\mu^l, (\sigma^l)^2] = g_\phi(\mathbf{H}^l_a)$, with mean and variance partitioned along the output dimension. The MLP is applied exclusively to audio token positions, leaving visual and text representations unchanged. Samples are drawn via the reparameterization trick:

At inference, only $\mu^l$ is used (no sampling). To balance compression with the retention of speech-discriminative content, the bottleneck output is interpolated with the pre-bottleneck representation:

where $\alpha = 0.5$ is fixed for all main experiments. The interpolated representation $\hat{Z}^l_a$ replaces $\mathbf{H}^l_a$ and is propagated to all subsequent LLM layers. In practice, VIB modules are inserted after multiple LLM layers, each with its own independent prior.

Experimental Setup

Dataset

All training and evaluation is conducted on the LRS2 dataset, consisting of BBC program clips with transcribed English speech.

Model Architecture

Following Llama-AVSR, the system uses:

• Audio encoder: Whisper-medium (frozen throughout training)
• Video encoder: AV-HuBERT (fine-tuned with LoRA, rank 16)
• LLM backbone: Llama-3.2-1B (fine-tuned with LoRA, rank 64)
• Projectors: Modality-specific linear projectors mapping encoder features to the LLM embedding space
• Token compression rate: 3× for both audio and video tokens

VIB modules are inserted after selected LLM layers and trained jointly with the LoRA modules. Each VIB module $g_\phi^l$ is a two-layer MLP: a linear layer followed by GeLU activation and a second linear layer with $2H$ output neurons, from which mean and log-variance of the variational posterior are obtained. The same model configuration is used for both Llama-AVSR and VIB-AVSR for fair comparison.

Training Paradigms

Two training paradigms are evaluated:

• Clean Paradigm: No noise is added to audio during training. This tests whether the VIB regularization generalizes to noisy test conditions even without noise augmentation.
• Noisy Paradigm: A random noise type is added to training audio with SNR level sampled uniformly at random per instance. This evaluates whether VIB further improves robustness on top of noise augmentation.

Evaluation Protocol

Word Error Rate (WER) is reported under two noise types — babble and speech noise — sampled from the MUSAN dataset at five SNR levels: $-10$, $-5$, $-2$, $0$, $5$ dB, plus a noise-free condition ($\infty$ dB). The metric Avg ($N > S$) is introduced as the average WER over extreme noise conditions ($-10$, $-5$, $-2$ dB), where noise dominates the signal.

Main Results

The table below summarizes WER for Llama-AVSR and VIB-AVSR across both noise types and training paradigms.

Noisy Training

Under noisy training, VIB-AVSR almost always reduces WER relative to Llama-AVSR, with the largest gains at lower SNRs — particularly under babble noise. The average WER under babble noise improves from 18.85% to 17.39%, and the extreme-noise average (Avg $N > S$) improves from 26.84% to 24.59%. Under speech noise, improvements are smaller but VIB-AVSR still reduces WER across all conditions except 5 dB. Critically, VIB-AVSR also achieves lower WER under noise-free evaluation ($\infty$: 2.38% vs. 2.72%), indicating that the bottleneck acts as an effective regularizer even beyond the noisy conditions themselves.

Clean Training

When trained without any noise augmentation, VIB-AVSR still yields substantial WER reductions at test time. Under babble noise, improvements appear across all SNR levels. The babble average WER drops from 23.07% to 21.09%, and the extreme-noise average from 33.85% to 30.64%. Under speech noise, consistent gains appear in the low-SNR regime. Notably, the bottleneck is never exposed to noisy samples during training, yet the learned representations generalize more effectively to unseen noisy conditions. This suggests that variational compression promotes robustness through a mechanism fundamentally distinct from data augmentation. Models trained on clean data degrade more sharply under extreme noise than noisy-trained counterparts, but VIB-AVSR consistently narrows this gap relative to Llama-AVSR, particularly at the lowest SNRs. Performance on clean speech ($\infty$) remains comparable (2.42% vs. 2.34%), confirming that robustness gains do not come at the expense of clean-condition performance.

Ablation Studies

Effect of VIB Layer Placement

The placement of VIB modules determines at which depth noise-corrupted representations are compressed. Inserting too early may interfere with low-level features; inserting too late leaves noise unmitigated. The study evaluates single, dual, and triple bottleneck configurations at various layer positions, using noisy training paradigm and $\beta = \frac{0.1}{H}$. Layer index $-1$ denotes placement immediately after the audio encoder (at the projection layer), with $\beta = 10^{-7}$ following prior work; no interpolation is applied here due to shape mismatch.

Key findings:

• Single-layer configurations perform comparably across positions, with no single placement offering a consistent advantage. The $-1$ configuration (early compression at the encoder-LLM interface) achieves the best single-layer average WER at 15.03%, but still falls short of the best dual configuration.
• Dual configuration (4, 8) yields the best overall performance, achieving the lowest average WER (14.86%) and the strongest gains under extreme noise, while remaining competitive at higher SNR levels. The advantage of (4, 8) over (8, 12) at low SNR suggests that compressing representations at an earlier intermediate layer is beneficial when noise is most severe.
• Triple configurations degrade performance across all conditions including clean speech, indicating that stacking too many compression stages leads to over-regularization. This motivates the use of dual placement as the default.

Effect of $\beta$

$\beta$ controls the strength of the KL regularization term. Too small a value reduces the regularization effect; too large a value risks discarding task-relevant features. Four values normalized by hidden dimension $H$ are ablated using the (4, 8) dual configuration:

At $\beta = 1$, performance degrades sharply across all SNR levels, with average WER rising to 17.68% — confirming that excessive compression is harmful. Among the remaining values, $\beta = 0.1/H$ achieves the best average WER of 14.86% and is the most consistent across conditions, and is therefore used in all other experiments.

Role of Interpolation Coefficient $\alpha$

Three configurations of $\alpha$ are compared using the (4, 8) dual setup with $\beta = 0.1/H$:

• Setting $\alpha = 0$ replaces the hidden state entirely with the sampled representation, leading to the worst performance across all conditions — consistent with over-regularization and excessive information loss.
• The cosine-scheduled variant ($0 \rightarrow 0.5$) improves over $\alpha = 0$ but still shows higher WER relative to the fixed setting. The authors attribute this to the period of near-complete compression early in training, which the LLM must recover from, leaving a lasting negative imprint on learned representations.
• Fixed $\alpha = 0.5$ consistently achieves the best performance and is used for all main experiments.

Summary of Key Design Choices

Based on the ablation studies, the following configuration is identified as most effective and used throughout the main experiments:

• VIB insertion layers: After LLM layers 4 and 8 (dual configuration)
• Regularization strength: $\beta = 0.1 / H$ where $H$ is the LLM hidden dimension
• Interpolation coefficient: $\alpha = 0.5$ (fixed throughout training)
• Inference: Use $\mu^l$ only (no stochastic sampling at test time)
• Audio encoder: Frozen — only projection layers and LoRA are trained alongside the VIB modules

Contributions and Novelty

The paper makes the following contributions:

• Proposes VIB-AVSR, a lightweight method that integrates VIB layers into the LLM backbone of an AVSR model to improve noise robustness at the representation level, without requiring architectural redesign or additional training data.
• Demonstrates that VIB-AVSR improves noise robustness under both noisy and clean training paradigms, showing that variational compression promotes generalization to noisy conditions even without noise augmentation exposure.
• Provides extensive ablation studies on VIB layer placement, regularization strength $\beta$, and interpolation coefficient $\alpha$, with empirical evidence supporting each design choice.
• Shows WER reductions over Llama-AVSR across multiple noise types (babble and speech) and SNR levels, with gains that widen under extreme noise conditions, while preserving recognition performance on clean speech.

Limitations and Discussion

Several limitations and boundary conditions are noted or implied by the results:

• Experiments are conducted exclusively on the LRS2 dataset with Llama-3.2-1B as the LLM backbone. Generalization to other benchmarks (e.g., LRS3, VoxCeleb2) or larger LLMs is not demonstrated in this paper.
• The method is evaluated with babble and speech noise types from MUSAN. Generalizability to other noise types (e.g., music, environmental noise) is not explicitly verified.
• The gains under speech noise are smaller than those under babble noise. Speech noise is spectrally similar to the target signal, making it harder to compress away with the information bottleneck.
• Adding a third VIB module leads to over-regularization and degraded performance, suggesting that the benefit of the bottleneck saturates quickly and that careful placement with limited depth is important.
• The approach requires careful selection of $\beta$ and $\alpha$ through ablation, which may require additional tuning effort when adapting to new architectures or datasets.
• The clean-condition performance under clean training is marginally higher for Llama-AVSR (2.34% vs. 2.42%), indicating a small trade-off when no noise is seen during training — though this is within a very small absolute margin.

Conclusion

VIB-AVSR introduces a principled, lightweight approach to improving noise robustness in LLM-based audio-visual speech recognition. By inserting Variational Information Bottleneck modules into intermediate layers of the LLM backbone — specifically after layers 4 and 8 in a Llama-3.2-1B backbone — and applying variational compression exclusively to audio token representations, the model is encouraged to retain acoustically relevant content while discarding noise-correlated features. The method requires no architectural redesign, no additional training data, and introduces negligible computational overhead.

Experiments under babble and speech noise show consistent WER reductions over the Llama-AVSR baseline across SNR levels and training paradigms. Notably, the gains hold even when no noise augmentation is used during training, demonstrating that variational compression promotes noise-robust representations through a mechanism fundamentally distinct from data augmentation. The gains are largest under extreme noise conditions (SNR $\leq -2$ dB), precisely where robust acoustic representations matter most for practical deployment of conversational and audio-visual AI systems.