SALMONN-2 Pushes General-Purpose Audio Language Models Further
Today's digest spotlights SALMONN-2 from Tsinghua University, an audio language model that leverages self-supervised representations and multi-layer fusion to achieve broad, balanced hearing abilities across speech, audio, music, and paralinguistics.
Performance of SALMONN-2 and existing open-weight ALLMs. SALMONN-2 achieves competitive performance on audio understanding tasks (ASR, AAC, ER, etc.) and benchmarks (MMSU, MMAR and MMAU-Pro), while exhibiting new capabilities on audio analysis tasks that were previously overlooked, such as speech quality assessment, SED, and spoofing detection. From Tsinghua University.
Today's digest is a focused one — a single but substantial contribution from Tsinghua University that pushes the frontier of general-purpose audio language models, tackling the long-standing challenge of building a single system that truly hears across all audio domains.
SpeechLLMs & Audio Understanding
Self-supervised encoders meet large language models for richer, more versatile audio understanding.
One of the persistent headaches in audio-language model design is that no single encoder captures everything — speech intelligibility, acoustic events, musical structure, and speaker affect all live at different levels of representation. SALMONN-2 addresses this head-on by replacing task-specific supervised encoders with self-supervised encoders paired with a multi-layer fusion adapter that explicitly harvests hierarchical representations across encoder layers. The result is a model that doesn't have to choose between low-level acoustic detail and high-level semantic content — it gets both. On top of this, SALMONN-2 introduces multimodal in-context learning via contextual biasing, allowing the model to condition on provided examples at inference time without retraining, a capability that meaningfully closes the gap between audio LMs and their text-only counterparts. The payoff is balanced, competitive performance spanning speech recognition, audio captioning, music understanding, and paralinguistic tasks within a single unified model.
SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations
SALMONN-2 advances audio language models using self-supervised encoders with a multi-layer fusion adapter to capture hierarchical representations. It introduces multimodal in-context learning capabilities through contextual biasing, achieving balanced performance across speech, audio, music, and paralinguistic tasks.
Abstract
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
Trending on Hugging Face
Moshi: a speech-text foundation model for real-time dialogue
A speech-text foundation model for real-time dialogue via end-to-end speech-to-speech generation instead of cascaded components. By jointly modeling overlapping audio streams with text-token prediction, it achieves 200ms latency while preserving emotion and handling natural conversational dynamics like interruptions.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
A novel GPT-based model, OmniFlatten, enables real-time natural full-duplex spoken dialogue through a multi-stage post-training technique that integrates speech and text without altering the original model's architecture.
Qwen3-TTS Technical Report
The Qwen3-TTS series presents advanced multilingual text-to-speech models with voice cloning and controllable speech generation capabilities, utilizing dual-track LM architecture and specialized speech tokenizers for efficient streaming synthesis.
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
IndexTTS, an enhanced text-to-speech system combining XTTS and Tortoise models, offers improved naturalness, enhanced voice cloning, and controllable usage through hybrid character-pinyin modeling and optimized vector quantization.
VibeVoice Technical Report
VibeVoice synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer, achieving superior performance and fidelity.