FSA-GRPO
FSA-GRPO: Teaching Auditory LLMs to Use Few-shot Demonstrations
FSA-GRPO uses RL-based post-training to teach auditory LLMs to leverage few-shot demonstrations through semantic alignment rewards. Unlike inference-time prompting, it explicitly optimizes models to benefit from demonstrations while preserving zero-shot capability for low-resource speech tasks.
Links
Paper & demos
Abstract
Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large language models are not explicitly trained to perform inference in this demonstration-conditioned format, limiting the extent to which they can benefit from few-shot prompting. To address this limitation, we introduce Few-Shot Aware GRPO (FSA-GRPO), an RL-based post-training recipe that uses a specially designed reward to encourage the model to leverage few-shot demonstrations, thereby strengthening its few-shot adaptation ability. Notably, training with only high-resource adult ASR data improves the model's general few-shot adaptation ability, yielding gains not only in children's speech recognition but also in speech translation and audio understanding. We further study data selection and auxiliary reward weighting to identify an effective training recipe. Our experiments show that when in-domain data are unavailable or cannot be used for training, FSA-GRPO is more effective than direct tuning on related out-of-domain data.
Introduction and Motivation
Auditory large language models (LLMs) have demonstrated impressive capabilities across speech recognition, translation, and audio understanding. However, their strong performance is largely driven by training on large-scale, high-resource data. When applied to low-resource settings — where labeled in-domain data are scarce, expensive to collect, or privacy-restricted — these models can fail substantially. Children's speech recognition is a canonical example: child speech differs from adult speech in acoustic and linguistic patterns, and despite various domain adaptation techniques, child ASR lags significantly behind adult ASR performance.
In-Context Learning (ICL) offers an attractive alternative to gradient-based adaptation. By conditioning a model on a small set of labeled in-domain demonstrations at inference time, it is possible to guide the model without any parameter updates. This few-shot prompting approach is especially appealing because the demonstrations can be swapped out as the target domain changes, providing a dynamic and flexible adaptation mechanism. However, most auditory LLMs are not explicitly trained in a demonstration-conditioned format, meaning their ability to leverage in-context examples is limited and often unreliable.
A natural solution is to post-train auditory LLMs on few-shot formatted data, but this introduces two risks: the model may lose its zero-shot capability, or it may learn to mimic the prompt structure without actually using the demonstrations to guide generation. This motivates the paper's central question:
Can few-shot adaptation be optimized without catastrophic forgetting of zero-shot inference?
To answer this, the authors propose Few-Shot Aware GRPO (FSA-GRPO), a reinforcement learning-based post-training strategy that teaches auditory LLMs to genuinely use few-shot demonstrations while preserving their direct inference capability. The method combines a standard ASR accuracy reward with a novel semantic alignment reward that explicitly encourages the model to attend to the retrieved in-context examples during generation.
Related Work
Few-Shot Adaptation for Speech and Audio
Speech ICL has been studied primarily as an inference-time mechanism. Prior work adapted Whisper for Chinese dialect recognition by conditioning on a few paired speech-transcript examples. Subsequent research explored better demonstration selection, including Bayesian example selection and token-level retrieval (M2R-Whisper). More recent work in the LMM (Large Multimodal Model) framework — including TICL and TICL+ — extends ICL to accented, multilingual, and children's ASR with text-embedding-based and acoustic reranking strategies.
Beyond ASR, few-shot prompting has been applied to speech translation (COSMIC, SALM), speech emotion recognition, and general audio tasks (UniAudio 1.5). However, all of these approaches focus on inference-time prompting or retrieval, without explicitly training the model to make better use of demonstrations.
Training Models to Use Demonstrations
MetaICL meta-trains text LLMs on diverse NLP tasks in a few-shot format, showing that ICL can be strengthened through demonstration-conditioned training. SMILE applies this idea to Whisper using high-resource ASR data in an ICL-style format. Omnilingual ASR explicitly trains the language decoder in a few-shot format for multilingual recognition. However, these approaches are ASR-specific and do not generalize across auditory LLM architectures. MetaSICL (the most closely related prior work) post-trains auditory LLMs with demonstration-conditioned speech tasks and shows improvements on held-out tasks, but it relies on supervised fine-tuning and does not explicitly verify that the model learns to use the demonstrations rather than merely adapting to the prompt format.
RL Post-Training for Speech and Audio LLMs
Group Relative Policy Optimization (GRPO) was originally introduced in DeepSeekMath as a memory-efficient variant of PPO and later adopted in DeepSeek-R1 for rule-based RL post-training. GRPO is well-suited for optimizing non-differentiable sequence-level objectives such as WER or BLEU through automatic rewards. Recent work has applied GRPO to audio question answering, spoken QA, speech translation, and ASR, often showing stronger performance than supervised fine-tuning under limited data or domain shift.
Method: FSA-GRPO
Training Data and Preparation
Training data is drawn exclusively from the English subset of Common Voice. This design choice is deliberate: it highlights the key advantage of FSA-GRPO, which is its ability to improve few-shot adaptation behavior using only high-resource, out-of-domain data, without access to scarce or privacy-sensitive target-domain recordings.
To mirror the few-shot inference format, training instances are constructed following the TICL protocol. The Common Voice development split serves as the query set $\mathcal{D}_\text{query}$ and the training split as the demonstration pool $\mathcal{D}_\text{pool}$. At each training step, a query utterance-transcript pair $(x_q, y_q)$ is sampled from the query set, and $k = 3$ in-context demonstrations $\{(x_j, y_j)\}_{j=1}^{k}$ are retrieved from the pool. The model is then trained to generate the query transcript conditioned on the concatenated demonstrations and the query audio:
$$P_{\theta}(y_q \mid x_1, y_1, \ldots, x_k, y_k, x_q)$$
This format directly aligns training with the demonstration-conditioned inference setting. Because GRPO training is computationally expensive, the authors sample 2,000 query instances using a data-selection strategy described in the ablation section.
Reward Design
The GRPO framework samples a group of $G$ candidate responses per prompt. Each response $y_i$ is assigned a scalar reward $\mathbb{R}_i$, and the group-normalized advantage is computed as:
$$\hat{A}_i = \frac{\mathbb{R}_i - \frac{1}{G}\sum_{j=1}^{G}\mathbb{R}_j}{\sigma\left(\{\mathbb{R}_j\}_{j=1}^{G}\right)}$$
ASR Reward
The primary reward measures transcription accuracy using Word Error Rate (WER). Both the generated response and the reference are normalized before comparison:
$$r_\text{ASR}(x, y_i) = \max\left(0,\ 1 - \text{WER}(\tilde{y}_i, \tilde{y}^\star)\right)$$
where $\tilde{y}_i = \mathcal{N}(y_i)$ and $\tilde{y}^\star = \mathcal{N}(y^\star)$ are the normalized prediction and reference. This reward allows the model to use or ignore demonstrations as appropriate, thereby avoiding catastrophic forgetting of zero-shot capability.
Semantic Alignment (SA) Reward
Prior work has shown that auditory LLMs benefit most from semantically similar in-context examples. To encourage the model to actively leverage the retrieved demonstrations rather than ignoring them, a semantic alignment reward is introduced. Let $\mathcal{I}(x)$ denote the set of retrieved demonstration transcripts for query $x$. Both the generated transcript and each retrieved transcript are encoded with a sentence encoder $f_\phi$ (SentenceBERT) and L2-normalized:
$$z_i = \frac{f_\phi(y_i)}{\|f_\phi(y_i)\|_2}, \quad e_j = \frac{f_\phi(y_j)}{\|f_\phi(y_j)\|_2}, \quad j \in \mathcal{I}(x)$$
The semantic similarity score uses the best-matching demonstration (to account for unequal informativeness across demonstrations):
$$s_i = \max_{j \in \mathcal{I}(x)} z_i^\top e_j$$
This similarity is converted to a bounded reward using a group-specific linear ramp defined by thresholds $\tau_\text{low}^{g(x)}$ (25th percentile) and $\tau_\text{full}^{g(x)}$ (50th percentile) of the within-group similarity distribution:
$$r_\text{SA}(x, y_i) = \operatorname{clip}\!\left(\frac{s_i - \tau_\text{low}^{g(x)}}{\tau_\text{full}^{g(x)} - \tau_\text{low}^{g(x)}},\ 0,\ 1\right)$$
Similarities below $\tau_\text{low}$ receive zero SA reward; similarities above $\tau_\text{full}$ receive maximum SA reward. Bounding the reward is intentional: the SA signal is meant as a lightweight auxiliary signal, not the dominant optimization target, to prevent the model from over-copying retrieved labels.
Combined Reward
The final per-response reward linearly combines transcription accuracy and demonstration alignment:
$$\mathbb{R}_i = \lambda \cdot r_\text{SA}(x, y_i) + r_\text{ASR}(x, y_i)$$
where $\lambda$ is a tunable weight controlling the strength of the SA reward.
GRPO Training Objective
The policy is optimized with the clipped GRPO objective:
$$\mathcal{L}_\text{GRPO}(\theta) = -\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_i}\sum_{t=1}^{T_i}\left[\min\!\left(\rho_{i,t}(\theta)\hat{A}_i,\ \operatorname{clip}(\rho_{i,t}(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_i\right) - \beta D_\text{KL}^{i,t}\right]$$
where the probability ratio is:
$$\rho_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} \mid p_q, y_{i, and the KL divergence penalty against the frozen reference model is: $$D_\text{KL}^{i,t} = \frac{\pi_\text{ref}(y_{i,t} \mid p_q, y_{i, FSA-GRPO is applied to two auditory LLM families: Qwen2.5-Omni and Audio-Flamingo-Next. To keep training lightweight and reduce overfitting, the audio encoder tower is frozen entirely, and only LoRA adapters inserted into all linear layers of the language backbone are updated. Unless otherwise noted, LoRA rank 8 and alpha 32 are used throughout. For GRPO rollouts, group size $G = 8$ candidate responses are sampled per prompt, with temperature 1.0, top-$p$ = 1.0, and no top-$k$ filtering (to maximize output diversity given the constrained ASR response space). The KL reference policy $\pi_\text{ref}$ is the frozen base model with LoRA disabled. Key hyperparameters: KL coefficient $\beta = 0.001$, clipping $\epsilon = 0.2$, learning rate $5 \times 10^{-5}$, AdamW optimizer with betas $(0.9, 0.95)$, $\epsilon = 10^{-8}$, weight decay 0.1, cosine scheduler without warmup. The final model is the checkpoint with the best training reward. All training is on NVIDIA A100 40 GB GPUs. Two child speech datasets are used. MyST (My Science Tutor) contains spontaneous conversational speech from Grades 3–5 students. RSR (Redmond Sentence Recall) contains scripted sentence-recall recordings from children aged 5–9. Following prior work, bounded WER is reported: the word-level edit count is capped at the reference length before normalization to reduce the effect of severe hallucinations on the corpus-level metric. MMAU evaluates a wide range of audio skills spanning speech, environmental sound, and music, emphasizing perception and domain-specific reasoning. MMAR covers speaker, environment, and content reasoning; audio quality comparison; music understanding; anomaly detection; spatial and temporal reasoning; and general reasoning. Accuracy on the public test splits (using official evaluation scripts) is reported. Multilingual ASR is evaluated on CommonVoice subsets in German, French, and Chinese (WER for German and French; CER for Chinese). Speech translation is evaluated on CoVoST2 English-to-Japanese and Japanese-to-English translation, using BLEU (up to 4-gram precision). All of these language pairs and directions are held-out from training, testing transfer of few-shot adaptation behavior beyond English ASR. Three main baselines are compared against FSA-GRPO: The full numerical results from the appendix table are summarized below for Qwen2.5-Omni: For multilingual ASR (CommonVoice de/zh/fr) and speech translation (CoVoST2 en→ja, ja→en), FSA-GRPO also improves over baselines in few-shot evaluation. On German ASR, FSA-GRPO achieves 5.29 WER (3-shot) vs. 7.09 for MetaSICL and 6.97 for SFT. On en→ja translation (BLEU), FSA-GRPO reaches 37.25 vs. 34.48 for MetaSICL and 33.49 for SFT. Key observations: A practical low-resource scenario is studied: RSR child speech recognition when no RSR training data can be used. The question is whether FSA-GRPO's few-shot approach is more effective than directly fine-tuning on related or generic out-of-domain data. Baselines include: SFT on CV 2k (Common Voice, out-of-domain), SFT on CV 20k, SFT on MyST 2k (child speech, related domain but not target), and GRPO on MyST 2k. Two inference settings are tested: RSR in-context examples (in-domain ICE) and MyST in-context examples (out-of-domain ICE). For Audio-Flamingo-Next, FSA-GRPO also consistently improves across MyST and RSR in zero-shot, 3-shot, and out-of-domain ICE settings (RSR 3-shot: from 19.32 to 17.45; RSR OoD ICE: from 23.37 to 22.35). Analysis of the results: The same FSA-GRPO recipe (including GRPO configuration and reward hyperparameters tuned for Qwen2.5-Omni) is applied to Audio-Flamingo-Next without any model-specific tuning. Improvements are consistent across MyST and RSR, in both zero-shot and 3-shot settings and under both in-domain and out-of-domain ICE conditions. The gains on Audio-Flamingo-Next are more modest than on Qwen2.5-Omni, which the authors attribute to the non-optimized hyperparameters for this model family. These results are presented as preliminary evidence of cross-model transferability rather than a fully tuned outcome. Since GRPO training is expensive, an ablation is conducted to identify which training instances provide the most useful learning signal under a fixed budget of 1,000 samples. Instances are characterized along two axes: This yields four subsets: random baseline, good-context/good-performance, bad-context/bad-performance, and good-context/bad-performance. Results confirm the hypothesis: good-context/bad-performance examples are the most valuable. They expose failure cases where the model has not learned to leverage informative demonstrations, providing strong GRPO signal. Good-context/good-performance examples (largely already solved) underperform the random baseline. Bad-context/bad-performance examples improve over random but underperform the good-context/bad-performance subset, since noisy retrieved demonstrations can mislead training. Based on this, the main training recipe prioritizes good-context/bad-performance instances. All nonzero SA reward weights outperform the ASR-only control ($\lambda = 0$), confirming the value of the semantic alignment signal. The sweet spot is $\lambda = 0.04$: at this value, the SA reward acts as a moderate regularizer that steers the model toward using in-context information without dominating the training objective. When $\lambda$ is too large ($0.08$), performance degrades slightly, suggesting the model begins to over-copy from retrieved demonstrations at the expense of processing the query audio. The paper explicitly discusses several limitations: FSA-GRPO's success in improving both zero-shot and few-shot performance simultaneously is noteworthy and arguably its most important property. Standard post-training approaches tend to improve one at the cost of the other: SFT can overfit the training format and lose flexibility, while zero-shot-only training provides no advantage when demonstrations are available. GRPO's reward structure resolves this tension naturally. The ASR reward allows the model to benefit from demonstrations when they help and ignore them when they don't — this is a form of demonstration-conditional self-selection driven by the reward signal. The SA reward provides an additional gentle push toward attending to in-context information, but by design it is kept subordinate to the ASR reward to prevent pathological copying behavior. The data-selection findings also have practical implications beyond FSA-GRPO. The insight that good-context/bad-performance examples are more valuable than either good-performance examples (saturated) or bad-context examples (noisy) provides a principled, computationally cheap way to curate GRPO training data. The two diagnostic scores — gold-to-ICE similarity and base model few-shot WER — can be computed without any additional model training, making this strategy easy to apply in practice. The transfer of ASR-trained few-shot adaptation to audio understanding and reasoning (MMAU, MMAR) is perhaps the most striking finding. These tasks have very different output formats, reasoning requirements, and audio types compared to ASR. The fact that training the model to use ASR demonstrations effectively also improves its use of demonstrations for general audio understanding suggests that the underlying mechanism — learning to condition generation on contextual patterns in the few-shot prompt — is indeed a general skill rather than a task-specific behavior. FSA-GRPO introduces a reinforcement learning-based post-training recipe that teaches auditory LLMs to genuinely use few-shot demonstrations during inference. By combining a WER-based ASR accuracy reward with a novel semantic alignment reward — applied to a LoRA-augmented model trained in the same demonstration-conditioned format used at test time — FSA-GRPO strengthens the model's in-context learning ability without sacrificing zero-shot performance. Training with only English adult ASR data generalizes to children's ASR, multilingual ASR, speech translation, and audio understanding/reasoning. In a controlled low-resource evaluation on child speech recognition, FSA-GRPO reduces RSR bounded WER by 53.9% relative without using any RSR training data and outperforms all direct fine-tuning baselines. Ablations confirm that good-context/bad-performance data selection and a moderate SA reward weight ($\lambda = 0.04$) are key practical design choices. Overall, the work establishes demonstration-aware RL post-training as a promising alternative to direct fine-tuning for adapting auditory LLMs to low-resource and unseen speech and audio tasks.Model Architecture and LoRA Configuration
Evaluation Setup
Child's ASR
Audio Understanding and Reasoning (AU/AR)
Multilingual ASR and Speech Translation
Experiments
Main Baselines Comparison
Method
MyST WER (0-shot)
MyST WER (3-shot)
RSR WER (0-shot)
RSR WER (3-shot)
MMAU Acc (0-shot)
MMAU Acc (3-shot)
MMAR Acc (0-shot)
MMAR Acc (3-shot)
Qwen2.5-Omni
23.05
22.72
35.65
27.86
65.80%
67.30%
49.20%
53.80%
+ MetaSICL
26.76
17.42
33.52
22.16
68.10%
72.10%
53.00%
54.50%
+ SFT w. same data
22.63
17.50
36.49
22.28
62.70%
70.10%
45.90%
54.50%
+ FSA-GRPO (Ours)
16.94
11.23
28.73
16.32
66.80%
72.80%
49.50%
56.40%
Comparison to Direct Tuning (Low-Resource Scenario)
Method
MyST WER (0-shot)
MyST WER (3-shot)
RSR WER (0-shot)
RSR WER (3-shot)
RSR WER (3-shot, OoD ICE)
Qwen2.5-Omni
23.05
22.72
35.42
27.29
38.91
+ SFT w. CV 2k
15.14
17.02
29.81
23.78
33.50
+ SFT w. CV 20k
13.99
15.54
28.66
21.56
30.45
+ SFT w. MyST 2k
14.34
20.86
29.43
24.38
33.88
+ GRPO w. MyST 2k
11.66
18.09
26.29
22.69
30.54
+ FSA-GRPO w. CV 2k (Ours)
16.94
11.23
28.73
16.32
25.28
Transferability
Data Selection Ablation
Training Subset (1k)
Bounded WER (RSR 3-shot)
Qwen2.5-Omni (no training)
27.83
+ Random
22.14
+ Good ctx, good perf
23.59
+ Bad ctx, bad perf
21.37
+ Good ctx, bad perf
20.56
Auxiliary SA Reward Weight Ablation
SA Reward Weight $\lambda$
Bounded WER (RSR 3-shot)
0 (ASR-only control)
21.58
0.01
19.78
0.02
18.52
0.04
16.56
0.08
17.31
Key Contributions and Novelty
Limitations
Discussion
Conclusion