Agentic ASR

Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation

Agentic ASR frames speech recognition as a multi-turn interactive process using a closed-loop system with large language models for semantic correction and transcript refinement. This enables more accurate and human-like dialogue alignment compared to traditional single-pass ASR.

• asr
• llm
• agent
• dialogue

Demos

These demos showcase the Agentic ASR system's novel human-like interaction paradigm for speech recognition, where iterative multi-turn corrections reduce semantic errors beyond traditional metrics. Watch how corrections refine transcription accuracy and semantic coherence over multiple loops, illustrating the semantic error rate improvements and reasoning-based editing in complex multilingual, named-entity, and code-switching contexts. Visual frameworks clarify the system's closed-loop architecture and benchmarking approach.

Authors: Zixuan Jiang, Yanqiao Zhu, Peng Wang, Qinyuan Chen, Xinjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao, Xiangang Li, Kai Yu, Xie Chen

Categories: cs.AI, cs.CL

Published 2026-05-28 · Updated 2026-05-28

Links

Paper & demos

• arXiv
• Project page

Code & resources

• GitHub4 stars0 watchers0 forksMIT

Impact

• Semantic Scholar0 citations

Abstract

Automatic speech recognition (ASR) is a core component of human--computer interaction and an increasingly important front-end for LLM-based assistants and agents. However, most current ASR systems still follow a single-pass paradigm, which is poorly aligned with human communication, where misunderstandings are resolved through iterative clarification and refinement. This mismatch makes it difficult to correct meaning-critical errors once they occur. Meanwhile, token-level metrics such as WER or CER cannot adequately reflect such a problem. To address these limitations, we formulate \emph{Interactive ASR} as a multi-turn refinement task and propose \textbf{Agentic ASR}, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing. We further introduce the \textbf{Sentence-level Semantic Error Rate} ($S^2ER$), an LLM-based semantic evaluation metric, together with an \textbf{Interactive Simulation System} for scalable and reproducible benchmarking. Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors, with much larger gains in $S^2ER$ than in conventional token-level metrics. Human--AI alignment and ablation studies further validate the reliability of the semantic judge and the robustness of the proposed framework. The code is available at: https://interactiveasr.github.io/ and the live demo is available at https://i-asr.sjtuxlance.com/

---

Introduction

This paper argues that the standard ASR pipeline is mismatched with how humans actually communicate and correct misunderstandings. Conventional ASR is a single-pass, open-loop recognizer: it maps an input utterance $I$ to one transcription $Y = \mathrm{ASR}(I)$ and then stops. In human conversation, however, recognition errors are often repaired through multi-turn clarification, especially when the error concerns a name, an entity, or another meaning-critical detail. The authors’ central claim is that future ASR for assistants and agents should be treated as a stateful interactive refinement problem, not just a one-shot decoding problem.

The paper makes two closely related contributions. First, it defines Interactive ASR as a multi-turn task in which the model progressively updates its transcription state using both the current speech input and the history of prior turns. Second, it proposes Agentic ASR, a closed-loop framework that combines a single-pass ASR front end with LLM-based semantic correction, intent routing, and reasoning-based editing. In parallel, the paper introduces a sentence-level semantic metric, $S^2ER$ (Sentence-level Semantic Error Rate), and an Interactive Simulation System (ISS) for repeatable multi-turn evaluation.

Problem Formulation: Interactive ASR

The authors formalize the task by contrasting standard ASR with an interaction-aware variant. In conventional ASR, the output depends only on the current acoustic input:

$$Y = \mathrm{ASR}(I).$$

In Interactive ASR, the system maintains a transcription history $Y_{[:t-1]} = \{Y_0, \ldots, Y_{t-1}\}$ and updates its state at each turn $t$ using the new speech input $I_t$ together with the prior interaction context:

$$Y_0 = \mathrm{InteractiveASR}(\emptyset, I_0),$$

$$Y_t = \mathrm{InteractiveASR}(Y_{[:t-1]}, I_t) \quad \text{for } t > 0.$$

This framing is important because it turns ASR from an independent decoding problem into a recurrent state-update problem. The practical motivation is that many ASR failures are not merely local token errors; they are errors that break intent, entity identity, or downstream tool execution. A system that can revise its hypothesis under feedback is therefore better aligned with human repair behavior and with agent-oriented applications.

Agentic ASR: Architecture and Correction Loop

The proposed Agentic ASR system instantiates Interactive ASR with a modular closed loop. At each turn $t$, the user provides speech $I_t$. A standard ASR model first produces a hypothesis

$$H_t = \mathrm{ASR}(I_t).$$

An LLM then refines this hypothesis using the current transcript history:

$$Y_t = \mathrm{LLM}(H_t, Y_{[:t-1]}; \mathcal{P}_{\mathrm{refine}}).$$

The paper decomposes this refinement process into three explicit stages to make it more interpretable and controllable:

• Semantic correction: rewrite the raw ASR output into an explicit instruction that is consistent with the previous transcript history, producing $H_t' = \mathrm{SemanticCorrection}(H_t, Y_{[:t-1]})$.
• Intent routing: classify the corrected instruction into one of three intents: confirmation, new input, or correction.
• Reasoning-based correction: if the intent is correction, locate the editable span, infer the intended edit from the utterance and context, and apply the modification to update the transcription state.

The state update rule is explicitly defined as:

$$Y_t = \begin{cases} Y_{t-1}, & c_t = \mathrm{confirmation},\\ H_t', & c_t = \mathrm{new\ input},\\ H_t^{\mathrm{corr}}, & c_t = \mathrm{correction}. \end{cases}$$

For correction turns, the reasoning component is written as a composition of three operations:

$$\mathrm{ReasoningCorrector} = \mathrm{Modify} \circ \mathrm{Reason} \circ \mathrm{Locate}.$$

Conceptually, this is the paper’s key architectural choice. Instead of letting an LLM freely rewrite the transcript in a single step, the model is constrained to (1) find where the correction should be applied, (2) infer what change is intended, and (3) apply only the necessary modification. The authors present this as a way to better match how humans issue partial repair instructions in dialogue.

Sentence-level Semantic Error Rate ($S^2ER$)

The paper argues that token-level metrics such as Word Error Rate (WER), Character Error Rate (CER), Named-Entity Error Rate (NER), and Mixed Error Rate (MER) are insufficient for interactive ASR, because they count all local mismatches equally even when some mismatches are semantically harmless. To address this, the authors introduce $S^2ER$, a binary semantic metric that asks whether a transcription preserves enough meaning for correct intent execution.

The metric is defined over $N$ utterances as:

$$S^2ER = \frac{1}{N}\sum_{i=1}^{N}(1-\hat{z}_i),$$

where $\hat{z}_i \in \{0,1\}$ indicates whether hypothesis $Y_i$ is semantically equivalent to reference $Y_{GT,i}$. Here, $\hat{z}_i=1$ means the meaning is preserved, and $\hat{z}_i=0$ means a meaning-critical error occurred. The judge is instructed to focus on whether the main intent and key content are preserved, especially proper nouns, named entities, and other task-critical information, while ignoring fillers, disfluencies, and punctuation.

To improve stability, the paper uses a three-round bidirectional voting protocol. In each round, the LLM judge evaluates the pair twice with swapped order:

$$z_{i,r}^{(1)} = \mathrm{LLM}_{\mathrm{judge}}(Y_i, Y_{GT,i}; \mathcal{P}_{\mathrm{judge}}),$$

$$z_{i,r}^{(2)} = \mathrm{LLM}_{\mathrm{judge}}(Y_{GT,i}, Y_i; \mathcal{P}_{\mathrm{judge}}),$$

where $z_{i,r}^{(1)}, z_{i,r}^{(2)} \in \{0,1\}$. A round is counted as positive only if both judgments indicate semantic equivalence, and the final label is determined by majority vote across three rounds:

$$\hat{z}_i = \mathbf{1}\!\left(\sum_{r=1}^{3}\bigl(z_{i,r}^{(1)} \land z_{i,r}^{(2)}\bigr) \ge 2\right).$$

The paper motivates this design with two examples. In one case, a transcription with many filler/discourse-word differences can have a high WER but still preserve meaning; in another, a small substitution affecting a key entity can have a lower WER but fail semantically. This is the paper’s core argument for moving beyond token-level metrics in interactive ASR.

The authors also explain why the metric is appropriate for agentic settings: downstream success often depends on whether the transcription preserves the user’s intent and key entities, not whether it matches the reference word-for-word. In that sense, $S^2ER$ is designed to measure functional usability rather than lexical similarity.

Interactive Simulation System (ISS)

Human-in-the-loop evaluation of interactive ASR is expensive and difficult to reproduce, so the paper introduces an automatic Interactive Simulation System (ISS). ISS simulates multi-round user–system interaction and uses the $S^2ER$ judge as the stopping criterion.

At each round $t$, the evaluated system produces a transcription $Y_{i,t}$ for sample $i$, and the $S^2ER$ judge checks whether it is semantically equivalent to the ground truth. If equivalence is reached, the interaction terminates early; otherwise, a user simulator generates a corrective spoken instruction for the next round. The round-wise semantic error rate is:

$$S^2ER_t = \frac{1}{N}\sum_{i=1}^{N}(1-\hat{z}_{i,t}),$$

where $\hat{z}_{i,t}=1$ means sample $i$ is semantically correct at round $t$.

The user simulator itself is described as a composition of an LLM-based corrector and a TTS vocalizer:

$$X_{i,t+1} = \mathrm{TTS}\big(\mathrm{LLM}(Y_{i,t}, Y_{i,\mathrm{GT}}, \mathcal{P}_{\mathrm{corr}})\big).$$

The LLM identifies the key semantic discrepancy between the current hypothesis and the reference, generates a concise correction, and TTS converts that correction into speech. The reference audio for each sample is used as the acoustic prompt to preserve speaker consistency across turns.

Experimental Setup

The experimental section evaluates both the interactive framework and the semantic evaluation protocol. The default backbone ASR model is Qwen3-ASR-1.7B. The unified LLM used for three roles—reasoning in Agentic ASR, correction generation in the user simulator, and semantic judging in $S^2ER$ evaluation—is Qwen3-32B. For speech synthesis in the simulation loop, the paper uses Index-TTS-1.5.

The evaluation covers three types of benchmarks:

• Multilingual speech: GigaSpeech Test for English and WenetSpeech Test_Net for Mandarin.
• Named-entity-intensive speech: AISHELL-NER Dev$^\dagger$ and AISHELL-NER Test$^\dagger$, created by filtering AISHELL-1 dev/test utterances that contain named entities.
• Code-switching speech: ASRU2019 Test and CS-Dialogue Test$^\dagger$, the latter constructed by selecting code-switching utterances from the original CS-Dialogue corpus.

The paper evaluates on representative English, Mandarin, named-entity-heavy, and Mandarin–English code-switching conditions in order to test whether interactive semantic correction generalizes beyond a single language or speaking style.

Human–AI Alignment for $S^2ER$

Before using $S^2ER$ as a benchmark metric, the authors verify that the LLM judge aligns with human judgments. They sample 40 utterances each from GigaSpeech, WenetSpeech, and ASRU2019, yielding 120 validation examples spanning English, Chinese, and code-switching conditions. Semantic consistency is annotated independently by 25 non-expert annotators and 5 domain experts using a binary protocol, and the average human score serves as reference.

The paper reports Pearson correlation coefficients $r$ between the LLM judge, expert judgments, and the human reference scores. The LLM evaluation is repeated five times to measure stability.

The main takeaway is that the judge tracks human semantic decisions reliably across all three datasets, with correlations above $0.8$ and small standard deviations across repeated runs. The paper highlights that the LLM judge is not only stable but also slightly better correlated with human labels than the domain experts on these samples. This supports $S^2ER$ as a practical semantic metric for interactive ASR evaluation.

Main Results: Multi-Turn Interaction Improves Semantic Correctness

The primary result is that iterative interaction steadily reduces semantic errors across all benchmark families. The paper reports representative checkpoints at Loop 0, 1, 3, and 10, and also shows full trajectories from Loop 0 to Loop 10. The most important trend is that $S^2ER$ decreases monotonically with more interaction, and the biggest improvements appear in the first few turns.

The reported values show a clear early-turn effect. For example, GigaSpeech Test drops from $21.47\%$ at Loop 0 to $12.35\%$ after one interaction and to $3.49\%$ by Loop 10. ASRU2019 Test drops from $28.57\%$ to $10.32\%$ after one loop and to $1.36\%$ by Loop 10. The paper emphasizes that the largest semantic gains happen immediately, meaning the framework can recover a substantial fraction of meaning-critical errors with only a small interaction budget.

The qualitative conclusion from the curves is that semantic repair happens faster than lexical cleanup. In the figures, $S^2ER$ falls much more sharply than WER, CER, or MER. This supports the paper’s claim that iterative interaction primarily fixes the errors that matter for intent and downstream execution, rather than simply polishing local token discrepancies.

Ablation Studies

Different Base ASR Models

The first ablation checks whether the interactive framework depends on a particular ASR backbone. The default Qwen3-ASR-1.7B backbone is replaced with two alternatives: the larger FireRedASR2-LLM-8.3B and the much weaker Whisper. All other parts of the interactive pipeline are kept fixed. The paper uses three representative benchmarks for this test: GigaSpeech Test, AISHELL-NER Test$^\dagger$, and ASRU2019 Test.

The conclusion is that the interactive gains are not tied to one specific recognizer. All three backbones show consistent decreases in $S^2ER$ as the number of interaction loops increases. This is important because it suggests that the benefit comes from the interaction mechanism itself, not from a special compatibility between Agentic ASR and Qwen3-ASR.

The paper also reports that even Whisper, despite starting much worse, benefits substantially from the closed loop. Its final $S^2ER$ values after multiple loops are reported as $3.79\%$ on GigaSpeech Test, $6.82\%$ on AISHELL-NER Test$^\dagger$, and $3.75\%$ on ASRU2019 Test. Stronger backbones still achieve better final floors—for example, FireRedASR2-LLM-8.3B reaches $0.55\%$ on AISHELL-NER Test$^\dagger$ compared with $2.02\%$ for Qwen3-ASR-1.7B and $6.82\%$ for Whisper—but the key point is that even weak ASR can be made meaningfully more usable through interaction.

Size of the LLM Reasoner

The second ablation replaces the $32$B reasoning model with a smaller $8$B variant, while also using the same smaller model in the user simulator’s correction generator. This isolates the effect of LLM scale on both reasoning quality and simulation quality.

The main result is that the $8$B model still preserves the monotonic benefit of interaction, but it consistently underperforms the $32$B default on all datasets. The reported final $S^2ER$ at Loop 10 is:

The paper notes an important nuance: while the smaller model still reduces semantic errors, conventional token-level metrics can become less stable, and may even worsen on some benchmarks. The authors interpret this as a sign that weaker LLMs are less precise at span identification and edit control. In other words, the $8$B model is still usable, but the $32$B model produces cleaner and more reliable corrections.

LLM-as-a-Judge Strategy

The third ablation studies the voting strategy used for $S^2ER$ judgment. The paper compares a single bidirectional judgment with majority voting over $K \in \{3,5,7\}$ rounds. The goal is to measure whether repeated judgments improve agreement with human references and whether there is an efficiency–robustness trade-off.

The takeaway is that limited repetition helps: a single bidirectional judgment already correlates strongly with human labels, and majority-3 improves the overall correlation on the full validation set from $0.8543$ to $0.8628$. However, more voting is not always better. Majority-5 is best on GigaSpeech, but not overall, and majority-7 shows diminishing returns. The authors therefore recommend majority-3 as the default trade-off between reliability and cost.

Interpretation of the Results

Across all experiments, the paper consistently finds that interaction mainly repairs meaning-critical errors. This is why $S^2ER$ falls much more sharply than WER, CER, NER, or MER. Conventional token-level metrics underestimate the value of the framework because they are sensitive to many local changes that do not alter downstream meaning, while the proposed semantic metric captures whether the transcript is actually usable for intent execution.

The results also show that the framework is broadly robust. It works on English, Mandarin, named-entity-heavy utterances, and Mandarin–English code-switching. It also remains effective across strong and weak ASR backbones. The most important empirical pattern is not simply that performance improves, but that it improves in a way that is aligned with semantic success. That is, the system is not just producing cleaner text; it is becoming more faithful to the user’s intended meaning.

Stated Contributions and Novelty

• The paper defines Interactive ASR as a stateful multi-turn refinement task rather than a one-shot transcription problem.
• It proposes Agentic ASR, which combines a single-pass ASR front end with semantic correction, intent routing, and structured locate–reason–modify editing.
• It introduces $S^2ER$, a binary sentence-level semantic metric focused on whether the transcription preserves sufficient meaning for correct intent execution.
• It develops an Interactive Simulation System for scalable and reproducible multi-turn evaluation using an LLM-based user simulator and semantic stopping criterion.
• It validates the judge against human annotations and shows that iterative interaction yields the largest gains in semantic correctness, especially early in the interaction process.

Limitations and Future Work

The paper does not present a separate limitations section, but it does identify two future directions that effectively bound the current work. First, the authors note that richer interactive supervision would be valuable, including real user correction traces or automatically constructed interaction data, to improve robustness under realistic deployment conditions. Second, they suggest post-training a smaller task-specific refinement model as a promising direction, since compact models already show some correction ability but still lag behind larger models in stability and precision.

From the reported experiments, another practical implication is that semantic gains can be realized without many turns, but the cost of interaction, the dependence on LLM reasoning quality, and the fidelity of simulated user feedback remain important factors for deployment. These are not framed as explicit limitations in the paper, but they are the main constraints visible from the described setup and ablations.

Conclusion

The paper’s main message is that ASR for assistants and agents should look more like human conversation: when an error matters, it should be repairable through interaction. By formalizing Interactive ASR, proposing Agentic ASR, and introducing $S^2ER$ plus an automatic simulation framework, the authors provide both a mechanism and an evaluation paradigm for that goal. Their experiments show that iterative correction consistently lowers semantic error rates across multilingual, named-entity-intensive, and code-switching settings, and that the resulting improvements are substantially better reflected by semantic evaluation than by token-level metrics alone.