CTC-Seeded Edit Flow

CTC-Seeded Token Edit Refinement for Non-Autoregressive Speech Recognition

Refines greedy CTC transcripts in non-autoregressive ASR by predicting insertions, deletions, and substitutions using an acoustic-conditioned Edit Flow decoder. This approach uses the CTC output as a strong seed, improving accuracy efficiently with fewer refinement steps by leveraging acoustic and confidence cues.

• asr
• low-latency
• streaming

Authors: Wanting Huang, Weiran Wang

Categories: eess.AS

Comment: Submitted to IEEE SLT 2026

Published 2026-06-27 · Updated 2026-06-27

Links

Paper & demos

• arXiv

Abstract

Non-autoregressive automatic speech recognition (ASR) enables parallel decoding, but many refinement-based methods begin from random, fully masked, or fixed-length token sequences, requiring multiple iterations to reconstruct the complete transcript. We instead formulate ASR decoding as a variable-length edit refinement of a greedy connectionist temporal classification (CTC) hypothesis. An acoustic-conditioned Edit Flow decoder operates directly on the collapsed CTC hypothesis, predicting insertion, deletion, and substitution operations in parallel. The Edit Flow decoder is jointly trained with a CTC model using a continuous-time discrete diffusion loss. During inference, we find that just two edit steps yield substantial Word Error Rate (WER) reductions, and classifier-free guidance (CFG) further enhances recognition quality by focusing the model on audio features. We also constrain edit proposals using CTC confidence to improve accuracy. Finally, ablation studies validate our design choices, while decoder pretraining and pretrained encoder integration yield significant additional performance gains.

---

Overview and motivation

This paper addresses a central tradeoff in non-autoregressive automatic speech recognition (ASR): CTC-style systems decode in parallel and are fast, but their conditional-independence assumption limits accuracy, while many refinement-based non-autoregressive methods begin from an uninformative starting point such as random tokens, fully masked sequences, or fixed-length latent states. The authors argue that ASR should not reconstruct a transcript from scratch when a greedy CTC pass already provides a strong first approximation. Instead, they reformulate decoding as variable-length token edit refinement of the collapsed greedy CTC hypothesis.

The central idea is to treat the first-pass CTC output as a seed transcript and then correct it with an acoustic-conditioned Edit Flow decoder that predicts insertions, deletions, and substitutions in parallel. This gives a diffusion-style iterative decoder that operates directly in token space, rather than on frame-level CTC alignments or on a fully masked sequence. The paper emphasizes three practical advantages of this design: it avoids reconstructing already-correct tokens, it naturally supports length changes, and it can leverage both acoustic evidence and confidence cues from the CTC model.

The method is evaluated on LibriSpeech using the standard 960-hour training set and development/test splits. The authors report that the strongest configuration combines a large Edit Flow decoder, edit-aware text pretraining, classifier-free guidance on the audio condition, and confidence-based gating of edits derived from CTC alignments. Under these settings, the system improves the first-pass CTC baseline substantially, and two refinement steps are enough to obtain the best results in the reported experiments.

Core contribution

• Reframes ASR decoding as CTC hypothesis-to-transcript edit refinement instead of generation from noise or masking.
• Introduces an acoustic-conditioned Edit Flow decoder that edits token sequences with insertion, deletion, and substitution operations in parallel.
• Trains the decoder jointly with CTC using a continuous-time discrete diffusion objective.
• Adds two inference-time controls: audio classifier-free guidance and CTC-confidence gating of proposed edits.
• Designs an edit-aware text pretraining scheme based on deletion, substitution, and insertion corruptions.

Method

First-pass CTC hypothesis

The ASR encoder maps an acoustic input $A$ to hidden states $H = \operatorname{Enc}(A) = (h_1, \ldots, h_T)$. A CTC softmax layer produces frame-level posteriors over the vocabulary and blank symbol. The greedy alignment is obtained by selecting the most likely symbol at each frame,

$$ a_t = \arg\max_{v \in \mathcal{V} \cup \{\varnothing\}} p_{\mathrm{ctc}}(v \mid h_t), \quad t=1,\ldots,T, $$

and then collapsing repeated labels and blanks:

$$ \hat{Y}^{(0)} = \mathcal{B}(a_{1:T}). $$

This collapsed hypothesis is the initial sequence that the Edit Flow decoder refines. The paper explicitly motivates this choice by noting that the collapsed CTC output is already much shorter than a frame-level alignment because blanks are removed, so the refinement state is compact and focused on token corrections rather than alignment reconstruction.

Variable-length edit path

The decoder is trained to transform the noisy CTC hypothesis into the ground-truth transcript $Y = (y_1, \ldots, y_N)$. The paper prepends a protected beginning-of-sequence token and defines

$$ x_0 = [\langle \mathrm{bos} \rangle ; \hat{Y}^{(0)}], \quad x_1 = [\langle \mathrm{bos} \rangle ; Y]. $$

Because $x_0$ and $x_1$ can have different lengths, the method computes a Levenshtein alignment in an auxiliary space that allows a gap symbol $\epsilon$. This alignment induces a sequence of primitive edit events: identity when aligned symbols match, insertion when the source side is a gap, deletion when the target side is a gap, and substitution when both are non-gap tokens but differ.

A continuous refinement time $t \in [0,1]$ is sampled through a monotonic noise scheduler $\kappa(t)$ with $\kappa(0)=0$ and $\kappa(1)=1$. An intermediate aligned state is formed by choosing source or target symbols according to $\kappa(t)$, and the decoder consumes the gap-removed sequence $x_t = \operatorname{RemoveGap}(z_t)$. The gaps are therefore only a device for defining the edit path; they never enter the decoder input.

Acoustic-conditioned Edit Flow decoder

The Edit Flow decoder is a bidirectional Transformer conditioned on the current token sequence $x_t$, the acoustic memory $H$, and the flow time $t$. It predicts position-wise intensities for insertion, deletion, and substitution, along with token distributions for the token-producing operations. In the paper’s notation, insertion is allowed at every position including the protected beginning-of-sequence slot, while deletion and substitution are only defined at real transcript-token positions.

The model parameterizes the rate of a concrete edit proposal $e_i$ as

$$ u_\theta(e_i \mid x_t, H, x_0, t) = \begin{cases} \lambda_i^{\mathrm{ins}} q_i^{\mathrm{ins}}(v), & e_i = \operatorname{Ins}(v),\ i \in \mathcal{P}_t, \\ \lambda_i^{\mathrm{sub}} q_i^{\mathrm{sub}}(v), & e_i = \operatorname{Sub}(v),\ i \in \mathcal{I}_t, \\ \lambda_i^{\mathrm{del}}, & e_i = \operatorname{Del},\ i \in \mathcal{I}_t, \end{cases} $$

where $\lambda_i^{\mathrm{ins}}$, $\lambda_i^{\mathrm{sub}}$, and $\lambda_i^{\mathrm{del}}$ are operation intensities and $q_i^{\mathrm{ins}}(v)$ and $q_i^{\mathrm{sub}}(v)$ are token distributions. This formulation directly edits token sequences and changes length without introducing a separate length predictor or a frame-level editable alignment state.

Continuous-time diffusion objective

The Edit Flow loss is written as a continuous-time discrete diffusion objective over the remaining target edits in the aligned auxiliary space. The total edit intensity is

$$ U_\theta(x_t, H, x_0, t) = \sum_{i \in \mathcal{P}_t} \lambda_i^{\mathrm{ins}} + \sum_{i \in \mathcal{I}_t} \left(\lambda_i^{\mathrm{sub}} + \lambda_i^{\mathrm{del}}\right). $$

The training objective is

$$ \mathcal{L}_{\mathrm{EF}} = \mathbb{E}_{t, z_t}\left[ U_\theta(x_t, H, x_0, t) - \frac{\dot{\kappa}(t)}{1-\kappa(t)} \sum_{m \in \mathcal{M}_t} \log u_\theta(e_m^\star \mid x_t, H, x_0, t) \right], $$

where $\mathcal{M}_t$ indexes the remaining target edits and $e_m^\star$ denotes the corresponding correct edit action. For supervised ASR, the paper combines this with the standard CTC loss:

$$ \mathcal{L} = \alpha \mathcal{L}_{\mathrm{CTC}} + (1-\alpha) \mathcal{L}_{\mathrm{EF}}, \quad \alpha = 0.3. $$

This joint objective trains the encoder, CTC head, and Edit Flow decoder end-to-end.

Edit-aware text pretraining

The paper optionally pretrains the decoder on text-only data before speech fine-tuning. Given a clean transcript $Y$, it constructs a noisy source $\widetilde{Y}$ by independently applying deletion, substitution, and insertion corruptions to each token with probabilities $\rho_{\mathrm{del}}$, $\rho_{\mathrm{sub}}$, and $\rho_{\mathrm{ins}}$. The source-target pair $(\widetilde{Y}, Y)$ is aligned with the same Levenshtein procedure and trained using the Edit Flow objective, but with the acoustic memory set to zero during pretraining.

The intent is to teach the decoder an explicit correction behavior before it is asked to condition on speech. The ablations show that the corruption rate matters: the best setting uses a relatively small noise level, consistent with the fact that the pretraining distribution should resemble the residual error pattern of the CTC seed rather than arbitrary heavy corruption.

Inference strategies

Parallel iterative refinement

Inference treats the decoder as a diffusion process that moves from the CTC seed $x_0$ toward the final transcript $x_1$ in a small number of parallel steps. At step $k$ with step size $h_k$, the model uses a Poisson tau-leaping approximation to derive the probability of an insertion event and the probability of a delete-or-substitute event at each position:

$$ p_{i,k}^{\mathrm{ins}} = 1 - \exp(-h_k \lambda_{i,k}^{\mathrm{ins}}), $$

$$ p_{i,k}^{\mathrm{d/s}} = 1 - \exp\!\left[-h_k\left(\lambda_{i,k}^{\mathrm{del}} + \lambda_{i,k}^{\mathrm{sub}}\right)\right]. $$

The paper reports that deterministic decoding works better than stochastic sampling. An edit is accepted when its probability exceeds an operation-specific threshold of $0.1$, and the highest-scoring token is chosen for insertion or substitution. The best-performing configuration uses only two refinement steps with $h_k = 0.5$.

Audio classifier-free guidance

To make the refinement more strongly adhere to the acoustic condition, the paper applies classifier-free guidance on the audio memory. During training, with probability $0.1$, the acoustic memory $H$ is replaced by zero while the sequence inputs are kept unchanged. At inference, the audio-conditioned and audio-dropped outputs are combined with guidance scale $w$.

For operation type $o \in \{\mathrm{ins}, \mathrm{sub}, \mathrm{del}\}$, the guided rate is

$$ \widetilde{\lambda}_i^o = \exp\left((1+w)\log \lambda_{i,\mathrm{audio}}^o - w \log \lambda_{i,\mathrm{drop}}^o\right). $$

The token distributions for insertion and substitution are combined analogously with the same guidance scale. In the experiments, $w=0.3$ is the best value for both encoder families considered.

CTC confidence guidance

A key practical refinement is to restrict edits to acoustically uncertain regions. The CTC model provides token-level confidence scores by averaging the posterior over the aligned frames for each token $\hat{y}_i$:

$$ c_i = \frac{1}{|\mathcal{R}_i|} \sum_{t \in \mathcal{R}_i} p_{\mathrm{ctc}}(\hat{y}_i \mid h_t), $$

where $\mathcal{R}_i$ is the set of repeated frames aligned to token $\hat{y}_i$. Boundary confidence for insertions is then approximated by

$$ b_j = \begin{cases} c_1, & j = 0, \\ \min(c_j, c_{j+1}), & 0 < j < N, \\ c_N, & j = N. \end{cases} $$

For the first refinement step, confidence can be read directly from the greedy CTC alignment. Once the hypothesis changes, the paper recomputes a forced alignment of the current hypothesis with the original CTC posteriors, using that forced alignment for later confidence estimates. An edit proposal is accepted only if both criteria are met: its event probability exceeds $0.1$ and the CTC confidence at the targeted token or boundary is below $0.7$. The paper reports that a greedy-to-forced schedule for confidence estimation is best.

Experimental setup

The evaluation is performed on LibriSpeech, using the standard 960-hour training set for ASR training and the text-only LibriSpeech data for decoder pretraining. The metric is word error rate (WER) on the standard development and test sets. The implementation is built in ESPnet.

Two acoustic encoder configurations are considered in the main experiments: an ESPnet Transformer encoder trained from scratch on LibriSpeech and a frozen pretrained Whisper encoder. The paper trains the full system for 35 epochs and averages the last 5 checkpoints for evaluation. The ablation tables report model scaling with small, medium, and large settings; the large setting corresponds to 403M total parameters with an ESPnet encoder, while the test table also reports frozen Whisper Base and Whisper Medium variants.

The final comparison table places the method against several non-autoregressive and diffusion-based ASR systems under comparable settings, including TransFusion, CASS-NAT, FDDM, Whisper-LLaDA, Drax, dLLM-ASR, and MDM-ASR.

Ablation study

Model capacity

Increasing the size of the hybrid CTC/Edit Flow system improves WER consistently on LibriSpeech development sets. The table below uses the ESPnet encoder, disables edit-aware pretraining, and evaluates with two refinement steps but without audio CFG or CTC confidence guidance.

The trend is monotonic: larger models do better, and Edit Flow consistently improves over the joint CTC baseline even before adding the stronger inference-time controls.

Edit-aware pretraining

Pretraining on text-only corrupted transcripts helps, but only when the corruption strength is well matched to the residual error profile of the CTC seed. The paper sets $\rho_{\mathrm{del}} = \rho_{\mathrm{sub}} = \rho_{\mathrm{ins}} = \rho$ in this experiment.

The best noise level is $\rho=0.03$. The paper’s interpretation is that the pretraining corruption should be small enough to resemble the CTC-seed editing regime, but large enough to teach the decoder to correct realistic insertions, deletions, and substitutions.

Inference steps and confidence guidance

With the pretrained large decoder, the number of refinement steps matters most at the beginning of the denoising process. The reported results show that two steps are best, while four steps slightly degrade performance. Confidence guidance also matters: using greedy alignment confidence in the first step and forced-alignment confidence afterwards is better than using one alignment strategy exclusively or turning guidance off.

The paper concludes from this ablation that the best operational recipe is: two edit steps, a greedy-to-forced confidence schedule, deterministic proposal acceptance, and a thresholded edit policy.

Audio classifier-free guidance

Guidance scale also matters. The best scale is $w=0.3$ across both encoder families tested. Larger values begin to hurt performance, suggesting that overemphasizing the guidance term can become too aggressive.

Final test results

The final test table reports the method with the hyperparameters selected on the development set. The main pattern is consistent across encoders: Edit Flow improves the first-pass CTC output, and edit-aware pretraining adds another gain. The strongest result in the paper is obtained with the Whisper Medium encoder and a pretrained Edit Flow decoder.

Relative to the CTC seed, the paper highlights reductions from 3.5/7.9 to 2.6/5.8 with the ESPnet encoder and from 2.6/6.1 to 2.0/4.7 with the Whisper Medium encoder. The authors also note that their best system is competitive with recent non-autoregressive and diffusion-based ASR methods while using only two refinement steps, moderate model size, and no reliance on very large external supervised initialization or multilingual pretraining corpora.

Qualitative behavior

The paper includes examples showing that two iterations are often enough to recover the reference transcript exactly. The first edit step usually performs coarse corrections such as fixing obvious substitutions or inserting missing high-confidence words, while the second step handles smaller residual edits. This behavior is consistent with the reported ablations: most of the gain comes from the first two parallel passes, and additional steps offer diminishing returns.

Limitations and scope

• Dataset scope: the experimental validation is limited to LibriSpeech, a read-English benchmark. The paper does not report multilingual or cross-domain results, although multilingual ASR is mentioned as future work.
• Dependence on the first-pass CTC hypothesis: the approach assumes that the greedy CTC seed is informative enough for edit refinement. The need for confidence gating and the sensitivity to pretraining corruption level both suggest that the method benefits from a reasonably accurate initial CTC pass.
• Reported efficiency: the paper clearly motivates two-step decoding and parallel edits, but it does not present a detailed latency, throughput, or memory benchmark beyond step count and model size.
• Comparative setting caveat: the paper notes that some competing systems use much larger external resources or stronger initialization, so raw WER comparisons should be interpreted with the training-data and initialization differences in mind.

Bottom line

The paper’s main technical contribution is a CTC-seeded, variable-length edit refinement framework for non-autoregressive ASR. By starting from a greedy CTC transcript and learning to apply token-level edits under a diffusion-style objective, the method avoids full transcript generation from noise and focuses computation on residual errors. The combination of edit-aware pretraining, audio classifier-free guidance, and confidence-based edit gating is empirically effective, and the best reported setting reaches 2.6/5.8 WER on LibriSpeech test-clean/test-other with the ESPnet encoder and 2.0/4.7 with the Whisper Medium encoder.