Akapulu Labs logo Akapulu Labs Research

UniSAE

UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling

UniSAE — method overview

UniSAE unifies speaker, emotion, and content editing in one framework with control from sub-phoneme to word level. It introduces Discrete Phonetic PosteriorGrams that factorize speech into discrete tokens, enabling direct phonetic manipulation—a capability absent in prior work treating these attributes separately.

  • speech-to-speech
  • tts
  • voice-conversion
  • emotion
  • prosody
  • asr

Authors: Chuanbo Zhu, Wuyou Zhou, Rongxiu Zhong, Shilei Zhang, Kun Qian, Yike Guo, Wei Xue

Categories: cs.SD, cs.AI, cs.CL, eess.AS

Published 2026-06-30 · Updated 2026-06-30

Links

Paper & demos

Impact

Abstract

Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility. We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture. UniSAE introduces a Discrete Phonetic PosteriorGram (DPPG) representation that factorizes speech content into discrete tokens encoding phoneme identity, pronunciation variants, and duration, enabling direct phoneme- and sub-phoneme-level editing. For higher-level modifications, an autoregressive content transformer predicts edited DPPG sequences for word-level content editing. The edited sequences are rendered into speech by a diffusion-based acoustic decoder, conditioned on disentangled speaker and emotion representations. Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.

Introduction and Motivation

Speech editing systems have traditionally focused on a narrow slice of the editing space: replacing words or short phrases in an utterance while keeping the rest of the recording intact. Recent neural approaches based on self-supervised learning (SSL) tokens or neural codec representations have made this kind of content editing increasingly realistic. At the same time, voice conversion (VC) and emotional voice conversion (EVC) have matured as separate research threads, giving practitioners the ability to change who is speaking or how they feel. Yet no single framework has bridged these capabilities into a unified, composable editing pipeline.

The paper argues that practical editing scenarios — such as a sound director correcting a pronunciation, adjusting speaker timbre, or modifying emotional expression — require simultaneous and independent control of linguistic content, speaker identity, and emotion. Two concrete technical challenges stand in the way:

  • Granularity of content editing. Existing SSL and codec representations do not explicitly encode phoneme boundaries, pronunciation variants, or durations. Phoneme-level modifications can be achieved only indirectly through sequence generation, and sub-phoneme editing (pronunciation variants, duration adjustment) is essentially unsupported.
  • Reliable multi-attribute disentanglement. Inpainting-based frameworks couple content, speaker identity, and emotion inside a shared latent space, making independent manipulation difficult. Compounding the problem, large-scale emotional speech corpora with diverse speakers and matched linguistic content are scarce, and speaker identity and emotion are intrinsically entangled in the acoustic signal.

To address these gaps, the authors propose UniSAE (Unified Speech Attribute Editing), a single architecture that supports composable editing of speaker identity, emotion, and content from the sub-phoneme level up to the word level. The paper also introduces UniEditCorpus, a 580-hour synthetic emotional speech corpus constructed via Manifold Distillation to provide counterfactual supervision for disentanglement learning.

Problem Formulation: Speech Attribute Editing

The paper extends conventional speech editing to a broader task called Speech Attribute Editing (SAE). An utterance is represented as a triplet $(c, s, e)$, where $c$ denotes linguistic content, $s$ denotes speaker identity, and $e$ denotes emotion. Any SAE operation is a state transition:

$$( c,\, s,\, e ) \;\rightarrow\; ( c',\, s',\, e' )$$

where one or more attributes are changed while the rest are preserved. The paper defines a taxonomy of primitive editing operations:

  • Speaker editing: $(c,s,e) \rightarrow (c, s', e)$
  • Emotion editing: $(c,s,e) \rightarrow (c, s, e')$
  • Word-level content editing: $(c,s,e) \rightarrow (c'_w, s, e)$
  • Phoneme-level content editing: $(c_{p,v,d}, s, e) \rightarrow (c_{p',v,d}, s, e)$ — changes phoneme identity $p$ while preserving variant $v$ and duration $d$
  • Sub-phoneme variant editing: $(c_{p,v,d}, s, e) \rightarrow (c_{p,v',d}, s, e)$ — changes pronunciation variant $v$ while keeping identity and duration
  • Sub-phoneme duration editing: $(c_{p,v,d}, s, e) \rightarrow (c_{p,v,d'}, s, e)$ — adjusts duration $d$
  • Joint editing: $(c,s,e) \rightarrow (c', s', e')$ — simultaneous modification of any subset of attributes

This formulation subsumes all prior task-specific systems as special cases and provides a clean interface for composing operations.

UniEditCorpus: Manifold Distillation

Learning to disentangle speaker and emotion is fundamentally limited by data scarcity. The authors construct UniEditCorpus — a fully crossed content × speaker × emotion corpus — through a process they call Manifold Distillation. The key idea is to use a zero-shot TTS model to synthesize all speaker–emotion combinations for a fixed set of text contents, generating abundant counterfactual training pairs (e.g., identical content from the same speaker expressed with different emotions, or identical content and emotion from different speakers).

Construction pipeline:

  1. Emotional speech prompts are collected from six public corpora: ESD, JLcorpus, RAVDESS, MEAD, TESS, and EMNS. For datasets with emotion intensity labels (RAVDESS, MEAD), only high-intensity utterances are retained for clarity. The prompt pool covers 87 speakers (46 male, 41 female) and five emotion categories: neutral, happy, sad, angry, and surprised.
  2. 2,000 text utterances from VCTK are used as the content pool.
  3. A zero-shot TTS model (IndexTTS) synthesizes every speaker–emotion combination, yielding 870,000 utterances (approximately 581 hours).
  4. Automatic quality filtering: samples are regenerated using an emotion recognition model until the predicted emotion matches the target label, with up to 50 retries per sample.

Corpus statistics and quality: The final corpus achieves a Character Error Rate (CER) of 1.16%, speaker consistency (SpkSim) of 0.759, emotion consistency (EmoSim) of 0.802, and an average UTMOS score of 3.785. These numbers are comparable to the human-recorded ESD dataset, confirming that the synthetic corpus approaches real-speech quality.

Data splits: 1,800 unique text contents are used for training (693,000 utterances, 77 speakers, ~462 hours); 100 for validation (88,500 utterances, 82 speakers including 5 held-out unseen speakers, ~58 hours); 100 for test (88,500 utterances, 82 speakers, ~61 hours). The five unseen speakers appear only in validation and test to assess generalization to out-of-distribution identities.

UniSAE Architecture Overview

Overview of the proposed UniSAE framework, Content Transformer training and inference.
Overview of the proposed UniSAE framework, Content Transformer training and inference.

UniSAE adopts a two-stage generation pipeline:

  1. Content sequence modeling: Given source speech and an editing request, the DPPG encoder and Content Transformer produce an edited sequence of discrete phonetic tokens.
  2. Acoustic rendering: An Acoustic Diffusion Decoder synthesizes mel-spectrograms conditioned on the edited token sequence and disentangled speaker and emotion embeddings.

The four key components are: (1) the Discrete Phonetic PosteriorGram (DPPG) representation, (2) the Content Transformer, (3) speaker and emotion encoders, and (4) the Acoustic Diffusion Decoder.

Component 1: Discrete Phonetic PosteriorGram (DPPG)

DPPG construction and phoneme, sub-phoneme editing operations.
DPPG construction and phoneme, sub-phoneme editing operations.

The DPPG is the central content representation in UniSAE, designed to explicitly factorize phonetic content into three independent components: phoneme identity ($p$), pronunciation variant ($v$), and duration ($d$).

Construction

Frame-level continuous Phonetic PosteriorGrams (PPGs) are extracted from source speech using a pretrained PPG encoder. Each frame is a posterior distribution over 40 phoneme categories from CMUDict. These continuous PPGs are then discretized via per-phoneme MiniBatch K-Means clustering:

$$I(k) = \sum_{i=1}^{n} \min_{\mu_j \in \mathcal{C}} \| x_i - \mu_j \|_2^2$$

where $x_i$ is a PPG frame and $\mu_j$ is a cluster centroid. The number of clusters per phoneme is chosen automatically using the elbow criterion (Kneedle algorithm). Given a PPG frame $x$ associated with phoneme $p$, the variant index is:

$$v = \arg\min_j \| x - \mu_j^{(p)} \|_2^2$$

Each frame is discretized into a token <p_v>. The total vocabulary size is 203 duration-free DPPG (DF-DPPG) tokens across all phoneme categories. The number of sub-phoneme variants per phoneme ranges from 2 (e.g., /ae/, /ah/, /ih/, /k/, /l/, /r/) up to 8 (e.g., /hh/, /n/), reflecting each phoneme's intrinsic acoustic variability.

Dual DPPG Formats

Two forms of DPPG are used:

  • Duration-Free DPPG (DF-DPPG), <p_v>: Frame-level representation; used as input to the Acoustic Diffusion Decoder.
  • Duration-Aware DPPG (DA-DPPG), <p_v_d>: Obtained by merging consecutive identical DF-DPPG frames and recording their duration count; used by the Content Transformer for efficient autoregressive sequence modeling. Conversion between the two forms is fully reversible via merge/demerge operations.

An illustrative example: for phoneme /n/, the four DPPG variants encode qualitatively different realizations — the canonical alveolar nasal (<n_0>, 98.9% /n/ posterior), a vowel-coarticulated variant (<n_1>, 59.6% /n/ and 35.0% /ah/), a voiced stop-influenced nasal (<n_2>, 60.6% /n/ and 32.8% /d/), and a pause-adjacent variant (<n_3>, 60.9% /n/ and 32.4% silence).

Editing Operations

The DPPG factorization directly supports the editing taxonomy:

  • Phoneme-level editing: replace $p$ in one or more tokens
  • Sub-phoneme variant editing: replace $v$ while keeping $p$ and $d$
  • Duration editing: modify $d$ while keeping $p$ and $v$

Component 2: Content Transformer

The Content Transformer is a GPT-2-style decoder-only Transformer that autoregressively models DA-DPPG token sequences to perform word-level content editing and indirect phoneme modification. It uses a causal masking and rearrangement strategy inspired by VoiceCraft to support arbitrary-span editing under left-to-right decoding.

The model has 12 Transformer decoder layers and operates over a vocabulary of 5,440 tokens (DA-DPPG tokens plus canonical phoneme tokens).

Training

Masked spans in the DA-DPPG sequence are selected using a mixed masking strategy:

  • With probability $\lambda$: word-level masking aligned to word boundaries; canonical phonemes from G2P are used as the conditioning sequence.
  • With probability $1-\lambda$: token-level masking of random contiguous spans; only phoneme identity is retained in the conditioning sequence (variant and duration fields are stripped: <p_v_d> → <p>).

Selected $N$ spans are replaced by mask tokens (<M0> … <Mn>), and their corresponding canonical phoneme sequences are appended to the input after mask identifiers. Correspondence between masked DA-DPPG spans and canonical phonemes is determined by a DTW-based forced alignment algorithm (described below). An end-of-utterance token <EOU> separates the conditioning sequence from the prediction target, after which the full target sequence is concatenated.

The training objective is a causal language modeling loss restricted to tokens after <EOU>:

$$\mathcal{L}_{\text{CLM}} = - \sum_{t \in \mathcal{T}} \log P\!\left(Y_t \;\middle|\; \mathbf{Y}_{where $\mathcal{T}$ is the set of token positions after <EOU>. The model attends to the entire sequence but loss is computed only on the target portion.

Inference

  • Word insertion / substitution: mask tokens are placed at target positions; canonical phoneme sequences of the new content are appended; the model generates the corresponding DA-DPPG tokens.
  • Word deletion: aligned DA-DPPG tokens are removed directly without any model prediction step.

DTW-Based Phoneme-to-DPPG Forced Alignment

To construct conditioning sequences during training, the paper proposes a lightweight phoneme-to-DPPG forced alignment algorithm. Each DA-DPPG token <p_v_d> is reduced to its phoneme identity by discarding variant and duration. Direct phoneme matching is unreliable due to coarticulation and pronunciation variability, so a Phoneme-Aware Matching Cost (PAMC) is defined:

$$d_{\text{pamc}}(p, q) = \begin{cases} 0 & p = q \\ 0.25 & p, q \in \mathcal{G}_k \\ 0.5 & p, q \in \mathcal{V} \\ 0.75 & p, q \in \mathcal{C} \\ 1.0 & p \in \mathcal{V},\, q \in \mathcal{C} \text{ or vice versa} \\ 1.5 & p = \texttt{sil} \text{ or } q = \texttt{sil} \end{cases}$$

where $\mathcal{G}_k$ denotes a fine-grained articulatory phoneme group (e.g., {/iy/, /ih/}, {/t/, /d/}, {/s/, /z/}, {/n/, /ng/}), $\mathcal{V}$ is the set of vowels, and $\mathcal{C}$ is the set of consonants. Monotonic, forced DTW is then performed using PAMC as the local cost. After backtracking, each canonical phoneme is assigned to a span of DA-DPPG tokens; spans are aggregated by word boundaries to produce the word-to-DPPG-span mapping used for training.

Component 3: Speaker and Emotion Disentanglement

A key advantage of UniEditCorpus is that every utterance is observed under multiple speaker–emotion combinations, providing explicit counterfactual supervision. The authors adapt the GE2E loss to a dual-attribute setting. Each mini-batch contains $S \times E \times K$ utterances, where $K$ samples are drawn for every speaker–emotion pair.

Speaker and emotion encoders are optimized separately using the same batch. Given normalized embeddings $\mathbf{h}$ and class centroids $\mathbf{c}$, the loss for a target attribute (speaker or emotion) is:

$$\mathcal{L}_{\text{attr}} = - \sum_{s,e,k} \log \frac{\exp\!\left(w \cdot \cos(\mathbf{h}, \mathbf{c}_{\text{true}}) + b\right)}{\sum_{j=1}^{N} \exp\!\left(w \cdot \cos(\mathbf{h}, \mathbf{c}_j) + b\right)}$$

where $w$ and $b$ are learnable scaling parameters and $N$ is the number of attribute classes. The speaker encoder is trained with $\mathcal{L}_{\text{spk}}$ and the emotion encoder with $\mathcal{L}_{\text{emo}}$, encouraging attribute-discriminative embeddings while remaining invariant to the non-target attribute.

Both encoders are 4-layer Transformers operating on wav2vec 2.0 features and producing 256-dimensional embeddings.

Component 4: Acoustic Diffusion Decoder

The Acoustic Diffusion Decoder is a diffusion probabilistic model (DPM) with velocity parameterization ($v$-prediction). Before decoding, DA-DPPG tokens <p_v_d> are expanded back to frame-level DF-DPPG sequences by repeating each token according to its duration field. The denoising network $f_\theta$ predicts the velocity $\mathbf{v}_t$ at diffusion timestep $t$, conditioned on:

  • DF-DPPG embeddings (phonetic content)
  • Speaker embeddings (from the speaker encoder)
  • Emotion embeddings (from the emotion encoder)

The architecture is a U-Net diffusion model that generates 24 kHz mel-spectrograms. A BigVGAN vocoder converts the mel-spectrograms to waveforms. This two-stage design cleanly separates content editing (handled by the Content Transformer over compact DA-DPPG tokens) from paralinguistic rendering (handled by the diffusion decoder).

Experiments

Datasets and Benchmarks

  • Training data: LibriTTS-R and UniEditCorpus for the DPPG tokenizer and Content Transformer; UniEditCorpus additionally for the speaker/emotion encoders and Acoustic Diffusion Decoder.
  • Evaluation — speaker/emotion editing: UniEditCorpus test set (10 seen + 5 unseen speakers) and ESD test set (5 seen + 2 unseen speakers), each with 1,000 utterances (50 used for subjective evaluation).
  • Evaluation — word-level content editing: ESDEdit, a new benchmark derived from 100 ESD utterances with single-word insertion, deletion, and substitution operations.
  • Evaluation — phoneme-level editing: 8 phoneme substitution pairs × 20 utterances each = 160 utterances; covers nasals, liquids, fricatives, stops, and vowels.
  • Evaluation — sub-phoneme editing: 20 utterances with the <n_0> → <n_1> variant swap.
  • Evaluation — joint editing: ESDEdit utterances paired with speaker and emotion prompts from 2 seen speakers across 5 emotions.

Baselines

  • Speaker/emotion editing: EmoConv-Diff (diffusion-based EVC) and ZEST (HiFi-GAN-based EVC).
  • Word-level content editing: VoiceCraft and SSR-Speech (both codec-token inpainting systems).
  • Phoneme/sub-phoneme and joint editing are evaluated only for UniSAE, as no prior framework supports these capabilities.

Evaluation Metrics

  • CER (Character Error Rate): content preservation, measured by a pretrained wav2vec 2.0 ASR model.
  • SpkSim: cosine similarity of Resemblyzer speaker embeddings between output and speaker prompt.
  • EmoSim: cosine similarity of emotion2vec+ embeddings between output and emotion prompt.
  • UTMOS: predicted Mean Opinion Score for naturalness.
  • nMOS, sMOS, eMOS: subjective naturalness, speaker similarity, and emotion similarity MOS (rated on a 5-point scale).
  • TPD / SPD / OPD: Target / Source / Other Phoneme Detection rates for phoneme-level editing evaluation (sum to 100%).

Results

Speaker and Emotion Editing

Results on UniEditCorpus and ESD are summarized in the main results table. Key findings:

  • Content preservation: UniSAE achieves CER comparable to EmoConv-Diff on both datasets. ZEST attains the lowest CER (relying on direct waveform filtering via HiFi-GAN without discretization), while UniSAE's DPPG discretization introduces little information loss.
  • Speaker and emotion control: UniSAE consistently outperforms both EmoConv-Diff and ZEST on SpkSim, EmoSim, and all naturalness scores on UniEditCorpus under both seen and unseen speaker settings. On UniEditCorpus seen speakers: SpkSim 0.710 vs. 0.707 / 0.555, EmoSim 0.773 vs. 0.721 / 0.758, nMOS 4.053 vs. 3.895 / 2.579, sMOS 3.474 vs. 3.000 / 2.263, eMOS 3.947 vs. 2.842 / 3.684.
  • ESD generalization: On ESD seen speakers, UniSAE maintains competitive performance (CER 8.447, SpkSim 0.709, eMOS 4.158). Under ESD unseen speakers, SpkSim drops to 0.605 (vs. EmoConv-Diff's 0.648), revealing a remaining challenge in speaker generalization to out-of-domain identities.

Word-Level Content Editing

On ESDEdit, VoiceCraft achieves the best CER (6.556), SpkSim (0.894), EmoSim (0.933), and UTMOS (3.615), followed closely by SSR-Speech. UniSAE scores CER 6.882, SpkSim 0.738, EmoSim 0.811, UTMOS 3.485 — slightly below specialized inpainting systems. The paper attributes this gap to the fundamental design difference: VoiceCraft and SSR-Speech directly preserve source audio attributes via codec inpainting, while UniSAE explicitly separates content from speaker/emotion and renders from scratch. The competitive performance nonetheless validates that DPPG-based phonetic modeling is effective for word-level editing while providing substantially richer controllability.

Phoneme- and Sub-Phoneme-Level Editing

Mel spectrograms of the word nine under phoneme, sub-phoneme, and duration editing. (a) Original utterance; (b) phoneme editing; (c) sub-phoneme editing; (d) duration editing.
Mel spectrograms of the word "nine" under phoneme, sub-phoneme, and duration editing. (a) Original utterance with <n_0>; (b) phoneme editing (<n_0> → <l_0>); (c) sub-phoneme editing (<n_0> → <n_1>); (d) duration editing by lengthening /aa/ from 12 frames to 17 frames.

Phoneme identity editing achieves an average TPD of 83.75% with SPD 7.50% and OPD 8.75% across eight phoneme substitution pairs spanning diverse phoneme categories. Individual pair results:

  • /n/ → /l/: TPD 100%, SPD 0%, OPD 0%
  • /m/ → /n/: TPD 95%, SPD 5%, OPD 0%
  • /t/ → /d/: TPD 100%, SPD 0%, OPD 0%
  • /s/ → /sh/: TPD 85%, SPD 5%, OPD 10%
  • /k/ → /g/: TPD 85%, SPD 10%, OPD 5%
  • /aa/ → /ey/: TPD 75%, SPD 10%, OPD 15%
  • /r/ → /l/: TPD 70%, SPD 20%, OPD 10%
  • /p/ → /b/: TPD 60%, SPD 10%, OPD 30% (failures tend to produce an ambiguous realization between /p/, /b/, and silence)

The authors note that editing success depends on both the phoneme pair and lexical plausibility. Pairs that produce valid words (e.g., "nine" → "line", "tom" → "dom") succeed more often than pairs producing non-words (e.g., "please" → "blease"). The /p/ → /b/ case illustrates that even when phoneme recognition is ambiguous, the edited speech often remains intelligible due to lexical context.

Sub-phoneme variant editing: Replacing <n_0> with <n_1> (a vowel-coarticulated variant of /n/) results in 45% SPD and 55% OPD — the canonical /n/ realization is suppressed, with outputs predominantly recognized as vowels. This is consistent with enhanced vowel coarticulation rather than a phoneme identity change, demonstrating that DPPG variants encode context-dependent phonetic realizations while preserving the underlying phoneme category.

Mel spectrogram analysis (Figure above) visualizes the effects of all three editing modes on the word "nine":

  • Phoneme editing <n_0> → <l_0>: changes initial consonant from /n/ to /l/ and shifts the following vowel from /aa/ to a diphthong-like /ay ah/ sequence — globally consistent pronunciation change.
  • Variant editing <n_0> → <n_1>: weakens the nasal onset and advances the vowel transition, consistent with enhanced coarticulation.
  • Duration editing: lengthens /aa/ from 12 to 17 frames while preserving surrounding phonetic content.

Joint Speech Attribute Editing

Joint editing results on ESDEdit demonstrate that combining content, speaker, and emotion editing causes little degradation compared with individual tasks:

  • Content + Speaker: CER 7.565, SpkSim 0.714, EmoSim 0.762, UTMOS 3.426
  • Content + Emotion: CER 6.562, SpkSim 0.704, EmoSim 0.789, UTMOS 3.421
  • Content + Speaker + Emotion: CER 7.432, SpkSim 0.729, EmoSim 0.824, UTMOS 3.582

The three-attribute setting performs comparably to two-attribute editing, confirming that the disentanglement design enables reliable composable control without substantial cross-attribute interference.

Ablation Study

Two ablation conditions are evaluated on UniEditCorpus seen-speaker setting:

  1. DPPG → continuous PPG: Replacing the discrete DPPG with continuous PPGs yields CER 3.577 (slightly better than UniSAE's 4.015), SpkSim 0.722, EmoSim 0.760, UTMOS 3.508. Differences are marginal, confirming that discretization introduces limited information loss. Crucially, continuous PPGs cannot support explicit sub-phoneme or phoneme-level token manipulation.
  2. Disentangled embeddings → off-the-shelf (OTS) embeddings: Replacing the proposed dual-GE2E disentangled speaker/emotion embeddings with off-the-shelf d-vector and emotion2vec+ representations causes a dramatic degradation: SpkSim drops from 0.710 to 0.691, EmoSim drops from 0.773 to 0.582, UTMOS drops from 3.499 to 3.120. The paper attributes this to residual speaker–emotion entanglement in pretrained representations, which encourages the model to exploit shortcut cues rather than genuinely conditioning on the intended attribute.

Comparison with Prior Work

The paper positions UniSAE against four representative systems in a capability comparison table:

  • ZEST: supports speaker and emotion editing; no content editing at any granularity.
  • EmoConv-Diff: supports speaker and emotion editing; no content editing.
  • VoiceCraft: supports word-level content editing; phoneme-level only indirectly (no explicit control); no sub-phoneme editing; no speaker or emotion editing.
  • SSR-Speech: same capability profile as VoiceCraft.
  • UniSAE: all five capabilities — speaker, emotion, word, phoneme, and sub-phoneme editing — within a single unified framework.

Implementation Details

  • DPPG encoder: Pretrained PPG encoder from Churchwell et al. (2024); 40 phoneme categories (CMUDict); 203 DF-DPPG tokens total after per-phoneme K-Means clustering.
  • Content Transformer: 12-layer GPT-2-style decoder-only Transformer; vocabulary of 5,440 DA-DPPG and phoneme tokens.
  • Speaker/emotion encoders: 4-layer Transformers on wav2vec 2.0 features; 256-dimensional output embeddings.
  • Acoustic decoder: U-Net DPM with $v$-prediction generating 24 kHz mel-spectrograms; BigVGAN vocoder for waveform synthesis.

Stated Contributions and Novelty

  1. Unified framework: UniSAE is the first system to jointly support speaker editing, emotion editing, word-level content editing, phoneme-level content editing, and sub-phoneme editing (variant and duration) within a single architecture.
  2. DPPG representation: A novel discrete content representation that explicitly factorizes speech into phoneme identity, pronunciation variant, and duration — enabling direct token-level manipulation at multiple linguistic granularities while reducing sequence length for efficient autoregressive modeling.
  3. Manifold Distillation and UniEditCorpus: A principled methodology for constructing a large-scale, fully crossed content × speaker × emotion corpus using zero-shot TTS, providing the counterfactual supervision needed for robust speaker–emotion disentanglement.
  4. Dual-attribute GE2E disentanglement: A training scheme that adapts the GE2E loss to simultaneously train speaker and emotion encoders with independent attribute-discriminative objectives on a shared batch, exploiting the counterfactual structure of UniEditCorpus.
  5. DTW-based phoneme-to-DPPG alignment: A phoneme-aware forced alignment algorithm with PAMC that handles coarticulation and pronunciation variability, enabling robust construction of conditioning sequences for the Content Transformer.

Limitations

The paper is explicit about several remaining challenges:

  • Unseen speaker generalization: Performance on ESD unseen speakers degrades compared to seen speakers, particularly for SpkSim (0.605 vs. 0.709 on ESD), indicating that speaker representations do not fully generalize to out-of-domain identities.
  • Word-level editing quality: UniSAE's CER and UTMOS scores for word-level editing are slightly below those of specialized inpainting systems (VoiceCraft, SSR-Speech), reflecting the trade-off between generality and task-specific optimization.
  • Phoneme editing difficulty: Some phoneme substitution pairs (e.g., /p/ → /b/) produce ambiguous realizations, and editing success depends on both the phoneme pair and the lexical plausibility of the resulting word. The authors acknowledge that a more systematic investigation of these contextual factors is left for future work.
  • Synthetic training data: UniEditCorpus is entirely synthetic; while its quality is validated quantitatively, the domain gap to natural human speech may still limit performance, particularly for out-of-domain emotion categories or speaking styles not covered by the prompt pool.

Conclusion

UniSAE presents a principled and practical unification of speech editing capabilities. By introducing the DPPG representation — which explicitly disentangles phoneme identity, pronunciation variant, and duration — and combining it with a masked autoregressive Content Transformer, disentangled speaker/emotion encoders trained via dual-attribute GE2E loss, and a diffusion-based acoustic decoder, the framework enables composable manipulation of all three fundamental speech attributes. The UniEditCorpus, constructed through Manifold Distillation, provides the large-scale counterfactual supervision required for robust disentanglement that would be impossible to obtain from naturally recorded corpora alone. Experimental results confirm state-of-the-art speaker and emotion controllability, competitive word-level editing performance, and the first demonstrated system-level support for explicit phoneme- and sub-phoneme-level editing.