CORTIS
Continual Speaker Identity Unlearning with Minimal Interference
CORTIS enables continual speaker identity unlearning in zero-shot text-to-speech without reviving previously forgotten speakers. It combines contrastive Fisher-information masking with orthogonal projection to handle sequential privacy-removal requests without retaining past speaker data.
Links
Paper & demos
Abstract
Machine unlearning removes designated concepts or knowledge from pre-trained models. Recent work has extended this paradigm to speaker identity unlearning in zero-shot text-to-speech (ZS-TTS), the task of selectively erasing a model's ability to replicate a speaker's voice. Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present Cumulative ORThogonal Identity Suppression (CORTIS), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. With VoiceBox, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .
Introduction and Motivation
Modern Zero-Shot Text-to-Speech (ZS-TTS) systems can clone a target speaker's voice from only a few seconds of reference audio. This capability poses serious privacy risks: a short clip from a phone call, podcast, or voicemail is sufficient to synthesize convincing speech attributed to someone who never produced it. Data-protection regulations such as the GDPR and CCPA grant users the Right to Be Forgotten (RTBF), creating a concrete demand for speaker identity unlearning in deployed ZS-TTS systems.
Machine unlearning provides the algorithmic framing for removing designated knowledge from pre-trained models. Prior work on speaker identity unlearning in ZS-TTS (specifically the "Do Not Mimic" framework) has shown that a model can be induced to produce random or non-identifying output when prompted with a forgotten speaker's reference. However, these methods assume all unlearning requests arrive simultaneously and are handled in a single training pass — an assumption that is fundamentally at odds with real-world deployment, where removal requests from different users arrive sequentially over time.
The paper identifies a critical failure mode when existing methods are applied sequentially: unlearning a new speaker causes measurable recovery of speakers that were already unlearned — a phenomenon the authors call catastrophic re-learning. This is the inverse of catastrophic forgetting in continual learning: instead of the model forgetting a useful skill when learning something new, it relearns a previously erased capability when performing a new erasure task.
Furthermore, there is a fundamental data-retention paradox in the sequential setting. If a provider batches requests and waits to process them all simultaneously, it must retain each requester's data until the batch run — directly contradicting the right being exercised. The natural alternative (processing each request on arrival and discarding data immediately) is precisely what existing methods cannot handle.
Problem Formulation
The paper formalizes continual speaker identity unlearning under two hard constraints:
- (C1) Sequential arrival: Unlearning requests arrive one at a time. At step $i$, a request to forget speakers $f_i$ produces a cumulative forget set $\mathcal{F}_i = \mathcal{F}_{i-1} \cup f_i$, with the system updating $\theta_{i-1} \to \theta_i$ without knowledge of future requests.
- (C2) Forget non-retention: Once step $i$ is processed, the forget data $\mathcal{D}^{f_i}$ is discarded and cannot be accessed at any subsequent step. Only the current model state $\theta_i$ and remain data $\mathcal{D}^{\mathcal{R}_i}$ can persist.
The objective at each step $i$ is:
$$\theta_i(x^r, y) \approx \hat{x}^{spk=r}_{y} \quad \forall r \in \mathcal{R}_i, \quad \text{and} \quad \theta_i(x^f, y) \not\approx \hat{x}^{spk=f}_{y} \quad \forall f \in \mathcal{F}_i$$That is, the model must preserve zero-shot synthesis quality for all remain speakers while refusing to replicate any speaker in the cumulative forget set $\mathcal{F}_i$ — not merely the most recently requested one. The challenge under (C1) and (C2) is maintaining the second condition for $\mathcal{F}_{i-1}$ while unlearning $f_i$, without any access to $\mathcal{D}^{\mathcal{F}_{i-1}}$.
Why Catastrophic Re-learning Happens
In ZS-TTS systems, the retain loss that protects model utility on subsequent requests provides no supervision on parameters previously responsible for forgetting earlier speakers, leaving those parameters free to drift back toward their original values. Because ZS-TTS models are designed to be highly generalizable across unseen speakers, even simple regularization that protects retain-set utility (without an active forget loss on previously unlearned speakers) is sufficient to revive those identities during the next unlearning sequence.
One might attempt to re-introduce an active forget loss on every previously unlearned speaker at each new request, but this approach has two compounding problems: (1) it is no longer meaningful continual unlearning — the training cost grows with each step ($1K \to 2K \to 3K \to \cdots$ steps), and (2) retaining a forget loss on previous speakers requires retaining their data, directly violating RTBF compliance and creating the data retention paradox.
Method: CORTIS
The paper proposes CORTIS (Cumulative ORThogonal Identity Suppression), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines two complementary mechanisms: contrastive Fisher-information parameter masking and orthogonal projection against a cumulative unlearning subspace.
Component 1: Contrastive Parameter Localization
State-of-the-art ZS-TTS models exhibit highly entangled representations — components are not explicitly modularized to separate speaker identity from content generation. Without constraint, the optimizer can update parameters unrelated to speaker identity and, more critically, can overwrite parameters that drove successful unlearning on previous speakers.
For each unlearning sequence $i$, the diagonal Fisher Information Matrix $F_{f_i}$ of the forget loss on speaker $f_i$'s data is computed, and a contrastive saliency map is constructed:
$$\text{saliency}_i = \frac{F_{f_i} + \epsilon}{\max\!\left(F_{\mathcal{R}_{i}},\, F_{f_1},\, \dots,\, F_{f_{i-1}}\right) + \epsilon}$$The element-wise max in the denominator acts as a soft guard: any parameter important for retain quality or for any prior forget speaker is pushed toward the bottom of the saliency ranking. The top-$k\%$ of $\text{saliency}_i$ globally defines the trainable mask $M_i$; remaining parameters are frozen during updates at sequence $i$. In experiments, $k=30$ is used, freezing 70% of model parameters at each unlearning step.
This contrastive formulation serves multiple purposes: it guards against interference with previously unlearned speakers, protects the current speaker against being overwritten in future updates, and concentrates each unlearning sequence efficiently on the most relevant parameters.
Empirical Validation of Mask Disjointness
The authors verify empirically that per-speaker masks localize to largely disjoint subsets of model parameters. Off-diagonal Jaccard indices range from $0.049$ to $0.197$, confirming that the contrastive denominator successfully decorrelates each request's mask from prior ones.
Component 2: Orthogonal Projection on the Cumulative Forget Subspace
Parameter masking concentrates each unlearning step on a forget-relevant region, but it does not by itself constrain how parameters within that region are updated. Even when the mask successfully isolates speaker-identity-relevant weights, sequential updates within the masked region can still move along directions that revert prior speakers' unlearning.
After completing the unlearning run for speaker $f_i$, a rank-$R$ orthonormal basis $U_i$ is extracted to summarize the directions the optimizer used during that run, following the Gradient Projection Memory (GPM) recipe. Gradient snapshots are collected at fixed intervals throughout training, stacked, and a truncated SVD is computed; the top-$R$ left singular vectors with their singular values $\Sigma_i$ form the per-speaker artifact. To ensure each new basis captures only directions not already covered by prior speakers, the prior subspace is subtracted from each gradient snapshot before SVD, so $U_i$ is orthogonal to $U_1, \dots, U_{i-1}$ by construction.
Rather than directly accumulating $[U_1 \mid \cdots \mid U_i]$ (which would grow unboundedly), the authors maintain a fixed-rank merged basis. At sequence $i+1$, the energy-weighted column stack is formed:
$$\Phi_i = [U_1\Sigma_1 \mid \cdots \mid U_i\Sigma_i]$$where each column is scaled by its singular value to encode how much that direction was used during the corresponding speaker's run. The rank-$R_{\text{merge}}$ truncated SVD of $\Phi_i$ yields a merged basis $U_{ After every optimizer step, the weight delta $\delta$ within the trainable mask is projected onto the orthogonal complement of $U_{ With a 30% Fisher mask on a 24-layer transformer, the union of prior masks spans approximately 30M parameters. To keep the SVD merge tractable, the authors form the Gram matrix $G_i = M_i^\top M_i \in \mathbb{R}^{C \times C}$ where $C = \sum_{j=1}^{i} r_j$ is the total column count. For typical $r_j \leq 40$ and $i \leq 5$, $C \leq 200$, so $G_i$ fits in tens of kilobytes regardless of parameter dimension. The Gram matrix is accumulated by streaming row-chunks from CPU to GPU, keeping peak GPU memory bounded. All experiments use VoiceBox as the ZS-TTS backbone — a non-autoregressive model for speech generation that parameterizes a continuous-time transport from a Gaussian prior $p_0$ to the speech distribution $p_1$ via Conditional Flow Matching (CFM). A neural network with parameters $\theta$ predicts the conditional vector field $v_t(w, y, x_{\text{ctx}}; \theta)$ driving this transport, where: The trajectory satisfies: The acoustic generator is a Transformer with U-Net-style skip connections between symmetric layers, convolutional positional embeddings, and ALiBi attention biases. The configuration is: 24 layers, 16 attention heads, embedding dimension 1024, FFN inner dimension 4096. The duration predictor mirrors the acoustic model but is smaller (8 layers, 8 heads, embedding dimension 512, FFN dimension 2048). Mel-spectrograms are converted to waveforms with a Diffwave vocoder trained on the English subset of LibriHeavy. The base checkpoint is pretrained for 500K steps on LibriHeavy with Adam in mixed-precision FP16. The evaluation follows the sequential unlearning scenario formalized in the problem formulation. At each step $i$, only data from the current forget speaker $f_i$ is accessible; data for previously forgotten speakers $\mathcal{F}_{i-1}$ is discarded. The model at each sequence is initialized from the previous sequence ($\theta_{i-1}$). The five forget speakers are acoustically well-separated: pairwise cosine similarities between their WavLM-TDCNN speaker embeddings range from $-0.120$ to $0.233$, all well within the different-speaker regime (empirical same-speaker lower bound: $0.46$). Four baselines are evaluated: Interpretive bounds are calibrated against real-world speaker similarity distributions on LibriSpeech test-clean (200 randomly sampled pairs): Key finding: CORTIS is the only method that keeps every previously forgotten speaker forgotten across the full sequence while retaining competitive remain-set quality. After Request 3, all three forget-speaker similarities remain below $0.18$ (S-$f_1{=}0.172$, S-$f_2{=}0.148$, S-$f_3{=}0.124$), while S-R holds at $0.557$. Every baseline collapses on at least one axis. TGU (catastrophic re-learning): TGU achieves strong unlearning at Request 1 (S-$f_1{=}0.164$), but the previously unlearned speaker reverts sharply once a new request is processed: S-$f_1$ jumps to $0.612$ after Request 2 and remains at $0.603$ after Request 3. The same pattern recurs for $f_2$ (S-$f_2{=}0.546$ at Request 3). SGU (catastrophic forgetting of retain capability): SGU avoids re-learning — all forget-speaker similarities stay below $0.24$ — but retain-set quality degrades monotonically, with S-R falling from $0.479$ to $0.315$ over the three-request sequence. UN and SelFT (parameter drift insufficient): Both continual-learning regularizers succeed at limiting drift on the retain set (UN: S-R=$0.580$; SelFT: S-R=$0.548$ after Request 3), but neither prevents forgotten speakers from re-emerging. UN's S-$f_1$ rises to $0.638$ at Request 3; SelFT's reaches $0.553$. Limiting parameter drift relative to the previous checkpoint preserves remain-set performance but is not a sufficient mechanism for protecting previously forgotten identities. CORTIS is extended to 5 sequential requests. Three key observations: Detailed numerical results for Requests 4 and 5 (from the appendix): Masking by itself partially mitigates re-learning — it soft-excludes parameters important for prior speakers from the trainable set — but the effect is incomplete. Within the masked region, gradient updates can still move along directions previously used to forget $f_1, \dots, f_{i-1}$. Without projection, S-$f_1$ and S-$f_2$ revert to $0.334$ and $0.397$ at Request 3. This demonstrates that parameter-level localization and direction-level protection are complementary. Both configurations achieve comparable forget-set similarity at every request — protection of prior identities is not contingent on a particular mask budget. However, a smaller mask ($k{=}20$) degrades S-R from $0.557$ to $0.523$ at Request 3, reflecting that an over-restricted mask leaves the forget loss with too few parameters, inducing larger updates within the projected subspace. Rank $R=40$ is used as the default. A small $R$ may leave the protected subspace under-specified; a large $R$ over-constrains later sequences. $R=40$ achieves the best balance between forget suppression and retain quality across the 3-speaker sequence. CORTIS adds three overhead sources beyond sequential TGU: (i) Fisher information computation on the forget speaker's data — approximately 30 minutes per request; (ii) truncated SVD on collected gradient snapshots — approximately 3 seconds; (iii) per-step projection — approximately 0.5 s per optimizer step (from 8.48 s to 8.76 s with 4-step gradient accumulation, a 3.3% overhead). In aggregate, CORTIS requires 3.5 hours per unlearning request — substantially below cumulative TGU (87.5 hours after 3 requests, scaling linearly with $i$). The cumulative TGU baseline is the only sequential alternative that does not catastrophically re-learn, but it (a) violates forget-data non-retention constraint (C2), and (b) scales quadratically across the model's lifetime. CORTIS achieves comparable durability at constant per-request cost. The appendix also compares against a model-merging baseline (TIES-Merge) in which each speaker is independently unlearned from the pretrained model $\theta_0$, yielding per-speaker checkpoints whose task vectors $\tau_f = \tilde{\theta}_f - \theta_0$ are combined via TIES-Merging: This baseline is excluded from the main table because it fundamentally violates constraint (C2): it requires retaining the pretrained model $\theta_0$ indefinitely (which already contains full speaker synthesis capabilities for all forgotten identities), plus either all per-speaker checkpoints or the original forget datasets. Despite this advantage, CORTIS substantially outperforms TIES-Merge even without retaining any of these resources. Across all methods: AdamW in FP16, gradient clipping at 0.2, EMA decay 0.9999, masking ratio sampled from [0.7, 1.0], conditioning dropout probability 0.8, mel-spectrogram normalized with mean $-5.884$ and std $2.261$. The paper explicitly identifies several limitations: The motivation for this work is fundamentally protective. Zero-shot TTS systems can clone a person's voice from seconds of reference audio, with harms that disproportionately affect individuals who never consented to having their voices replicable. Continual speaker identity unlearning gives service providers a concrete procedure for honoring removal requests as they arrive, without retaining the voice data the requester asked to have erased. The authors note that unlearning durability is a necessary but not sufficient condition for responsible ZS-TTS deployment; robust evaluation, transparent RTBF handling policies, and adversarial robustness studies are complementary directions. CORTIS is the first framework to address continual speaker identity unlearning in ZS-TTS under realistic deployment constraints — sequential request arrival and forget-data non-retention. The key insight is that catastrophic re-learning is the primary failure mode for continual speaker identity unlearning, and it requires two complementary countermeasures: parameter-level localization through contrastive Fisher saliency masking, and direction-level protection through orthogonal projection onto the complement of the cumulative unlearning subspace. Together, these mechanisms achieve consistent suppression of all previously forgotten speakers across long request sequences — maintaining S-$f$ values below $0.2$ across five sequential speakers — while preserving competitive remain-set quality, at a constant per-request cost of 3.5 hours on two A100 GPUs compared to 87.5+ hours for the only alternative that avoids catastrophic re-learning (but violates RTBF compliance).Practical Implementation of the Cumulative Subspace
Backbone Architecture: VoiceBox
Experimental Setup
Evaluation Scenario and Datasets
Baselines
Evaluation Metrics
Results
Main Results (3-Speaker Sequence)
Method
After Request 1
After Request 2
After Request 3
W-R↓ W-F↓ S-R↑ S-f₁↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓ S-f₃↓
Original
2.1 2.6 0.649 0.721
2.1 2.5 0.649 0.721 0.674
2.1 2.5 0.649 0.721 0.674 0.866
SGU
2.7 2.5 0.479 0.165
2.8 2.6 0.348 0.178 0.075
2.7 2.2 0.315 0.233 0.101 0.192
TGU
2.3 2.5 0.624 0.164
2.5 3.0 0.563 0.612 0.198
3.0 2.6 0.582 0.603 0.546 0.180
UN
2.8 2.6 0.565 0.229
2.8 2.7 0.545 0.344 0.140
3.0 2.5 0.580 0.638 0.555 0.106
SelFT
2.7 2.5 0.592 0.154
2.8 2.6 0.585 0.482 0.077
2.7 2.3 0.548 0.553 0.434 0.110
CORTIS
2.9 2.6 0.602 0.162
2.9 2.6 0.553 0.185 0.122
2.8 2.6 0.557 0.172 0.148 0.124
Analysis of Baseline Failures
Scalability to Longer Sequences (5 Speakers)
Step W-R↓ S-R↑ W-F↓ S-f₁↓ S-f₂↓ S-f₃↓ S-f₄↓ S-f₅↓
After Request 4
2.8 0.562 2.8 0.198 0.109 0.167 0.193 —
After Request 5
2.7 0.527 2.3 0.178 0.096 0.170 0.114 0.033
Ablation Studies
Projection vs. Mask-Only
Variant
After Request 2
After Request 3
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓ S-f₃↓
w/o Projection
2.7 2.4 0.528 0.200 0.103
2.8 2.3 0.546 0.334 0.397 0.120
w. Projection (CORTIS)
2.9 2.6 0.553 0.185 0.122
2.8 2.6 0.557 0.172 0.148 0.124
Mask Budget Ablation ($k$ Parameter)
Variant
After Request 1
After Request 2
After Request 3
W-R↓ W-F↓ S-R↑ S-f₁↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓ S-f₃↓
k=20
2.8 2.5 0.625 0.158
2.7 2.6 0.601 0.194 0.112
2.8 2.6 0.523 0.155 0.082 0.132
k=30 (default)
2.9 2.6 0.602 0.162
2.9 2.6 0.553 0.185 0.122
2.8 2.6 0.557 0.172 0.148 0.124
Projection Subspace Rank Ablation
Rank $R$
After Request 2
After Request 3
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓
W-R↓ W-F↓ S-R↑ S-f₁↓ S-f₂↓ S-f₃↓
20
2.7 2.6 0.558 0.169 0.093
2.6 2.6 0.568 0.187 0.090 0.144
30
2.7 2.6 0.556 0.156 0.108
2.8 2.6 0.476 0.140 0.086 0.163
40 (default)
2.9 2.6 0.553 0.185 0.122
2.8 2.6 0.557 0.172 0.148 0.124
Computational Efficiency
Method Steps Memory (GB) Time (hours) TGU (sequential) 10K 30.8 29 TGU (cumulative) 30K 30.8 87.5 UN 10K 30.8 29 SelFT 11K 48.5 22 CORTIS 3K 49.3 3.5 RTBF-Incompliant Comparison: TIES-Merging
Implementation Details
CORTIS Training
Baseline Training Configurations
Limitations
Societal Impact
Conclusion