Why multilingual unlearning is harder than it looks

If you’re shipping safety‑constrained or policy‑constrained agents, “unlearning” is quickly moving from paper concept to operational requirement: remove a piece of knowledge while keeping the rest of the model useful.

The core problem tackled in *Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning* is that unlearning is not monolingual by default. The authors show that:

Unlearning a fact in one language does not guarantee its removal in others; changing the query or even the requested answer language can reopen seemingly forgotten knowledge — a cross‑lingual loophole. [\[1\]](http://arxiv.org/abs/2609.40286v1)

So if your pipeline “forgets” a fact in English, the same fact can remain accessible via:

  • A query expressed in another language
  • A prompt that requests the answer in a different language

From an attacker’s perspective, that’s just an alternate prompt template. From a compliance perspective, it means your safety guarantees are language‑conditional.

The naive fix — “just unlearn in all languages” — is explicitly called out as not viable:

The most straightforward solution — unlearning in all languages — is neither scalable nor desirable as it amplifies damage to unrelated model capabilities. [\[1\]](http://arxiv.org/abs/2609.40286v1)

So the central engineering question becomes:

Given a limited budget of languages I can directly supervise unlearning on, which languages should I choose to maximize cross‑lingual forgetting?

The paper formalizes this as *language‑budgeted multilingual unlearning* and builds both a benchmark and a selection strategy around it. [\[1\]](http://arxiv.org/abs/2609.40286v1)

---

The Cross‑Lingual Unlearning Tensor: what you’re actually forgetting

To make this concrete, the authors introduce the Cross‑Lingual Unlearning Tensor, a benchmark designed to measure when forgetting in one linguistic form transfers to others.

Documented properties:

  • It spans 174 language–script pairs.
  • It covers 25 atomic paraphrase types for expressing the same knowledge. [\[1\]](http://arxiv.org/abs/2609.40286v1)

Functionally, you can think of it as an evaluation grid with at least three axes:

  1. Language / script pair – different ways of writing and speaking across 174 configurations.
  2. Paraphrase type – 25 distinct ways to rephrase the same underlying fact.
  3. Forget vs. test split – a set of expressions you train the unlearning procedure on, and held‑out expressions you probe afterward.

The tensor lets you ask structured questions like:

  • If I unlearn fact F in language A with paraphrase type P₁, how much does that reduce access to F in:
  • Language A with other paraphrase types P₂…P₂₅?
  • A *different* language B, with any paraphrase type?
  • Conversely, are there languages or scripts where unlearning *doesn’t* generalize well, indicating a persistent loophole?

Crucially, this is about *generalization of forgetting* across both languages and paraphrasing patterns, not just “did the logit for the training string go down.”

For builders, the takeaway is straightforward:

  • You can’t reason about unlearning robustness without a cross‑lingual + paraphrase‑sensitive evaluation grid.
  • The Cross‑Lingual Unlearning Tensor is an explicit attempt to provide that grid at large scale (174×25). [\[1\]](http://arxiv.org/abs/2609.40286v1)

---

Language‑budgeted unlearning: picking where to push

Given such a benchmark, the paper defines language‑budgeted multilingual unlearning:

We introduce the task of language budgeted multilingual unlearning where the goal is to select a subset of languages that maximizes cross‑lingual erasure. [\[1\]](http://arxiv.org/abs/2609.40286v1)

The setting is:

  • You have a budget: a small number of languages where you can afford to:
  • Build unlearning datasets
  • Run fine‑tuning or other parameter updates
  • Conduct intensive evaluation
  • The rest of the languages get no direct forget supervision — they’re “held‑out” targets.
  • You want to minimize residual access to the forgotten facts in *all* languages, not just the supervised ones.

A naive strategy would be:

  • Pick languages where the model is strongest (e.g., high‑resource languages) and hope forgetting transfers.

The paper shows that doesn’t work reliably:

Surprisingly, we find naively selecting strong individual sources does not reliably compose into strong source sets, motivating our development of COVER. [\[1\]](http://arxiv.org/abs/2609.40286v1)

In other words, “best single sources” are not “best sets of sources.” Interactions between languages matter: two source languages might cover very similar parts of the cross‑lingual space, leaving large gaps elsewhere.

---

COVER: coverage‑aware language selection

To address that combinatorial mess, the authors propose COVER:

We further propose COVER, which selects source languages to maximize predicted COVERage of languages receiving no forget supervision, enabling unlearning on a language budget. [\[1\]](http://arxiv.org/abs/2609.40286v1)

Key properties that *are* documented:

  • Goal: choose a subset of *source* languages (those you’ll directly unlearn on) that maximizes predicted coverage of *target* languages (those with no unlearning training).
  • Inputs at deployment time:
  • Benign calibration data
  • Access to the frozen model
  • No mention of needing access to original training data or sensitive forget examples. [\[1\]](http://arxiv.org/abs/2609.40286v1)
  • Evaluation:
  • Tested on three model families.
  • Uses two disjoint forget sets, i.e., separate groups of facts to erase.
  • On held‑out languages (no direct unlearning), COVER reduces mean residual access by 7.8–27.3% relative to uniform source selection. [\[1\]](http://arxiv.org/abs/2609.40286v1)

The mechanism details (how coverage is computed, what the calibration procedure looks like) are *not* spelled out in the abstract, but you can infer the operational role:

  1. Score candidate source languages using only benign data and the frozen model.
  2. Select a set of sources that jointly maximize some notion of “coverage” of other languages.
  3. Run your unlearning procedure on those sources only.
  4. Evaluate residual access on held‑out languages and paraphrases, ideally using the Cross‑Lingual Unlearning Tensor.

The headline result is that just being smart about which languages you use for unlearning gives a substantial reduction in residual knowledge leakage compared to naive uniform sampling over languages. [\[1\]](http://arxiv.org/abs/2609.40286v1)

For an agentic system that must comply with “forget this class of facts globally,” that’s the difference between:

  • “We tested English, looks good, trust us on the rest.”
  • “We systematically optimized which languages to retrain on, and we can show lower residual access across a wide language set.”

---

Beyond synthetic setups: low‑resource incidents

A common criticism of unlearning work is that it stays in synthetic, high‑resource corners. The paper does take a step toward more realistic conditions:

We find these gains extend beyond synthetic benchmarks to real news documents in low-resource language settings using human translated data from the Low Resource Languages for Emergent Incidents (LORELEI) corpus. [\[1\]](http://arxiv.org/abs/2609.40286v1)

So:

  • The LORELEI corpus supplies human‑translated news documents in low‑resource languages related to emergent incidents.
  • COVER’s improvements in reducing held‑out residual access carry over to this kind of real, low‑resource data, not just controlled benchmarks. [\[1\]](http://arxiv.org/abs/2609.40286v1)

For builders who deal with crisis‑related or politically sensitive content in multiple languages, this is the regime that actually matters: text that users care about, in languages your base model is weaker on, where complete removal is important.

---

How this fits into an unlearning stack

The paper does not specify a particular unlearning algorithm (e.g., which loss, optimizer, or parameter subset), but it does define the *selection* piece:

  • Whatever forgetting procedure you already use (fine‑tuning, editing, etc.) operates on a chosen set of languages.
  • COVER is a front‑end that helps you pick that set using only benign data and a frozen model. [\[1\]](http://arxiv.org/abs/2609.40286v1)
  • The Cross‑Lingual Unlearning Tensor is a back‑end evaluation framework that tells you how well forgetting transfers across 174 language–script pairs and 25 paraphrase types. [\[1\]](http://arxiv.org/abs/2609.40286v1)

If you’re designing an agentic system with unlearning requirements, you can think in terms of three components:

  1. Unlearning engine
  • Whatever you use to actually suppress or erase knowledge on a supervised dataset.
  • Not specified in [\[1\]](http://arxiv.org/abs/2609.40286v1).
  1. Language selection layer (COVER)
  • Chooses which languages to feed into the unlearning engine given a fixed budget.
  • Uses benign calibration data and the frozen model at deployment to predict cross‑lingual coverage. [\[1\]](http://arxiv.org/abs/2609.40286v1).
  1. Cross‑lingual evaluation (Unlearning Tensor + LORELEI)
  • Measures residual access to the forgotten facts across:
  • 174 language–script pairs
  • 25 paraphrase types
  • Real low‑resource news documents via LORELEI. [\[1\]](http://arxiv.org/abs/2609.40286v1)

Plugging this into an orchestration stack for agents:

  • Policy change arrives (“forget set S globally”).
  • Planner:
  • Maps S to multilingual representations using your existing translation / localization tools (not specified in [\[1\]](http://arxiv.org/abs/2609.40286v1)).
  • Invokes COVER to choose a subset of languages as unlearning sources.
  • Training subsystem:
  • Runs your unlearning procedure restricted to those languages.
  • Evaluation subsystem:
  • Probes for residual access using Cross‑Lingual Unlearning Tensor style evaluations and LORELEI‑like real corpora where relevant.

The crucial conceptual upgrade is treating language choice in unlearning as an optimization problem, not an implementation detail.

---

Why this matters now for builders

Even without the algorithmic details, [\[1\]](http://arxiv.org/abs/2609.40286v1) crystallizes several operationally important facts:

  • Monolingual audits are unsafe: checking unlearning only in the language you trained on misses cross‑lingual loopholes. [\[1\]](http://arxiv.org/abs/2609.40286v1)
  • Language coverage is structured: some languages provide better “forgetting coverage” to others, but picking them greedily doesn’t compose well. [\[1\]](http://arxiv.org/abs/2609.40286v1)
  • Selection pays off: a principled selector like COVER yields 7.8–27.3% reductions in mean held‑out residual access over uniform source selection across three model families and two forget sets. [\[1\]](http://arxiv.org/abs/2609.40286v1)
  • Real‑world extension: these gains are not confined to synthetic benchmarks, but carry into low‑resource incident news via LORELEI. [\[1\]](http://arxiv.org/abs/2609.40286v1)

From a systems perspective, this changes how you should think about unlearning:

  • It’s no longer “apply procedure U to dataset D and call it done.”
  • It’s “jointly optimize:
  • Which *facts* to forget
  • Which *languages* to supervise on
  • How you *evaluate* transfer of forgetting to unseen languages and paraphrases.”

COVER explicitly targets the second piece, under realistic constraints of compute and data. [\[1\]](http://arxiv.org/abs/2609.40286v1)

---

What is not documented

The abstract and metadata of [\[1\]](http://arxiv.org/abs/2609.40286v1) leave several implementation‑critical details unspecified:

  • The exact construction of the Cross‑Lingual Unlearning Tensor beyond its size (174 language–script pairs, 25 paraphrase types):
  • How many facts are included
  • How facts are chosen
  • How paraphrase types are operationally defined
  • The internal mechanics of COVER:
  • The precise objective used to define “coverage”
  • How benign calibration data is selected and labeled
  • Whether COVER is model‑agnostic or tuned to specific architectures
  • The identity and size of the “three model families” and “two disjoint forget sets” used in experiments:
  • Model architectures
  • Parameter counts
  • Training data scale
  • The exact metric definition of “mean held‑out residual access” and how it is computed.
  • The unlearning algorithm applied once languages are selected:
  • Whether it is fine‑tuning, editing, or another technique
  • What loss functions or constraints are used
  • The detailed setup of the LORELEI experiments:
  • Which languages and incidents are used
  • How news documents are sampled and labeled
  • How success is measured beyond the statement that COVER’s gains “extend” to this setting

Any engineering extrapolation beyond these documented points (e.g., specific training recipes, API‑level designs, or exact numbers apart from those cited above) would require the full paper or additional sources and is not established by [\[1\]](http://arxiv.org/abs/2609.40286v1).