Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
2026-06-02 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models forget specific facts in multiple languages without retraining from scratch. They found that forgetting facts in one language sometimes causes forgetting in related languages, especially those with similar scripts or language families. Their analysis shows that the models don't truly erase the knowledge but only suppress it in later layers. They also discovered a way to partially reverse this forgetting by adjusting the model during use, recovering much of the hidden information across languages.
large language modelsunlearningmultilinguallanguage transferTOFU benchmarklatent spacedecoder layersinference-time steeringcross-lingual models
Authors
Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein, Lea Frermann
Abstract
Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.