Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
2026-07-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to make large language models "unlearn" specific facts without messing up everything else they know, which is important for privacy and safety. They point out that current tests don't catch when models still remember facts if asked in tricky ways or accidentally forget unrelated knowledge. To fix this, the authors created SUITE, a new way to test and train models on what to forget and what to keep. They also developed JensUn++, an improved method that better balances forgetting targeted info while retaining other knowledge in various models.
Machine unlearningLarge language modelsPrivacySafetyKnowledge retentionBenchmarkingSUITEJensUn++Forgetting evaluationRetain evaluation
Authors
Amit Peleg, Naman Deep Singh, Naama Pearl, Bibhabasu Mohapatra, Matthias Hein
Abstract
Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond exact training prompts. Retain evaluation must probe a far larger and implicitly defined set, namely every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on the obtained insights, we introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. Code and datasets are available at https://amitpeleg.github.io/forget-narrowly-retain-broadly