Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models
2026-04-09 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors study how large language models store and update complex rules, like those in math and physics, rather than just simple facts. They find that rule knowledge is spread across different parts of the model and can't be changed by fixing just one layer. To solve this, they create a method called Distributed Multi-Layer Editing (DMLE) that updates different model layers depending on the type of rule information. Their approach improves the model's ability to understand and apply rules better than previous methods.
large language modelsmodel editingrule-level knowledgetransformer layerscausal tracingformula representationsinstance portabilityGPT-J-6BLLaMA-3-8Bbenchmark
Authors
Yating Wang, Wenting Zhao, Yaqi Zhao, Yongshun Gong, Yilong Yin, Haoliang Sun
Abstract
Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances. Yet most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention. This assumption does not hold for rule-level knowledge, where a single rule must remain consistent across multiple interdependent forms. We investigate this problem through a mechanistic study of rule-level knowledge editing. To support this study, we extend the RuleEdit benchmark from 80 to 200 manually verified rules spanning mathematics and physics. Fine-grained causal tracing reveals a form-specific organization of rule knowledge in transformer layers: formulas and descriptions are concentrated in earlier layers, while instances are more associated with middle layers. These results suggest that rule knowledge is not uniformly localized, and therefore cannot be reliably edited by a single-layer or contiguous-block intervention. Based on this insight, we propose Distributed Multi-Layer Editing (DMLE), which applies a shared early-layer update to formulas and descriptions and a separate middle-layer update to instances. While remaining competitive on standard editing metrics, DMLE achieves substantially stronger rule-level editing performance. On average, it improves instance portability and rule understanding by 13.91 and 50.19 percentage points, respectively, over the strongest baseline across GPT-J-6B, Qwen2.5-7B, Qwen2-7B, and LLaMA-3-8B. The code is available at https://github.com/Pepper66/DMLE.