PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs
2026-03-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created PathMem, a new computer system that helps analyze pathology images by combining visual patterns with organized medical knowledge, like diagnostic rules and disease classifications. Current models can look at images and read text but have trouble using detailed medical knowledge correctly. PathMem mimics how human pathologists think by using a special memory system that remembers long-term facts and updates working memory based on the image context. This approach helps the system give better reports and diagnoses on pathology tests compared to earlier methods.
Computational pathologyMultimodal large language modelsMemory TransformerLong-term memoryWorking memoryDiagnostic reasoningWhole Slide Imaging (WSI)Knowledge integrationGrading criteria
Authors
Jinyue Li, Yuci Liang, Qiankun Li, Xinheng Lyu, Jiayu Qian, Huabao Chen, Kun Wang, Zhigang Zeng, Anil Anthony Bharath, Yang Liu
Abstract
Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.