ReforMe: Re-Shaping Documents with Contextual Prompting and Layout-Aware Propagation
2026-06-02 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors created a new system to help turn tricky documents with handwriting and messy layouts into digital text. It combines layout detection, OCR, and language models, letting users fix mistakes by editing directly or giving simple instructions. The system also spreads corrections across similar parts of the document to save time. In tests with real documents, their approach made fixing errors faster and less repetitive.
Optical Character RecognitionLarge Language ModelsDocument Layout AnalysisHandwritten Text RecognitionUser InteractionError CorrectionHuman-in-the-Loop SystemsDigital Document Processing
Authors
Nabin Khanal, Tongyan Wang, Jui-Cheng Chiu, Ningning Nicole Kong, Hannah Yanhua Zong, Yingjie Victor Chen
Abstract
Digitizing complex documents with handwritten content, irregular tables, and heterogeneous layouts remains challenging, as traditional Optical Character Recognition (OCR) systems fail to capture writing nuances, author-specific conventions, and document structure, and recent LLM-based approaches lack mechanisms for precise, scalable correction. We present an interactive document digitization system that integrates layout-aware parsing, OCR, and LLM-based reconstruction with user-driven refinement. The system is informed by a formative study that identifies key challenges and interaction needs in real-world digitization workflows. It supports both direct edits and natural-language instructions, and introduces a layout-aware propagation mechanism that generalizes user corrections across structurally similar regions. This enables not only efficient error correction but also document re-shaping into structured, analyzable representations. We evaluate the system through a within-subjects user study (n=12) on real-world documents. Results show improved correction efficiency and reduced repetitive effort, demonstrating more effective and controllable document digitization procedure.