AI summaryⓘ
The authors developed PlanSightRAG, a new system that helps engineers check if civil infrastructure plans meet rules by directly analyzing images of the plans instead of relying on text extracted by OCR, which loses important layout details. Their system combines several AI components to search and reason visually and keeps track of evidence with heatmaps. They created a large dataset from state transportation plans to test their method and found it retrieves relevant information accurately without prior examples. Additionally, their AI can automatically understand numeric rules from specifications without humans setting the rules in advance. This shows a step forward in automating compliance checks using visual plan data.
Civil infrastructure compliance2D plan imageryOCR limitationsMultimodal retrievalRetrieval-Augmented Generation (RAG)ColNomic-3BMaxSim heatmapsZero-shot retrievalVisual rule groundingSpecification corpus
Authors
Nabaraj Subedi, Shuvo Dip Datta, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar
Abstract
Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the geometry and layout essential for interpreting these plans. We present a Visual-First Multimodal Retrieval-Augmented Generation (RAG) framework called PlanSightRAG. It indexes and reasons directly over plan imagery, integrates a ColNomic-3B multi-vector retrieval, an agentic Planner-Retriever-Auditor-Synthesizer, and MaxSim heatmaps as an evidence trail. We introduce a 4,056-pair benchmark from five state Departments of Transportation (DOT) standard plans (1,898 pages). PlanSightRAG achieves 91.47% Recall@5 on zero-shot retrieval, while on a held-out Michigan DOT corpus, it achieves 91.40%. On synthetic, parametrically-generated compliance drawings, our Qwen2.5-VL-72B pipeline reaches 100% verdict accuracy only when supplied a pre-resolved rule threshold, a controlled ceiling that a non-VLM OCR baseline already reaches at 76.4%. Finally, we demonstrate autonomous visual rule-grounding by extracting numeric limits directly from a specification corpus without any human-supplied rules.