Can LLMs Perform Deep Technical Comprehension of Computer Architecture Papers?
2026-07-13 • Computers and Society
Computers and SocietyHardware ArchitectureMultiagent Systems
AI summaryⓘ
The authors studied whether large language models can deeply understand and critique computer architecture research papers instead of just summarizing them. They tested Gauntlet, a system that uses multiple expert-like reviewers and a final synthesis step to analyze papers. When compared with human critiques on 20 papers, Gauntlet's analyses were generally preferred, especially for thoroughness and critical insight. The key advantage came from using multiple reviewer personas and combining their views, rather than running a single model alone. The authors shared all their data and methods to help the community build on this work.
large language modelscomputer architectureautomated critiquemulti-agent systemanalysis pipelineISCA conferenceHPCA conferencemodel synthesiscritical rigorevaluation metrics
Authors
Nishant Aggarwal, Ayushi Dubal, Sreeraj Kannakarankodi, Ian McDougall, Adarsh Mittal, Vishnu Ramadas, Noah Scott, Ranganath Selagamsetty, Weichu Yang, Karthikeyan Sankaralingam
Abstract
Can large language models perform deep technical comprehension of computer architecture papers -- not summarization, but structured critique that names the core mechanism, surfaces buried assumptions, and connects a contribution beyond its own scope? We study Gauntlet, an open-source pipeline that analyzes a paper through five independent expert-persona reviewers and an adversarial synthesis stage. On 20 ISCA 2025 and HPCA 2026 papers, ten researchers each wrote their own analyses and then judged, for papers other than their own, the human analysis against Gauntlet's. Across the 20 comparisons evaluators preferred Gauntlet in 15 (human in 4, one tie); its advantage is significant on per-analyst totals (paired Wilcoxon, p < 0.01) and largest on Critical Rigor, vanishing only on Calibration. Where humans win, it is on trust and usefulness rather than depth: a confident wrong claim, a mechanism described but not taught, or unprioritized breadth. A 98-paper automated ablation shows the gain comes from the multi-agent structure -- the pipeline beats the same model run as a single rich-persona agent on 96% of papers -- and specifically from its synthesis pass. We release all analyses, scores, and the rubric as a community resource.