AI summaryⓘ
The authors created 3D-DefectBench, a testing system to check how well automated tools find problems in 3D models without needing humans to inspect everything. They studied many parts of the evaluation process, like the choice of AI model, camera angles, and input types, finding that which AI model is used matters most but other factors also influence results. They found that using six regular color images works almost as well as more complex and costly setups. Even the best AI judges they tested still don't match trained humans, especially when using less accurate labels. Their work shows that evaluating these AI tools should consider the whole process and how human labels are used, not just the AI alone.
Automated evaluationGenerative 3D systemsVision-language models (VLM)3D defect detectionCamera protocolPrompt schemaMulti-view imagingHuman reference labelsBenchmarkingFactorial design
Authors
Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng
Abstract
Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. However, the reliability of an automated judge depends on the entire evaluation pipeline, not only the underlying vision-language model (VLM), but also how assets are rendered, what visual evidence is provided, how the task is specified, and how human reference labels are constructed. We introduce 3D-DefectBench, a benchmark and framework for systematic analysis of VLM-based 3D defect detection pipelines. It complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and prompt adherence, providing actionable diagnostics for generator development and judge evaluation. Using a balanced factorial design, we vary four pipeline factors, VLM, camera protocol, visual input, and prompt schema, across 84 inference designs and approximately 3.2 million scored defect decisions, followed by staged validation on a broader set of frontier models. Model choice is the largest determinant of agreement with human labels, but the remaining factors also affect performance, interact with model selection, and can change the best configuration. Within the evaluated design space, a compact six-view RGB protocol performs comparably to denser multi-view settings and inputs augmented with depth or surface normals, making it a strong cost-effective default. Under this standardized pipeline, the best of 12 VLM judges still lag behind trained human labelers, while texture agreement drops sharply when expert-consensus labels are replaced by noisier silver labels. These findings show that automated judges should be evaluated as complete pipelines and calibrated across human reference regimes, rather than benchmarked only as standalone models. We release labels, prompts, predictions, and Croissant metadata on Hugging Face.