3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

A new benchmark, 3D-DefectBench, tests how well vision-language models catch fine-grained defects in generated 3D assets. Across 84 pipeline designs and about 3.2 million defect judgments, model choice mattered most, but camera setup, input type, and prompt wording also shifted results. The takeaway: automated judges need to be evaluated as full pipelines, not just as standalone models.