3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

3D asset teams are getting a more rigorous yardstick for automated QA. 3D-DefectBench is built to probe vision-language model judges on nine binary defect categories, covering geometry, texture, and prompt adherence, so teams can see where a generator is actually failing instead of relying on a single holistic score.

The study uses a balanced factorial design across four pipeline factors: the VLM itself, camera protocol, visual input, and prompt schema. That produced 84 inference designs and roughly 3.2 million scored defect decisions, then a broader validation pass on frontier models. The headline result is that model choice is the biggest driver of agreement with human labels, but the surrounding pipeline still meaningfully changes outcomes and can even change which configuration looks best.

One practical finding stands out for production use: a compact six-view RGB setup held up surprisingly well against denser multi-view captures and against inputs augmented with depth or surface normals. That makes it a strong default when compute, storage, or render time matter. Even so, the best of 12 evaluated VLM judges still trailed trained human labelers under the standardized setup.

The benchmark also shows how fragile these systems can be when the reference labels get noisier. Texture agreement dropped sharply when expert-consensus labels were replaced with silver labels, which is a useful warning for teams trying to bootstrap evaluation data cheaply. The released labels, prompts, predictions, and Croissant metadata on Hugging Face should make it easier for studios and tool...