MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding

MRD, short for metamers rendered differentiably, applies physically based differentiable rendering to a familiar ML problem: figuring out what a vision model really encodes about a 3D scene. Instead of comparing pixels alone, it optimizes scene parameters until two physically different scenes land on the same internal activation, creating model metamers grounded in geometry, materials, and lighting.

That matters because many vision systems are trained on flat images but are still expected to behave as if they understand 3D structure. MRD gives researchers a way to probe those assumptions directly, including whether a model is sensitive to object shape while holding material and illumination constant. For game developers working with vision-driven tools, robotics-adjacent systems, or AI-assisted content pipelines, this kind of analysis helps reveal where a model is robust and where it may be relying on shortcuts.

The initial evaluation looks at multiple models and focuses on recovering scene parameters for geometry and bidirectional reflectance distribution function material properties. The optimized scenes often match the target model activations closely, but the visual results can differ substantially, which is exactly the point: the method exposes which physical attributes a model treats as interchangeable and which ones it cannot ignore.

The broader implication is that differentiable rendering is becoming more than a graphics research tool; it’s a way to interrogate perception systems with scene-level controls that artists and graphics programmers already think in....