AI summaryⓘ
The authors explain that differentiable renderers can show which parts of a 3D scene most affect different image quality measures by calculating gradients, similar to how gradient-based saliency methods highlight important input features in neural networks. These gradient maps, called metric saliency maps, track how changes in scene elements impact specific metrics like brightness or glare. Unlike traditional saliency that follows learned weights, metric saliency follows the physical rendering process itself. The authors found that what matters most depends on the metric chosen, meaning saliency is tied to the measurement, not just the scene. This suggests differentiable renderers provide useful insights about scenes beyond just generating images.
differentiable renderergradient-based saliencymetric saliency mapreverse-mode differentiationlight transportscene parameterspsychovisual glareperceptual scoresimage formationmodel interpretability
Authors
Linas Beresna, Eugene Fiume
Abstract
Gradient-based saliency methods reveal which input features most influence a neural network's output, and are a standard tool for model interpretability. We observe that differentiable renderers, which are conventionally used for parameter optimisation, produce an analogous form of saliency: given any scalar metric evaluated on a rendered image, a single reverse-mode differentiation pass yields per-parameter gradients that identify which scene elements most influence the metric. We call these gradient fields metric saliency maps. Unlike neural saliency, which propagates attribution through learned weights, metric saliency propagates through the image formation process itself, including multi-bounce light transport, capturing parameter dependencies that are semi-opaque to manual inspection. We compute metric saliency maps for qualitatively different objectives: psychovisual glare indices, mean scene luminance, and neural perceptual scores. The saliency rankings differ substantially across metrics for the same scene, with parameters that dominate one objective being negligible for another. The saliency map is specific to the metric, not an intrinsic property of the scene. Our results suggest that differentiable renderers produce derivative images that are as informative for scene understanding as the primal images they were designed to generate.