Overview of differentiable inverse rendering with interpretable basis BRDFs
Overview of differentiable inverse rendering with interpretable basis BRDFs
Inverse rendering seeks to reconstruct both geometry and spatially varying BRDFs (SVBRDFs) from captured images. To address the inherent ill-posedness of inverse rendering, basis BRDF representations are commonly used, modeling SVBRDFs as spatially varying blends of a set of basis BRDFs. However, existing methods often yield basis BRDFs that lack intuitive separation and have limited scalability to scenes of varying complexity. In this paper, we introduce a differentiable inverse rendering method that produces interpretable basis BRDFs. Our approach models a scene using 2D Gaussians, where the reflectance of each Gaussian is defined by a weighted blend of basis BRDFs. We efficiently render an image from the 2D Gaussians and basis BRDFs using differentiable rasterization and impose a rendering loss with the input images. During this analysis-by-synthesis optimization process of differentiable inverse rendering, we dynamically adjust the number of basis BRDFs to fit the target scene while encouraging sparsity in the basis weights. This ensures that the reflectance of each Gaussian is represented by only a few basis BRDFs. This approach enables the reconstruction of accurate geometry and interpretable basis BRDFs that are spatially separated. Consequently, the resulting scene representation, comprising basis BRDFs and 2D Gaussians, supports physically-based novel-view relighting and intuitive scene editing.
Given a set of multi-view photometric images, we initialize point cloud and extract base color for basis BRDFs. We jointly optimize 2D Gaussians and basis BRDFs by comparing the differentiably-rendered images and the input images. Our method enables obtaining interpretable basis BRDFs with spatially-separated basis-BRDF weights and the number of basis BRDFs adapts to the scene.
During the analysis-by-synthesis optimization, we compute the values of each basis BRDF for sampled half-way angles $\theta_{h}$ from which radiometric difference is obtained. We compute the geometric difference between point clouds of basis BRDFs. If two basis BRDFs are radiometrically and geometrically similar, we merge them. If the rendered weight map $W_i$ has few valid pixels, we remove the basis BRDF.
@article{chung2024differentiable,
title={Differentiable Inverse Rendering with Interpretable Basis BRDFs},
author={Chung, Hoon-Gyu and Choi, Seokjun and Baek, Seung-Hwan},
journal={arXiv preprint arXiv:2411.17994},
year={2024}
}