Matthias Niessner releases GenRecon, reformulating multi-view 3D scene reconstruction as conditional 3D generation via Trellis2
It produces relightable PBR meshes from sparse RGB images.
📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢
Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior!
We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views.
GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion.
https://youtu.be/Tp-i06DPXa0 https://kasothaphie.github.io/GenRecon/
Amazing work by @katha_schmid, @nicolasvluetzow, Jozef, @angelaqdai
