Abstract
We introduce MV-CoLight, a two-stage framework for illumination-consistent object compositing in both 2D images and 3D scenes. Our novel feed-forward architecture models lighting and shadowsdirectly, avoiding the iterative biases of diffusion-based methods. We employ a Hilbert curve-based mapping to align 2D image inputs with 3D Gaussian scene representations seamlessly. To facilitate training and evaluation, we further introduce a large-scale 3D compositing dataset. Experiments demonstrate state-of-the-art harmonized results across standard benchmarks and our dataset, as well as casually captured real-world scenes demonstrate the framework’s robustness and wide generalization.
Method
Overview of MV-CoLight. (a) We insert a white puppy as the composite object onto the table between basketballs, and render multi-view inharmonious images, background-only images, and depth maps using a camera trajectory moving from distant to close-up positions. (b) We input a single-view data into the 2D object compositing model, which processes the data through multiple Swin Transformer blocks to output the harmonized result. (c) We project the multi-view features from 2D models into Gaussian space via , combine them with the original inharmonious Gaussian colors projected into 2D Gaussian color space through , and then feed them into the 3D object compositing model. The model outputs harmonized Gaussian colors and computes rendering loss by incorporating Gaussian shape attributes.
Results
Single-view qualitative comparisons. Compared to baselines, our method successfully generates coherence illumination and plausible shadows while decoupling highlights from inserted objects.
Multi-view qualitative comparisons. Our approach synthesizes multi-view consistent illumination and shadows while strictly preserving the original scene geometry, scale and object placement.
Real-world scene visualization. We evaluate our method on real-world scenes and achieve both color harmonization and realistic lighting/shadow generation.
