Search

Measuring the material, geometry, and ambient lighting of surfaces is a key technology in the object’s appearance reconstruction. In this article, we propose a novel deep learning-based method to extract such information to reconstruct the object’s appearance from an RGB image. Firstly, we design new deep convolutional neural network architectures to improve the performance by fusing complementary features from hierarchical layers and different tasks. Then we generate a synthetic dataset to train the proposed model to tackle the problem of the absence of the ground-truth. To transfer the domain from the synthetic data to the specific real image, we introduce a self-supervised test-time training strategy to finetune the trained model. The proposed architecture only requires one image as input when inferring the material, geometry, and ambient lighting. The experiments are conducted to evaluate the proposed method on both the synthetic data and real data. The results show that our trained model outperforms the existing baselines in each task and presents obvious improvement in final appearance reconstruction, which verifies the effectiveness of the proposed methods.

Measuring the material reflectance of surfaces is a key technology in inverse rendering, which can be used in object appearance reconstruction. In this paper we propose a novel deep learning-based method to extract material information represented by a physically-based bidirectional reflectance distribution function from an RGB image of an object. Firstly, we design new deep convolutional neural network architectures to regress material parameters by self-supervised training based on a differentiable image-based renderer. Then we generate a synthetic dataset to train the model as the initialization of the self-supervised system. To transfer the domain from the synthetic data to the real image, we introduce a test-time training strategy to finetune the pretrained model to improve the performance. The proposed architecture only requires one image as input and the experiments are conducted to evaluate the proposed method on both the synthetic data and real data. The results show that our trained model presents dramatic improvement and verifies the effectiveness of the proposed methods.

In this paper, we propose a framework to generate stereoscopic content from a single image using the relative depth label predicted from deep residual network. Specifically, our framework first obtains a coarse relative depth label from the network and refines it to painting depth by sampling and interpolation, then an unsupervised clustering algorithm is employed to separate pixels of different depths into different layers to generate stereoscopic images. Experimental results with good visual effects demonstrate that the proposed method can be generally applied in both outdoor and indoor scenes. Meanwhile the quantitative results on relative depth estimation from a single image are comparable to state-of-the-art. Further experiments show the application possibility of our method in VR and panorama.

Depth estimation is a classical topic in computer vision, however, inferring the depth of a scene from a single image remains an extremely difficult problem. In this paper, a non-parametric method is adopted to obtain the depth of a single image. To this end, RGB-D datasets are exploited as the inference basis. Given a query image, a global scene-level retrieval is performed against the dataset, followed by a superpixel-level matching. The superpixels-based scene representation is introduced to model the depth jointly in terms of superpixel centroid. The depth estimation is formulated as contextual inference and the depth propagation. The contextual inference is expressed as a Markov random field (MRF) energy function defined on a sparse depth map obtained by the matching process and implemented in a graphical model whose edges encode the interactions between the superpixel centroids. Then the depth propagation generates the final dense depth map from the inferred result. The benefits of the proposed method is demonstrated on the standard dataset.