Search

When using Monte Carlo Path Tracing (MCPT) method to render 3D scenes, artifacts may occur due to insufficient sampling, and directly increasing the number of samples can increase the time cost of the rendering algorithm. An effective strategy is to adaptively allocate the number of samples per pixel in an iterative manner. However, iterative operations introduce additional computational overhead during the rendering process. To solve this problem, we proposed a rendering acceleration method that does not require iterative computation. This method combines the Just Noticeable Difference (JND) information and uses a neural network to predict the sampling matrix of the scene, which is adjusted based on the lighting information in the pre-rendered image. First, we extract the JND information of pre-rendered images during the preprocessing and estimate the fast convergence regions in the scene (such as environment map regions, light source regions, etc.). Then, we use Conv-LSTM to estimate the JND features of high-quality rendered images. We design a multi-feature fusion network to predict the number of samples required for each pixel during the rendering process. The network takes the preprocessed pre-rendered images as the input of the encoder, which are fused with the output of Conv-LSTM as the input of the decoder to output the corresponding sampling matrix. In addition, we noticed that areas with darker lighting are more difficult to converge during the rendering process. Therefore, we calculated the lighting clustering results of the pre-rendered image and adjusted the sampling matrix output by the sampling prediction model based on the lighting clustering results. The experimental results indicate that our method has better performance compared to the current methods. [ABSTRACT FROM AUTHOR]

The geometric distortion in panoramic images significantly mediates the performance of saliency detection method based on traditional CNN. The strategy of dynamically expanding convolution kernel can achieve good results, but it also produces a lot of computational overhead in the process of reading the adjacency list, which decreases the computational efficiency. The appearance of graph convolution provides a new way to solve such problems. Although using graph convolution can effectively extract the structural features of the graph, it reduces the accuracy of the model resulting from ignoring the spatial features of the image signal. To this end, this paper proposes a construction method of the multi-scale graph structure of the panoramic image and a panoramic image saliency detection model composed of an image saliency feature extraction network and multi-scale saliency feature fusion network combining the image structure information and spatial information in the panoramic image. First, we establish a graph structure consisting of root and leaf nodes obtained by super-pixel segmentation at different scales and spherical Fibonacci sampling, respectively. Then, a feature extraction network composed of two graph convolution layers and two one-dimensional auto-encoders with the same parameterization is used to extract the salient features of the multi-scale graph structure. Finally, the U-Net network fuses the multi-scale saliency features to get the final saliency map. The results show that the proposed model performs better than the state-of-the-art models in terms of calculation speed and accuracy. [ABSTRACT FROM AUTHOR]

The geometric distortion of the panoramic image makes the saliency detection method based on traditional 2D convolution invalid. "Mapped Convolution" can effectively solve this problem, which accepts a task- or domain-specific mapping function in the form of an adjacency list that dictates where the convolutional filters sample the input. However, when applied to panorama saliency detection, the method results in additional computational overhead due to repeatedly sampling overlapping regions of adjacent convolution positions along the longitude. In order to solve this problem, we improved the calculation process of "Mapped Convolution". Rather than accessing adjacency list during the convolution, we first sample the panorama based on the adjacency list for only once and obtain a sampled map. This sampling process is called the decoupled sampling of "Mapped Convolution". And then the map is convoluted in traditional 2D way, thus avoiding repeatedly sampling. In this paper, an interpolation method based on the Softmax function is also proposed and applied to the interpolation calculation of decoupled sampling. Compared with common interpolation methods such as linear interpolation, this interpolation method makes our network more efficient during training. We additionally introduce a new adaptive equator bias algorithm allowing for different attention distributions at different longitudes, which is more consistent with viewer's visual behavior. Combining the U-Autoencoder network containing the decoupled sampling with the adaptive equator bias algorithm, we construct a 360-degree visual saliency detection model. We map the original panorama into a cube, and then use the the cube isometric mapping method to remap it into a panorama and input it into the network for training. Then, the crude saliency map output by the decoder is combined with the equator bias map to obtain the final saliency map. The results show that the model proposed is superior to recent state-of-the-art models in terms of computational speed and saliency-map prediction. [ABSTRACT FROM AUTHOR]

Monte Carlo path tracing is a widely used method for generating realistic rendering results in multimedia applications, but often suffers from poor convergence and heavy sampling budget. Insufficient path samples may lead to noisy results. Some noises are hidden in textures, and the human visual system cannot detect them all. Just noticeable difference (JND) quantifies this limitation as a full-reference perception threshold. In rendering, the reference is unavailable and a surrogate is required. This paper proposed a perception-JND-driven path tracing method for reducing sampling budget. We tested and verified the surrogate JND thresholds derived from current rendering results. Then, we introduced difference pooling module and shading restart module to control perceptual convergence. Further, to improve accuracy, we developed the strategy for optimizing sampling steps. Experiments showed that the proposed method outperformed the state-of-the-art method at moderately low sampling levels, offering a lightweight and efficient solution to reducing sample budget while improving visual quality. [ABSTRACT FROM AUTHOR]

The quality of images generated by computer graphics rendering algorithms is mainly affected by visible distortion at some pixel locations. Image quality assessment (IQA) metrics are commonly utilized to assess the quality of rendered images, but their results are a global difference value, which does not provide pixel‐wise differences to optimize the renderings. In contrast, visibility difference models including visual perception models and deep learning models can calculate pixel‐wise visibility difference between distorted images and reference images. However, they either are only applied to a single type of visible distortion or are seriously dependent on datasets. To this end, the authors propose a novel model, dubbed Human Visual Perception and Deep Learning Image Difference Metric (HPDL‐IDM), which combines the Human Visual System (HVS) model and deep learning. HPDL‐IDM primarily consists of two modules: (i) the visual perception feature calculation module, which calculates difference maps between various kinds of features extracted from the reference image and the distorted image according to the visual characteristics of human eyes and concatenates them, and (ii) the deep learning module, which utilizes a neural network of encoder–decoder structure to train on the LocvisVC and VisTexRes datasets whose input and output are these concatenated feature difference maps and the final image distortion visibility difference map respectively. Additionally, the authors pool the final difference map into a global difference value between 0 and 1 to apply their model to many image processing tasks related to Image quality metrics (IQMs). Experimental results show that HPDL‐IDM's generalization capacity and accuracy are improved by a large margin compared to other models. [ABSTRACT FROM AUTHOR]