Search

Specular reflections pose great challenges on various multimedia and computer vision tasks, e.g., image segmentation, detection and matching. In this paper, we build a large-scale Paired Specular-Diffuse (PSD) image dataset, where the images are carefully captured by using real-world objects and the ground-truth specular-free diffuse images are provided. To the best of our knowledge, this is the first real-world benchmark dataset for specular highlight removal task, which is useful for evaluating and encouraging new deep learning-based approaches. Given this dataset, we present a novel Generative Adversarial Network (GAN) for specular highlight removal from a single image by introducing the detection of specular reflection information as a guidance. Our network also makes full use of the attention mechanism and is able to directly model the mapping relation between the diffuse area and the specular highlight area without any explicit estimation of the illumination. Experimental results demonstrate that the proposed network is more effective to remove specular reflection components with the guidance of specular highlight detection than recent state-of-the-art methods.

A major challenges of deep learning (DL) is the necessity to collect huge amounts of training data. Often, the lack of a sufficiently large dataset discourages the use of DL in certain applications. Typically, acquiring the required amounts of data costs considerable time, material and effort. To mitigate this problem, the use of synthetic images combined with real data is a popular approach, widely adopted in the scientific community to effectively train various detectors. In this study, we examined the potential of synthetic data-based training in the field of intelligent transportation systems. Our focus is on camera-based traffic sign recognition applications for advanced driver assistance systems and autonomous driving. The proposed augmentation pipeline of synthetic datasets includes novel augmentation processes such as structured shadows and gaussian specular highlights. A well-known DL model was trained with different datasets to compare the performance of synthetic and real image-based trained models. Additionally, a new, detailed method to objectively compare these models is proposed. Synthetic images are generated using a semi-supervised errors-guide method which is also described. Our experiments showed that a synthetic image-based approach outperforms in most cases real image-based training when applied to cross-domain test datasets (+10% precision for GTSRB dataset) and consequently, the generalization of the model is improved decreasing the cost of acquiring images.

We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images for any desired scene. Our technique includes foreground estimation via alpha matting, relighting, and compositing. We demonstrate that each of these stages can be tackled in a sequential pipeline without the use of priors (e.g. known background or known illumination) and with no specialized acquisition techniques, using only a single RGB portrait image and a novel, target HDR lighting environment as inputs. We train our model using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes. To perform realistic relighting for compositing, we introduce a novel per-pixel lighting representation in a deep learning framework, which explicitly models the diffuse and the specular components of appearance, producing relit portraits with convincingly rendered non-Lambertian effects like specular highlights. Multiple experiments and comparisons show the effectiveness of the proposed approach when applied to in-the-wild images.

This paper addresses the task of estimating spatially-varying reflectance (i.e., SVBRDF) from a single, casually captured image. Central to our method is a highlight-aware (HA) convolution operation and a two-stream neural network equipped with proper training losses. Our HA convolution, as a novel variant of standard (ST) convolution, directly modulates convolution kernels under the guidance of automatically learned masks representing potentially overexposed highlight regions. It helps to reduce the impact of strong specular highlights on diffuse components and at the same time, hallucinates plausible contents in saturated regions. Considering that variation of saturated pixels also contains important cues for inferring surface bumpiness and specular components, we design a two-stream network to extract features from two different branches stacked by HA convolutions and ST convolutions, respectively. These two groups of features are further fused in an attention-based manner to facilitate feature selection of each SVBRDF map. The whole network is trained end to end with a new perceptual adversarial loss which is particularly useful for enhancing the texture details. Such a design also allows the recovered material maps to be disentangled. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to recover clear SVBRDFs from a single casually captured image, and performs favorably against state-of-the-arts. Since we impose very few constraints on the capture process, even a non-expert user can create high-quality SVBRDFs that cater to many graphical applications.

Specular highlights often exist in real-world images due to the material property of objects and the capturing environments. High light in real-life scenes can reduce image quality and cause loss of image information, to overcome this, we propose an effective specular reflection image enhancement algorithm. First, the scene depth map of specular reflection image was obtained by the color attenuation prior method; then, two adaptive adjustment factors were developed to optimize the transmission map for specular reflection images and $L_{0}$ gradient minimization filter was used to eliminate halo artifacts. Finally, the nonuniform illumination compensation strategy based on the YUV color space was used to expand the dynamic range of the image and increase the saturation of the image. Experimental results via subjective and objective evaluations demonstrate that the proposed method can provide better visual quality than other methods and can effectively enhance the information in the specular reflection image.

Obstacle distance measurement is one of the key technologies for autonomous navigation of high-voltage transmission line inspection robots. To address the robustness of obstacle distance measurement under varying illumination conditions, this article develops a research method that fuses image enhancement with robot monocular vision so that the robot can adapt to various levels of illumination running along the transmission line. During the inspection of high-voltage transmission lines in such an overexposed (excessively bright) environment, a specular highlight suppression method is proposed to suppress the specular reflections in an image; when scene illumination is insufficient, a robust low-light image enhancement method based on a tone mapping algorithm with weighted guided filtering is presented. Based on the monocular vision measurement principle, the error generation mechanism is analyzed through experiments, and we introduce the parameter modification mechanism. The two proposed image enhancement methods outperform other state-of-the-art enhancement algorithms in qualitative and quantitative analyses. The experimental results show that the measurement error is less than 3% for static distance measurements and less than 5% for dynamic distance measurements within 6 m. The proposed method can meet the requirements of high-accuracy positioning, real-time performance and strong robustness. This method greatly contributes to the sustainable development of inspection robots in the power industry.

In this paper, we propose a specular highlight image enhancement algorithm based on a dark channel prior to solve the problem of information loss in specular highlight images in real scenes. First, the algorithm is based on the dark channel prior algorithm, and a moving window minimum filter is used to estimate the global illumination component. Then, a weighted function based on the local pixel color difference is introduced to solve the halo artifacts in the image. Then, the improved guided filter algorithm is used to optimize the transmittance, which improves the computational efficiency of the algorithm. Finally, the brightness of the image is adjusted by the CLAHE algorithm to enhance the local details of the image. Experimental results show that this method can effectively enhance the information in specular highlight images, and its processing results are obviously better than those of other algorithms.

In scenes lit with sharp point-like light sources, light can bounce several times on specular materials before getting into our eyes, forming purely specular light paths. However, to our knowledge, rendering such multi-bounce pure specular paths has not been handled in previous work: while many light transport methods have been devised to sample various kinds of light paths, none of them are able to find multi-bounce pure specular light paths from a point light to a pinhole camera. In this paper, we present path cuts to efficiently render such light paths. We use a path space hierarchy combined with interval arithmetic bounds to prune non-contributing regions of path space, and to slice the path space into regions small enough to empirically contain at most one solution. Next, we use an automatic differentiation tool and a Newton-based solver to find an admissible specular path within a given path space region. We demonstrate results on several complex specular configurations, including RR, TT, TRT and TTTT paths.

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects., Comment: Siggraph Asia 2020

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.

As an important branch of computational optics, the light field can simultaneously record the spatial intensity and propagation direction of light, which provides a new way to realize three-dimensional (3D) measurement. Although light field measuring system (LFMS) can quickly detect nontransparent surfaces, specular light affects the overall measurement result. In this paper, an LFMS based on a diffuse light source and polarization technology is proposed to eliminate specular light and obtain a more accurate 3D reconstruction result in one exposure. During the measurement, a series of images corresponding to continuously varying polarization angle are captured, in which the value(V)-channel value distribution is used as the criterion of specular light removal efficiency, and hence, the image at the optimal polarization angle is obtained, in which the influence of specular light is removed. The depth map and specular light removal efficiency are experimentally compared with those of the traditional LFMS. The experimental results show that the depth map error can be greatly reduced, which further verifies the feasibility of the proposed method and system. The present study can be extended to the traditional optical system or other machine vision measurement methods involving specular light.

In natural rainy scenes, visibility is significantly degraded by two types of phenomena: specular highlights of nearby individual rain streaks and atmospheric veiling effect caused by distant accumulated rain. However, most existing deraining methods only take the first kind of degradation into consideration, which limits their potential application in heavy rain. In this study, a joint rain and atmospheric veil removal framework is proposed to address this problem. Since rain streaks and rain accumulation are entangled with each other, which is intractable to simulate, causing clean/rainy image pairs of real-world are hard to generate. Hence, after introducing a generalised rain model, which can represent both rain streaks and atmospheric veil physically, the authors do not learn the mapping function between image pairs using deep-learning architecture, but estimate the rain streaks, transmission, and atmospheric light via Gaussian mixture model patch prior and dark channel prior to solve the rain model instead. According to the comprehensive experimental evaluations, the proposed method outperforms other state-of-the-art methods in terms of both high visibility and vivid colour, especially in natural heavy rain scenario.

An improved spectral reflectance estimation method was developed to transform captured RGB images to spectral reflectance. The novelty of our method is an iteratively reweighted regulated model that combines polynomial expansion signals, which was developed for spectral reflectance estimation, and a cross-polarized imaging system, which is used to eliminate glare and specular highlights. Two RGB images are captured under two illumination conditions. The method was tested using ColorChecker charts. The results demonstrate that the proposed method could make a significant improvement of the accuracy in both spectral and colorimetric: it can achieve 23.8% improved accuracy in mean CIEDE2000 color difference, while it achieves 24.6% improved accuracy in RMS error compared with classic regularized least squares (RLS) method. The proposed method is sufficiently accurate in predicting the spectral properties and their performance within an acceptable range, i.e., typical customer tolerance of less than 3 DE units in the graphic arts industry.

This work addresses the reflection removal with flash and no-flash image pairs to separate reflection from transmission. When objects are covered by glass, the no-flash image usually contains reflection, and thus flash is used to enhance transmission details. However, the flash image suffers from the specular highlight on the glass surface caused by flash. In this paper, we propose a siamese dense network (SDN) for reflection removal with flash and no-flash image pairs. SDN extracts shareable and complementary features via concatenated siamese dense blocks. We utilize an image fusion block for the SDN to fuse the intermediate output of two branches. Since severe information loss occurs in the specular highlight, we detect the specular highlight in the flash image based on gradient of the maximum chromaticity. Through observations, flash causes various artifacts such as tone distortion and inhomogeneous brightness. Thus, with synthetic datasets we collect 758 pairs of real flash and no-flash image pairs (including their ground truth) by different cameras to gain generalization. Various experiments show that the proposed method successfully removes reflections using flash and no-flash image pairs and outperforms state-of-the-art ones in terms of visual quality and quantitative measurements. Besides, we apply the SDN to color/depth image pairs and achieve both color reflection removal and depth filling.

Rendering specular material appearance is a core problem of computer graphics. While smooth analytical material models are widely used, the high-frequency structure of real specular highlights requires considering discrete, finite microgeometry. Instead of explicit modeling and simulation of the surface microstructure (which was explored in previous work), we propose a novel direction: learning the high-frequency directional patterns from synthetic or measured examples, by training a generative adversarial network (GAN). A key challenge in applying GAN synthesis to spatially varying BRDFs is evaluating the reflectance for a single location and direction without the cost of evaluating the whole hemisphere. We resolve this using a novel method for partial evaluation of the generator network. We are also able to control large-scale spatial texture using a conditional GAN approach. The benefits of our approach include the ability to synthesize spatially large results without repetition, support for learning from measured data, and evaluation performance independent of the complexity of the dataset synthesis or measurement.

Endoscopy images show part of the gastrointestinal tract or other parts of the human body. Due to the complex environment in which this tract is located in the human body and the limitations of the image acquisition equipment, endoscopy images may present blur and specular highlights that are common types of degradation. In this paper, we present a blind deblurring approach for endoscopy images that estimate the blur kernel from the fusion of the specular highlights. The proposed method can be divided into two phases. First, the specular highlights are precisely extracted from the original image using a sparse and low-rank image decomposition approach. Then, these highlights are automatically clustered and fused according to the spatial and intensity features. In this way, potential kernels are generated and used for the deblurring process. The experimental results show that the proposed method presents an improved and comparable performance when considered other state-of-the-art methods that use highlights segmentation and image deblurring.

The internal structure of hair consists of three layers: the cuticle, the cortex, and the medulla, and there are multiple membrane structures inside the cuticle. Light incident on these subwavelength structures is subject to scattering and interference phenomena, and the reflected highlights vary depending on the viewing angle. This phenomenon of light coloration due to the microstructure is called structural coloring or photonic coloration. It shows a unique specular highlight and determines the magnitude of the specular highlight, which is not observable in the linear optics of CG. In the case of hairs, this structural coloring or photonic coloration occurs in addition to the simple surface reflections and the backscattering lights, which penetrate the hair and are absorbed by the melanin pigment. In the present report, we mainly discuss the effects of the structural coloring first caused by the multi-layered structure in the cuticle region on the hair surface, simulating the electromagnetic field using the NS-FDTD (Non-Standard Finite Difference in Time Domain) method. In addition, we also discuss the backscattering phenomena inside the hair fibers.

Several recent works have presented image-based methods for creating high-fidelity immersive virtual environments from photographs of real-world scenes. In this paper, we provide a user-centered evaluation of such methods by way of a user study investigating their impact on viewers’ perception of visual realism and sense of presence. In particular, we focus on two specific elements commonly introduced by image-based approaches. First, we investigate the extent to which using dedicated image-based rendering algorithms to render the scene with view-dependent effects (such as specular highlights) causes users to perceive it as being more realistic. Second, we study whether making the scene fade out beyond a fixed volume in 3D space significantly reduces participants’ feeling of being there, examining different sizes for this viewing volume. To provide details on the virtual environment used in the study, we also describe how we recreated a museum gallery for room-scale virtual reality using a custom-built multi-camera rig. The results of our study show that using image-based rendering to render view-dependent effects can effectively enhance the perception of visual realism and elicit a stronger sense of presence, even when it implies constraining the viewing volume to a small range of motion.

Sophisticated generative adversary network (GAN) models are now able to synthesize highly realistic human faces that are difficult to discern from real ones visually. In this work, we show that GAN synthesized faces can be exposed with the inconsistent corneal specular highlights between two eyes. The inconsistency is caused by the lack of physical/physiological constraints in the GAN models. We show that such artifacts exist widely in high-quality GAN synthesized faces and further describe an automatic method to extract and compare corneal specular highlights from two eyes. Qualitative and quantitative evaluations of our method suggest its simplicity and effectiveness in distinguishing GAN synthesized faces.

Specular highlight detection and removal are fundamental and challenging tasks. Although recent methods have achieved promising results on the two tasks by training on synthetic training data in a supervised manner, they are typically solely designed for highlight detection or removal, and their performance usually deteriorates significantly on real-world images. In this paper, we present a novel network that aims to detect and remove highlights from natural images. To remove the domain gap between synthetic training samples and real test images, and support the investigation of learning-based approaches, we first introduce a dataset with about 16K real images, each of which has the corresponding ground truths of highlight detection and removal. Using the presented dataset, we develop a multi-task network for joint highlight detection and removal, based on a new specular highlight image formation model. Experiments on the benchmark datasets and our new dataset show that our approach clearly outperforms state-of-the-art methods for both highlight detection and removal.

We introduce a neural network-based method to denoise pairs of images taken in quick succession, with and without a flash, in low-light environments. Our goal is to produce a high-quality rendering of the scene that preserves the color and mood from the ambient illumination of the noisy no-flash image, while recovering surface texture and detail revealed by the flash. Our network outputs a gain map and a field of kernels, the latter obtained by linearly mixing elements of a per-image low-rank kernel basis. We first apply the kernel field to the no-flash image, and then multiply the result with the gain map to create the final output. We show our network effectively learns to produce high-quality images by combining a smoothed out estimate of the scene's ambient appearance from the no-flash image, with high-frequency albedo details extracted from the flash input. Our experiments show significant improvements over alternative captures without a flash, and baseline denoisers that use flash no-flash pairs. In particular, our method produces images that are both noise-free and contain accurate ambient colors without the sharp shadows or strong specular highlights visible in the flash image., CVPR 2021. Project page at https://www.cse.wustl.edu/~zhihao.xia/deepfnf/

In this paper, a novel, to the best of our knowledge, iterative approach for highlight removal is proposed using lenselet-based plenoptic cameras without multiple exposures. An unsupervised k-means clustering approach that relates unsaturated pixels to chromatic dispersion based on the intrinsic decomposition and dichromatic reflection model is proposed to recover unsaturated highlights. Meanwhile, an adaptive direction method along with a Gaussian probability distribution model is designed to recover the saturated highlights. Finally, a method that combines the specular residual ratio with information entropy is built to quantitatively evaluate the quality of highlight removal. Generally, our method not only fully removes specular highlights, but also has low spatial complexity of image acquisition, more stability, and outstanding restoration for complex scenes.

Specular highlights are commonplace in images, however, methods for detecting them and removing the phenomenon are particularly challenging. A reason for this is the difficulty in creating a dataset for training or evaluation, as in the real world, we lack the necessary control over the environment. Therefore, we propose a novel physically-based rendered LIGHT Specularity (LIGHTS) Dataset for the evaluation of the specular highlight detection task. Our dataset consists of 18 high-quality architectural scenes, where each scene is rendered with multiple views. In total, the dataset contains 2, 603 views with an average of 145 views per scene. Additionally, we propose a simple aggregation based method for specular highlight detection that outperforms prior work by 3.6% in two orders of magnitude less time on our dataset.

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery. Practically, specular reflection (a.k.a. highlight) always exists in endoscopic images and significantly affects surgeons’ observation and judgment. Motivated by the fact that the values of the red channel in nonhighlight area of endoscopic images are higher than those of the green and blue ones, this paper proposes an adaptive specular highlight detection method for endoscopic images. Specifically, for each pixel, we design a criterion for specular highlight detection based on the ratio of the red channel to both the green and blue channels. With the designed criteria, we take advantage of image segmentation and then develop an adaptive threshold with respect to the differences between the red channel and the other ones of neighboring pixels. To validate the proposed method, we conduct experiments on clinical data and CVC-ClinicSpec open database. The experimental results demonstrate that the proposed method yields an averaged Precision, Accuracy, and F1-score rate of 88.76%, 99.60% and 72.56%, respectively, and outperforms the state-of-the-art approaches based on color distribution reported for endoscopic highlight detection.

International audience; Dans ce papier, nous introduisons et résolvons le problème de stéréophotométrie pour les caméras 360◦ accessibles au grand public. Nous présentons une équation de l’irradiance pour les images sphériques, spécifiquement adaptée aux caméras twin-fisheye cameras ainsi qu’un algorithme pour l’estimation de la direction des éclairages, en s’appuyant sur la tache spéculaire qui apparait sur des sphères miroir. Plusieurs expérimentations sont réalisées à partir d’images de synthèse et d’images réelles acquises avec une caméra Ricoh Theta V, afin de vérifier le fonctionnement et la robustesse de l’approche proposée. Pour permettre et faciliter la reproduction des résultats, les jeux de données ainsi que les programmes dé- veloppés sont publiquement disponibles à cette adresse :home.mis.u-picardie.fr/∼fabio/PhotoSphere.html

Removing undesirable specular highlight from a single input image is of crucial importance to many computer vision and graphics tasks. Existing methods typically remove specular highlight for medical images and specific-object images, however, they cannot handle the images with text. In addition, the impact of specular highlight on text recognition is rarely studied by text detection and recognition community. Therefore, in this paper, we first raise and study the text-aware single image specular highlight removal problem. The core goal is to improve the accuracy of text detection and recognition by removing the highlight from text images. To tackle this challenging problem, we first collect three high-quality datasets with fine-grained annotations, which will be appropriately released to facilitate the relevant research. Then, we design a novel two-stage network, which contains a highlight detection network and a highlight removal network. The output of highlight detection network provides additional information about highlight regions to guide the subsequent highlight removal network. Moreover, we suggest a measurement set including the end-to-end text detection and recognition evaluation and auxiliary visual quality evaluation. Extensive experiments on our collected datasets demonstrate the superior performance of the proposed method.

Previous methods of highlight removal in image processing have exclusively addressed images taken in specific illumination environments. However, most of these methods have limitations in natural scenes and thus, introduce artifacts to nature images. In this work, we propose a specular highlight removal method that is applicable to natural scene image. Firstly, we decompose the input image into a reflectance image and an illumination image based on Retinex theory, and show that the illumination image of natural scene is obviously different from that of commonly used experimental scene. Then, the smooth features of the input image are extracted to help estimate the specular reflection coefficient in chromaticity space. Finally, the space transformation is used to calculate the specular components and the highlight removal is achieved by subtracting the specular reflection component from the original image. The experimental results show that our method outperforms most of existing methods on natural scene images, especially in some challenging scenarios with saturated pixels and complex textures.

Controlled capture of real-world material appearance yields tabulated sets of highly realistic reflectance data. In practice, however, its high memory footprint requires compressing into a representation that can be used efficiently in rendering while remaining faithful to the original. Previous works in appearance encoding often prioritised one of these requirements at the expense of the other, by either applying high-fidelity array compression strategies not suited for efficient queries during rendering, or by fitting a compact analytic model that lacks expressiveness. We present a compact neural network-based representation of BRDF data that combines high-accuracy reconstruction with efficient practical rendering via built-in interpolation of reflectance. We encode BRDFs as lightweight networks, and propose a training scheme with adaptive angular sampling, critical for the accurate reconstruction of specular highlights. Additionally, we propose a novel approach to make our representation amenable to importance sampling: rather than inverting the trained networks, we learn to encode them in a more compact embedding that can be mapped to parameters of an analytic BRDF for which importance sampling is known. We evaluate encoding results on isotropic and anisotropic BRDFs from multiple real-world datasets, and importance sampling performance for isotropic BRDFs mapped to two different analytic models.

In this paper, we present a technique for estimating the geometry and reflectance of objects using only a camera, flashlight, and optionally a tripod. We propose a simple data capture technique in which the user goes around the object, illuminating it with a flashlight and capturing only a few images. Our main technical contribution is the introduction of a recursive neural architecture, which can predict geometry and reflectance at 2^{k}*2^{k} resolution given an input image at 2^{k}*2^{k} and estimated geometry and reflectance from the previous step at 2^{k-1}*2^{k-1}. This recursive architecture, termed RecNet, is trained with 256x256 resolution but can easily operate on 1024x1024 images during inference. We show that our method produces more accurate surface normal and albedo, especially in regions of specular highlights and cast shadows, compared to previous approaches, given three or fewer input images. For the video and code, please visit the project website http://dlichy.github.io/ShapeAndMaterialAtHome/., Comment: Accepted to CVPR 2021

Particles are widely used in movie production rendering for various different effects. Blobbies (aka metaballs) are a very useful primitive to bridge the intermediate regime between the bulk of a fluid and fast-moving spray particles as well as providing geometric variation to droplets. The use of anisotropy allows one to represent thin line structures better than classic isotropic shapes. Tessellation of such fine geometric structures is prone to geometric artifacts, especially under strong motion blur, which may heavily distort the surface during the shutter or because topology changes can't be represented well. Intersecting rays with the isosurface analytically has robustness and precision advantages. Operating on the original representation provides highly accurate spatial and temporal derivatives that are useful for filtering specular highlights. In this chapter we describe some algorithmic tools to robustly and efficiently intersect blobby surfaces supporting anisotropy and higher-order motion blur.

Specular highlight removal is a challenging task. We present a novel data-driven approach for automatic specular highlight removal from a single image. To this end, we build a new dataset of real-world images for specular highlight removal with corresponding ground-truth diffuse images. Based on the dataset, we also present a specular highlight removal network by introducing the detection of specular reflections information as guidance. The experimental evaluations indicate that the proposed approach outperforms recent state-of-the-art methods.

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination (“daylight priors”). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy. Observers made achromatic settings within three-dimensional rendered scenes containing matte or glossy shapes, illuminated by either daylight or nondaylight illuminations. We assessed both the variability of these settings and their accuracy, in terms of the standard color constancy index (CCI). When a spectrally uniform background was present, neither CCIs nor variability improved with specular highlights or daylight illuminants (Experiment 1). When a Mondrian background was introduced, CCIs decreased overall but were higher for scenes containing glossy, as opposed to matte, shapes (Experiments 2 and 3). There was no overall reduction in variability of settings and no benefit for scenes illuminated by daylights. Taken together, these results suggest that the human visual system indeed uses specular highlights to improve color constancy but only when other cues, such as from the local surround, are weakened.

Different lighting conditions surrounding an object can cause specular reflections, resulting in specular highlights in the captured image. These can interfere with image processing algorithms, leading to false interpretation of results. Hence, applications that demand consistent object appearance require that such highlights are detected and localized for subsequent processing. Using a slightly modified U-Net architecture, we propose a semantic segmentation model to localize specular highlights. To evaluate the segmentation performance, the IoU metric is used. This measures how good the segmentation prediction matches the corresponding reference annotation by dividing their intersection by their union (IoU). The model achieved a frequency-weighted and mean IoU of 0.83 and 0.75 respectively when tested on a benchmark dataset. Furthermore, the proposed network was also trained and tested on a custom dataset, focusing on flash light reflections in an underground tunnel environment. For this custom dataset, the model achieved a frequency-weighted and mean IoU of 0.98 and 0.80 respectively.

Specular highlight detection is a challenging problem, and has many applications such as shiny object detection and light source estimation. Although various highlight detection methods have been proposed, they fail to disambiguate bright material surfaces from highlights, and cannot handle non-white-balanced images. Moreover, at present, there is still no benchmark dataset for highlight detection. In this paper, we present a large-scale real-world highlight dataset containing a rich variety of material categories, with diverse highlight shapes and appearances, in which each image is with an annotated ground-truth mask. Based on the dataset, we develop a deep learning-based specular highlight detection network (SHDNet) leveraging multi-scale context contrasted features to accurately detect specular highlights of varying scales. In addition, we design a binary cross-entropy (BCE) loss and an intersection-over-union edge (IoUE) loss for our network. Compared with existing highlight detection methods, our method can accurately detect highlights of different sizes, while effectively excluding the non-highlight regions, such as bright materials, non-specular as well as colored lighting, and even light sources.

Purpose Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure. Methods We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos. Results We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods. Conclusion The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.

In this work, we present a method to turn still life paintings with global illumination effects into dynamic paintings with moving lights. Our method specifically focuses on still life images containing (1) glass objects, which can have specular highlights with Fresnel effect and (2) fruits, which can have reflection and subsurface scattering. Our goal is to preserve the original look-and-feel of the still-life paintings while allowing the user to move the light source anywhere in the scene, causing the shadows, diffuse shading, as well as reflections and specular highlights to move according to the new light position. We have provided a proof of concept based on an original digital painting. This method can be used to turn any similar still life painting into a dynamic painting.

We propose a model of plane ovals based on two foci defined in the 3D space. Geometrically, some of these ovals are the isocontours of a planar specular highlight for a point light source according to Phong’s reflectance model. We, hence, call them the highlight ovals. More precisely, we define a geometric highlight oval as the locus of points p whose product of distances to the foci is proportional to the dot product of the vectors from p to the foci. When the proportionality is taken up to sign, this gives a quartic, whose roots define the algebraic highlight oval. Similarly to the Cartesian oval, the algebraic formulation hence includes a pair of geometric highlight ovals. Similarly to the Cassinian oval, the geometric definition includes non-convex closed curves. The highlight ovals have four intrinsic parameters, namely the height of the two foci relative to the oval’s plane, the distance between the two foci’s projection on the oval’s plane and a scale factor fixing the proportionality and related to the isocontour’s intensity. We thoroughly study the topology and convexity of the family of algebraic highlight ovals for any combination of its four intrinsic parameters and show how this characterizes the geometric highlight ovals, and thus the isocontours of specular highlights. We report an extensive experimental evaluation, showing that the highlight ovals form a reliable model of specular highlights.

Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Code/models: https://github.com/nadeemlab/CEP., CVPR 2020. **First two authors contributed equally to this work

Specularity or highlight problem exists widely in hyperspectral images, provokes reflectance deviation from its true value, and can hide major defects in food objects or detecting spurious false defects causing failure of inspection and detection processes. In this study, a new non-iterative method based on the dichromatic reflection model and principle component analysis (PCA) was proposed to detect and remove specular highlight components from hyperspectral images acquired by various imaging modes and under different configurations for numerous agro-food products. To demonstrate the effectiveness of this approach, the details of the proposed method were described and the experimental results on various spectral images were presented. The results revealed that the method worked well on all hyperspectral and multispectral images examined in this study, effectively reduced the specularity and significantly improves the quality of the extracted spectral data. Besides the spectral images from available databases, the robustness of this approach was further validated with real captured hyperspectral images of different food materials. By using qualitative and quantitative evaluation based on running time and peak signal to noise ratio (PSNR), the experimental results showed that the proposed method outperforms other specularity removal methods over the datasets of hyperspectral and multispectral images. info:eu-repo/semantics/acceptedVersion

The purpose of this research is to create new automatic methods for endoscopic image digital processing, ensuring their high ergonomics and the possibility of effective use in clinical decision support systems. As the result of investigation, the following methods were proposed: the detection and removal of specular highlights; compensation of radial and tangential geometric distortions; the mosaic panorama creation from the input video stream with low level of detail; brightness and contrast enhancement, providing simultaneous successful correction of both dark and bright areas of the image (uneven contrast) without significant underlining of the noise component typical for the existing nonlinear contrasting methods, especially in low-detail image areas; custom color correction based on linear transformation matrix taking into account endoscopic image characteristics and making it possible to customize color palette according to the physician individual preferences. The methods considered were successfully tested on real endoscopic images at the department of innovative medical devices of the Korean Electrotechnological Research Institute. The test results demonstrate their effectiveness and applicability in clinical decision support systems.

© 2013 IEEE. The presence of specular highlight is a critical issue for both natural and medical images such as those produced by laparoscopes, which can lead to erroneous visual tracking, stereo reconstruction, and image segmentation. Specular highlight removal from a single image is necessary for image analysis and applications. Due to the differences between natural and medical image scenes, existing literature to address this issue has only been effective on natural images or medical images with textureless regions. To overcome this limitation, we propose a global optimization method for specular highlight removal from a single image based on a dichromatic reflection model. In addition to introducing modified illumination chromaticity, the proposed method consists of two novel steps: one for estimating diffuse chromaticity by correcting hue and saturation on highlighted regions, and the other for estimating diffuse and specular reflection coefficients using convex optimization with double regularization. The estimated diffuse chromaticity is proven to approximate the true diffuse chromaticity and the proposed optimization algorithm is guaranteed to find the optimal diffuse coefficients. Experimental results show that the proposed method can effectively remove specular highlights from both natural images and endoscopic images with texture detail preservation. To further demonstrate the efficacy of our proposed method, an application of stereo reconstruction using a public dataset illustrates that our highlight removal method can enhance surface reconstruction accuracy from 1.10mm RMSD to 0.69mm RMSD.