Search

Specular reflections pose great challenges on various multimedia and computer vision tasks, e.g., image segmentation, detection and matching. In this paper, we build a large-scale Paired Specular-Diffuse (PSD) image dataset, where the images are carefully captured by using real-world objects and the ground-truth specular-free diffuse images are provided. To the best of our knowledge, this is the first real-world benchmark dataset for specular highlight removal task, which is useful for evaluating and encouraging new deep learning-based approaches. Given this dataset, we present a novel Generative Adversarial Network (GAN) for specular highlight removal from a single image by introducing the detection of specular reflection information as a guidance. Our network also makes full use of the attention mechanism and is able to directly model the mapping relation between the diffuse area and the specular highlight area without any explicit estimation of the illumination. Experimental results demonstrate that the proposed network is more effective to remove specular reflection components with the guidance of specular highlight detection than recent state-of-the-art methods.

We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images for any desired scene. Our technique includes foreground estimation via alpha matting, relighting, and compositing. We demonstrate that each of these stages can be tackled in a sequential pipeline without the use of priors (e.g. known background or known illumination) and with no specialized acquisition techniques, using only a single RGB portrait image and a novel, target HDR lighting environment as inputs. We train our model using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes. To perform realistic relighting for compositing, we introduce a novel per-pixel lighting representation in a deep learning framework, which explicitly models the diffuse and the specular components of appearance, producing relit portraits with convincingly rendered non-Lambertian effects like specular highlights. Multiple experiments and comparisons show the effectiveness of the proposed approach when applied to in-the-wild images.

Specular highlights often exist in real-world images due to the material property of objects and the capturing environments. High light in real-life scenes can reduce image quality and cause loss of image information, to overcome this, we propose an effective specular reflection image enhancement algorithm. First, the scene depth map of specular reflection image was obtained by the color attenuation prior method; then, two adaptive adjustment factors were developed to optimize the transmission map for specular reflection images and $L_{0}$ gradient minimization filter was used to eliminate halo artifacts. Finally, the nonuniform illumination compensation strategy based on the YUV color space was used to expand the dynamic range of the image and increase the saturation of the image. Experimental results via subjective and objective evaluations demonstrate that the proposed method can provide better visual quality than other methods and can effectively enhance the information in the specular reflection image.

Obstacle distance measurement is one of the key technologies for autonomous navigation of high-voltage transmission line inspection robots. To address the robustness of obstacle distance measurement under varying illumination conditions, this article develops a research method that fuses image enhancement with robot monocular vision so that the robot can adapt to various levels of illumination running along the transmission line. During the inspection of high-voltage transmission lines in such an overexposed (excessively bright) environment, a specular highlight suppression method is proposed to suppress the specular reflections in an image; when scene illumination is insufficient, a robust low-light image enhancement method based on a tone mapping algorithm with weighted guided filtering is presented. Based on the monocular vision measurement principle, the error generation mechanism is analyzed through experiments, and we introduce the parameter modification mechanism. The two proposed image enhancement methods outperform other state-of-the-art enhancement algorithms in qualitative and quantitative analyses. The experimental results show that the measurement error is less than 3% for static distance measurements and less than 5% for dynamic distance measurements within 6 m. The proposed method can meet the requirements of high-accuracy positioning, real-time performance and strong robustness. This method greatly contributes to the sustainable development of inspection robots in the power industry.

In this paper, we propose a specular highlight image enhancement algorithm based on a dark channel prior to solve the problem of information loss in specular highlight images in real scenes. First, the algorithm is based on the dark channel prior algorithm, and a moving window minimum filter is used to estimate the global illumination component. Then, a weighted function based on the local pixel color difference is introduced to solve the halo artifacts in the image. Then, the improved guided filter algorithm is used to optimize the transmittance, which improves the computational efficiency of the algorithm. Finally, the brightness of the image is adjusted by the CLAHE algorithm to enhance the local details of the image. Experimental results show that this method can effectively enhance the information in specular highlight images, and its processing results are obviously better than those of other algorithms.

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects., Comment: Siggraph Asia 2020

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.

This work addresses the reflection removal with flash and no-flash image pairs to separate reflection from transmission. When objects are covered by glass, the no-flash image usually contains reflection, and thus flash is used to enhance transmission details. However, the flash image suffers from the specular highlight on the glass surface caused by flash. In this paper, we propose a siamese dense network (SDN) for reflection removal with flash and no-flash image pairs. SDN extracts shareable and complementary features via concatenated siamese dense blocks. We utilize an image fusion block for the SDN to fuse the intermediate output of two branches. Since severe information loss occurs in the specular highlight, we detect the specular highlight in the flash image based on gradient of the maximum chromaticity. Through observations, flash causes various artifacts such as tone distortion and inhomogeneous brightness. Thus, with synthetic datasets we collect 758 pairs of real flash and no-flash image pairs (including their ground truth) by different cameras to gain generalization. Various experiments show that the proposed method successfully removes reflections using flash and no-flash image pairs and outperforms state-of-the-art ones in terms of visual quality and quantitative measurements. Besides, we apply the SDN to color/depth image pairs and achieve both color reflection removal and depth filling.

An improved spectral reflectance estimation method was developed to transform captured RGB images to spectral reflectance. The novelty of our method is an iteratively reweighted regulated model that combines polynomial expansion signals, which was developed for spectral reflectance estimation, and a cross-polarized imaging system, which is used to eliminate glare and specular highlights. Two RGB images are captured under two illumination conditions. The method was tested using ColorChecker charts. The results demonstrate that the proposed method could make a significant improvement of the accuracy in both spectral and colorimetric: it can achieve 23.8% improved accuracy in mean CIEDE2000 color difference, while it achieves 24.6% improved accuracy in RMS error compared with classic regularized least squares (RLS) method. The proposed method is sufficiently accurate in predicting the spectral properties and their performance within an acceptable range, i.e., typical customer tolerance of less than 3 DE units in the graphic arts industry.

Endoscopy images show part of the gastrointestinal tract or other parts of the human body. Due to the complex environment in which this tract is located in the human body and the limitations of the image acquisition equipment, endoscopy images may present blur and specular highlights that are common types of degradation. In this paper, we present a blind deblurring approach for endoscopy images that estimate the blur kernel from the fusion of the specular highlights. The proposed method can be divided into two phases. First, the specular highlights are precisely extracted from the original image using a sparse and low-rank image decomposition approach. Then, these highlights are automatically clustered and fused according to the spatial and intensity features. In this way, potential kernels are generated and used for the deblurring process. The experimental results show that the proposed method presents an improved and comparable performance when considered other state-of-the-art methods that use highlights segmentation and image deblurring.

The purpose of this research is to create new automatic methods for endoscopic image digital processing, ensuring their high ergonomics and the possibility of effective use in clinical decision support systems. As the result of investigation, the following methods were proposed: the detection and removal of specular highlights; compensation of radial and tangential geometric distortions; the mosaic panorama creation from the input video stream with low level of detail; brightness and contrast enhancement, providing simultaneous successful correction of both dark and bright areas of the image (uneven contrast) without significant underlining of the noise component typical for the existing nonlinear contrasting methods, especially in low-detail image areas; custom color correction based on linear transformation matrix taking into account endoscopic image characteristics and making it possible to customize color palette according to the physician individual preferences. The methods considered were successfully tested on real endoscopic images at the department of innovative medical devices of the Korean Electrotechnological Research Institute. The test results demonstrate their effectiveness and applicability in clinical decision support systems.

© 2013 IEEE. The presence of specular highlight is a critical issue for both natural and medical images such as those produced by laparoscopes, which can lead to erroneous visual tracking, stereo reconstruction, and image segmentation. Specular highlight removal from a single image is necessary for image analysis and applications. Due to the differences between natural and medical image scenes, existing literature to address this issue has only been effective on natural images or medical images with textureless regions. To overcome this limitation, we propose a global optimization method for specular highlight removal from a single image based on a dichromatic reflection model. In addition to introducing modified illumination chromaticity, the proposed method consists of two novel steps: one for estimating diffuse chromaticity by correcting hue and saturation on highlighted regions, and the other for estimating diffuse and specular reflection coefficients using convex optimization with double regularization. The estimated diffuse chromaticity is proven to approximate the true diffuse chromaticity and the proposed optimization algorithm is guaranteed to find the optimal diffuse coefficients. Experimental results show that the proposed method can effectively remove specular highlights from both natural images and endoscopic images with texture detail preservation. To further demonstrate the efficacy of our proposed method, an application of stereo reconstruction using a public dataset illustrates that our highlight removal method can enhance surface reconstruction accuracy from 1.10mm RMSD to 0.69mm RMSD.

Sophisticated generative adversary network (GAN) models are now able to synthesize highly realistic human faces that are difficult to discern from real ones visually. In this work, we show that GAN synthesized faces can be exposed with the inconsistent corneal specular highlights between two eyes. The inconsistency is caused by the lack of physical/physiological constraints in the GAN models. We show that such artifacts exist widely in high-quality GAN synthesized faces and further describe an automatic method to extract and compare corneal specular highlights from two eyes. Qualitative and quantitative evaluations of our method suggest its simplicity and effectiveness in distinguishing GAN synthesized faces.

We introduce a neural network-based method to denoise pairs of images taken in quick succession, with and without a flash, in low-light environments. Our goal is to produce a high-quality rendering of the scene that preserves the color and mood from the ambient illumination of the noisy no-flash image, while recovering surface texture and detail revealed by the flash. Our network outputs a gain map and a field of kernels, the latter obtained by linearly mixing elements of a per-image low-rank kernel basis. We first apply the kernel field to the no-flash image, and then multiply the result with the gain map to create the final output. We show our network effectively learns to produce high-quality images by combining a smoothed out estimate of the scene's ambient appearance from the no-flash image, with high-frequency albedo details extracted from the flash input. Our experiments show significant improvements over alternative captures without a flash, and baseline denoisers that use flash no-flash pairs. In particular, our method produces images that are both noise-free and contain accurate ambient colors without the sharp shadows or strong specular highlights visible in the flash image., CVPR 2021. Project page at https://www.cse.wustl.edu/~zhihao.xia/deepfnf/

Specular highlights are commonplace in images, however, methods for detecting them and removing the phenomenon are particularly challenging. A reason for this is the difficulty in creating a dataset for training or evaluation, as in the real world, we lack the necessary control over the environment. Therefore, we propose a novel physically-based rendered LIGHT Specularity (LIGHTS) Dataset for the evaluation of the specular highlight detection task. Our dataset consists of 18 high-quality architectural scenes, where each scene is rendered with multiple views. In total, the dataset contains 2, 603 views with an average of 145 views per scene. Additionally, we propose a simple aggregation based method for specular highlight detection that outperforms prior work by 3.6% in two orders of magnitude less time on our dataset.

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery. Practically, specular reflection (a.k.a. highlight) always exists in endoscopic images and significantly affects surgeons’ observation and judgment. Motivated by the fact that the values of the red channel in nonhighlight area of endoscopic images are higher than those of the green and blue ones, this paper proposes an adaptive specular highlight detection method for endoscopic images. Specifically, for each pixel, we design a criterion for specular highlight detection based on the ratio of the red channel to both the green and blue channels. With the designed criteria, we take advantage of image segmentation and then develop an adaptive threshold with respect to the differences between the red channel and the other ones of neighboring pixels. To validate the proposed method, we conduct experiments on clinical data and CVC-ClinicSpec open database. The experimental results demonstrate that the proposed method yields an averaged Precision, Accuracy, and F1-score rate of 88.76%, 99.60% and 72.56%, respectively, and outperforms the state-of-the-art approaches based on color distribution reported for endoscopic highlight detection.

International audience; Dans ce papier, nous introduisons et résolvons le problème de stéréophotométrie pour les caméras 360◦ accessibles au grand public. Nous présentons une équation de l’irradiance pour les images sphériques, spécifiquement adaptée aux caméras twin-fisheye cameras ainsi qu’un algorithme pour l’estimation de la direction des éclairages, en s’appuyant sur la tache spéculaire qui apparait sur des sphères miroir. Plusieurs expérimentations sont réalisées à partir d’images de synthèse et d’images réelles acquises avec une caméra Ricoh Theta V, afin de vérifier le fonctionnement et la robustesse de l’approche proposée. Pour permettre et faciliter la reproduction des résultats, les jeux de données ainsi que les programmes dé- veloppés sont publiquement disponibles à cette adresse :home.mis.u-picardie.fr/∼fabio/PhotoSphere.html

Removing undesirable specular highlight from a single input image is of crucial importance to many computer vision and graphics tasks. Existing methods typically remove specular highlight for medical images and specific-object images, however, they cannot handle the images with text. In addition, the impact of specular highlight on text recognition is rarely studied by text detection and recognition community. Therefore, in this paper, we first raise and study the text-aware single image specular highlight removal problem. The core goal is to improve the accuracy of text detection and recognition by removing the highlight from text images. To tackle this challenging problem, we first collect three high-quality datasets with fine-grained annotations, which will be appropriately released to facilitate the relevant research. Then, we design a novel two-stage network, which contains a highlight detection network and a highlight removal network. The output of highlight detection network provides additional information about highlight regions to guide the subsequent highlight removal network. Moreover, we suggest a measurement set including the end-to-end text detection and recognition evaluation and auxiliary visual quality evaluation. Extensive experiments on our collected datasets demonstrate the superior performance of the proposed method.

Previous methods of highlight removal in image processing have exclusively addressed images taken in specific illumination environments. However, most of these methods have limitations in natural scenes and thus, introduce artifacts to nature images. In this work, we propose a specular highlight removal method that is applicable to natural scene image. Firstly, we decompose the input image into a reflectance image and an illumination image based on Retinex theory, and show that the illumination image of natural scene is obviously different from that of commonly used experimental scene. Then, the smooth features of the input image are extracted to help estimate the specular reflection coefficient in chromaticity space. Finally, the space transformation is used to calculate the specular components and the highlight removal is achieved by subtracting the specular reflection component from the original image. The experimental results show that our method outperforms most of existing methods on natural scene images, especially in some challenging scenarios with saturated pixels and complex textures.

In this paper, we present a technique for estimating the geometry and reflectance of objects using only a camera, flashlight, and optionally a tripod. We propose a simple data capture technique in which the user goes around the object, illuminating it with a flashlight and capturing only a few images. Our main technical contribution is the introduction of a recursive neural architecture, which can predict geometry and reflectance at 2^{k}*2^{k} resolution given an input image at 2^{k}*2^{k} and estimated geometry and reflectance from the previous step at 2^{k-1}*2^{k-1}. This recursive architecture, termed RecNet, is trained with 256x256 resolution but can easily operate on 1024x1024 images during inference. We show that our method produces more accurate surface normal and albedo, especially in regions of specular highlights and cast shadows, compared to previous approaches, given three or fewer input images. For the video and code, please visit the project website http://dlichy.github.io/ShapeAndMaterialAtHome/., Comment: Accepted to CVPR 2021

Specular highlight removal is a challenging task. We present a novel data-driven approach for automatic specular highlight removal from a single image. To this end, we build a new dataset of real-world images for specular highlight removal with corresponding ground-truth diffuse images. Based on the dataset, we also present a specular highlight removal network by introducing the detection of specular reflections information as guidance. The experimental evaluations indicate that the proposed approach outperforms recent state-of-the-art methods.

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination (“daylight priors”). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy. Observers made achromatic settings within three-dimensional rendered scenes containing matte or glossy shapes, illuminated by either daylight or nondaylight illuminations. We assessed both the variability of these settings and their accuracy, in terms of the standard color constancy index (CCI). When a spectrally uniform background was present, neither CCIs nor variability improved with specular highlights or daylight illuminants (Experiment 1). When a Mondrian background was introduced, CCIs decreased overall but were higher for scenes containing glossy, as opposed to matte, shapes (Experiments 2 and 3). There was no overall reduction in variability of settings and no benefit for scenes illuminated by daylights. Taken together, these results suggest that the human visual system indeed uses specular highlights to improve color constancy but only when other cues, such as from the local surround, are weakened.

Different lighting conditions surrounding an object can cause specular reflections, resulting in specular highlights in the captured image. These can interfere with image processing algorithms, leading to false interpretation of results. Hence, applications that demand consistent object appearance require that such highlights are detected and localized for subsequent processing. Using a slightly modified U-Net architecture, we propose a semantic segmentation model to localize specular highlights. To evaluate the segmentation performance, the IoU metric is used. This measures how good the segmentation prediction matches the corresponding reference annotation by dividing their intersection by their union (IoU). The model achieved a frequency-weighted and mean IoU of 0.83 and 0.75 respectively when tested on a benchmark dataset. Furthermore, the proposed network was also trained and tested on a custom dataset, focusing on flash light reflections in an underground tunnel environment. For this custom dataset, the model achieved a frequency-weighted and mean IoU of 0.98 and 0.80 respectively.

Purpose Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure. Methods We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos. Results We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods. Conclusion The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.

The capsule endoscopy robot can only use monocular vision due to the dimensional limit. To improve the depth perception of the monocular capsule endoscopy robot, this paper proposes a photometric stereo-based depth map reconstruction method. First, based on the characteristics of the capsule endoscopy robot system, a photometric stereo framework is established. Then, by combining the specular property and Lambertian property of the object surface, the depth of the specular highlight point is estimated, and the depth map of the whole object surface is reconstructed by a forward upwind scheme. To evaluate the precision of the depth estimation of the specular highlight region and the depth map reconstruction of the object surface, simulations and experiments are implemented with synthetic images and pig colon tissue, respectively. The results of the simulations and experiments show that the proposed method provides good precision for depth map reconstruction in monocular capsule endoscopy.

We introduce a novel algorithm that turns a flash selfie taken with a smartphone into a studio-like photograph with uniform lighting. Our method uses a convolutional neural network trained on a set of pairs of photographs acquired in a controlled environment. For each pair, we have one photograph of a subject's face taken with the camera flash enabled and another one of the same subject in the same pose illuminated using a photographic studio-lighting setup. We show how our method can amend lighting artifacts introduced by a close-up camera flash, such as specular highlights, shadows, and skin shine.

This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis. We further demonstrate that GCNet can be directly integrated with existing calibrated methods to achieve improved results on surface normal estimation. Our code and model can be found at https://guanyingc.github.io/UPS-GCNet.

Capturing the 3D geometry of transparent objects is a challenging task, ill-suited for general-purpose scanning and reconstruction techniques, since these cannot handle specular light transport phenomena. Existing state-of-the-art methods, designed specifically for this task, either involve a complex setup to reconstruct complete refractive ray paths, or leverage a data-driven approach based on synthetic training data. In either case, the reconstructed 3D models suffer from over-smoothing and loss of fine detail. This paper introduces a novel, high precision, 3D acquisition and reconstruction method for solid transparent objects. Using a static background with a coded pattern, we establish a mapping between the camera view rays and locations on the background. Differentiable tracing of refractive ray paths is then used to directly optimize a 3D mesh approximation of the object, while simultaneously ensuring silhouette consistency and smoothness. Extensive experiments and comparisons demonstrate the superior accuracy of our method., Comment: 13 pages, 21 figures

The separation of reflection components is an important issue in computer graphics, computer vision, and image processing. This is a highly ill-posed problem since the number of unknowns to solve is much larger than the number of equations. We present a method to reduce the difficulty of this problem by assuming that surface geometry is known. A novel objective function based on robust principal component analysis is proposed to simultaneously separate specularities and estimate the position of light source. We develop an Augmented Lagrangian Multiplier based algorithm to solve the objective function efficiently and effectively. Experimental results on real-world and synthetic data demonstrate the effectiveness of our method.

Door detection is becoming an increasingly important subject in building indoor modelling owing to its value in scan-to-BIM processes. This paper presents an original approach that detects open, semi-open and closed doors in 3D laser scanned data of indoor environments. The proposed technique is unique in that it integrates the information regarding both the geometry (i.e. XYZ coordinates) and colour (i.e. RGB or HSV) provided by a calibrated set of 3D laser scanner and a colour camera. In other words, our technique is developed in a 6D-space framework. The geometry-colour integration and other characteristics of our method make it robust to occlusion and variations in colours resulting from varying lighting conditions at each scanning location (e.g. specular highlights) and from different scanning locations. In addition to this paper, the authors also contribute a public dataset of real scenes along with an annotated ground truth. The dataset has varying levels of challenges and will help to assess the performance of new and existing contributions in the field. The approach proposed in this paper is tested against that dataset, yielding encouraging results.

Endoscopy provides a convenient way to access the inner structures of various organs. The endoscopic images provide an immediate observation and help diagnosis and therapy. Shape reconstruction from endoscopic images further provides real scale factor for image-guided navigation. However, specular highlights, bright patches of light appearing on the imaged surface, mask the real image texture and result in erroneous reconstruction. Therefore, the detection of specular highlights is essential for accurate reconstruction. In this study, we divide the images into homogeneous regions by color quantization and spatial segmentation. Then, a thresholding technique based on histogram of the pixel intensity values is used. Finally, we check the gray level consistency for each region to avoid over segmentation. The experimental results show that the proposed method can achieve successfully detection.

A novel specular highlights detection method in colonoscopy videos is presented. The method is based on a model of appearance defining specular highlights as bright spots which are highly contrasted with respect to adjacent regions. Our approach proposes two stages: segmentation and then classification of bright spot regions. The former defines a set of candidate regions obtained through a region growing process with local maxima as initial region seeds. This process creates a tree structure which keeps track, at each growing iteration, of the region frontier contrast; final regions provided depend on restrictions over contrast value. Non-specular regions are filtered through a classification stage performed by a linear SVM classifier using model-based features from each region. We introduce a new validation database with more than 25, 000 regions along with their corresponding pixel-wise annotations. We perform a comparative study against other approaches. Results show that our method is superior to other approaches, with our segmented regions being closer to actual specular regions in the image. Finally, we also present how our methodology can also be used to obtain an accurate prediction of polyp histology.

The presence of specular highlights can hide underlying features of a scene within an image and can be problematic in many application scenarios. In particular, this poses a significant challenge for applications where image stitching is used to create a single static image of a scene from inspection footage of pipes, gas tubes, train tracks and concrete structures. Furthermore, they can hide small defects in the images causing them to be missed during inspection. We present a method which exploits additional information in neighbouring frames from video footage to reduce specularity from each frame. The technique first automatically determines frames which contain overlapping regions before the relationship that exists between them is exploited in order to suppress the effects of specular reflections. This results in an image that is free from specular highlights provided there is at least one frame present in the sequence where a given pixel is present in a diffuse form. The method is shown to work well on greyscale as well as colour images and effectively reduces specularity and significantly improves the quality of the stitched image, even in the presence of noise. While applied to the challenge of reducing specularity in inspection videos, the method improves upon the state-of-the-art in specularity removal, and, its applications are wider ranging as a general purpose pre-processing tool.

The visual system is considered to employ various image cues from an object image to perceive its glossiness. It has been reported that, surprisingly, even for object images without specular highlights we can perceive glossiness by relying on low-luminance specular components (Kim, Marlow, & Anderson, 2012). This type of perceptual glossiness is referred to as dark gloss. However, it is still unclear whether dark gloss is observed commonly across various objects, and what image features are cues for dark gloss. To address these issues, we performed several psychophysical experiments. First, we measured perceived glossiness for a number of computer-graphics object images with natural specular reflection components (Full condition) and for those without high-luminance components of specular reflections (Dark condition). The results showed that dark gloss (glossiness perception in the Dark condition) was generally observed on almost all object images, while its intensity was rather different across the images. Then we psychologically or computationally measured several image features for the stimulus images, such as luminance edge number, recognizability of reflection images, and some highlight-related features, to examine their relations to perceived glossiness with a multiple regression analysis. The results demonstrated that luminance edge number was most strongly related to glossiness scores among the measured features only for object images with potent dark gloss. These results suggest that luminance edges are an effective cue for dark gloss under certain stimulus conditions.

Colonoscopy examinations are widely used for detecting colon cancer and many other colon abnormalities. Unfortunately, the resulting colon videos often have artifacts caused by camera motion and specular highlights caused by light reflections from the wet colon surface. To address these problems, we have developed a method for motion compensated colonoscopy image restoration. Our approach utilizes RANSAC-based image registration to align sequences of N consecutive images in the colonoscopy video and restores each frame of the video using information from these aligned images. We compare image alignment quality when N adjacent images are registered to each other versus registering images with larger step sizes between them. Three types of image pre processing were evaluated in our work. We found that the removal of non-informative images prior to image registration produced better alignment results and reduced processing time. We also evaluated the effects of image smoothing and resizing as a pre processing step for image registration.

In this paper, we present an efficient framework for editing the appearance of measured materials. First, we separate the material data into diffuse and specular parts, which is important for measured BRDFs since editing without distinguishing the two parts makes the results less predictive. Additionally, the albedo of each part is calculated for later color editing. Next, we use a novel method to cluster the materials by their gloss levels and select the representative material of every level. With this at hand, the roughness, i.e., shape of the specular highlights can be adjusted among different levels. Finally, we reconstruct the edited materials which can be directly used in both online and offline rendering. We test the framework on the MERL dataset to validate the effectiveness of our method.

Relighting is an essential step in realistically transferring objects from a captured image into another environment. For example, authentic telepresence in Augmented Reality requires faces to be displayed and relit consistent with the observer's scene lighting. We investigate end-to-end deep learning architectures that both de-light and relight an image of a human face. Our model decomposes the input image into intrinsic components according to a diffuse physics-based image formation model. We enable non-diffuse effects including cast shadows and specular highlights by predicting a residual correction to the diffuse render. To train and evaluate our model, we collected a portrait database of 21 subjects with various expressions and poses. Each sample is captured in a controlled light stage setup with 32 individual light sources. Our method creates precise and believable relighting results and generalizes to complex illumination conditions and challenging poses, including when the subject is not looking straight at the camera., CVPR 2020 (Oral)

The acquisition of translucent objects has become a very common task thanks to the progress of 3D scanning technology. Since the characteristic soft appearance of translucent objects is due to subsurface scattering, the details are naturally left out in this appearance. For objects that have complex shapes, this lack of detail is obviously more prominent. In this paper, we propose a method to preserve the details of surface geometry by adding highlight effects. In generating highlight effects, our method employs a local orthonormal frame and combines, in a novel way, the incoming and outgoing light in approximating the subsurface scattering process. When the incident illuminant direction changes from nearly overhead to nearly horizontal, our method effectively preserves complex surface geometry details in the appearance of translucent materials. Through experiments, we show that our method can store surface features as well as maintain the translucency of the original materials and even enhance the perception of translucency. By numerically comparing the generated highlight effects with those generated by the traditional Bidirectional Reflectance Distribution Function (BRDF) models with different bandwidth parameters, we demonstrate the validity of our proposed method.

We present a method for turning a flash selfie taken with a smartphone into a photograph as if it was taken in a studio setting with uniform lighting. Our method uses a convolutional neural network trained on a set of pairs of photographs acquired in an ad-hoc acquisition campaign. Each pair consists of one photograph of a subject’s face taken with the camera flash enabled and another one of the same subject in the same pose illuminated using a photographic studio-lighting setup. We show how our method can amend defects introduced by a close-up camera flash, such as specular highlights, shadows, skin shine, and flattened images.

Automobile surface defects like scratches or dents occur during the process of manufacturing and cross-border transportation. This will affect consumers&rsquo, first impression and the service life of the car itself. In most worldwide automobile industries, the inspection process is mainly performed by human vision, which is unstable and insufficient. The combination of artificial intelligence and the automobile industry shows promise nowadays. However, it is a challenge to inspect such defects in a computer system because of imbalanced illumination, specular highlight reflection, various reflection modes and limited defect features. This paper presents the design and implementation of a novel automatic inspection system (AIS) for automobile surface defects which are the located in or close to style lines, edges and handles. The system consists of image acquisition and image processing devices, operating in a closed environment and noncontact way with four LED light sources. Specifically, we use five plane-array Charge Coupled Device (CCD) cameras to collect images of the five sides of the automobile synchronously. Then the AIS extracts candidate defect regions from the vehicle body image by a multi-scale Hessian matrix fusion method. Finally, candidate defect regions are classified into pseudo-defects, dents and scratches by feature extraction (shape, size, statistics and divergence features) and a support vector machine algorithm. Experimental results demonstrate that automatic inspection system can effectively reduce false detection of pseudo-defects produced by image noise and achieve accuracies of 95.6% in dent defects and 97.1% in scratch defects, which is suitable for customs inspection of imported vehicles.

The manual postmortem inspection of broilers and their viscera is becoming a bottleneck as the slaughter rate increases. Computer vision can assist veterinarians during the inspection, but specular highlights can hide crucial details when inspecting for diseases on the viscera set. This study aims to restore details behind these specular highlights by capturing two images of the same viscera using shifting light positions. The dataset consists of images captured in-line at a poultry processing plant. The method achieves an average SSIM score of 0.96 over a test set of 100 image sets. The result is visually pleasing images with correct textural information instead of specular highlights.

A novel specular highlight restoration algorithm has been proposed to remove the specular highlight in a real-time vision system. In this paper, the specular highlight region has been detected and it has been restored by image inpainting method. Most of specular highlight detection algorithms are effective only for a single image. And the auto-threshold algorithm has been implemented in real time, but, it has still low reliability with a heavy computational cost. The proposed system detects pixels corresponding to the specular highlight region in the HSI color space through a newly defined classification table. So, Specular highlight can be detected quickly with these simplified classification table. In addition, it has versatility because it can be combined with various existing high-performance image inpainting method. The superiority of the proposed algorithm is compared with the conventional specular highlight removal algorithm by combining proposed algorithm with two high-performance image inpainting techniques.

Specular highlights on the surface of an object can serve as cues for estimating illumination color, because highlights on many objects directly reflect illumination colors. In addition, we experience that illumination colors typically have chromaticities near the daylight locus in daily life, and this experience may possibly affect our estimation of the illumination color. In this study, we investigated the effects of specular highlights and object colors on the estimation of the illumination colors by human observers using images of a single object with specular highlights as stimuli. In the experiments, the colors of all pixels on an object with specular highlights changed from the reference D65 chromaticity to a test chromaticity in a trial. Observers judged whether this chromaticity change appeared to be induced by changing the object or by changing the illumination. In the control experiment, observers performed the same task for a stimulus of a mat object without specular highlights. Different values of hue and saturation were used as the test chromaticities. In the results, the chromaticity changes appeared more likely to be caused by changing the illumination when the test color had lower saturation. In addition, the ratios of “illumination changed” responses were slightly higher for objects with specular highlights than for objects without specular highlight. However, the ratios of “illumination changed” responses averaged across observers were not largely different across color directions. These results suggest that specular highlights slightly contribute to the estimation of illumination color, and that the daylight locus does not largely affect the estimation of the illumination color.

In structured light 3D measurement field, when the object has smooth surface, it can form a highlight area due to the specular reflection, and the distortion of the object will make a large measurement error. In order to solve this problem, this paper use seven steps sine-phase shift combined with linear diffuser to remove highlight. Firstly, the principle of removing specular with diffuser is analyzed, then the overall design of the system is introduced, which includes 3D reconstruction and system calibration method. Finally the reconstructed experiments are carried out with ceramic plate. Experimental results show that the proposed method can significantly reduce the highlights area of reconstructed image compared with the highlights area without diffuser. The diffuser can obviously inhibited by highlights, although it can not completely remove the highlights, it plays a very important role in reconstructing specular object with more accurate and better quality.

A method for light sources estimation is proposed in this paper. The method utilizes feature points in cast shadows to estimate near light source positions from estimated source directions using specular highlights. There are several methods that can be used to estimate light sources from scene images, using either cast shadows or specular highlights. However, most of them are limited to directional light sources. The proposed method can estimate the positions and intensities of multiple near point light sources. Specular highlights on an object of known geometry are first used for light source direction estimation. Then, a discontinuity point in the object shape and the corresponding cast shadow on a ground plane are used for light source position estimation. Feature points obtained from an image of the cast shadow, however, can be inaccurate due to various factors. Information on diffused light reflected from Lambertian ground-plane surface is subsequently used to improve estimation accuracy. Experimental results were used to evaluate the performance of the proposed method. Key words: Light source estimation, light source recovery, augmented reality.

The intensity value recorded by terrestrial laser scanning (TLS) systems is significantly influenced by incidence angles. Most existing models focus on the diffuse reflection of rough surfaces and ignore the specular reflection, despite that both reflections simultaneously exist in all natural surfaces. At large incidence angles, specular reflection can be neglected. However, laser detectors can receive a portion of specular reflection at small incidence angles. Specular reflection can lead to additional increase in the original intensity data and even highlight phenomenon on scanned targets, especially those with a relatively smooth or highly reflective surface. In this paper, a new empirical method is proposed to correct the intensities of highlight regions caused by the specular reflection. The intensity from the specular reflection is obtained by subtracting the intensity caused by diffuse reflection and instrumental effects from the original intensity. The proposed method is tested and validated on different targets scanned by Faro Focus3D 120. Results imply that the proposed method can effectively eliminate the highlight phenomenon in TLS for 3-D point cloud representation by intensity and intensity image interpretation.