Search

Humans implicitly rely on the properties of materials to guide our interactions. Grasping smooth materials, for example, requires more care than rough ones. We may even visually infer non-visual properties (e.g., softness is a physical material property). We refer to visually-recognizable material properties as visual material attributes. Recognizing these attributes in images can provide valuable information for scene understanding and material recognition. Unlike typical object and scene attributes, however, visual material attributes are local (i.e., “fuzziness” does not have a shape). Given full supervision, we may accurately recognize such attributes from purely local information (small image patches). Obtaining consistent full supervision at scale, however, is challenging. To solve this problem, we probe the human visual perception of materials. By asking simple yes/no questions comparing pairs of image patches, we obtain the weak supervision required to build a set of classifiers for attributes that, while unnamed, function similarly to the attributes with which we describe materials. Furthermore, we integrate this method in the end-to-end learning of a CNN that simultaneously recognizes materials and their visual attributes. Experiments show that visual material attributes serve as both a useful representation for known material categories and as a basis for transfer learning. [ABSTRACT FROM AUTHOR]

We introduce view birdification, the problem of recovering ground-plane movements of people in a crowd from an ego-centric video captured from an observer (e.g., a person or a vehicle) also moving in the crowd. Recovered ground-plane movements would provide a sound basis for situational understanding and benefit downstream applications in computer vision and robotics. In this paper, we formulate view birdification as a geometric trajectory reconstruction problem and derive a cascaded optimization method from a Bayesian perspective. The method first estimates the observer's movement and then localizes surrounding pedestrians for each frame while taking into account the local interactions between them. We introduce three datasets by leveraging synthetic and real trajectories of people in crowds and evaluate the effectiveness of our method. The results demonstrate the accuracy of our method and set the ground for further studies of view birdification as an important but challenging visual understanding problem. [ABSTRACT FROM AUTHOR]

Our world is full of objects with complex reflectances situated in rich illumination environments. Though stunning, the diversity of appearance that arises from this complexity is also daunting. For this reason, past work on geometry recovery has tried to frame the problem into simplistic models of reflectance (such as Lambertian, mirrored, or dichromatic) or illumination (one or more distant point light sources). In this work, we directly tackle the problem of joint reflectance and geometry estimation under known but uncontrolled natural illumination by fully exploiting the surface orientation cues that become embedded in the appearance of the object. Intuitively, salient scene features (such as the sun or stained glass windows) act analogously to the point light sources of traditional geometry estimation frameworks by strongly constraining the possible orientations of the surface patches reflecting them. By jointly estimating the reflectance of the object, which modulates the illumination, the appearance of a surface patch can be used to derive a nonparametric distribution of its possible orientations. If only a single image exists, these strongly constrained surface patches may then be used to anchor the geometry estimation and give context to the less-descriptive regions. When multiple images exist, the distribution of possible surface orientations becomes tighter as additional context is given, though integrating the separate views poses additional challenges. In this paper we introduce two methods, one for the single image case, and another for the case of multiple images. The effectiveness of our methods is evaluated extensively on synthetic and real-world data sets that span the wide range of real-world environments and reflectances that lies between the extremes that have been the focus of past work. [ABSTRACT FROM PUBLISHER]

The appearance of an object in an image encodes invaluable information about that object and the surrounding scene. Inferring object reflectance and scene illumination from an image would help us decode this information: reflectance can reveal important properties about the materials composing an object; the illumination can tell us, for instance, whether the scene is indoors or outdoors. Recovering reflectance and illumination from a single image in the real world, however, is a difficult task. Real scenes illuminate objects from every visible direction and real objects vary greatly in reflectance behavior. In addition, the image formation process introduces ambiguities, like color constancy, that make reversing the process ill-posed. To address this problem, we propose a Bayesian framework for joint reflectance and illumination inference in the real world. We develop a reflectance model and priors that precisely capture the space of real-world object reflectance and a flexible illumination model that can represent real-world illumination with priors that combat the deleterious effects of image formation. We analyze the performance of our approach on a set of synthetic data and demonstrate results on real-world scenes. These contributions enable reliable reflectance and illumination inference in the real world. [ABSTRACT FROM PUBLISHER]

Abstract: We present a novel 3D reassembly method for fragmented, thin objects with unknown geometry. Unlike past methods, we do not make any restrictive assumptions about the overall shape of the object, or its painted texture. Our key observation is that regardless of the object''s shape, matching fragments will have similar geometry and photometry along and across their adjoining regions. We begin by encoding the scale variability of each fragment''s boundary contour in a multichannel, 2D image representation. Using this multichannel boundary contour representation, we identify matching sub-contours via 2D partial image registration. We then align the fragments by minimizing the distance between their adjoining regions while simultaneously ensuring geometric continuity across them. The configuration of the fragments as they are incrementally matched and aligned form a graph structure that we use to improve subsequent matches. By detecting cycles in this graph, we identify subsets of fragments with interdependent alignments. We then minimize the error within the subsets to achieve a globally optimal alignment. We leverage user feedback to cull the otherwise exponential search space; after each new match is found and aligned, it is presented to a user for confirmation or rejection. Using ceramic pottery as the driving example, we demonstrate the accuracy and efficiency of our method on six real-world datasets. [Copyright &y& Elsevier]

We introduce a novel neural network-based BRDF model and a Bayesian framework for object inverse rendering, i.e., joint estimation of reflectance and natural illumination from a single image of an object of known geometry. The BRDF is expressed with an invertible neural network, namely, normalizing flow, which provides the expressive power of a high-dimensional representation, computational simplicity of a compact analytical model, and physical plausibility of a real-world BRDF. We extract the latent space of real-world reflectance by conditioning this model, which directly results in a strong reflectance prior. We refer to this model as the invertible neural BRDF model (iBRDF). We also devise a deep illumination prior by leveraging the structural bias of deep neural networks. By integrating this novel BRDF model and reflectance and illumination priors in a MAP estimation formulation, we show that this joint estimation can be computed efficiently with stochastic gradient descent. We experimentally validate the accuracy of the invertible neural BRDF model on a large number of measured data and demonstrate its use in object inverse rendering on a number of synthetic and real images. The results show new ways in which deep neural networks can help solve challenging radiometric inverse problems. [ABSTRACT FROM AUTHOR]

We introduce a novel neural network-based BRDF model and a Bayesian framework for object inverse rendering, i.e., joint estimation of reflectance and natural illumination from a single image of an object of known geometry. The BRDF is expressed with an invertible neural network, namely, normalizing flow, which provides the expressive power of a high-dimensional representation, computational simplicity of a compact analytical model, and physical plausibility of a real-world BRDF. We extract the latent space of real-world reflectance by conditioning this model, which directly results in a strong reflectance prior. We refer to this model as the invertible neural BRDF model (iBRDF). We also devise a deep illumination prior by leveraging the structural bias of deep neural networks. By integrating this novel BRDF model and reflectance and illumination priors in a MAP estimation formulation, we show that this joint estimation can be computed efficiently with stochastic gradient descent. We experimentally validate the accuracy of the invertible neural BRDF model on a large number of measured data and demonstrate its use in object inverse rendering on a number of synthetic and real images. The results show new ways in which deep neural networks can help solve challenging radiometric inverse problems. [ABSTRACT FROM AUTHOR]

Atmospheric conditions induced by suspended particles, such as fog and haze, severely alter the scene appearance. Restoring the true scene appearance from a single observation made in such bad weather conditions remains a challenging task due to the inherent ambiguity that arises in the image formation process. In this paper, we introduce a novel Bayesian probabilistic method that jointly estimates the scene albedo and depth from a single foggy image by fully leveraging their latent statistical structures. Our key idea is to model the image with a factorial Markov random field in which the scene albedo and depth are two statistically independent latent layers and to jointly estimate them. We show that we may exploit natural image and depth statistics as priors on these hidden layers and estimate the scene albedo and depth with a canonical expectation maximization algorithm with alternating minimization. We experimentally evaluate the effectiveness of our method on a number of synthetic and real foggy images. The results demonstrate that the method achieves accurate factorization even on challenging scenes for past methods that only constrain and estimate one of the latent variables. [ABSTRACT FROM AUTHOR]

We present a method for simultaneously estimating the illumination of a scene and the reflectance property of an object from single view images—a single image or a small number of images taken from the same viewpoint. We assume that the illumination consists of multiple point light sources, and the shape of the object is known. First, we represent the illumination on the surface of a unit sphere as a finite mixture of von Mises-Fisher distributions based on a novel spherical specular reflection model that well approximates the Torrance-Sparrow reflection model. Next, we estimate the parameters of this mixture model including the number of its component distributions and the standard deviation of them, which correspond to the number of light sources and the surface roughness, respectively. Finally, using these results as the initial estimates, we iteratively refine the estimates based on the original Torrance-Sparrow reflection model. The final estimates can be used to relight single-view images such as altering the intensities and directions of the individual light sources. The proposed method provides a unified framework based on directional statistics for simultaneously estimating the intensities and directions of an unknown number of light sources, as well as the specular reflection parameter of the object in the scene. [ABSTRACT FROM AUTHOR]

In the use of the three-dimensional data obtained from a stereo system, laser range finder, and other equipment, merge processing is important. This paper proposes a new merge method for range images, based on the volume-based technique. In the proposed method, in contrast to the conventional method, the voxels are adaptively segmented according to the curvature of the surface shape to be reconstructed. This helps to represent the geometrical shape efficiently, effectively utilizing the computation resources. In the framework of the proposed merge process, additional features such as color and the laser reflectivity can be added to the three-dimensional geometrical information, which helps to maintain sharp edges in the texture and to perform rendering and the texture mapping effectively. By examining the consensus for both the geometrical shape and the color in the whole framework, the merge is made robust to noise. In this paper, the system is described first, and then the results of application to several kinds of actual data are presented. © 2007 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 90(11): 50–60, 2007; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20353 [ABSTRACT FROM AUTHOR]

Several techniques have been developed for recovering reflectance properties of real surfaces under unknown illumination. However, in most cases, those techniques assume that the light sources are located at inifinity, which cannot be applied safely to, for example, reflectance modeling of indoor environments. In this paper, we propose two types of methods to estimate the surface reflectance property of an object, as well as the position of a light source from a single view without the distant illumination assumption, thus relaxing the conditions in the previous methods. Given a real image and a 3D geometric model of an object with specular reflection as inputs, the first method estimates the light source position by fitting to the Lambertian diffuse component, while separating the specular and diffuse components by using an iterative relaxation scheme. Our second method extends that first method by using as input a specular component image, which is acquired by analyzing multiple polarization images taken from a single view, thus removing its constraints on the diffuse reflectance property. This method simultaneously recovers the reflectance properties and the light source positions by optimizing the linearity of a log-transformed Torrance-Sparrow model. By estimating the object's reflectance property and the light source position, we can freely generate synthetic images of the target object under arbitrary lighting conditions with not only source direction modification but also source-surface distance modification. Experimental results show the accuracy of our estimation framework. [ABSTRACT FROM AUTHOR]

In this paper, we tackle the problem of geometric and photometric modeling of large intricately shaped objects. Typical target objects we consider are cultural heritage objects. When constructing models of such objects, we are faced with several important issues that have not been addressed in the past-issues that mainly arise due to the large amount of data that has to be handled. We propose two novel approaches to efficiently handle such large amounts of data: A highly adaptive algorithm for merging range images and an adaptive nearest-neighbor search to be used with the algorithm. We construct an integrated mesh model of the target object in adaptive resolution, taking into account the geometric and/or photometric attributes associated with the range images. We use surface curvature for the geometric attributes and (laser) reflectance values for the photometric attributes. This adaptive merging framework leads to a significant reduction in the necessary amount of computational resources. Furthermore, the resulting adaptive mesh models can be of great use for applications such as texture mapping, as we will briefly demonstrate. Additionally, we propose an additional test for the k-d tree nearest-neighbor search algorithm. Our approach successfully omits back-tracking, which is controlled adaptively depending on the distance to the nearest neighbor. Since the main consumption of computational cost lies in the nearest-neighbor search, the proposed algorithm leads to a significant speed-up of the whole merging process. In this paper, we present the theories and algorithms of our approaches with pseudo code and apply them to several real objects, including large-scale cultural assets. [ABSTRACT FROM AUTHOR]

As part of a project for the electronic preservation of cultural objects, this paper proposes a new technique for position alignment of multiple range images. In particular, the study focuses on accuracy of alignment and robustness to noise in the input data. Using the conjugate gradient method based on the M-estimator, simultaneous alignment is performed by minimization of an error function that handles all range images at the same time. Furthermore, for more efficient and more accurate search for matching points, the KD-tree data structure is utilized, and additional attributes such as the laser reflection brightness and the brightness of the three-dimensional point itself are introduced into the error measure. This paper proposes such a method and presents the results of experiments with real data. © 2004 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 87(8): 61–74, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.20109 [ABSTRACT FROM AUTHOR]

Proposes the Eigen-Texture method for generating virtual images of objects from their real images. Presentation of sample appearances of a real object under various illumination and viewing conditions; Compression of the sample images in the two-dimensional coordinate system defined on the three-dimensional model surface.

Computational surface modeling that underlies material recognition has transitioned from reflectance modeling using in-lab controlled radiometric measurements to image-based representations based on internet-mined single-view images captured in the scene. We take a middle-ground approach for material recognition that takes advantage of both rich radiometric cues and flexible image capture. A key concept is differential angular imaging, where small angular variations in image capture enables angular-gradient features for an enhanced appearance representation that improves recognition. We build a large-scale material database, Ground Terrain in Outdoor Scenes (GTOS) database, to support ground terrain recognition for applications such as autonomous driving and robot navigation. The database consists of over 30,000 images covering 40 classes of outdoor ground terrain under varying weather and lighting conditions. We develop a novel approach for material recognition called texture-encoded angular network (TEAN) that combines deep encoding pooling of RGB information and differential angular images for angular-gradient features to fully leverage this large dataset. With this novel network architecture, we extract characteristics of materials encoded in the angular and spatial gradients of their appearance. Our results show that TEAN achieves recognition performance that surpasses single view performance and standard (non-differential/large-angle sampling) multiview performance. [ABSTRACT FROM AUTHOR]

This paper introduces a novel depth recovery method based on light absorption in water. Water absorbs light at almost all wavelengths whose absorption coefficient is related to the wavelength. Based on the Beer-Lambert model, we introduce a bispectral depth recovery method that leverages the light absorption difference between two near-infrared wavelengths captured with a distant point source and orthographic cameras. Through extensive analysis, we show that accurate depth can be recovered irrespective of the surface texture and reflectance, and introduce algorithms to correct for nonidealities of a practical implementation including tilted light source and camera placement, nonideal bandpass filters and the perspective effect of the camera with a diverging point light source. We construct a coaxial bispectral depth imaging system using low-cost off-the-shelf hardware and demonstrate its use for recovering the shapes of complex and dynamic objects in water. We also present a trispectral variant to further improve robustness to extremely challenging surface reflectance. Experimental results validate the theory and practical implementation of this novel depth recovery paradigm, which we refer to as shape from water. [ABSTRACT FROM AUTHOR]

Reflectance and shape are two important components in visually perceiving the real world. Inferring the reflectance and shape of an object through cameras is a fundamental research topic in the field of computer vision. While three-dimensional shape recovery is pervasive with varieties of approaches and practical applications, reflectance recovery has only emerged recently. Reflectance recovery is a challenging task that is usually conducted in controlled environments, such as a laboratory environment with a special apparatus. However, it is desirable that the reflectance be recovered in the field with a handy camera so that reflectance can be jointly recovered with the shape. To that end, we present a solution that simultaneously recovers the reflectance and shape (i.e., dense depth and normal maps) of an object under natural illumination with commercially available handy cameras. We employ a light field camera to capture one light field image of the object, and a 360-degree camera to capture the illumination. The proposed method provides positive results in both simulation and real-world experiments. [ABSTRACT FROM AUTHOR]

Video analysis has been moving towards more detailed interpretation (e.g., segmentation) with encouraging progress. These tasks, however, increasingly rely on densely annotated training data both in space and time. Since such annotation is labor-intensive, few densely annotated video data with detailed region boundaries exist. This work aims to resolve this dilemma by learning to automatically generate region boundaries for all frames of a video from sparsely annotated bounding boxes of target regions. We achieve this with a Volumetric Graph Convolutional Network (VGCN), which learns to iteratively find keypoints on the region boundaries using the spatio-temporal volume of surrounding appearance and motion. We show that the global optimization of VGCN leads to more accurate annotation that generalizes better. Experimental results using three latest datasets (two real and one synthetic), including ablation studies, demonstrate the effectiveness and superiority of our method. [ABSTRACT FROM AUTHOR]

Tracking pedestrians is a vital component of many computer vision applications, including surveillance, scene understanding, and behavior analysis. Videos of crowded scenes present significant challenges to tracking due to the large number of pedestrians and the frequent partial occlusions that they produce. The movement of each pedestrian, however, contributes to the overall crowd motion (i.e., the collective motions of the scene's constituents over the entire video) that exhibits an underlying spatially and temporally varying structured pattern. In this paper, we present a novel Bayesian framework for tracking pedestrians in videos of crowded scenes using a space-time model of the crowd motion. We represent the crowd motion with a collection of hidden Markov models trained on local spatio-temporal motion patterns, i.e., the motion patterns exhibited by pedestrians as they move through local space-time regions of the video. Using this unique representation, we predict the next local spatio-temporal motion pattern a tracked pedestrian will exhibit based on the observed frames of the video. We then use this prediction as a prior for tracking the movement of an individual in videos of extremely crowded scenes. We show that our approach of leveraging the crowd motion enables tracking in videos of complex scenes that present unique difficulty to other approaches. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a novel method for reconstructing surface normals and depth of dynamic objects in water. Past shape recovery methods have leveraged various visual cues for estimating shape (e.g., depth) or surface normals. Methods that estimate both compute one from the other. We show that these two geometric surface properties can be simultaneously recovered for each pixel when the object is observed underwater. Our key idea is to leverage multi-wavelength near-infrared light absorption along different underwater light paths in conjunction with surface shading. Our method can handle both Lambertian and non-Lambertian surfaces. We derive a principled theory for this surface normals and shape from water method and a practical calibration method for determining its imaging parameters values. By construction, the method can be implemented as a one-shot imaging system. We prototype both an off-line and a video-rate imaging system and demonstrate the effectiveness of the method on a number of real-world static and dynamic objects. The results show that the method can recover intricate surface features that are otherwise inaccessible. [ABSTRACT FROM AUTHOR]

Recognizing wet surfaces and their degrees of wetness is essential for many computer vision applications. Surface wetness can inform us slippery spots on a road to autonomous vehicles, muddy areas of a trail to humanoid robots, and the freshness of groceries to us. The fact that surfaces darken when wet, i.e., monochromatic appearance change, has been modeled to recognize wet surfaces in the past. In this paper, we show that color change, particularly in its spectral behavior, carries rich information about surface wetness. We first derive an analytical spectral appearance model of wet surfaces that expresses the characteristic spectral sharpening due to multiple scattering and absorption in the surface. We present a novel method for estimating key parameters of this spectral appearance model, which enables the recovery of the original surface color and the degree of wetness from a single multispectral image. Applied to a multispectral image, the method estimates the spatial map of wetness together with the dry spectral distribution of the surface. To our knowledge, this is the first work to model and leverage the spectral characteristics of wet surfaces to decipher its appearance. We conduct comprehensive experimental validation with a number of wet real surfaces. The results demonstrate the accuracy of our model and the effectiveness of our method for surface wetness and color estimation. [ABSTRACT FROM AUTHOR]

We introduce RGB road scene material segmentation, i.e. , per-pixel segmentation of materials in real-world driving views with pure RGB images, as a novel computer vision task by building a benchmark dataset and by deriving a new method. Our dataset, KITTI-Materials, is based on the well-established KITTI dataset and consists of 1000 frames covering 24 different road scenes of urban/suburban landscapes, carefully annotated with one of 20 material categories for every pixel. It is the first dataset for RGB material segmentation in real driving scenes. Through careful analysis of KITTI-Materials, we identify the extraction and fusion of texture and image context as the key to accurate modeling of road scene material appearance. For this, we introduce R oad scene M aterial S egmentation Net work (RMSNet) as a baseline method for this challenging task. RMSNet encodes multi-scale hierarchical features with efficient Transformer layers. We construct the decoder of RMSNet based on a novel efficient self-attention model, which we refer to as SAMixer which adaptively fuses texture and context cues across multiple feature levels. Extensive experiments on KITTI-Materials validate the effectiveness of our RMSNet. We believe our work lays a solid foundation for further studies on RGB road scene material segmentation. [Display omitted] • Materials with informative properties are critical for finer road scene understanding. • Despite its significance, RGB Road scene Material Segmentation (RMS) remains unstudied. • We propose KITTI-Materials – the first benchmark dataset focused on RGB RMS. • We present RMSNet – a novel network for effective and efficient RGB RMS. [ABSTRACT FROM AUTHOR]

Despite their ubiquitous presence, little has been investigated about the scale variability-the relative variations in the spatial extents of local structures-of 3D geometric data. In this paper we present a comprehensive framework for exploiting this 3D geometric scale variability in range images that provides rich information for characterizing the overall geometry. We derive a sound scale-space representation, which we refer to as the geometric scale-space, that faithfully encodes the scale variability of the surface geometry, and derive novel detectors to extract prominent features and identify their natural scales. The result is a hierarchical set of features of different scales which we refer to as scale-dependent geometric features. We then derive novel local shape descriptors that represent the surface structures that give rise to those features by carving out and encoding the local surface that fall within the support regions of the features. This leads to scale-dependent or scale-invariant local shape descriptors that convey significant discriminative information of the object geometry. We demonstrate the effectiveness of geometric scale analysis on range images, and show that it enables novel applications, in particular, fully automatic registration of multiple objects from a mixed set of range images and 3D object recognition in highly cluttered range image scenes. [ABSTRACT FROM AUTHOR]

• Introduced HOI-MO problem that has been overlooked in the past. • Two new benchmark test sets for HOI-MO based on commonly used public datasets. • A novel HOI-MO method that leverages global scene context. • Demonstration of the effectiveness of the proposed model with extensive experiments. Human–object interaction (HOI) detection is an important vision task that requires the detection of individual object instances and reasoning of their relations. Despite encouraging advancement in recent years, past methods are still limited to relatively simple images where the human and object instances can be detected without difficulties. HOI in the wild should work even when the objects that a person is interacting with are not visible or hard to detect in the image. In this paper, we formulate HOI with missing objects (HOI-MO) as a research problem, and show that it is indeed critical as many such instances can be found even in the commonly-used public HOI detection datasets. We label these to compose new test sets for the proposed method. To our knowledge, we introduce the first method for such challenging HOI detection that incorporates global scene information. The effectiveness and superiority of the proposed method are demonstrated through extensive experiments and comparisons. [ABSTRACT FROM AUTHOR]

This paper presents an overview of our research project on digital preservation of cultural heritage objects and digital restoration of the original appearance of these objects. As an example of these objects, this project focuses on the preservation and restoration of the Great Buddhas. These are relatively large objects existing outdoors and providing various technical challenges. Geometric models of the great Buddhas are digitally achieved through a pipeline, consisting of acquiring data, aligning multiple range images, and merging these images. We have developed two alignment algorithms: a rapid simultaneous algorithm, based on graphics hardware, for quick data checking on site, and a parallel alignment algorithm, based on a PC cluster, for precise adjustment at the university. We have also designed a parallel voxel-based merging algorithm for connecting all aligned range images. On the geometric models created, we aligned texture images acquired from color cameras. We also developed two texture mapping methods. In an attempt to restore the original appearance of historical objects, we have synthesized several buildings and statues using scanned data and a literature survey with advice from experts. [ABSTRACT FROM AUTHOR]