Search

Semantic video segmentation is a key challenge for various applications. This paper presents a new model named Noisy-LSTM, which is trainable in an end-to-end manner, with convolutional LSTMs (ConvLSTMs) to leverage the temporal coherence in video frames, together with a simple yet effective training strategy that replaces a frame in a given video sequence with noises. Our training strategy spoils the temporal coherence in video frames and thus makes the temporal links in ConvLSTMs unreliable; this may consequently improve the ability of the model to extract features from video frames and serve as a regularizer to avoid overfitting, without requiring extra data annotations or computational costs. Experimental results demonstrate that the proposed model can achieve state-of-the-art performances on both the CityScapes and EndoVis2018 datasets. The code for the proposed method is available at https://github.com/wbw520/NoisyLSTM.

Several recent studies on compressive video sensing realized scene capture beyond the fundamental trade-off limit between spatial resolution and temporal resolution using random space-time sampling. However, most of these studies obtained results for higher-frame-rate video that was produced in simulation experiments or using an optically simulated random sampling camera, because there are currently no commercially available image sensors with random exposure or sampling capabilities. We fabricated a prototype complementary metal-oxide-semiconductor (CMOS) image sensor with quasi pixel-wise exposure timing that can realize nonuniform space-time sampling. The prototype sensor resets exposures independently by columns and fixes these exposures by rows for each 8 × 8-pixel block. This CMOS sensor is not fully controllable the pixels and has line-dependent control, but offers greater flexibility when compared with regular CMOS or charge-coupled device sensors with global or rolling shutters. We propose a method of realizing pseudo-random sampling for high-speed video acquisition that uses the flexibility of the CMOS sensor. We reconstruct the high-speed video sequence from images produced in pseudo-random sampling using a pre-learned decoder.

International audience; The rise of digitization of cultural documents offers large-scale contents, opening the road for development of AI systems in order to preserve, search, and deliver cultural heritage. To organize such cultural content also means to classify them, a task that is very familiar to modern computer science. Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph. Such a knowledge graph, combined with content analysis, enhances the notion of proximity between artworks so it improves the performances in classification tasks. In this paper, we propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data. With label propagation, we boost artwork classification by training a model using a graph convolutional network, relying on the relationships between entities of the knowledge graph. Following a transductive learning framework, our experiments show that relying on a knowledge graph modeling the relations between labeled data and unlabeled data allows to achieve state-of-the-art results on multiple classification tasks on a dataset of paintings, and on a dataset of Buddha statues. Additionally, we show state-of-the-art results for the difficult case of dealing with unbalanced data, with the limitation of disregarding classes with extremely low degrees in the knowledge graph.

Transparent object segmentation can be very useful in computer vision applications. However, because the transparent objects borrow texture from their background and have a similar appearance to their surroundings, they are not handled well by regular image segmentation methods. In this paper, we propose a method that overcomes these problems using the consistency and distortion properties in a light-field image. The light-field linearity is used to estimate the likelihood of a pixel belonging to the transparent object or Lambertian background, and the occlusion detector is used to find the occlusion boundary. Graph-cut optimization is applied for the pixel labeling problem. We acquire light field datasets from both camera array and lenslet camera for the transparent object, and use these datasets to evaluate our method. The results demonstrate that the proposed method successfully segments transparent objects from the background under various conditions.

Reflectance and shape are two important components in visually perceiving the real world. Inferring the reflectance and shape of an object through cameras is a fundamental research topic in the field of computer vision. While three-dimensional shape recovery is pervasive with varieties of approaches and practical applications, reflectance recovery has only emerged recently. Reflectance recovery is a challenging task that is usually conducted in controlled environments, such as a laboratory environment with a special apparatus. However, it is desirable that the reflectance be recovered in the field with a handy camera so that reflectance can be jointly recovered with the shape. To that end, we present a solution that simultaneously recovers the reflectance and shape (i.e., dense depth and normal maps) of an object under natural illumination with commercially available handy cameras. We employ a light field camera to capture one light field image of the object, and a 360-degree camera to capture the illumination. The proposed method provides positive results in both simulation and real-world experiments.

Fall is one of the leading causes of injury for the elderly individuals. Systems that automatically detect falls can significantly reduce the delay of assistance. Most of commercialized fall detection systems are based on wearable devices, which elderly individuals tend to forget wearing. Using surveillance cameras to detect falls based on computer vision is ideal, because anyone in the monitoring scopes can be under protection. However, the privacy protection issue using surveillance cameras has been bothering people. To effectively protect the privacy, we proposed an optical level anonymous image sensing system, which can protect the privacy by hiding the facial regions optically at the video capturing phase. We apply the system to fall detection. In detecting falls, we propose a neural network by combining a 3D convolutional neural network for feature extraction and an autoencoder for modelling the normal behaviors. The learned autoencoder reconstructs the features extracted from videos with normal behaviors with smaller average errors than those extracted from videos with falls. We evaluated our neural network by a hold-out validation experiment, and showed its effectiveness. In field tests, we showed and discussed the applicability of the optical level anonymous image sensing system for privacy protection and fall detection.

International audience; Buddha statues are a part of human culture, especially of the Asia area, and they have been alongside human civilisation for more than 2,000 years. As history goes by, due to wars, natural disasters, and other reasons, the records that show the built years of Buddha statues went missing, which makes it an immense work for historians to estimate the built years. In this paper, we pursue the idea of building a neural network model that automatically estimates the built years of Buddha statues based only on their face images. Our model uses a loss function that consists of three terms: an MSE loss that provides the basis for built year estimation; a KL divergence-based loss that handles the samples with both an exact built year and a possible range of built years (e.g., dynasty or centuries) estimated by historians; finally a regularisation that utilises both labelled and unlabelled samples based on manifold assumption. By combining those three terms in the training process, we show that our method is able to estimate built years for given images with 37.5 years of a mean absolute error on the test set. CCS CONCEPTS • Computing methodologies → Computer vision; Neural networks; • Applied computing → Fine arts.

We annotate 17,000 SNS posts with both the writer’s subjective emotional intensity and the reader’s objective one to construct a Japanese emotion analysis dataset. In this study, we explore the difference between the emotional intensity of the writer and that of the readers with this dataset. We found that the reader cannot fully detect the emotions of the writer, especially anger and trust. In addition, experimental results in estimating the emotional intensity show that it is more difficult to estimate the writer’s subjective labels than the readers’. The large gap between the subjective and objective emotions imply the complexity of the mapping from a post to the subjective emotion intensities, which also leads to a lower performance with machine learning models.

Topic modeling that can automatically assign topics to legal documents is very important in the domain of computational law. The relevance of the modeled topics strongly depends on the legal context they are used in. On the other hand, references to laws and prior cases are key elements for judges to rule on a case. Taken together, these references form a network, whose structure can be analysed with network analysis. However, the content of the referenced documents may not be always accessed. Even in that case, the reference structure itself shows that documents share latent similar characteristics. We propose to use this latent structure to improve topic modeling of law cases using document homophily. In this paper, we explore the use of homophily networks extracted from two types of references: prior cases and statute laws, to enhance topic modeling on legal case documents. We conduct in detail, an analysis on a dataset consisting of rich legal cases, i.e., the COLIEE dataset, to create these networks. The homophily networks consist of nodes for legal cases, and edges with weights for the two families of references between the case nodes. We further propose models to use the edge weights for topic modeling. In particular, we propose a cutting model and a weighting model to improve the relational topic model (RTM). The cutting model uses edges with weights higher than a threshold as document links in RTM; the weighting model uses the edge weights to weight the link probability function in RTM. The weights can be obtained either from the co-citations or from the cosine similarity based on an embedding of the homophily networks. Experiments show that the use of the homophily networks for topic modeling significantly outperforms previous studies, and the weighting model is more effective than the cutting model.

Retinal vessel segmentation is of great interest for diagnosis of retinal vascular diseases. To further improve the performance of vessel segmentation, we propose IterNet, a new model based on UNet, with the ability to find obscured details of the vessel from the segmented vessel image itself, rather than the raw input image. IterNet consists of multiple iterations of a mini-UNet, which can be 4$\times$ deeper than the common UNet. IterNet also adopts the weight-sharing and skip-connection features to facilitate training; therefore, even with such a large architecture, IterNet can still learn from merely 10$\sim$20 labeled images, without pre-training or any prior knowledge. IterNet achieves AUCs of 0.9816, 0.9851, and 0.9881 on three mainstream datasets, namely DRIVE, CHASE-DB1, and STARE, respectively, which currently are the best scores in the literature. The source code is available., Accepted in 2020 Winter Conference on Applications of Computer Vision (WACV 20)

This paper presents a method for reconstructing 3D image from multi-focus microscopic images captured with different focuses. We model the multi-focus imaging by a microscopy and produce the 3D image of a target object based on the model. The 3D image reconstruction is done by minimizing the difference between the observed images and the simulated images generated by the imaging model. Simulation and experimental result shows that the proposed method can generate the 3D image of a transparent object efficiently and reliably.

We investigate the problem of compressive acquisition of a dynamic light field. A promising solution for compressive light field acquisition is to use a coded aperture camera, with which an entire light field can be computationally reconstructed from several images captured through differently-coded aperture patterns. With this method, it was assumed that the scene should not move throughout the complete acquisition process, which restricted real applications. In this study, however, we assume that the target scene may change over time, and propose a method for acquiring a dynamic light field (a moving scene) using a coded aperture camera and a convolutional neural network (CNN). To successfully handle scene motions, we develop a new configuration of image observation, called V-shape observation, and train the CNN using a dynamic-light-field dataset with pseudo motions. Our method is validated through experiments using both a computer-generated scene and a real camera.

Generating models to handle new visual tasks requires additional datasets, which take considerable effort to create. We propose a method of domain adaptation for merging multiple models with less effort than creating an additional dataset. This method merges pre-trained models in different domains using glue layers and a generative model, which feeds latent features to the glue layers to train them without an additional dataset. We also propose a generative model that is created by distilling knowledge from pre-trained models. This enables the dataset to be reused to create latent features for training the glue layers. We apply this method to object detection in a low-light situation. The YOLO-in-the-Dark model comprises two models, Learning-to-See-in-the-Dark model and YOLO. We present the proposed method and report the result of domain adaptation to detect objects from RAW short-exposure low-light images. The YOLO-in-the-Dark model uses fewer computing resources than the naive approach.

We propose a framework for adaptively registering background models with an image for background subtraction with moving cameras. Existing methods search for a background model using a fixed window size, to suppress the number of false positives when detecting the foreground. However, these approaches result in many false negatives because they may use inappropriate window sizes. The appropriate size depends on various factors of the target scenes. To suppress false detections, we propose adaptively controlling the method parameters, which are typically determined heuristically. More specifically, the search window size for background registration and the foreground detection threshold are automatically determined using the re-projection error computed by the homography based camera motion estimate. Our method is based on the fact that the error at a pixel is low if it belongs to background and high if it does not. We quantitatively confirmed that the proposed framework improved the background subtraction accuracy when applied to images from moving cameras in various public datasets.

Detection followed by projection in conventional privacy cameras is vulnerable to software attacks that threaten to expose image sensor data. By multiplexing the incoming light with a coded mask, a FlatCam camera removes the spatial correlation and captures visually protected images. However, FlatCam imaging suffers from poor reconstruction quality and pays no attention to the privacy of visual information. In this paper, we propose a deep learning-based compressive sensing approach to reconstruct and protect sensitive regions from secured FlatCam measurements. We predict sensitive regions via facial segmentation and separate them from the captured measurements. Our deep compressive sensing network was trained with simulated data, and was tested on both simulated and real FlatCam data.

Deep convolutional neural networks (CNNs) have established their feet in the ground of computer vision and machine learning, used in various applications. In this work, an attempt is made to learn a CNN for a task of facial expression recognition (FER). Our network has convolutional layers linked with an FC layer with a skip-connection to the classification layer. Motivation behind this design is that lower layers of a CNN are responsible for lower level features, and facial expressions can be mainly encoded in low-to-mid level features. Hence, in order to leverage the responses from lower layers, all convo-lutional layers are integrated via FC layers. Moreover, a network with shared parameters is used to extract landmark motion trajectory features. These visual and landmark features are fused to improve the performance. Our method is evaluated on the CK+ and Oulu-CASIA facial expression datasets.

We propose an efficient pipeline from input to output for a tensor light-field display. Conventionally, a dense light field (i.e., tens of images taken with narrow viewpoint intervals) is required as an input in such displays. However, obtaining dense light fields is a challenging task for real scenes. To make the acquisition process more efficient, we adopted a coded-aperture camera as an input device, which is suitable for acquiring dense light fields in a compressive manner. Moreover, we modeled the entire process from acquisition to display using a convolutional neural network. As a result of training the network on a massive light field data, we can reproduce the whole light field on the display from only a few images taken with the camera. Both simulative and real experiments were conducted to show the effectiveness of our method.

Commercially available light field cameras have difficulty in capturing 5D (4D + time) light field videos. They can only capture still light filed images or are excessively expensive for normal users to capture the light field video. To tackle this problem, we propose a deep learning-based method for synthesizing a light field video from a monocular video. We propose a new synthetic light field video dataset that renders photorealistic scenes using UnrealCV rendering engine because no light field dataset is available. The proposed deep learning framework synthesizes the light field video with a full set (9$\times$9) of sub-aperture images from a normal monocular video. The proposed network consists of three sub-networks, namely, feature extraction, 5D light field video synthesis, and temporal consistency refinement. Experimental results show that our model can successfully synthesize the light field video for synthetic and actual scenes and outperforms the previous frame-by-frame methods quantitatively and qualitatively. The synthesized light field can be used for conventional light field applications, namely, depth estimation, viewpoint change, and refocusing.

A new change detection method based on background modeling is proposed.A new concept of light field sensing is introduced to evaluate spatio-temporal consistency.Performance of change detection outperformed the state-of-the-arts in severe surveillance condition. This paper is an extension of the work that was originally reported in Shimada et al. (2013). This paper proposes a change detection method based on spatio-temporal light ray consistency. The proposed method introduces light field sensing, which is used to generate an arbitrary in-focus plane. Change detection is performed in a surveillance scene, where the background region can be filtered out by an out-focusing process. This approach resolves a longstanding issue in background modeling-based object detection, which often suffers from false positives in the background regions. To realize this new change detection method, a new feature representation, called the local ray pattern (LRP), is introduced. The LRP evaluates the spatial consistency of the light rays, and this plays an important role in distinguishing whether the light rays come from the in-focus plane or elsewhere. A combination of the LRP and Gaussian mixture model (GMM)-based background modeling realizes change detection in the in-focus plane. Experimental results demonstrate the proposed method's effectiveness and its applicability to video surveillance.

Most conventional digital video cameras face a fundamental trade-off between spatial resolution, temporal resolution and dynamic range (i.e., brightness resolution) because of a limited bandwidth for data transmission. A few recent studies have shown that with non-uniform space-time sampling, such as that implemented with pixel-wise coded exposure, one can go beyond this trade-off and achieve high efficiency for scene capture. However, in these studies, the sampling schemes were pre-defined and independent of the target scene content. In this paper, we propose an adaptive space-time-brightness sampling method to further improve the efficiency of video capture. The proposed method adaptively updates a pixel-wise coded exposure pattern using the information analyzed from previously captured frames. We built a prototype camera that enables adaptive coding of patterns online to show the feasibility of the proposed adaptive coded exposure method. Simulation and experimental results show that the adaptive space-time-brightness sampling scheme achieves more accurate video reconstruction results and high dynamic range with less computational cost, than previous method. To the best of our knowledge, our prototype is the first implementation of an adaptive pixel-wise coded exposure camera.

The photometric stereo method enables estimation of surface normals from images that have been captured using different but known lighting directions. The classical photometric stereo method requires at least three images to determine the normals in a given scene. However, this method cannot be applied to dynamic scenes because it is assumed that the scene remains static while the required images are captured. In this work, we present a dynamic photometric stereo method for estimation of the surface normals in a dynamic scene. We use a multi-tap complementary metal-oxide-semiconductor (CMOS) image sensor to capture the input images required for the proposed photometric stereo method. This image sensor can divide the electrons from the photodiode from a single pixel into the different taps of the exposures and can thus capture multiple images under different lighting conditions with almost identical timing. We implemented a camera lighting system and created a software application to enable estimation of the normal map in real time. We also evaluated the accuracy of the estimated surface normals and demonstrated that our proposed method can estimate the surface normals of dynamic scenes.

We propose a learning-based framework for acquiring a light field through a coded aperture camera. Acquiring a light field is a challenging task due to the amount of data. To make the acquisition process efficient, coded aperture cameras were successfully adopted; using these cameras, a light field is computationally reconstructed from several images that are acquirToshiakied with different aperture patterns. However, it is still difficult to reconstruct a high-quality light field from only a few acquired images. To tackle this limitation, we formulated the entire pipeline of light field acquisition from the perspective of an auto-encoder. This auto-encoder was implemented as a stack of fully convolutional layers and was trained end-to-end by using a collection of training samples. We experimentally show that our method can successfully learn good image-acquisition and reconstruction strategies. With our method, light fields consisting of 5 \(\times \) 5 or 8 \(\times \) 8 images can be successfully reconstructed only from a few acquired images. Moreover, our method achieved superior performance over several state-of-the-art methods. We also applied our method to a real prototype camera to show that it is capable of capturing a real 3-D scene.

In this paper, we address the problem of image annotation with incomplete labeling, where multiple objects in each training image are not fully labeled. The conventional one-versus-all support vector machine (OVA-SVM), which performs fairly well on full labeling, decays drastically under the setting of incompleteness. Recently, a structured output learning method termed OVA-SSVM was proposed to boost the performance of OVA-SVM by modeling the structured associations of labels and show efficiency under the setting of incompleteness. OVA-SSVM assumes that each training sample includes a single label and adopts an loss measure of classification style where, as long as one of the predicted label is correct, the overall prediction should be considered correct. However, this may not be appropriate for the multilabel annotation task. Therefore, we extend the OVA-SSVM method to the multilabel situation and design a novel image-specific structured loss to account for the dependences between predicted labels relying on image label associations. The superiority of the proposed image-specific structured loss is that it can directly learn the semantic relationships of labels from training data without predefined semantic hierarchy. Extensive empirical results on a variety of benchmark datasets show that the proposed method performs significantly better than OVA-SSVM on image annotation tasks with incomplete labeling and achieves competitive performance compared to other state-of-the-art methods. © 2015 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

The goal of image annotation is to automatically assign a set of textual labels to an image to describe the visual contents thereof. Recently, with the rapid increase in the number of web images, nearest neighbor (NN) based methods have become more attractive and have shown exciting results for image annotation. One of the key challenges of these methods is to define an appropriate similarity measure between images for neighbor selection. Several distance metric learning (DML) algorithms derived from traditional image classification problems have been applied to annotation tasks. However, a fundamental limitation of applying DML to image annotation is that it learns a single global distance metric over the entire image collection and measures the distance between image pairs in the image-level. For multi-label annotation problems, it may be more reasonable to measure similarity of image pairs in the label-level. In this paper, we develop a novel label prediction scheme utilizing multiple label-specific local metrics for label-level similarity measure, and propose two different local metric learning methods in a multi-task learning (MTL) framework. Extensive experimental results on two challenging annotation datasets demonstrate that 1) utilizing multiple local distance metrics to learn label-level distances is superior to using a single global metric in label prediction, and 2) the proposed methods using the MTL framework to learn multiple local metrics simultaneously can model the commonalities of labels, thereby facilitating label prediction results to achieve state-of-the-art annotation performance.

A thermal camera captures the temperature distribution of a scene as a thermal image. In thermal images, facial appearances of different people under different lighting conditions are similar. This is because facial temperature distribution is generally constant and not affected by lighting condition. This similarity in face appearances is advantageous for face detection. To detect faces in thermal images, cascade classifiers with Haar-like features are generally used. However, there are few studies exploring the local features for face detection in thermal images. In this paper, we introduce two approaches relying on local features for face detection in thermal images. First, we create new feature types by extending Multi-Block LBP. We consider a margin around the reference and the generally constant distribution of facial temperature. In this way, we make the features more robust to image noise and more effective for face detection in thermal images. Second, we propose an AdaBoost-based training method to get cascade classifiers with multiple types of local features. These feature types have different advantages. In this way we enhance the description power of local features. We did a hold-out validation experiment and a field experiment. In the hold-out validation experiment, we captured a dataset from 20 participants, comprising 14 males and 6 females. For each participant, we captured 420 images with 10 variations in camera distance, 21 poses, and 2 appearances (participant with/without glasses). We compared the performance of cascade classifiers trained by different sets of the features. The experiment results showed that the proposed approaches effectively improve the performance of face detection in thermal images. In the field experiment, we compared the face detection performance in realistic scenes using thermal and RGB images, and gave discussion based on the results.

A light field, which is often understood as a set of dense multi-view images, has been utilized in various 2D/3D applications. Efficient light field acquisition using a coded aperture camera is the target problem considered in this paper. Specifically, the entire light field, which consists of many images, should be reconstructed from only a few images that are captured through different aperture patterns. In previous work, this problem has often been discussed from the context of compressed sensing (CS). In contrast, we formulated this problem from the perspective of principal component analysis (PCA) to derive optimal non-negative aperture patterns and a straight-forward reconstruction algorithm. Even though it is based on a conventional technique, our method has proven to be more accurate and much faster than a state-of-the-art CS-based method.

Several recent studies in compressive video sensing have realized scene capture beyond the fundamental trade-off limit between spatial resolution and temporal resolution using random space-time sampling. However, most of these studies showed results for higher frame rate video that were produced by simulation experiments or using an optically simulated random sampling camera, because there are currently no commercially available image sensors with random exposure or sampling capabilities. We fabricated a prototype complementary metal oxide semiconductor (CMOS) image sensor with quasi pixel-wise exposure timing that can realize nonuniform space-time sampling. The prototype sensor can reset exposures independently by columns and fix these amount of exposure by rows for each 8×8 pixel block. This CMOS sensor is not fully controllable via the pixels, and has line-dependent controls, but it offers flexibility when compared with regular CMOS or charge-coupled device sensors with global or rolling shutters. We propose a method to realize pseudo-random sampling for high-speed video acquisition that uses the flexibility of the CMOS sensor. We reconstruct the high-speed video sequence from the images produced by pseudo-random sampling using an over-complete dictionary. The proposed method also removes the rolling shutter effect from the reconstructed video.

In this paper, we describe a supervised four-dimensional (4D) light field segmentation method that uses a graph-cut algorithm. Since 4D light field data has implicit depth information and contains redundancy, it differs from simple 4D hyper-volume. In order to preserve redundancy, we define two neighboring ray types (spatial and angular) in light field data. To obtain higher segmentation accuracy, we also design a learning-based likelihood, called objectness, which utilizes appearance and disparity cues. We show the effectiveness of our method via numerical evaluation and some light field editing applications using both synthetic and real-world light fields., 2016 IEEE International Conference on Computational Photography (ICCP),Evanston, IL, USA,13-15 May 2016

International audience; Image-based modeling methods for generating 3D models from an image sequence have been widely studied. Most of these methods, however, require huge redundant spatio-temporal images to estimate scene depth. This is not an effective use of capturing higher resolution texture. On the other hand, a route panorama, which is a continuous panoramic image along a path, is an efficient way of consolidating information from multiple viewpoints into a single image. A route panorama captured by a line camera also has the advantage of capturing higher resolution easily. In this paper, we propose a method for estimating the depth of an image from a route panorama using color drifts. The proposed method detects color drift by deformable window matching of the color channels. It also uses a hierarchical belief propagation to estimate the depth stably and decrease the computation cost thereof.