Search

The ability to sense and understanding the driving environment is a key technology for ADAS and autonomous driving. Human drivers have to pay more visual attention to important or target elements and ignore unnecessary ones present in their field of sight. A model that computes this visual attention of targets in a specific driving environment is essential and useful in supporting autonomous driving, object-specific tracking & detection, driving training, car collision warning, traffic sign detection, etc. In this paper, we propose a new framework of visual attention that can predict important objects in the driving scene using a conditional generative adversarial network. A large scale Visual Attention Driving Database (VADD) of saliency heat-maps is built from existing driving datasets using a saliency mechanism. The proposed framework model takes its strength from these saliency heat-maps as conditioning label variables. The results show that the proposed approach makes us able to predict heat-maps of most important objects in a driving environment.

The complexity of a scene in addition to the need for real-time processing are the main challenges that face any background/foreground separation approach for maritime environment. Recent studies on Low-rank and Sparse Separation (LSS) achieved good performance when compared to traditional background subtraction techniques in segregating the foreground from a complex background. However, the issue of maintaining this type of a separation via an updating mechanism is not well addressed by the majority of LSS approaches. The study presents a tensor based singular value decomposition approach for background/foreground separation. The approach is uniquely designed to deal with most challenges related to a maritime environment such as sea dynamics, boat wakes, variety of foreground objects, and camera jitter. Furthermore, the proposed approach operates incrementally via updating the separation components as opposed to reperforming the decomposition on the entire video sequence when a set of frames arrives. Additionally, a forgetting mechanism is employed in the proposed approach to efficiently handle challenges such as Stationary Foreground Objects (SFOs) and ghost effects. The performance of the proposed method with several state-of-the-art LSS and Non-LSS techniques on videos with complex maritime scenarios are evaluated. The results exhibit better performance over most of the tested challenges and also demonstrate the capability of the proposed method to perform the separation in less computational time.

This brief paper introduces a framework for supervised and semi-supervised learning by estimating a non-linear embedding that incorporates Spectral Graph Convolutions structure. The proposed algorithm exploits data-driven graphs in two ways. First, it integrates data smoothness over graphs. Second, its regression loss function jointly uses the data and its graph in the sense that the regressor model sees convolved data samples. The resulting framework can solve the problem of over-fitting on local neighborhood structures for image data having varied natures like outdoor scenes, faces and man-made objects. The proposed Graph Convolution based Semi-supervised Embedding (GCSE) not only provides a new perspective to non-linear embedding research but also induces the standpoint on Spectral Graph Convolutions methods. A series of experiments are conducted on four image datasets in order to compare the proposed method with some state-of-art semi-supervised methods. This evaluation demonstrates the effectiveness of the proposed embedding method.

This paper addresses the issue of image and video face retrieval. The aim of this work is to be able to retrieve images and/or videos of specific person from a dataset of images and videos if we have a query image of that person. The methods proposed so far either focus on images or videos and use hand crafted features. In this work we built an end-to-end pipeline for both image and video face retrieval where we use convolutional neural network (CNN) features from an off-line feature extractor. And we exploit the object proposals learned by a region proposal network (RPN) in the online filtering and re-ranking steps. Moreover, we study the impact of finetuning the networks, the impact of sum-pooling and max-pooling, and the impact of different similarity metrics. The results that we were able to achieve are very promising.

Traffic signs play an important role for Advanced Driver Assistance Systems (ADAS) as well as for autonomous driving vehicles. Most of the works done focus on recognizing symbol based signs leaving apart important information provided by other type of signs like complementary panels or text based signs. In this paper, we include detection and classification of both symbol and text based signs focusing on the most common ones found in European urban environments. The system consists of three stages, traffic sign detection, refinement and classification. The detection and refinement is performed using Mask R-CNN while the classification is achieved with a proposed Convolutional Neural Network (CNN) architecture. We introduced the extended version of the German Traffic Sign Detection Benchmark (GTSDB), labeled in a pixel manner (masks) with 164 classes grouped into 8 categories. It is used for the detection and classification steps. Experimental results on German environments show that our proposed system is capable of detecting all categories of traffic signs while at the same time recognizing them with high accuracy achieving comparable performance with the state of the art.

The background Initialization (BI) problem has attracted the attention of researchers in different image/video processing fields. Recently, a tensor-based technique called spatiotemporal slice-based singular value decomposition (SS-SVD) has been proposed for background initialization. SS-SVD applies the SVD on the tensor slices and estimates the background from low-rank information. Despite its efficiency in background initialization, the performance of SS-SVD requires further improvement in the case of complex sequences with challenges such as stationary foreground objects (SFOs), illumination changes, low frame-rate, and clutter. In this paper, a self-motion-assisted tensor completion method is proposed to overcome the limitations of SS-SVD in complex video sequences and enhance the visual appearance of the initialized background. With the proposed method, the motion information, extracted from the sparse portion of the tensor slices, is incorporated with the low-rank information of SS-SVD to eliminate existing artifacts in the initiated background. Efficient blending schemes between the low-rank (background) and sparse (foreground) information of the tensor slices is developed for scenarios such as SFO removal, lighting variation processing, low frame-rate processing, crowdedness estimation, and best frame selection. The performance of the proposed method on video sequences with complex scenarios is compared with the top-ranked state-of-the-art techniques in the field of background initialization. The results not only validate the improved performance over the majority of the tested challenges but also demonstrate the capability of the proposed method to initialize the background in less computational time.

Quantum computing offers a new approach to the problem modeling and solving. This paper deals with the quantum modeling of reactive agents. It also proposes a quantum algorithm to implement the subsumption architecture, widely used by reactive agents, particularly in robotics. This work shows the contribution of the formalism proposed by quantum mechanics to the modeling and the proof of certain properties of the agent behavior. After, the definition of the reactive agent state modeling, the paper suggests a behavior modeling approach based on two steps for subsumption architecture. The first one models the preset behavior that links each action to the perception states. The second one determines, among several actuated actions, the one that the robot must achieve. The subsumption architecture raises the challenge of modeling hierarchical priority of actions. To this end, a multipartite entanglement is used in the second step. More precisely, the paper proposes and generalizes a W-state circuit in order to be used for modeling hierarchical priority actions and controlling the robot accordingly. The result of both steps provides a formal model that links the robot’s perception (input) to the actions (output), with respect to the subsumption architecture. The proposed model of agent is simulated using IBM quantum computer. The simulation shows that the model can either be served as a control unit of the robot (CU) to obtain the suitable action or to simulate the robot behavior.

Stereo-vision is the most widely used technique in the development of environmental perception systems for intelligent transportation. The main requirement for the application of stereo vision on a vehicle is the processing time which must be very fast for autonomous driving in real time, whereas the computation of the correspondence of the images in the algorithm of stereo-vision requires a more computing power. This article presents an implementation of the stereo vision system to generate a scene disparity map using the sum of absolute differences (SAD), and the triangulation method for calculating the distance between the obstacle and the stereoscopic system. A parallel treatment is used to speed data processing and this algorithm is implemented in embedded platform. Several experiments are performed from a profiling analysis to have a statistical analysis for optimization.

Detecting moving objects from a moving vehicle is a challenging problem and crucial for autonomous driving, especially in urban scenarios. The current literature has focused on this task as many approaches have been dedicated to moving object detection. These approaches consist of multistage pipelines, including semantic segmentation and optical flow estimation, and require multiple sources of information from active and passive sensors. However, they fail to accurately segment moving objects due to the large ego- camera motion, in addition to the high processing time. In this work, we propose a novel approach to moving object detection by processing information only from a camera. Our approach is based on integrating an encoder-decoder network (EDNet) with a semantic segmentation model (Mask R-CNN), where Mask R-CNN detects the objects of interest and the EDNet classifies their motion (moving/static) over two consecutive frames. We compare the results of our proposed model with existing MOD models on three SOTA benchmarks. We achieved SOTA performance in terms of visual quality and accuracy with competitive speed.

Feature engineering forms an important component of machine learning and pattern recognition. It is a fundamental process for off-line writer identification of handwritten documents, which continues to be an interesting subject of research in various forensic and authentication areas. In this work, we propose an efficient, yet computationally and conceptually simple framework for off-line text independent writer identification using local textural features in characterizing the writing style of each writer. These include Local Binary Patterns (LBP), Local Ternary Patterns (LTP), and Local Phase Quantization (LPQ). Our approach focuses on exploiting the writing images at small observation regions where a set of connected component sub-images are cropped and extracted from each handwriting sample (document or set of word/text line images). These connected components are seen as texture images where each one of them is subjected to feature extraction using LBP, LPQ, or LTP. Then, a histogram sequence concatenation is applied to the feature image after dimensionality reduction followed by image subdivision into a number of non-overlapping regions. For classification, the 1-NN (Nearest Neighbor) classifier is used to identify the writer of the questioned samples based on the dissimilarity of feature vectors computed from all components in the writing. Experiments on IFN/ENIT (411 writers/Arabic), AHTID/MW (53 writers/Arabic), CVL (309 writers/English), and IAM (657 writers/English) databases demonstrate that our proposed system outperforms old and recent state-of-the-art writer identification systems on Arabic script, and demonstrates a competitive performance on English ones.

In tasks such as abandoned luggage detection and stopped car detection, stationary foreground objects (SFOs) need to be detected and properly classified in real time. Different methods have been proposed to detect SFOs, but they are mainly focused on certain types of objects. In this paper, an incremental singular value decomposition-based method is presented to detect all types of SFOs such as abandoned objects and removed objects. The proposed method decomposes the video tensor spatiotemporally and divides it into background and foreground components. An appropriate analysis is applied to the foreground tensor to define a pixel time series of each stationary foreground category. Such analysis leads to the fact that SFOs can be detected easily owing to their continuous persistence in the decomposed foreground tensor. Furthermore, the unique structure of the pixel time series of each category allows identifying the category of the detected objects, whether they are abandoned or removed, and detecting the exact time of the start and end of each event. The results demonstrate that the proposed method achieves a superior performance in detecting SFOs at both object and pixel levels. In addition, the proposed method is computationally simple, and its complexity is lower compared to other approaches; hence, it can adequately satisfy real-time requirements.

Semantic segmentation is a challenging task in computer vision systems. A lot of methods have been developed to tackle this problem ranging from autonomous vehicles, human-computer interaction, to robotics, medical research, agriculture and so on. Many of these methods have been built using the deep learning paradigm that has shown a salient performance. For this reason, we propose to survey these methods by, first categorizing them into ten different classes according to the common concepts underlying their architectures. Second, by providing an overview of the publicly available datasets on which they have been assessed. In addition, we present the common evaluation matrix used to measure their accuracy. Moreover, we focus on some of the methods and look closely at their architectures in order to find out how they have achieved their reported performances. Finally, we conclude by discussing some of the open problems and their possible solutions.

Graph-based embedding methods are very useful for reducing the dimension of high-dimensional data and for extracting their relevant features. In this paper, we introduce a novel nonlinear method called Flexible Discriminant graph-based Embedding with feature selection (FDEFS). The proposed algorithm aims to classify image sample data in supervised learning and semi-supervised learning settings. Specifically, our method incorporates the Manifold Smoothness, Margin Discriminant Embedding and the Sparse Regression for feature selection. The weights add l 2 , 1 -norm regularization for local linear approximation. The sparse regression implicitly performs feature selection on the original features of data matrix and of the linear transform. We also provide an effective solution method to optimize the objective function. We apply the algorithm on six public image datasets including scene, face and object datasets. These experiments demonstrate the effectiveness of the proposed embedding method. They also show that proposed the method compares favorably with many competing embedding methods.

Aiming at the defect of Local Binary Pattern (LBP) and its variants, this paper presents a new modeling of the conventional LBP operator for texture classification. Named Attractive-and-Repulsive Center-Symmetric Local Binary Patterns ( ACS-LBP and RCS-LBP ), the proposed new texture descriptors preserve the advantageous characteristics of uniform LBP. Based on local attractive-and-repulsive characteristics, the proposed local texture modeling can really inherit good properties from both gradient and texture operators than the Center-Symmetric Local Binary Patterns (CS-LBP) does. Different from CS-LBP , which considers four doublets around the center pixel, the proposed methods take into account the four triplets corresponding to the vertical and horizontal directions, and the two diagonal directions by including the value of the central pixel in the modeling. In addition, Average Local Gray Level ( ALGL ), Average Global Gray Level ( AGGL ) and the median value over 3 × 3 neighborhood are introduced to capture both microstructure and macrostructure texture information. To capture the coarse and fine information of the features and thus to make ACS-LBP and RCS-LBP more robust and stable, multiscale ARCS-LBP descriptor is proposed. There is no necessity to learn texton dictionary, as in methods based on clustering, and no tuning of parameters is required to deal with different datasets. Extensive experiments performed on thirteen challenging representative texture databases show that the proposed operators can achieve impressive classification accuracy. Furthermore, we clearly validate the feasibility of the proposed ACS-LBP , RCS-LBP and ARCS-LBP descriptors by comparing their results with those obtained with a large number of recent state-of-the-art texture descriptors including deep features. Statistical significance of achieved accuracy improvement is demonstrated through Wilcoxon signed rank test.

Effective features are important for visual tracking, and efficiency also needs to be considered especially for multi-object tracking. Thanks to the simplicity, we think compressive sensing features are suitable for this task. In this paper, we use compressive sensing features to improve the Markov decision process (MDP) multi-object tracking framework. First, we design a single object tracker which uses the compressive tracking to correct the optical flow tracking and apply this tracker into the MDP tracking framework. The appearance model constructed during compressive tracking also helps for data association. In order to validate our method, we firstly test the designed single object tracker with a common dataset. Then, we test our multi-object tracking method for vehicle tracking. Finally, we analyze and test our approach in the multi-object tracking (MOT) benchmark for pedestrian tracking. The results show our approach performs superiorly against several state-of-the-art online multi-object trackers.

Pedestrian detection has so far achieved great success in normal illumination, while pedestrians captured in extreme weather are often ignored. This paper investigates the importance of studying the effects of weather conditions on the recognition task, such as blurring and low contrast. Many image restoration techniques have recently been proposed, but are still insufficient to remove weather effects from images. We present our strong new pedestrian recognition system against climate situations, which is based on locating contours cues by applying multiple edge filters and extracting multiple features from images such as census transform (CT), modified census transform (MCT), and local gradient pattern (LGP) without performing any image restoration algorithm. The next stage involves finding the most discriminative characteristics using feature selection (FS) techniques. Finally, we use the final feature vector as an input to a radial basis function-based support vector machine classifier (RbfSVM) for pedestrian recognition. Experiments are performed on the daimler pedestrian classification benchmark dataset. Results show that the area under the curve (AUC) and the detection rate of our model are less affected by weather conditions compared to other common models like histogram of oriented gradients (HOG) and gabor filter bank (GFB) detectors.

Face recognition becomes an important task performed routinely in our daily lives. This application is encouraged by the wide availability of powerful and low-cost desktop and embedded computing systems, while the need comes from the integration in too much real world systems including biometric authentication, surveillance, human-computer interaction, and multimedia management. Moreover, face recognition technology is now adopted in new intelligent systems and devices like smart-phones, which impose some constraints related to the complexity and execution time of the recognition process. This fact brings new challenges and gives much more area to extend the ongoing researches. This research field experienced the development of many methods and architectures aiming at producing face recognition systems which are efficient in terms of precision, robustness and computation time. In the same context, this article proposes a new feature descriptor referred to as Mixed Neighborhood Topology Cross Decoded Patterns (MNTCDP) as an effective face descriptor, The proposed handcrafted descriptor fulfills the needs of current face recognition applications and can be integrated in different platforms, requiring simple, robust and computationally low algorithms. Instead of heuristic code constructions, MNTCDP is built using new neighborhood topology and new pattern encoding scheme, which have high ability to extract discriminative and stable face representation. The adopted face recognition system consists of three stages: (1) face detection and alignment to normalize the input images to a common form if needed; (2) feature extraction using the proposed MNTCDP descriptor and (3) face recognition through a supervised image classification task using the simple K-Nearest Neighbors classifier. Simulated experiments on ORL, YALE, Extended Yale B, FERET and AR datasets acquired under different illumination conditions or facial expressions show that the proposed MNTCDP descriptor presents high performance ability in classifying face images. MNTCDP demonstrates superior performance than a large number of recent state-of-the-art LBP variants and deep learning methods, as well as recent most promising works of the literature.

International audience; In this paper, we study the problem of autonomous and reliable positioning of a camera w.r.t. an object when only this latter is known but not the rest of the scene. We propose to combine the advantages and efficiency of a visual servoing scheme and the generalization ability of a generative adversarial network. The paper describes how to efficiently create a synthetic dataset in order to train a network that predicts an intermediate visual keyframe between two images. Subsequent predictions are used as visual features to autonomously converge towards the desired pose even for large displacements. We show that the proposed method can be used without any prior knowledge on the scene appearance except for the object itself, while being robust to various lighting conditions and specular surfaces. We provide experimental results, both in simulation and using a real service robot platform to validate and evaluate the effectiveness, robustness, and accuracy of our approach.

Depth perception is one of the most active research areas in computer vision. Passive stereo vision is a well known technique for obtaining 3-D depth information of objects seen by two or more video cameras from different viewpoints (Hartley & Zisserman, 2000|Brown et al., 2003). The difference of the viewpoint positions causes a relative displacement of the corresponding features in the stereo images. Such relative displacement, called disparity, encodes the depth information, which is lost when the three dimensional scene is projected on an image. The key problem, which is difficult to solve and computationally expensive (Barnard & Fisher, 1982), is hence to compare each feature extracted from one image with a number, generally large, of features extracted from the other image in order to find the corresponding one, if any. Once the matching process is established and the stereo vision system parameters are known, the depth computation is reduced to a simple triangulation (Jane & Haubecker, 2000|Dooze, 2001). This chapter presents some recent research works proposed to solve the stereo matching problem. The presented methods are based on a global approach, which can be viewed as a constraint satisfaction problem where the objective is to highlight a solution for which the matches are as compatible as possible with respect to specific constraints. These methods are tested and evaluated for real-time obstacle detection in front of a vehicle using linear stereo vision.

Classifying and modeling texture images, especially those with significant rotation, illumination, scale, and view-point variations, is a hot topic in the computer vision field. Inspired by local graph structure (LGS), local ternary patterns (LTP), and their variants, this paper proposes a novel image feature descriptor for texture and material classification, which we call Petersen Graph Multi-Orientation based Multi-Scale Ternary Pattern (PGMO-MSTP). PGMO-MSTP is a histogram representation that efficiently encodes the joint information within an image across feature and scale spaces, exploiting the concepts of both LTP-like and LGS-like descriptors, in order to overcome the shortcomings of these approaches. We first designed two single-scale horizontal and vertical Petersen Graph-based Ternary Pattern descriptors ( $PGTP_{h}$ and $PGTP_{v}$ ). The essence of $PGTP_{h}$ and $PGTP_{v}$ is to encode each $5\times 5$ image patch, extending the ideas of the LTP and LGS concepts, according to relationships between pixels sampled in a variety of spatial arrangements (i.e., up, down, left, and right) of Petersen graph-shaped oriented sampling structures. The histograms obtained from the single-scale descriptors $PGTP_{h}$ and $PGTP_{v}$ are then combined, in order to build the effective multi-scale PGMO-MSTP model. Extensive experiments are conducted on sixteen challenging texture data sets, demonstrating that PGMO-MSTP can outperform state-of-the-art handcrafted texture descriptors and deep learning-based feature extraction approaches. Moreover, a statistical comparison based on the Wilcoxon signed rank test demonstrates that PGMO-MSTP performed the best over all tested data sets.

Automatic facial expression recognition (FER) has been extensively studied owing to its wide range of applications, such as in e-learning platforms used to automatically collect the feedback of students regarding a particular content and to help children with autism have a better understanding of their environment. Owing to the advances made in the fields of machine learning and computational devices, researchers are developing more accurate and robust facial expression recognition frameworks. In this paper, we propose a completely new framework for person-independent FER based on combining textural and shape features from 49 detected landmarks in an input facial image. The shape information is extracted using the histogram of oriented gradients (HOG) applied on a binary patch generated by interpolating the locations of the 49 detected landmarks. The textural information is computed from 49 sub-images, each centered on one landmark, using a new handcrafted descriptor that we also propose herein and is referred to as Orthogonal and Parallel-based Directions Generic Quad Map Binary Patterns (OPD-GQMBP). OPD-GQMBP encodes the relevant information based on the orthogonality and parallelism of the geometries to select the prominent pixels within a n × n neighborhood. The proposed framework outperforms many previous state-of-the-art methods including deep-learning-based approaches on five widely used benchmarks: CK+, KDEF, JAFFE, Oulu-Casia VIS, and RaFD, through the Leave-One-Subject-Out evaluation protocol. In addition, the superiority of the OPD-GQMBP descriptor is fairly proven against 10 deep features (e.g., VGG, ResNets, DenseNet, GoogeLeNet, and Inception) and 12 recent and powerful LBP variants.

Autonomous driving has achieved rapid development over the last few decades, including the machine perception as an important issue of it. Although object detection based on conventional cameras has achieved remarkable results in 2D/3D, non-visual sensors such as 3D LiDAR still have incomparable advantages in the accuracy of object position detection. However, the challenge also exists with the difficulty in properly interpreting point cloud generated by LiDAR. This paper presents a multi-modal-based online learning system for 3D LiDAR-based object classification in urban environments, including cars, cyclists and pedestrians. The proposed system aims to effectively transfer the mature detection capabilities based on visual sensors to the new model learning based on non-visual sensors through a multi-target tracker (i.e. using one sensor to train another). In particular, it integrates the Online Random Forests (ORF) [1] method, which inherently has the abilities of fast and multi-class learning. Through experiments, we show that our system is capable of learning a high-performance model for LiDAR-based 3D object classification on-the-fly, which is especially suitable for robotics in-situ deployment while responding to the widespread challenge of insufficient detector generalization capabilities., Comment: 8 pages, 7 figures, accepted at IEEE ITSC 2021

International audience; In this paper, we propose combining between the transformed hand-crafted and deep features using PCA to recognize the six-basic facial expressions from static images. To evaluate our approach, we use three popular databases (CK+, CASIA and MMI). We introduce the use of the Pyramid Multi Level (PML) face representation for facial expression recognition. The hand-crafted features are obtained with such representations. Initially, we determine the optimal level of the PML features of three hand-crafted descriptors (HOG, LPQ and BSIF) using CK+, CASIA and MMI databases. After the optimal level of the PML is found for each descriptor, we combine them together with the transformed final VGG-face layers (FC6 and FC7) in order to get a compact image descriptor. In within-database experiments, our approach achieved higher accuracy than the state-of-art methods on both the CK+ and CASIA databases, and competitive result on the MMI database. Likewise, our approach outperformed the static methods in all six experiments of cross-databases. (C) 2020 Published by Elsevier Ltd.

International audience; Several human being traits can be used as a robust and distinctive identifier for a given person. The palm region of the hand is one of these features that researchers in biometric fields have given a huge consideration in recent years. Many works have been proposed in the literature to design palmprint (an image acquired of the palm region) recognition framework. Extraction of prominent image local features is a critical module in most of these approaches. Local Binary Patterns (LBP) like methods, have emerged as one of the most effective feature extraction techniques. Despite a period of remarkable evolution, neither extensive and comprehensive evaluation nor comparison has been performed to date on a large number of LBP variants and non-LBP texture methods in palmprint recognition problem. Motivated by this, this paper aims to fill that gap and provide a comprehensive comparative study of the performance of a large number of recent texture descriptors in palmprint recognition. Extensive experimental results on the well-known constrained and unconstrained challenging palmprint databases, indicate that a number of tested local texture descriptors, which are evaluated for the first time on palmprint recognition, achieve promising results. Classification results are statistically compared through Wilcoxon signed rank test.

International audience; Handwriting based writer identification is one of the reliable components of behavioral biometrics. A huge effort has been done in recent years to improve the writer identification performance. Our paper presents a new and effective off-line text-independent system for writer identification. Extracting features from handwriting substantially impacts the ability of the classification process to identify the query writers. With the use of suitable classifier, a well-designed and discriminative feature extraction improves the classification performance. For that, we introduce a discriminative yet simple feature method, referred to as Local gradient full-Scale Transform Patterns (LSTP). The proposed LSTP algorithm captures salient local writing structure at small regions of interest of the writing. These writing regions are termed as connected components. In the classification stage, we perform Hamming distance based NN classifier to compare and match LSTP feature vectors. The proposed framework is evaluated on 9 well-known handwritten benchmarks. Experimental results show high identification performance against the current state-of-the-art. (C) 2020 Elsevier B.V. All rights reserved.

Writer identification is experiencing a revival of activity in recent years and continues to attract great deal of attention as a challenging and important area of research in the field of forensic and authentication. In this work, we introduce a reliable off-line system for text-independent writer identification of handwritten documents. Feature engineering is an essential component of a pattern recognition system, which can enhance or decrease the classification performance. A well-designed and defined feature extraction method improves the classification task. This paper proposes, for feature extraction, an effective, yet high-quality and conceptually simple feature image descriptor referred to as Cross multi-scale Locally encoded Gradient Patterns (CLGP). The proposed CLGP feature extraction method, which is expected to better represent salient local writing structure, operates at small observation regions (i.e., connected component sub-images) of the writing sample. CLGP histogram feature vectors computed from all these observation regions in all writing samples are considered as classification inputs to identify query writers using the Nearest Neighbor Classifier (1-NN). Our system is evaluated on six standard databases (IFN/ENIT, AHTID/MW, CVL, IAM, Firemaker, and ICDAR2011) including handwritten samples in Arabic, English, French, Greek, German, and Dutch languages. Comparing the identification performance with old and recent state-of-the-art methods, the proposed system achieves the highest performance on IFN/ENIT, AHTID/MW, and ICDAR2011 databases, and demonstrates competitive performance on IAM, CVL, and Firemaker databases.

We address the problem of image-to-video face retrieval. Given a query image of a person, the aim is to retrieve videos of that same person. The methods proposed so far are based on hand-crafted features. In this work we investigate the use of an off-the-shelf object detection network as a feature extractor by building an image-to-video face retrieval pipeline composed of an offline feature extractor and online filtering and re-ranking steps that use the object proposals learned by a Region Proposal Network (RPN) and their associated representations taken from a CNN. Moreover we study the relevance of features from a fine-tuned network. In addition to that we explore the use of face detection before extracting the features and we test the impact of different similarity metrics. The results obtained are promising.

International audience; In this paper, we propose a method that removes raindrops with light field image using image inpainting. We first use the depth map generated from light field image to detect raindrop regions which are then expressed as a binary mask. The original image with raindrops is improved by refocusing on the far regions and filtering by a high-pass filter. With the binary mask and the enhanced image, image inpainting is then utilized to eliminate raindrops from the original image. We compare pre-trained models of several deep learning based image inpainting methods. A light field raindrop dataset is released to verify our method. Image quality analysis is performed to evaluate the proposed image restoration method. The recovered images are further applied to object detection and visual localization tasks.

Place recognition is an important step in intelligent driving, allowing the vehicle recognize where it is to plan its route. Obtaining distinguishable features can ensure the success of image-based place recognition. However, generating robust features across drastically appearance changing images is still a challenging problem. Deep features are frequently chosen instead of local features in the tasks of place recognition following the development of convolutional neural networks. But even the deep features generated by powerful neural models can cause unsatisfactory recognition results. This is perhaps due to a lack of information selecting process. The technology of semantic segmentation allows recognizing and classifying image information. Semantic segmentation followed by image inpainting provide a possibility of detecting, deleting and reconstructing annoying information.

The Codebook model is one of the popular real-time models for background subtraction to detect moving objects. In this paper, we propose two techniques to adapt the original Codebook algorithm to multispectral images: dynamic mechanism and fusion strategy. For each channel, only absolute spectral value is used to calculate the spectral similarity between the current frame pixel and reference average value in the matching process, which can simplify the matching equations. Besides, the deciding boundaries are obtained based on statistical information extracted from the data and always adjusting themselves to the scene changes. Results demonstrate that with the proposed techniques, we can acquire a comparable accuracy with other methods using the same multispectral dataset for background subtraction.

International audience; Pattern recognition and computer vision fields experienced the proposal of several architectures and approaches to deal with the demands of real world applications including face recognition. They have almost the same structure, based generally on a series of steps where the main ones are feature extraction and classification. The literature works interessted in face recognition problems, insist on the role of texture description as one of the key elements in face analysis, since it greatly affects recognition accuracy. Therefore, texture feature extraction has gained much attention and became a long-standing research topic thanks to its abilities to efficiently understand the face recognition process, especially in terms of face description. Recently, several literature researches in face application proposed new architectures based on pattern description proved by their discriminative power when extracting the feature information from facial images. These advantages combined with an outstanding performance in many classification applications, allowed the LBP-like descriptors to be one of the most prominent texture description method. Given this period of remarkable evolution, this research work includes a comprehensive analytical study of the face recognition performance of 64 LBP-like and 3 non-LBP texture descriptors recently proposed in the literature. To this end, we adopted a face recognition framework composed of four stages: 1) image pre-processing using gamma correction; 2) feature extraction using texture descriptors; 3) histogram calculation and 4) face recognition and classification based on the simple parameter-free Nearest Neighbors classifier (NN). The conducted comprehensive evaluations and experiments on the challenging and widely used benchmarks ORL, YALE, Extended YALE B and FERET databases presenting different challenges, indicate that a number of evaluated texture descriptors, which are tested for the first time on face recognition task, achieve better or competitive compared to several recent systems reported in face recognition literature.

This paper presents new modeling of local binary patterns for texture representation. Referred to as local binary gradient contours (LBGC), the proposed models are expected to better represent the salient local texture structure. Thanks to the flexibility of repulsive-attractive characteristics, which represent the cornerstone of the proposed descriptors, two distinct LBP-like descriptors are built: repulsive and attractive local binary gradient contours (RLBGC and ALBGC). Conventional methods such as LBP, the family of binary gradient contours (BGC1, BGC2 and BGC3), LBP by neighborhoods (nLBPd) and several other LBP-like methods, are based on pairwise comparison of adjacent pixels. Unlike these methods, the proposed RLBGC and ALBGC operators encode the differences between local intensity values within triplets of pixels, along a closed path around the central pixel of a 3x3 gray-scale image patch. In order to increase the robustness of the proposed RLBGC and ALBGC descriptors, the triplet formed by the average local and average global gray levels (ALGL and AGGL) and the central pixel is incorporated in the modeling. In order to make the proposed approach more robust and stable, the RLBGC and ALBGC are concatenated together to form multi-scale repulsive-and-attractive local binary gradient contour (RALBGC) descriptor. Extensive experimental results from 13 challenging representative texture datasets show that the proposed descriptors, applied on each dataset, can achieve interesting classification accuracy, which is competitive or better than a great number of state-of-the-art LBP variants and non-LBP methods. Furthermore, statistical hypothesis testing is performed to prove the statistical significance of the achieved accuracy improvement over all the tested datasets.

Extracting the background from a video in the presence of various moving patterns is the focus of several background-initialization approaches. To model the scene background using rank-one matrices, this paper proposes a background-initialization technique that relies on the singular-value decomposition (SVD) of spatiotemporally extracted slices from the video tensor. The proposed method is referred to as spatiotemporal slice-based SVD (SS-SVD). To determine the SVD components that best model the background, a depth analysis of the computation of the left/right singular vectors and singular values is performed, and the relationship with tensor-tube fibers is determined. The analysis proves that a rank-1 matrix extracted from the first left and right singular vectors and singular value represents an efficient model of the scene background. The performance of the proposed SS-SVD method is evaluated using 93 complex video sequences of different challenges, and the method is compared with state-of-the-art tensor/matrix completion-based methods, statistical-based methods, search-based methods, and labeling-based methods. The results not only show better performance over most of the tested challenges, but also demonstrate the capability of the proposed technique to solve the background-initialization problem in a less computational time and with fewer frames.

In this paper, the three level descriptions from LTP and the directional features from LDP are combined to form a new local feature descriptor, referred to as local directional ternary pattern (LDTP) for texture classification. LDTP is a framework, which consists in encoding both contrast information and directional pattern features in a compact way based on local derivative variations. To achieve robustness, the proposed operator first computes for each pixel within its 3 × 3 overlapping grayscale image patch, on the one hand, eight directional edge responses using the eight Frei–Chen masks, and on the other hand, central edge response through the 2nd derivative of Gaussian filter to capture more detailed information. This allows producing a more discriminative encoding than several state-of-the art methods based only on intensity information. Then, spatial relationships among the neighboring pixels through the edge responses are exploited independently with the help of both LDP’s and LTP’s concepts to enhance the discrimination capability. Indeed, the implicit utilization of both concepts of LTP and LDP encodes more information in comparison to the existing directional and derivative methods in less space, and simultaneously allows discriminating more textures. Finally, the resultant LDTP pattern is divided into two distinct parts: local directional ternary pattern upper (LDTPU) and local directional ternary pattern lower (LDTPL), and the final feature descriptor vector is obtained by linear concatenation of both LDTPU and LDTPL histograms. The experiments carried out on nine publicly available texture datasets demonstrated that the proposed LDTP descriptor achieves classification performance, which is competitive or better than several recent and old state-of-the-art LBP variants. Statistical significance of the achieved accuracy improvement by the proposed descriptor has been also demonstrated through the Wilcoxon signed rank test applied on all the tested datasets.

A formal definition of Concave and Convex Binary Thresholding Functions are introduced.Two new LBP-like descriptors: Local Concave and Convex Micro-Structures Pattern (LCvMSP and LCxMSP) descriptors are proposed.LCvMSP and LCxMSP are concatened into a single vector feature to obtain the multiscale LCCMSP descriptor.A statistical hypothesis testing based method for parameters optimization on several datasets is proposed.The proposed methods demonstrate superior performance to 79 LBP variants and non-LBP methods over 13 texture datasets. Motivated by researching new image texture modeling that improves state-of-the-art LBP variants and non-LBP descriptors, this paper proposes a novel approach for constructing local image descriptors, which are suitable for histogram based image representation. Instead of heuristic code constructions, the proposed approach is based on local concave-and-convex characteristics, which have high ability to extract discriminative and stable texture representation. Different from the majority of descriptors that only encode relationships between the pixels in doublets around central pixel (within 33 neighborhood), the proposed approach encodes relationships between the pixels in triplets by including the central pixel in the modeling. We build two distinct descriptors by dividing local features into two distinct groups, i.e., local concave and convex microstructure patterns (LCvMSP and LCxMSP), according to relationships between the pixels inside the triplets, formed along closed path around the central pixel of a 33-grayscale image patch. To make the descriptors more insensitive to noise and invariant to monotonic gray scale transformation, two supplementary triplets are added in the modeling. These triplets are formed using the central pixel and four virtual pixels set to the median of the grey-scale values of the 33 neighbourhood and the whole image and the average local and global gray levels respectively. The histograms obtained from the single scale descriptors LCvMSP and LCxMSP are concatenated together to build multi-scale histogram feature vector referred to as local concave-and-convex micro-structure pattern (LCCMSP), that is expected to better represent salient local texture structure. We evaluated the effectiveness of the proposed methods on thirteen challenging representative widely-used texture datasets, and found that the proposed LCvMSP, LCxMSP and LCCMSP operators achieve performances that are competitive or better than a large number of recent most promising state-of- the-art LBP variants and non-LBP descriptors. Statistical comparison based on Wilcoxon signed rank test demonstrated that the proposed methods are the top three over all the tested datasets.

Feature engineering is fundamental in applied machine learning. It plays a major role in writer identification of handwritten documents, which has been an active area of research in the literature. In this paper, we propose a conceptually simple, yet high-quality and computationally efficient descriptor referred to as block wise local binary count (BW-LBC) for offline text independent writer identification of handwritten documents. The proposed BW-LBC operator characterizes the writing style of each writer by a set of histograms calculated from all the connected components in the writing. Each histogram is constructed by calculating the occurrence distribution of pixels corresponding to the writing within small blocks in each connected component extracted and cropped from the input handwriting sample (document or set of words/text lines). Specifically, for a given connected component divided into N × N non-overlapping blocks, the appearance probability of writing pixels in the block number i corresponds to the histogram bin number i in the produced corresponding histogram of N × N bins. The samples are classified according to their normalized histogram feature vectors through the nearest-neighbor rule (1-NN) using the Hamming distance. Extensive experiments performed on four challenging handwritten databases (IFN/ENIT, AHTID/MW, CVL and IAM) containing handwritten texts in Arabic and English languages, show that the proposed system using the BW-LBC operator demonstrates superior performance on the Arabic databases (i.e., AHTID/MW and IFN/ENIT) and competitive performance on the English scripts compared to the old and recent state-of-the-art writer identification approaches.

In this paper, we follow the trend of deep learning and make an attempt to investigate the potential benefit of using multispectral images via convolutional neural networks for background subtraction task. The major contributions of this work lie in two aspects, based on the impressive algorithm FgSegNet _ v2. Firstly, we extract three channels out of the seven of the FluxData FD-1665 multispectral dataset to match the number of input channels of the VGG16 deep model. Some combinations of three-channel based multispectral images perform better than RGB images. Secondly, a new convolutional encoder is designed to use all the multispectral channels available to further explore the information of multispectral images. The results outperform the RGB images and also other approaches using the same multispectral dataset.

Automatic facial expression analysis is a challenging topic in computer vision due to its complexity and its important role in many applications such as humancomputer and social interaction. This paper presents a Facial Expression Recognition (FER) method based on an automatic and more efficient facial decomposition into regions of interest (ROI). First, seven ROIs, representing more precisely facial components involved in expression of emotions (left eyebrow, right eyebrow, left eye, right eye, between eyebrows, nose and mouth), are extracted using the positions of some landmarks detected by IntraFace (IF). Then, each ROI is resized and partitioned into blocks which are characterized using several texture and shape descriptors and their combination. Finally, a multiclass SVM classifier is used to classify the six basic facial expressions and the neutral state. In term of evaluation, the proposed automatic facial decomposition is compared with existing ones to show its effectiveness, using three public datasets. The experimental results showed the superiority of our facial decomposition against existing ones and reached recognition rates of 96.06%, 92.03% and 93.34% for the CK, FEED and KDEF datasets, respectively. Then, a comparison with state-of-the-art methods is carried out using CK+ dataset. The comparison analysis demonstrated that our method outperformed or competed the results achieved by the compared methods. An automatic and appropriate facial decomposition with ROIs for FER.Emotion representation using texture, shape-based descriptors and their combination.LTP descriptor analysis with new definitions and strategies for thresholding.Comprehensive evaluation and comparison to state of the art on 4 public databases.Our method outperforms all the compared methods, and is competitive with one method.

We present an unsupervised framework that automatically detects objects of interest in images by formulating the general problem of semantic segmentation as community detection problem in graphs. The framework broadly follows a four-step procedure. First, we perform an over-segmentation of the original image using the well-known statistical region merging algorithm which presents the advantage of not requiring any quantization or colour space transformations. Second, we compute the feature descriptors of the resulting segmented regions. For encoding colour and other textural information, each region is described by an hybrid descriptor based on colour histograms and covariance matrix descriptor. Third, from the set of descriptors we construct different weighted graphs using various graph construction algorithms. Finally, the resulting graphs are then divided into groups or communities using a community detection algorithm based on spectral modularity maximization. This algorithm makes use of the eigenspectrum of matrices such as the graph Laplacian matrix and the modularity matrix which are more likely to reveal the community structure of the graph. Experiments conducted on large orthophotos depicting several zones in the region of Belfort city situated on the north-eastern of France provide promising results. The proposed framework can be used by semi-automatic approaches to handle the challenging problems of scene parsing.

International audience; Facial expression conveys important signs about the human affective state, cognitive activity, intention and personality. In fact, the automatic facial expression recognition systems are getting more interest year after year due to its wide range of applications in several interesting fields such as human computer/robot interaction, medical applications, animation and video gaming. In this study, the authors propose to combine between different descriptors features (histogram of oriented gradients, local phase quantisation and binarised statistical image features) after applying principal component analysis on each of them to recognise the six basic expressions and the neutral face from the static images. Their proposed fusion method has been tested on four popular databases which are: JAFFE, MMI, CASIA and CK+, using two different cross-validation schemes: subject independent and leave-one-subject-out. The obtained results show that their method outperforms both the raw features concatenation and state-of-the-art methods.

Recently, several inductive and flexible nonlinear data projection methods for graph-based semi-supervised learning were proposed. These state-of-the art techniques have a good performance. However, they have not taken into account the relevance of the original features in their model estimation. In this paper, we propose a joint graph-based embedding and feature weighting for getting a flexible and inductive nonlinear data representation on manifolds. The proposed criterion explicitly estimates the feature weights together with the projected data and the linear transformation such that data smoothness and large margins are achieved in the projection space. Moreover, the paper introduces a kernel variant of the model in order to get an inductive nonlinear embedding that is close to a real nonlinear subspace for a good approximation of the embedded data. The proposed frameworks can be seamlessly used in semi-supervised and supervised settings. The resulting optimization problems can be solved efficiently. The proposed embedding methods are evaluated on six public scene and face datasets. Experiments on image classification, in a semi-supervised setting, show that our proposed methods can have a performance that is better than that of many state-of-the-art methods including linear and nonlinear methods.

JIF=2.479; International audience; Automatic age estimation from face images is a topic of growing interest nowadays, because of its great value in various applications. The main challenge in automatic facial age estimation task comes from the large intra-class facial appearance variations due to both gender and race attributes. To this end, in this paper we propose a complete approach for age estimation based on demographic classification. The proposed approach consists of three main parts: (1) Automatic face detection and alignment to extract only the regions of interest. (2) Feature extraction from facial region images using Multi-level face representation. (3) Two-Stages age Estimation (TSE). The main idea of TSE is to classify the input face image into one of demographic classes, then estimate age within the identified demographic class. The experimental results demonstrate that our proposed approach can offer better performance for age estimation when compared to the state-of-the-art methods on MORPH-II, PAL and a subset of LFW databases.

Convolutional Network (ConvNet), with its strong image representation ability, has achieved significant progress in the computer vision and robotic fields. In this paper, we propose a visual localization approach based on place recognition that combines the powerful ConvNet features and localized image sequence matching. The image distance matrix is constructed based on the cosine distance of extracted ConvNet features, and then a sequence search technique is applied on this distance matrix for the final visual recognition. To speed up the computational efficiency, the locality sensitive hashing (LSH) method is applied to achieve real-time performances with minimal accuracy degradation. We present extensive experiments on four real world data sets to evaluate each of the specific challenges in visual recognition. A comprehensive performance comparison of different ConvNet layers (each defining a level of features) considering both appearance and illumination changes is conducted. Compared with the traditional approaches based on hand-crafted features and single image matching, the proposed method shows good performances even in the presence of appearance and illumination changes.

This paper presents an efficient method for the implementation of an image search system invariant to rotation. This method consists of applying circular masks of different size on the image, and extracting the color descriptor from the visible region on the mask, and then combining a texture descriptor for more precision. In our work, we have used the color histogram and histogram of local binary patterns. The experimentation of this approach gave excellent results for color rotation-invariant image retrieval.

Feature extraction is a fundamental process in writer identification that requires efficient methods to characterize and handle local variations of the characters shape and writer individuality. This paper presents a reliable and novel learning-based approach for off-line text-independent writer identification of handwriting. Exploiting the local regions in the writing samples (document or set of word/text line images), we propose a simple, yet adaptive and discriminative feature representation based on handcrafted texture descriptors including Local Binary Patterns (LBP), Local Ternary Patterns (LTP) and Local Phase Quantization (LPQ) to describe characteristic features of the writing style variability. The overall system extracts from the input writing a set of connected components, which seen as texture sub-images where each one of them is subjected to LBP, LPQ or LTP to compute the feature matrix. This matrix is then fed to dimensionality reduction process followed by segmentation into a number of non-overlapping zones, which are afterward subjected to sub-histogram sequence concatenation to form the holistic concatenated component feature histogram. In the identification stage, the writing samples are recognized and classified through their feature histograms using the nearest neighbor 1-NN classifier. Experiments conducted on two handwritten representative databases (AHTID/MW and IAM) indicate that our system performs better than the state-of-the-art systems on the Arabic AHTID/MW database, and show competitive performance on the English IAM database.

This paper presents an efficient handcrafted texture operator for texture modeling and classification. The proposed descriptor, referred to as local directional decoded ternary pattern (LDDTP), consists in encoding both directional pattern features and contrast information in a compact way based on local derivative variations. The proposed operator first computes for each pixel within its $3\times 3$ overlapping square neighborhood, on the one hand, central edge response through the 2nd derivative of Gaussian filter, and on the other hand, eight directional edge responses using the eight Frei-Chen masks to capture more detailed information. Then, spatial relationships among the neighboring pixels through the generated edge responses are exploited independently with the help of the concepts of LTP and LDP operators to enhance the discriminative power. Finally, the produced LDDTP pattern is splitted into two distinct parts: local directional decoded ternary pattern lower ( $LDDTP_{L}$ ) and local directional decoded ternary pattern upper ( $LDDTP_{U}$ ), which are combined into hybrid distributions to form the final LDDTP feature descriptor. Experimental results on eight publicly available texture datasets showed that the proposed LDDTP descriptor achieves classification performance, which is competitive or better than several old and recent state-of-the-art LBP variants.

Feature extraction is fundamental in writer identification to characterize a given handwritten text by a set of discriminative measures, which are used to capture the large variability between handwritten samples. This paper presents an effective learning-based approach for off-line text-independent writer identification using Local Convex Micro-Structure Patterns (LCxMSP) descriptor. The LCxMSP algorithm, which characterizes the writing style of each writer, is applied to a set of connected component sub-images extracted from the handwriting samples (documents or set of words/text lines). Histogram features computed from all the labeled components in all the writing samples are fed to the 1NN (Nearest Neighbor) classifier in order to identify the writer of the tested handwritten samples. Experimental results show that our proposed system combined with the LCxMSP descriptor demonstrate superior performance on the Hybrid-language ICDAR2011 database compared to the state-of-the-art writer identification systems.