Search

Extensive research has been conducted on analyzing human movements, driven by its diverse practical applications such as human–robot interaction, human learning, and clinical diagnosis. However, the current state-of-the-art still encounters scientific challenges when it comes to modeling human movements. There are two key aspects that need to be addressed. Firstly, new models should consider the stochastic nature of human movement and the physical structure of the human body to accurately predict the patterns in full-body motion descriptors over time. Secondly, while deep learning algorithms have been utilized, they lack explainability in terms of predicting body posture sequences, making it essential to improve their comprehensible representation of human movement. This paper aims to tackle these challenges by presenting three innovative methods for creating explainable representations of human movement. The study formulates human body movement as a state-space model based on the Gesture Operational Model (GOM). Model parameters are estimated through either one-shot training employing Kalman Filters or data-intensive training utilizing artificial neural networks. The trained models are utilized for analyzing the dexterity of expert professionals in full-body movements, enabling the identification of dynamic associations between body joints and gesture recognition. Additionally, these models are employed to generate artificial professional movements. • Three innovative methods for explainable human movement models. • Body dexterity analysis: Models reveal how artisans and operators perform tasks. • Minimal motion descriptors to enhance task recognition. • Artificial generation of full-body professional movements. • Extensive experimentation with real-world datasets. [ABSTRACT FROM AUTHOR]

Active learning alleviates labeling costs by selecting and labeling the most informative examples from an unlabeled pool. However, most existing active learning approaches estimate informativeness with uncalibrated confidence, resulting in unreliable informativeness estimation. These approaches generally ignored two significant issues caused by uncalibrated confidence methods. Firstly, the average uncalibrated confidence generated by modern neural networks is usually higher than the accuracy. Secondly, examples located near the decision boundaries are unstable during prediction when the target model updates parameters in the last several epochs, even throughout the training process. This phenomenon, caused by the forgetting characteristic of neural networks, has a significant impact on some specific models that estimate the informativeness by predicted probability vectors or pseudo labels. To address these issues, in this paper, we propose a novel active learning approach to reliably estimate informativeness with calibrated confidence. Specifically, we integrate the intermediate predictions for each unlabeled example, generated by the target model during the training process, to generate calibrated confidence. The calibrated confidence can capture a tendentious label from an indecisive subset of the class space. We show that the calibrated confidence with tendentiousness can maintain the ability of correct predictions. The empirical results demonstrate that our approach outperforms the state-of-the-art active learning methods on image classification tasks. • The active learning algorithm selects unstable examples by calibrated confidence.. • The confidence calibration method only requires training a single model. • Extensive experiments show the competitive performance of the proposed approach. [ABSTRACT FROM AUTHOR]

Highlights • We propose a novel iteration deep learning which can improve the input image iteratively. • We apply the proposed iterative deep learning for document enhancement and binarization in two possible ways: recurrent refinement and stacked refinement. • Our proposed method provides a new, clean version of the degraded image, one that is suitable for visualization and which shows promising results for binarization using Otsu's global threshold. Abstract This paper presents a novel iterative deep learning framework and applies it to document enhancement and binarization. Unlike the traditional methods that predict the binary label of each pixel on the input image, we train the neural network to learn the degradations in document images and produce uniform images of the degraded input images, which in turn allows the network to refine the output iteratively. Two different iterative methods have been studied in this paper: recurrent refinement (RR) that uses the same trained neural network in each iteration for document enhancement and stacked refinement (SR) that uses a stack of different neural networks for iterative output refinement. Given the learned nature of the uniform and enhanced image, the binarization map can be easily obtained through use of a global or local threshold. The experimental results on several public benchmark data sets show that our proposed method provides a new, clean version of the degraded image, one that is suitable for visualization and which shows promising results for binarization using Otsu's global threshold, based on enhanced images learned iteratively by the neural network. [ABSTRACT FROM AUTHOR]

Highlights • We develop a deep local descriptor for cross-modality face recognition, which can learn discriminant information from image patches. • We propose an enumeration loss function to eliminate modality gap on local patch level, which is integrated into a convolutional neural network. • Extensive experiments show that DLFace outperforms existing methods, which demonstrate the effectiveness of our method. Abstract Cross-modality face recognition aims to identify faces across different modalities, such as matching sketches with photos, low resolution face images with high resolution images, and near infrared images with visual lighting images, which is challenging because of the modality gap caused by texture, resolution, and illumination variations. Existing approaches either utilized hand-crafted approaches which ignore inherent data distribution characteristic, or applied deep learning-based algorithms on holistic face images with facial local information ignored. In this paper, we propose a deep local descriptor learning framework for cross-modality face recognition, which aims to learn discriminant and compact local information directly from raw facial patches. A novel cross-modality enumeration loss is proposed to eliminate the modality gap on local patch level, which is then integrated into a convolutional neural networks for deep local descriptor extraction. The proposed deep local descriptor can be easily applied to any traditional face recognition systems, and we use Fisherface as an example in the paper. Extensive experiments on six widely used cross-modality face recognition datasets demonstrate the superiority of proposed method over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • We believe this is the first handwriting recognition paper to achieve state-of-the-art results on both dictionary based and arbitrary symbol based handwriting recognition benchmarks. • Uses fully convolutional neural networks with an elegant series of even tap filters to center align each character found in a word for a very simplistic and effective architecture. • A preprocessing step normalizes input blocks to a canonical representation; which negates the need for costly recurrent symbol alignment correction. • When a dictionary is known, we further introduce a probabilistic character error rate to correct errant word blocks. • Our fully convolutional method uses over 100 symbols such that a single model can decipher letters, numbers, and symbols from multiple Latin-based languages. Abstract The recognition of handwritten text is challenging as there are virtually infinite ways a human can write the same message. Deep learning approaches for handwriting analysis have recently demonstrated breakthrough performance using both lexicon-based architectures and recurrent neural networks. This paper presents a fully convolutional network architecture which outputs arbitrary length symbol streams from handwritten text. A preprocessing step normalizes input blocks to a canonical representation which negates the need for costly recurrent symbol alignment correction. When a lexicon is known, we further introduce a probabilistic character error rate to correct errant word blocks. Our multi-state convolutional method is the first to demonstrate state-of-the-art results on both lexicon-based and arbitrary symbol based handwriting recognition benchmarks. [ABSTRACT FROM AUTHOR]

Highlights • A new attention based CNN-LSTM network is proposed for script identification. • It is based on local and global feature extraction and dynamically weighting them. • Attention used twice to give priority to significant features and to focus on more relevant patches. • The method has been tested on SIW-13, CVSI-2015, ICDAR-17 and MLe2e datasets. Abstract Script identification plays a significant role in analysing documents and videos. In this paper, we focus on the problem of script identification in scene text images and video scripts. Because of low image quality, complex background and similar layout of characters shared by some scripts like Greek, Latin, etc., text recognition in those cases become challenging. In this paper, we propose a novel method that involves extraction of local and global features using CNN-LSTM framework and weighting them dynamically for script identification. First, we convert the images into patches and feed them into a CNN-LSTM framework. Attention-based patch weights are calculated applying softmax layer after LSTM. Next, we do patch-wise multiplication of these weights with corresponding CNN to yield local features. Global features are also extracted from last cell state of LSTM. We employ a fusion technique which dynamically weights the local and global features for an individual patch. Experiments have been done in four public script identification datasets: SIW-13, CVSI2015, ICDAR-17 and MLe2e. The proposed framework achieves superior results in comparison to conventional methods. [ABSTRACT FROM AUTHOR]

Contactless 3D fingerprint identification has gained significant attentions in recent years as it can offer more hygienic, accurate and ubiquitous personal identification. Despite such advantages, contactless 3D imaging often results in partial 3D fingerprints as it requires relatively higher cooperation from users during the contactless 3D imaging. Such contactless 3D fingerprint images significantly degrade matching accuracy due to partial 3D fingerprint imaging. This paper proposes an end-to-end contactless 3D fingerprint representation learning model based on convolutional neural network (CNN). The proposed model includes one fully convolutional network for fingerprint segmentation and three Siamese networks to learn multi-view 3D fingerprint feature representation. Contactless partial 3D fingerprint identification is a more challenging problem due to its high degree of freedom during contactless 3D fingerprint acquisition and is also addressed by using proposed model. We therefore investigate multi-view 3D fingerprint recognition and partial 3D fingerprint using proposed approach. Comparative experimental results, presented in this paper using state-of-the-art 3D fingerprint recognition method, demonstrate the effectiveness of the proposed multi-view approach and illustrate a significant improvement of state-of-the-art 3D fingerprint recognition methods. [ABSTRACT FROM AUTHOR]

• We analyze the limitations of directly applying the high-order multi-hop neighbors to graph neural networks, including computational inefficiency and limited representation ability of the multi-hop neighbor. • We propose a Markov Clustering Regularized Graph Neural Network (MCMGN) to overcome the limitations. By introducing iterative approximation and Regularized Markov Clustering strategies, the proposed model improves the performance of the multi-hop graph neural network for graph-level representation learning. • Extensive experimental results have verified the effectiveness of the proposed Markov Clustering Regularized Graph Neural Network for graph classification on eight graph benchmark datasets. • In this paper, we develop a Markov Clustering Regularized Multi-hop Graph Neural Network (MCMGN) to overcome the two limitations of higher-order multi-hop graph neural network, i.e., computational inefficiency and limited representation ability of multi-hop neighbor. We analyze the reasons of these two limitations in detail and give the corresponding solutions, i.e., iteration approximation and Markov Clustering regularized approaches, to achieve a excellent performance for graph-level representation learning. To evaluate the effectiveness of the proposed model, we carry out a series of experiments including the comparison with existing models, prune threshold comparison, impact of the number of iteration, impact of the normalization of node feature matrix, and computation cost comparison. Extensive experimental results strongly demonstrate the superiority of MCMGN in predictive performance for graph-level representation learning. [Display omitted] Graph Neural Networks (GNNs) have shown great potential for graph data analysis. In this paper, we focus on multi-hop graph neural networks and aim to extend existing models to a high-order multi-hop form for graph-level representation learning. However, such a directly extending method suffers from two limitations, i.e., computational inefficiency and limited representation ability of the multi-hop neighbor. For the former limitation, we utilize an iteration approach to approximate the power of a complex adjacency matrix to achieve linear computational complexity. For the latter limitation, we introduce the Regularized Markov Clustering (R-MCL) to regularize the flow matrix, i.e., the adjacency matrix, in each iteration step. With these two strategies, we construct Markov Clustering Regularized Multi-hop Graph Neural Network (MCMGN) for graph-level representation learning tasks. Specifically, MCMGN consists of a multi-hop message passing phase and a readout phase, where the multi-hop message passing phase aims to learn multi-hop node embedding, and then the readout phase aggregates multi-hop node representations to generate graph embedding for graph-level representation learning tasks. Extensive experiments on eight graph benchmark datasets strongly demonstrate the effectiveness of Markov Clustering Regularized Multi-hop Graph Neural Network, leading to superior performance on graph classification. [ABSTRACT FROM AUTHOR]

• This paper jointly exploits strong and weak augmentations for unsupervised image clustering. • Two levels of contrastive learning are conducted on three streams of weak/strong augmented views. • A novel end-to-end deep clustering approach termed SACC is proposed. • Extensive experimental results confirm the superiority of SACC over the state-of-the-art. Deep clustering has attracted increasing attention in recent years due to its capability of joint representation learning and clustering via deep neural networks. In its latest developments, the contrastive learning has emerged as an effective technique to substantially enhance the deep clustering performance. However, the existing contrastive learning based deep clustering algorithms mostly focus on some carefully-designed augmentations (often with limited transformations to preserve the structure), referred to as weak augmentations, but cannot go beyond the weak augmentations to explore the more opportunities in stronger augmentations (with more aggressive transformations or even severe distortions). In this paper, we present an end-to-end deep clustering approach termed S trongly A ugmented C ontrastive C lustering (SACC), which extends the conventional two-augmentation-view paradigm to multiple views and jointly leverages strong and weak augmentations for strengthened deep clustering. Particularly, we utilize a backbone network with triply-shared weights, where a strongly augmented view and two weakly augmented views are incorporated. Based on the representations produced by the backbone, the weak-weak view pair and the strong-weak view pairs are simultaneously exploited for the instance-level contrastive learning (via an instance projector) and the cluster-level contrastive learning (via a cluster projector), which, together with the backbone, can be jointly optimized in a purely unsupervised manner. Experimental results on five challenging image datasets have shown the superiority of our SACC approach over the state-of-the-art. The code is available at https://github.com/dengxiaozhi/SACC. [ABSTRACT FROM AUTHOR]

This paper presents a deep neural network called DIVA unfolding a baseline adaptive denoising algorithm (DeQuIP), relying on the theory of quantum many-body physics. Furthermore, it is shown that with very slight modifications, this network can be enhanced to solve more challenging image restoration tasks such as image deblurring, super-resolution and inpainting. Despite a compact and interpretable (from a physical perspective) architecture, the proposed deep learning network outperforms several recent algorithms from the literature, designed specifically for each task. The key ingredients of the proposed method are on one hand, its ability to handle non-local image structures through the patch-interaction term and the quantum-based Hamiltonian operator, and, on the other hand, its flexibility to adapt the hyperparameters patch-wisely, due to the training process. • A deep learning model for imaging uses a baseline algorithm and quantum physics theory. • Our network uses a patch-interaction term and a quantum-based Hamiltonian operator to handle non-local image structures. • Adapting hyperparameters patch-wise during training enhances versatility in restoration tasks. • Our model's flexibility improves performance across various image restoration challenges. [ABSTRACT FROM AUTHOR]

Estimating human pose from images is the key to enabling machines to understand human actions. Existing works on human pose estimation mainly focus on designing more resultful deep neural networks to regress the locations of human joints. Although the human pose is obedient to anatomy and shows rich anatomical features, reasoning human body structure by machine in a complex environment is still an open problem. This paper proposes AnatPose which can effectively capture the structural dependency among human body parts by both deep neural network architecture and learning objectives: (1) For the deep neural network architecture, a bidirectional learning paradigm is proposed to learn body-part proportions and dependencies by organizing human body parts as sequential data. This innovation enables the messages to pass in a bidirectional way and makes the human body exchange information about each part deeper during training. (2) For the learning objective, the proposed AnatPose learns a probabilistic representation of multi-scale anatomical features, including keypoint heatmaps, bone heatmaps, and symmetry heatmaps. This innovation enables the multi-scale anatomical features to successfully capture the structural dependency at both low-level joints and high-level associations from the anatomical priors of the human body. Extensive experimental results demonstrate that the proposed AnatPose shows state-of-the-art performance on three challenging datasets. It achieves a PCK@0.2 detection rate of 95.2% on the LSP dataset, a PCKh@0.5 detection rate of 92.9% on the MPII dataset, and an mAP of 76.6% on the Microsoft COCO dataset. Benefiting from its state-of-the-art accuracy, the proposed approach is expected to be widely used in various human pose estimation-driven applications. • AnatPose organizes human body parts as anatomically sequential data. • AnatPose learns the anatomical properties by a bidirectional recurrent neural network. • An anatomically significant objective is designed for optimizing the neural network. • Extensive results show superior performance on human pose estimation. [ABSTRACT FROM AUTHOR]

Gating modules have been widely explored in dynamic network pruning (DNP) to reduce the run-time computational cost of deep neural networks while keeping the features representative. Despite the substantial progress, existing methods remain ignoring the consistency between feature and gate distributions, which may lead to distortion of gated features. In this paper, we propose a feature-gate coupling (FGC) approach aiming to align distributions of features and gates. FGC is a plug-and-play module that consists of two steps carried out in an iterative self-supervised manner. In the first step, FGC utilizes the k -Nearest Neighbor algorithm in the feature space to explore instance neighborhood relationships, which are treated as self-supervisory signals. In the second step, FGC exploits contrastive self-supervised learning (CSL) to regularize gating modules, leading to the alignment of instance neighborhood relationships within the feature and gate spaces. Experimental results validate that the proposed FGC method improves the baseline approach with significant margins, outperforming state-of-the-art methods with a better accuracy-computation trade-off. Code is publicly available at github.com/smn2010/FGC-PR. • We propose the feature-gate coupling (FGC) method for dynamic channel pruning. • FGC alleviates the feature-gate distribution inconsistency using contrastive loss. • Experiments show FGC's effectiveness with significant pruning performance gains. [ABSTRACT FROM AUTHOR]

Recently, deep learning has achieved great success in visual tracking. The goal of this paper is to review the state-of-the-art tracking methods based on deep learning. First, we introduce the background of deep visual tracking, including the fundamental concepts of visual tracking and related deep learning algorithms. Second, we categorize the existing deep-learning-based trackers into three classes according to network structure, network function and network training. For each categorize, we explain its analysis of the network perspective and analyze papers in different categories. Then, we conduct extensive experiments to compare the representative methods on the popular OTB-100, TC-128 and VOT2015 benchmarks. Based on our observations, we conclude that: (1) The usage of the convolutional neural network (CNN) model could significantly improve the tracking performance. (2) The trackers using the convolutional neural network (CNN) model to distinguish the tracked object from its surrounding background could get more accurate results, while using the CNN model for template matching is usually faster. (3) The trackers with deep features perform much better than those with low-level hand-crafted features. (4) Deep features from different convolutional layers have different characteristics and the effective combination of them usually results in a more robust tracker. (5) The deep visual trackers using end-to-end networks usually perform better than the trackers merely using feature extraction networks. (6) For visual tracking, the most suitable network training method is to per-train networks with video information and online fine-tune them with subsequent observations. Finally, we summarize our manuscript and highlight our insights, and point out the further trends for deep visual tracking. [ABSTRACT FROM AUTHOR]

• A dynamic SPECT kidney image database was established by manually labeling clinical renal dynamic images. • A multi-scale spatial perception module was proposed to enhance the semantic information of high-level feature maps. • Decoding modules with attention connection were proposed to refine the details gradually. • RASNet achieved the optimal performance in most of the indicators in automatic kidney segmentation. For the treatment of renal disease, the application of radioactive equipment has become one of the important methods. Accurate segmentation of renal contour plays an important role in clinical diagnosis. However, manual renal contour drawing is not only inefficient but also prone to inaccurate outlining results due to different manual proficiency and fatigue caused by long-term work. There is little research on automatic renal segmentation with renal dynamic imaging. To address this issue, an improved model based on a deep neural network called Renal Automatic Segmentation Network (RASNet) is proposed, to aid in the automatic segmentation of renal contours. Besides, a multi-scale spatial perception module and a decoding module with attention connection are introduced to enrich the semantic information and further improve the accuracy of network segmentation. Extensive experiments were conducted on a renal dynamic medical image database established in this paper. Analysis results show the superiority of the proposed RASNet to several existing segmentation frameworks. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNNs) are vulnerable to invisible perturbations on the images generated by adversarial attacks, which raises researches on the adversarial robustness of DNNs. A series of methods represented by the adversarial training and its variants have proven as one of the most effective techniques in enhancing the DNN robustness. Generally, adversarial training focuses on enriching the training data by involving perturbed data. Such data augmentation effect of the involved perturbed data in adversarial training does not contribute to the robustness of DNN itself and usually suffers from clean accuracy drop. Towards the robustness of DNN itself, we in this paper propose a novel defense that aims at augmenting the model in order to learn features that are adaptive to diverse inputs, including adversarial examples. More specifically, to augment the model, multiple paths are embedded into the network, and an orthogonality constraint is imposed on these paths to guarantee the diversity among them. A margin-maximization loss is then designed to further boost such DIversity via Orthogonality (DIO). In this way, the proposed DIO augments the model and enhances the robustness of DNN itself as the learned features can be corrected by these mutually-orthogonal paths. Extensive empirical results on various data sets, structures and attacks verify the stronger adversarial robustness of the proposed DIO utilizing model augmentation. Besides, DIO can also be flexibly combined with different data augmentation techniques (e.g. , TRADES and DDPM), further promoting robustness gains. • A novel adversarial defense exploring model properties to improve DNNs' robustness • Multiple paths augment DNNs for diverse features adaptive to adversarial inputs • An orthogonality loss and a margin-maximization loss contribute to DNNs' diversity • The proposed method outperforms the non-data-augmented adversarial defenses • Compatibility with the data-augmented techniques, leading to improved robustness [ABSTRACT FROM AUTHOR]

Brain tumors are threatening the life and health of people in the world. Automatic brain tumor segmentation using multiple MR images is challenging in medical image analysis. It is known that accurate segmentation relies on effective feature learning. Existing methods address the multi-modal MR brain tumor segmentation by explicitly learning a shared feature representation. However, these methods fail to capture the relationship between MR modalities and the feature correlation between different target tumor regions. In this paper, I propose a multi-modal brain tumor segmentation network via disentangled representation learning and region-aware contrastive learning. Specifically, a feature fusion module is first designed to learn the valuable multi-modal feature representation. Subsequently, a novel disentangled representation learning is proposed to decouple the fused feature representation into multiple factors corresponding to the target tumor regions. Furthermore, contrastive learning is presented to help the network extract tumor region-related feature representations. Finally, the segmentation results are obtained using the segmentation decoders. Quantitative and qualitative experiments conducted on the public datasets, BraTS 2018 and BraTS 2019, justify the importance of the proposed strategies, and the proposed approach can achieve better performance than other state-of-the-art approaches. In addition, the proposed strategies can be extended to other deep neural networks. • A novel multi-modal brain tumor segmentation network is proposed. • A multi-modal fusion module is introduced to fuse multi-modalities. • Disentangled representation learning is proposed to learn tumor-related features. • Region-aware contrastive learning is designed to learn useful feature representations. • Extensive experiments demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

• This paper systematically analyzes the robustness of elimination-based defense under closed-set and open-set attacks. Experimental results show that the residual perturbations can exhibit attacking effects as strong as full perturbations. • This paper reveals that it is essential to improve the robustness by breaking the correlation between perturbations and legitimate information, together with compressing the attachment space of perturbations. • This paper proposes an efficient defense strategy, Frequency-Adaptive Compression and rEconstruction (FACE), to mitigate attacking effects. • Extensive experimental results demonstrate that the proposed method achieves a 27.9% improvement in robust accuracy on ImageNet over elimination-based defense methods, and reduces TASR and FPR under closed-set and open-set attacks. The increasing use of deep neural networks exposes themselves to adversarial attacks in the real world drawn from closed-set and open-set, which poses great threats to their application in safety-critical systems. Since adversarial attacks tend to mislead an original model by adding small perturbations into clean images, an intuitive idea of defensing adversarial attacks is eliminating perturbations as much as possible to mitigate attacking effects. However, such elimination-based strategies unfortunately fail to achieve satisfactory robustness. Aiming to investigate the intrinsic reasons for this phenomenon, systematic experiments are carried out in this paper to indicate that even a 20% residual perturbation can still preserve and exhibit attacking effects as strong as a full one. Our study also indicates that there are strong correlations between perturbations and legitimate images. Thus, breaking the correlation across multiple bands is more effective in mitigating attacking effects. Based on these findings, this paper proposes an efficient defense strategy called "Frequency-Adaptive Compression and rEconstruction (FACE)" to improve the robustness of the model to adversarial attacks. Specifically, low-frequency bands containing semantic information are compressed by a down-sampling operation, while the channel width of high-frequency bands is squeezed and further compressed by adding noise before the Tanh activating function. Meanwhile, attachment spaces of perturbations are also squeezed to the extent as much as possible. Finally, a clean output is obtained by upsampling together with expanded reconstruction. Experiments are extensively conducted on widely used datasets to demonstrate the effectiveness of the proposed method. For closed-set attacks, FACE outperforms the STOA elimination-based methods on ImageNet, achieving a 27.9% improvement. For the MNIST open-set attacks, it not only reduces the success rate of targeted attack by a large margin (from 100% to 24.7%), but also mitigates attacking effects with an FPR-95 value of 0.3. [ABSTRACT FROM AUTHOR]

• We introduce a general framework to incorporate attribute selection into GNNs to enhance the compacity and robustness of GNNs. • We develop two kinds of GNNs, i.e., AsGCN and AsGAT for attributed graph data representation and learning. • We derive a simple and effective algorithm to optimize the proposed AsGCN and AsGAT. Graph Neural Networks (GNNs) have been widely used for graph learning tasks. The main aspect of GNN's layer-wise message passing is conducting attribute/feature propagation on graph. Most existing GNNs generally conduct feature propagation across all feature dimensions. However, in many real applications, attributes usually contain irrelevant and redundant noise. In this case, attribute/feature selection is desired to extract meaningful features and eliminate noisy ones for GNN's layer-wise propagation. Based on this observation, in this paper, we combine ℓ 2 , 1 / ℓ 1 -norm regularized attribute selection and GNNs together and propose a novel Attribute selection guided GNNs (AsGNNs) for graph data representation. AsGNNs aim to adaptively select some desired meaningful features/attributes that best serve GNNs. Moreover, an effective optimization framework has also been derived to train the proposed AsGNNs. The proposed AsGNNs provide a general framework which can incorporate any GNNs to conduct feature selection for layer-wise propagation. In this paper, we implement AsGNNs on both graph convolutional network (GCN) and graph attention network (GAT) and develop AsGCN and AsGAT for graph learning. Experimental results on several benchmark datasets demonstrate the effectiveness of the proposed AsGNNs (AsGCN, AsGAT) on semi-supervised learning tasks. [ABSTRACT FROM AUTHOR]

We present new achievements on the use of deep convolutional neural networks (CNN) in the problem of pedestrian detection (PD). In this paper, we aim to address the following questions: (i) Given non-deep state-of-the-art pedestrian detectors (e.g. ACF, LDCF), is it possible to improve their top performances? ; (ii) is it possible to apply a pre-trained deep model to these detectors to boost their performances in the PD context? In this paper, we address the aforementioned questions by cascading CNN models (pre-trained on Imagenet) with state-of-the-art non-deep pedestrian detectors. Furthermore, we also show that this strategy is extensible to different segmentation maps (e.g. RGB, gradient, LUV) computed from the same pedestrian bounding box (i.e. the proposal). We demonstrate that the proposed approach is able to boost the detection performance of state-of-the-art non-deep pedestrian detectors. We apply the proposed methodology to address the pedestrian detection problem on the publicly available datasets INRIA and Caltech. [ABSTRACT FROM AUTHOR]

Although deep neural networks (DNNs) have performed well on many perceptual tasks, they are vulnerable to adversarial examples that are generated by adding slight but maliciously crafted perturbations to benign images. Adversarial detection is an important technique for identifying adversarial examples before they are entered into target DNNs. Previous studies that were performed to detect adversarial examples either targeted specific attacks or required expensive computation. Designing a lightweight unsupervised detector is still a challenging problem. In this paper, we propose an A uto E ncoder-based A dversarial E xamples (AEAE) detector that can guard DNN models by detecting adversarial examples with low computation in an unsupervised manner. The AEAE includes only a shallow autoencoder that performs two roles. First, a well-trained autoencoder has learned the manifold of benign examples. This autoencoder can produce a large reconstruction error for adversarial images with large perturbations, so we can detect significantly perturbed adversarial examples based on the reconstruction error. Second, the autoencoder can filter out small noises and change the DNN's prediction on adversarial examples with small perturbations. It helps to detect slightly perturbed adversarial examples based on the prediction distance. To cover these two cases, we utilize the reconstruction error and prediction distance from benign images to construct a two-tuple feature set and train an adversarial detector using the isolation forest algorithm. We show empirically that AEAE is an unsupervised and inexpensive detector against most state-of-the-art attacks. Through the detection in these two cases, there is nowhere to hide adversarial examples. • We observe that adversarial detection is sensitive to the perturbation level. • We train a shallow autoencoder to find two key features from adversarial examples. • We propose a lightweight and unsupervised adversarial detector. [ABSTRACT FROM AUTHOR]

• A deep capsule network model named DCapsNet is proposed to automatically extract the activity or gait features from sensors and classify them. • The proposed model combines a set of convolutional layers and capsule network. • The set of convolutional layers are suitable for processing temporal sequences and providing scalar output features. • These features are then passed through capsule network which is trained by a dynamic routing algorithm to provide probability vectors of the human activity/gait. • The proposed model achieves better performance than the state-of-the-art models for recognizing human activity and gait. Recently, deep neural networks are used to recognize human activity/gait through mobile sensors which have attracted a great attention. Although the existing deep neural networks that perform automatic feature extraction have achieved desirable performance, their classification accuracy needs to be improved. In this paper, a deep neural network that combines a set of convolutional layers and capsule network is proposed. The proposed architecture named DCapsNet is suited to automatically extract the activity or gait features through built in sensors and classify them. The convolutional layers of the DCapsNet are more suitable for processing temporal sequences and provide scalar outputs but not the equivariance. The capsule network (CapsNet) is then trained by a dynamic routing algorithm to capture the equivariance having a magnitude and orientation, which increases the efficiency of the model classification. The performance of the proposed model is evaluated on four public datasets: two HAR datasets (UCI-HAR and WISDM) and two gait datasets (WhuGAIT). The recognition accuracy of the proposed model for the UCI-HAR and WISDM datasets are 97.92 % and 99.30 %, respectively, and for the WhuGAIT Dataset #1 and Dataset #2 are 94.75 % and 97.16 %, respectively. Experimental results show that the proposed model achieves the highest recognition accuracy over the reported results of the state-of-the-art models. [ABSTRACT FROM AUTHOR]

Deep Neural Networks have received considerable attention in recent years. As the complexity of network architecture increases in relation to the task complexity, it becomes harder to manually craft an optimal neural network architecture and train it to convergence. As such, Neural Architecture Search (NAS) is becoming far more prevalent within computer vision research, especially when the construction of efficient, smaller network architectures is becoming an increasingly important area of research, for which NAS is well suited. However, despite their promise, contemporary and end-to-end NAS pipeline require vast computational training resources. In this paper, we present a comprehensive overview of contemporary NAS approaches with respect to image classification, object detection, and image segmentation. We adopt consistent terminology to overcome contradictions common within existing NAS literature. Furthermore, we identify and compare current performance limitations in addition to highlighting directions for future NAS research. [ABSTRACT FROM AUTHOR]

Offline-trained Siamese networks have realized very promising tracking precision and efficiency. However, the performance is still limited by the drawbacks in online update. Traditional strategies cannot tackle the irregular variations of object and the sampling noise, so it is quite risky to adopt them to update Siamese trackers. In this paper, we present a two-stage one-shot learner by exploring the learning scheme of Siamese network, which reveals there are two key issues during online update, i.e., feature fusion and feature comparison. Based on this finding, we propose an updatable Siamese tracker by introducing two independent transformers (SiamTOL). Concretely, a Cross-aware transformer is designed to combine the features of the initial and the dynamic templates, while a Decoder-favored transformer is exploited to compare the fusing template and the search region. By combining these transformers, our tracker is able to adequately model the feature dependencies between multi-frame object samples. Extensive experimental results on several popular benchmarks well manifest that the proposed approach achieves the leading performance, and outperforms other state-of-the-art trackers. • A two-stage one-shot learner is proposed to learn multi-time historic object features. • An updatable Siamese tracker is presented based on the two-stage one-shot learner. • Massive experimental results manifest the superiority of the proposed tracker. [ABSTRACT FROM AUTHOR]

Graph convolution is a fundamental building block for many deep neural networks on graph-structured data. In this paper, we introduce a simple, yet very effective graph convolutional network with skip connections for semi-supervised anomaly detection. The proposed layerwise propagation rule of our model is theoretically motivated by the concept of implicit fairing in geometry processing, and comprises a graph convolution module for aggregating information from immediate node neighbors and a skip connection module for combining layer-wise neighborhood representations. This propagation rule is derived from the iterative solution of the implicit fairing equation via the Jacobi method. In addition to capturing information from distant graph nodes through skip connections between the network's layers, our approach exploits both the graph structure and node features for learning discriminative node representations. These skip connections are integrated by design in our proposed network architecture. The effectiveness of our model is demonstrated through extensive experiments on five benchmark datasets, achieving better or comparable anomaly detection results against strong baseline methods. We also demonstrate through an ablation study that skip connection helps improve the model performance. • We propose a multi-layer graph neural network for semi-supervised anomaly detection. • We introduce a learnable skip-connection module to improve the node representations. • We analyze the complexity of our model and train it using a regularized loss function. • We demonstrate through experiments that our model achieves superior performance. [ABSTRACT FROM AUTHOR]

• The improved word embedding model helps to facilitate adequate-ly representation learning in pre-training. • The improved dropout method can reduce the number of iterations needed for training, accelerate the training of deep neural networks. • We expanded the original non-functional requirements dataset (PROMISE dataset) and designed a new dataset called SOFTWARE NFR. • This model can be applied in the automation of software engineering. Non-functional requirements are property that software products must have in order to meet the user's business requirements, and are additional constraints on the quality and characteristics of software systems. They are generally written by software designers and documented in various parts of requirements documentation. When developing systems, developers need to classify non-functional requirements from requirements documents, and classifying these non-functional requirements requires professional skills, experience, and domain knowledge, which is challenging and time-consuming for developers. It would be beneficial to implement automatic classification of non-functional requirements from requirements documents, which could reduce the manual, time, and mental fatigue involved in identifying specific non-functional requirements from a large number of requirements. In this paper, a deep neural network model called NFRNet is designed to automatically classify non-functional requirements from software requirement documents. The network consists of two parts. One is an improved BERT word embedding model based on N-gram masking for learning context representation of the requirement descriptions, and the other is a Bi-LSTM classification network for capture context information of the requirement descriptions. We use a Softmax classifier in the end to classify the requirement descriptions. At the same time, in order to accelerate the training and improve the generalization ability of the model, the network uses multi-sample dropout regularization technology. This new regularization technology can reduce the number of iterations needed for training, accelerate the training of deep neural networks, and the networks trained achieved lower error rates. In addition, we expanded the original non-functional requirements dataset (PROMISE dataset) and designed a new dataset called SOFTWARE NFR. The new dataset far exceeds the original dataset in terms of the number of requirement description sentences and the number of non-functional requirements categories. It can be taken as a new testbed for non-functional requirements classification. Through cross-validation on the new dataset, the experimental results show that the network designed in this paper is significantly better than the other 17 classification methods in terms of Precision, Recall, and F1-score. At the same time, for the training set and the validation set, using the multi-sample dropout regularization technology can accelerate the training speed, reduce the number of iterations, and achieve lower error rates and loss. [ABSTRACT FROM AUTHOR]

In this paper, we propose a deep neural network (DNN) architecture called Input Output Deep Architecture (IODA) for solving the problem of image labeling. IODA directly links a whole image to a whole label map, assigning a label to each pixel using a single neural network forward step. Instead of designing a handcrafted a priori model on labels (such as an atlas in the medical domain), we propose to automatically learn the dependencies between labels. The originality of IODA is to transpose DNN input pre-training trick to the output space, in order to learn a high level representation of labels. It allows a fast image labeling inside a fully neural network framework, without the need of any preprocessing such as feature designing or output coding. In this paper, IODA is applied on both a toy texture problem and a real-world medical image dataset, showing promising results. We provide an open source implementation of IODA. 1 1 http://mloss.org/software/view/562/ , 2 2 https://github.com/jlerouge/crino [ABSTRACT FROM AUTHOR]

Spiking neural networks are efficient computation models for low-power environments. Spike-based BP algorithms and ANN-to-SNN (ANN2SNN) conversions are successful techniques for SNN training. Nevertheless, the spike-base BP training is slow and requires large memory costs, while ANN2SNN needs many inference steps to obtain good performance. In this paper, we propose an A ctivation C onsistency C oupled A NN- S NN (A C 2 AS) framework to train the SNN in a fast and memory-efficient way. The A C 2 AS consists of two components: (a) a weight-shared architecture between ANN and SNN and (b) spiking mapping units. Firstly, the architecture trains the weight-shared parameters on the ANN branch, resulting in fast training and low memory costs for SNN. Secondly, the spiking mapping units are designed to ensure that the activation values of the ANN are the spiking features. As a result, the activation consistency is guaranteed, and the classification error of the SNN can be optimized by training the ANN branch. Besides, we design an adaptive threshold adjustment (ATA) algorithm to decrease the firing of noisy spikes. Experiment results show that our A C 2 AS-based models perform well on the benchmark datasets (CIFAR10, CIFAR100, and Tiny-ImageNet). Moreover, the A C 2 AS achieves comparable accuracy under 0. 625 × time steps, 0. 377 × training time, 0. 27 × GPU memory costs, and 0. 33 × spike activities of the Spike-based BP model. The code is available at https://github.com/TJXTT/AC2ASNN. • Proposed the A C 2 AS framework for fast and memory efficient SNN training. • Designed an adaptive threshold adjustment algorithm and two spiking mapping units to ensure that the SNN can be equivalently represented by the ANN under the A C 2 AS framework. • The SNNs trained by A C 2 AS achieve considerable performance, sparse spike activity, and fast inference. [ABSTRACT FROM AUTHOR]

Modern artificial intelligence systems rely heavily on deep learning techniques. However, deep neural networks are easily disturbed by adversarial objects. Adversarial examples are beneficial to improve the robustness of the 3D neural network model and enhance the stability of the artificial intelligence system. At present, most 3D adversarial attack methods perturb the entire point cloud to generate adversarial examples, which results in high perturbation costs and low operability in the physical world. In this paper, we propose an adaptive local adversarial attack method (AL-Adv) on 3D point clouds to generate adversarial point clouds. First, we analyze the vulnerability of the 3D network model and extract the salient regions of the input point cloud, namely the vulnerable regions. Second, we propose an adaptive gradient attack algorithm that targets salient regions. The proposed attack algorithm adaptively assigns different disturbances in different directions of the three-dimensional coordinates of the point cloud. Experimental results show that our proposed AL-Adv method achieves a higher attack success rate than the global attack method. Specifically, the adversarial examples generated by AL-Adv demonstrate good imperceptibility and small generation costs. • The proposed AL-Adv method focuses on local regions of point clouds. • We design a novel adaptive gradient attack algorithm for local regions. • The proposed method AL-Adv achieves a higher attack success rate. • The proposed AL-Adv method generates adversarial examples with good imperceptibility. • The proposed AL-Adv method outperforms the global attack method. [ABSTRACT FROM AUTHOR]

• Improving the results of Siamese networks using CP-CVV. • Improving One-Shot-Learning results by class partitioning. • Improving the results of CIFAR-FS with CP-CVV. • Tuning CP-CVV hyper-parameters for optimal results. • CP-CVV as an alternative to traditional ensemble models. One Shot Learning includes all those techniques that make it possible to classify images using a single image per category. One of its possible applications is the identification of food products. For a grocery store, it is interesting to record a single image of each product and be able to recognise it again from other images, such as photos taken by customers. Within deep learning, Siamese neural networks are able to verify whether two images belong to the same category or not. In this paper, a new Siamese network training technique, called CP-CVV, is presented. It uses the combination of different models trained with different classes. The separation of validation classes has been done in such a way that each of the combined models is different in order to avoid overfitting with respect to the validation. Unlike normal training, the test images belong to classes that have not previously been used in training, allowing the model to work on new categories, of which only one image exists. Different backbones have been evaluated in the Siamese composition, but also the integration of multiple models with different backbones. The results show that the model improves on previous works and allows the classification problem to be solved, an additional step towards the use of Siamese networks. To the best of our knowledge, there is no existing work that has proposed integrating Siamese neural networks using a class-based validation set separation technique so as to be better at generalising for unknown classes. Additionally, we have applied Cross-Validation-Voting with ConvNeXt to improve the existing classification results of a well-known Grocery Store Dataset. [ABSTRACT FROM AUTHOR]

• We present a comprehensive review and new perspectives for data-driven single image deraining (SID); • We re-examine the three factors (i.e., data, rain model and network architecture) for SID problem; • We analyze the factors by proposing new and more reasonable criteria (i.e., general vs. specific, synthetical vs. mathematical, black-box vs. white-box); • We also study the relationship of the three factors from new perspectives of data, and reveal two different solving paradigms (explicit vs. implicit) for the SID task; • We profoundly studied one of the three factors, i.e., data, and measured the performance of current methods on different datasets by extensive experiments to reveal the effectiveness of SID data. S ingle I mage D eraining (SID) aims at recovering the rain-free background from an image degraded by rain streaks. For the powerful fitting ability of deep neural networks and massive training data, data-driven deep SID methods have obtained significant improvement over traditional model/prior-based ones. Current studies usually focus on improving the deraining performance by proposing different categories of deraining networks, while neglecting the interpretation of the solving process. As a result, the generalization ability may still be limited in real-world scenarios, and the deraining results also cannot effectively improve the performance of subsequent high-level tasks (e.g., object detection). To explore these issues, we in this paper re-examine the three important factors (i.e., data, rain model and network architecture) for the SID problem, and specifically analyze them by proposing new and more reasonable criteria (i.e., general vs. specific, synthetical vs. mathematical, black-box vs. white-box). We also study the relationship of the three factors from a new perspective of data, and reveal two different solving paradigms (explicit vs. implicit) for the SID task. We further discuss the current mainstream data-driven SID methods from five aspects, i.e., training strategy, network pipeline, domain knowledge, data preprocessing, and objective function, and some useful conclusions are summarized by statistics. Besides, we profoundly studied one of the three factors, i.e., data, and measured the performance of current methods on different datasets through extensive experiments to reveal the effectiveness of SID data. Finally, with the comprehensive review and in-depth analysis, we draw some valuable conclusions and suggestions for future research. [ABSTRACT FROM AUTHOR]

• A theoretical framework for domain generalization (DG) from the perspective of information bottleneck (IB) principle is established. • Based on the framework, a feasible solution by class-wise instance discrimination (CID) combined with maximizing feature entropy (MFE) is proposed to learn the desired representation for DG. • The proposed method achieves excellent performance on multiple datasets without knowing domain labels. • The proposed regularization rule (MFE) improves other invariance-based DG methods consistently. Although deep neural networks have achieved superior performance on many classical tasks, they deteriorate in real applications due to the unpredictable distribution shift. Domain generalization (DG) focuses on improving the generalization ability of the predictive model in unseen domains by training on multiple available source domains. All these domains share the same categories but commonly obey different distributions. In this paper, we establish a new theoretical framework for domain generalization from the perspective of the information bottleneck (IB) principle, which links representation learning in DG with domain-invariant representation learning and maximizing feature entropy (MFE). Based on the theoretical framework, we provide a feasible solution by class-wise instance discrimination combined with inter-dimension decorrelation and intra-dimension uniformity to learn the desired representation for domain generalization, which achieves excellent performance on multiple datasets without knowing domain labels. Extensive experiments show that the proposed regularization rule (MFE) can improve invariance-based DG methods consistently. Moreover, as an extreme case of domain generalization, we also show that MFE is promising to improve adversarial robustness. [ABSTRACT FROM AUTHOR]

• The IVP framework computes probability intervals by categorizing labeled data. • The use of embedding representations allow application of IVP on high-dimensional data. • Collection and use of unlabeled data improves accuracy, calibration and efficiency. • Pseudo-labeling of new unlabeled data has minimal performance overhead in runtime. Learning enabled components are frequently used by autonomous systems and it is common for deep neural networks to be integrated in such systems for their ability to learn complex, non-linear data patterns and make accurate predictions in dynamic environments. However, their large number of parameters and their use as black boxes introduce risks as the confidence in each prediction is unknown and output values like softmax scores are not usually well-calibrated. Different frameworks have been proposed to compute accurate confidence measures along with the predictions but at the same time introduce a number of limitations like execution time overhead or inability to be used with high-dimensional data. In this paper, we use the Inductive Venn Predictors framework for computing probability intervals regarding the correctness of each prediction in real-time. We propose taxonomies based on distance metric learning to compute informative probability intervals in applications involving high-dimensional inputs. By assigning pseudo-labels to unlabeled input data during system deployment we further improve the efficiency of the computed probability intervals. Empirical evaluation on image classification and botnet attacks detection in Internet-of-Things (IoT) applications demonstrates improved accuracy and calibration. The proposed method is computationally efficient, and therefore, can be used in real-time. The code is available at https://github.com/dboursinos/Efficient-Probability-Intervals-Classification-Inductive-Venn-Predictors. [ABSTRACT FROM AUTHOR]

• We propose a new SN P system. Three new types of rules with related evolution of spikes on the hybrid structure of neural-like P systems are designed. • We designed two dynamic regulatory factors on new rules to help guide the optimization of the HN P systems automatically. • A segmentation model for BMs is developed based on the HN P system that integrates the high accuracy of functions in deep learning models in semantic segmentation into P systems with good parallelism and robustness. Neural-like P systems are membrane computing models inspired by natural computing. Spiking neurral (SN) P systems, a kind of neural-like P systems, are viewed as third-generation neural network models. Although real neurons have complex structures, classical SN P systems simplify the structures and corresponding mechanisms to stationary two-dimensional graphs and lack related evolution mechanisms on spikes and channels, which limits the real applications of these models. In this paper, we propose a new hybrid SN P system with evolutionary channels (HN P systems), including three new types of rules for dynamically generating or removing one-one and one-many/many-one channels with related evolutions of spikes on the hybrid neuron structures. Two dynamic regulatory factors are also presented on rules to help guide the optimization of the HN P systems automatically. Based on the new P system, a multiple brain metastases (BMs) segmentation model is developed. The experimental results indicate that the proposed models outperform the state-of-the-art methods on the BMs, which have large variations in sizes, positions and shapes, and low contrast with their surroundings. Performances on the head and neck segmentation dataset also verifies the effectiveness of the HN P system. [ABSTRACT FROM AUTHOR]

• We propose a joint training framework where the knowledge of ANN is consistently transferred to the SNN during training. • We factorize the parameters of both ANN and SNN and restrict them to have the same singular vectors while learning different singular values. • Extensive experiments are conducted on vision benchmarks, demonstrating that our method reaches SOTA. Emerged as a biology-inspired method, Spiking Neural Networks (SNNs) mimic the spiking nature of brain neurons and have received lots of research attention. SNNs deal with binary spikes as their activation and therefore derive extreme energy efficiency on hardware. However, it also leads to an intrinsic obstacle that training SNNs from scratch requires a re-definition of the firing function for computing gradient. Artificial Neural Networks (ANNs), however, are fully differentiable to be trained with gradient descent. In this paper, we propose a joint training framework of ANN and SNN, in which the ANN can guide the SNN's optimization. This joint framework contains two parts: First, the knowledge inside ANN is distilled to SNN by using multiple branches from the networks. Second, we restrict the parameters of ANN and SNN, where they share partial parameters and learn different singular weights. Extensive experiments over several widely used network structures show that our method consistently outperforms many other state-of-the-art training methods. For example, on the CIFAR100 classification task, the spiking ResNet-18 model trained by our method can reach to 77.39 % top-1 accuracy with only 4 time steps. [ABSTRACT FROM AUTHOR]

Abstract: A particle swarm optimization based simultaneous learning framework for clustering and classification (PSOSLCC) is proposed in this paper. Firstly, an improved particle swarm optimization (PSO) is used to partition the training samples, the number of clusters must be given in advance, an automatic clustering algorithm rather than the trial and error is adopted to find the proper number of clusters, and a set of clustering centers is obtained to form classification mechanism. Secondly, in order to exploit more useful local information and get a better optimizing result, a global factor is introduced to the update strategy update strategy of particle in PSO. PSOSLCC has been extensively compared with fuzzy relational classifier (FRC), vector quantization and learning vector quantization (VQ+LVQ3), and radial basis function neural network (RBFNN), a simultaneous learning framework for clustering and classification (SCC) over several real-life datasets, the experimental results indicate that the proposed algorithm not only greatly reduces the time complexity, but also obtains better classification accuracy for most datasets used in this paper. Moreover, PSOSLCC is applied to a real world application, namely texture image segmentation with a good performance obtained, which shows that the proposed algorithm has a potential of classifying the problems with large scale. [Copyright &y& Elsevier]

• A directed convolutional neural network model is proposed • Partial dilation-Global erosion generates resistance against information loss • Several recent approaches for the segmentation of graphemes/characters are discussed • The over-traces issue is addressed for the first time in the literature • MMOs can provide real solutions if a convenient strategy is adopted Due to the nature of Arabic handwriting, segmenting words into characters/graphemes is the most difficult and critical task of the recognition system. The present paper proposes an approach to segment handwritten Arabic words into graphemes based on a directed Convolutional Neural Network (CNN) and Mathematical Morphology Operations (MMO). Arabic script is cursive, which means that almost all graphemes are connected via horizontal links; therefore, a technique to remove links will facilitate the segmentation of graphemes. In general, an MMO such as erosion seems suitable for getting the job done, but since Arabic handwriting is difficult, MMOs cause information loss and suffer from many issues such as diacritics and over-traces, which lead to over/under/bad segmentations. To overcome limitations, the present paper addresses these issues in the following order: the over-traces issue is addressed for the first time in the literature; a robust algorithm for diacritics extraction is provided; and finally, the main segmentation algorithm adopts a strategy based on a Partial Dilation (PD)-Global Erosion (GE) technique to combat the information loss issue. The PD phase amplifies important regions, while GE eliminates links between graphemes. The complementarity between PD and GE facilitates the extraction of graphemes and creates resistance against information loss. To properly tackle these difficult problems, this article exploits the robustness of CNNs, so a new directed CNN model is suggested. The idea is to draw the model's attention to certain targeted features, which are selected according to the nature of the problem addressed. The proposed directed CNN is used in all phases of the segmentation process. The experimental results are very encouraging and show that the proposed directed CNN model outperformed basic CNN in many experiments. The results also reveal that the followed strategy improved the ability of MMOs to perform segmentation and to compete with other approaches in this research area. [ABSTRACT FROM AUTHOR]

Abstract: Incremental learning of neural networks has attracted much interest in recent years due to its wide applicability to large scale data sets and to distributed learning scenarios. Moreover, nonstationary learning paradigms have also emerged as a subarea of study in Machine Learning literature due to the problems of classical methods when dealing with data set shifts. In this paper we present an algorithm to train single layer neural networks with nonlinear output functions that take into account incremental, nonstationary and distributed learning scenarios. Moreover, it is demonstrated that introducing a regularization term into the proposed model is equivalent to choosing a particular initialization for the devised training algorithm, which may be suitable for real time systems that have to work under noisy conditions. In addition, the algorithm includes some previous models as special cases and can be used as a block component to build more complex models such as multilayer perceptrons, extending the capacity of these models to incremental, nonstationary and distributed learning paradigms. In this paper, the proposed algorithm is tested with standard data sets and compared with previous approaches, demonstrating its higher accuracy. [Copyright &y& Elsevier]

• Two new nonparametric methods are proposed to extend the application scope of the conventional Dirichlet process mixture model. • A deep learning based nonparametric algorithm is proposed for the image generation and clustering task, in which the cluster number will not be pre-specified but estimated by model. • A new deep neural network architecture named Graph-VAE has been presented for the manifold and posterior probability approximation. • A mean field variational inference algorithm is derived, which incorporates the learning procedure with the Graph-VAE to handle the non-conjugated parameter estimation. For multi-manifold clustering, it is still a challenging problem on how to learn the cluster number automatically from data. This paper presents a novel nonparametric Bayesian model to cluster the multi-manifold data and estimate the number of submanifolds simultaneously. Our model firstly assumes that every submanifold is a probability distribution defined in the manifold space. Then, we approximate the manifold distribution with a deep neural network. To maintain the data similarity among data, we regularize the data generation process with a modified k -nearest neighbor graph. Though the posterior inference is hard, our model leads to a very efficient deterministic optimization algorithm, which incorporates the mean field variational inference with the Graph regularized Variational Auto-Encoder (Graph-VAE). By applying the Graph-VAE, our model exhibits another advantage of realistic image generation which overcomes the conventional clustering methods. Furthermore, we expand our proposed manifold algorithm with the Dirichlet Process Mixture (DPM) to model the real datasets, in which the manifold data and non-manifold data are coexisting. Experiments on synthetic data verify our theoretical analysis. Clustering results on motion segmentation, coil20 and 3D pedestrian show that our approach can significantly improve the clustering accuracy. The handwritten database experiment demonstrates the image generation capability. [ABSTRACT FROM AUTHOR]

Highlights • A deep-based model for solving facial expression recognition is proposed. • Propose a Salient Expression Region Descriptor (SERD) to locate expression-sensitive regions and refine deep features. • Construct an encoder-decoder framework to disentangle multiple variations (e.g. ages, races, poses, etc.) from features. • Experiments on six famous expression databases verify the effectiveness of the proposed model. Abstract Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. In this paper, we present a novel model, named Deep Attentive Multi-path Convolutional Neural Network (DAM-CNN), for FER. Different from most existing models, DAM-CNN can automatically locate expression-related regions in an expressional image and yield a robust image representation for FER. The proposed model contains two novel modules: an attention-based Salient Expressional Region Descriptor (SERD) and the Multi-Path Variation-Suppressing Network (MPVS-Net). SERD can adaptively estimate the importance of different image regions for FER task, while MPVS-Net disentangles expressional information from irrelevant variations. By jointly combining SERD and MPVS-Net, DAM-CNN is able to highlight expression-relevant features and generate a variation-robust representation for expression classification. Extensive experimental results on both constrained datasets (CK+, JAFFE, TFEID) and unconstrained datasets (SFEW, FER2013, BAUM-2i) demonstrate the effectiveness of our DAM-CNN model. [ABSTRACT FROM AUTHOR]

Highlights • A weighted similarity matrix minimizing the disparity between the similarities in the original space and the similarities in the hashing space. • Hashing function obtained by multilayer neural network and optimized by an efficient way. • Quantization errors between the input feature and the learned compact binary codes reduced by the proposal. Abstract Learning based hashing technologies have been widely adopted in multimedia retrieval as they could afford efficient storage and extract semantic information for high-dimensional data. However, the major difficulty of learning based hashing is the discrete constraint imposed on the required hashing codes, which makes the optimization generally to be NP-hard. In this paper, a novel supervised learning based discrete hashing (SLDH) approach is proposed to learn compact binary codes under the deep learning framework for image retrieval. We adopt multilayer network to convert the original features into binary codes, while it should exploit the semantic relevance of manual labels and keep the semantic similarity. For this purpose, we propose the objective function to obtain the binary codes including: 1) making full use of manual labels to get implicit semantic information; 2) using the weighted similarity matrix to keep the semantic similarity; 3) relaxing the discrete constraint to a normalized optimization problem; 4) adding the orthogonality constraint on binary codes to reduce the information redundancy. The objective function is optimized with the alternating direction method and modified alternating direction of multipliers(ADMM) algorithm with efficient iteration. Experiments are conducted on three databases and the results demonstrate the superiority to several state-of-the-art hashing based image retrieval methods. [ABSTRACT FROM AUTHOR]

Highlights • A multi-model, multi-size and multi-view deep model for Brain MR Image Segmentation. • Using both 'deep' (U-Net) and 'shallow' (BPNN) models to segment Brain MR Images. • Using multi-size patches extracted on three view planes as the input of our model. • Using CAE for image restoration and using an atlas to include brain anatomy into U-Nets. • Outperforms widely used segmentation methods on simulated and clinical MR images. Graphical abstract Abstract Segmentation of the brain into gray matter, white matter, and cerebrospinal fluid (CSF) using magnetic resonance (MR) imaging plays a fundamental role in neuroimaging research and clinical settings. Due to the complexity of brain anatomy, low image quality, and insufficient training data, both traditional and deep learning segmentation methods have a limited performance. In this paper, we propose a multi-model, multi-size and multi-view deep neural network (M3Net) for brain MR image segmentation, which uses three identical modules to segment transaxial, coronal, and sagittal MR slices, respectively. Each module consists of multi-size U-Nets and multi-size back propagation neural networks. It also uses a probabilistic atlas to explore brain anatomy and a convolutional auto-encoder (CAE) to restore MR images. The proposed M3Net model was evaluated against widely used segmentation methods on both synthetic and clinical studies. Our results suggest that the proposed model is able to segment Brain MR Images more accurately. [ABSTRACT FROM AUTHOR]

Highlights • We propose a novel temporal pooling approach to aggregate the frame-level features of a video, which explores the importance of incorporating the temporal order information. • We propose to treat the temporal evolution of the feature value at each feature dimension as a 1D signal and learn a unique convolutional filter bank for each 1D signal. • The proposed pooling method achieves promising action recognition performance while maintaining a tractable amount of model parameters. Abstract Most video based action recognition approaches create the video-level representation by temporally pooling the features extracted at every frame. The pooling methods they adopt, however, usually completely or partially ignore the dynamic information contained in the temporal domain, which may undermine the discriminative power of the resulting video representation since the video sequence order could unveil the evolution of a specific event or action. To overcome this drawback and explore the importance of incorporating the temporal order information, in this paper we propose a novel temporal pooling approach to aggregate the frame-level features. Inspired by the capacity of Convolutional Neural Networks (CNN) in making use of the internal structure of images for information abstraction, we propose to apply the temporal convolution operation to the frame-level representations to extract the dynamic information. However, directly implementing this idea on the original high-dimensional feature will result in parameter explosion. To handle this issue, we propose to treat the temporal evolution of the feature value at each feature dimension as a 1D signal and learn a unique convolutional filter bank for each 1D signal. By conducting experiments on three challenging video-based action recognition datasets, HMDB51, UCF101, and Hollywood2, we demonstrate that the proposed method is superior to the conventional pooling methods. [ABSTRACT FROM AUTHOR]

Highlights • We propose an attention-based pooling network for BIQA. • A correlation constraint between the estimated local quality and attention weight in the network is introduced to regulate the training. • An effective and interpretable attention model can be learned merely from subjective quality scores, without accessing to attention/saliency supervision. • The learned attention-based pooling can be applied to other IQA metrics to help improve the performance. Abstract Many recent algorithms based on convolutional neural network (CNN) for blind image quality assessment (BIQA) share a common two-stage structure, i.e. , local quality measurement followed by global pooling. In this paper, we mainly focus on the pooling stage and propose an attention-based pooling network (APNet) for BIQA. The core idea is to introduce a learnable pooling that can model human visual attention in a data-driven manner. Specifically, the APNet is built by incorporating an attention module and allows for a joint learning of local quality and local weights. It can automatically learn to assign visual weights while generating quality estimations. Moreover, we further introduce a correlation constraint between the estimated local quality and attention weight in the network to regulate the training. The constraint penalizes the case in which the local quality estimation on a region attracting more attention differs a lot from the overall quality score. Experimental results on benchmark databases demonstrate that our APNet achieves state-of-the-art prediction accuracy. By yielding an attention weight map as by-product, our model gives a better interpretability on the learned pooling. [ABSTRACT FROM AUTHOR]

Highlights • We design a new locality-aware coding layer for CNNs. • Our model can be implemented under an end-to-end learning framework. • A powerful descriptor is obtained for texture classification. • Our approach notably outperforms the state-of-the-art methods. Abstract Recent studies have demonstrated advantages of the representations learned by Convolutional Neural Networks (CNNs) in providing an appealing paradigm for visual classification tasks. Most existing methods adopt activations from the last fully connected layer as the image representation. This paper advocates exploiting appropriately convolutional layer activations to constitute a powerful descriptor for texture classification under an end-to-end learning framework. The main component of our method is a new locality-aware coding layer conducted with the locality constraint, where the dictionary and the encoding representation are learned simultaneously. The layer is readily amenable to training via the backpropagation as the locality-aware coding process has an analytical solution. It is capable of capturing class-specific information which makes the learned convolutional features more robust. The resulting representation is particularly useful for texture classification. Comprehensive experiments on the DTD, FMD and KTH-T2b datasets show that our approach notably outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • We defined the sub-filters of a filter and visualized them to verify these sub-filters can recognize multiple meaningful patterns. • Filter-in-Filter was proposed to make full use of the sub-filters to enhance the expressibility of the filters in CNNs. • Filter-in-Filter does not increase the number of parameters and increases the computational cost only slightly as compared to the standard convolution. We verified that FIF is effective to improve the performance of CNNs by conducting extensive experiments. Abstract Increasing the number of parameters seems to have improved convolutional neural networks, e.g. increasing the depth or width of the networks. In this paper, we propose a scheme to improve CNNs by deriving the six sub-filters from a filter, which share parameters among them and enhance the expressibility of the filter. We first defined the sub-filters of a filter, and by visualizing a well-trained CNN, we verified that these sub-filters could recognize multiple meaningful patterns with different visual characteristics, even when the filter containing them was not activated. These findings revealed that the filter has the potential to recognize multiple patterns. Inspired by these findings, we proposed the filter-in-filter (FIF) scheme to enhance the expressibility of a filter, by making full use of its sub-filters to recognize multiple meaningful sub-patterns. We verified the effectiveness of FIF on three image classification benchmark datasets, namely Tiny ImageNet, CIFAR-100 and ImageNet. Our experimental results showed that our models achieved consistent improvement over the base CNNs on the benchmark datasets, e.g. AlexNet and VGG16 using FIF achieved approximately 1% improvement on ImageNet. The sub-filters share the parameters and most of the computational cost with the filter containing them; therefore, FIF does not increase the number of parameters and increases the computational cost only slightly. [ABSTRACT FROM AUTHOR]

Highlights • Orderless pooling can maintain spatial invariance in local information aggregation for indoor scene classification. • Intra-modality Attentive Pooling mines and pools discriminative local semantic cues in each modality. • Cross-modality Attentive Pooling learns to attend on different modalities in terms of different local cues to fuse the selected discriminative semantic cues across modalities. • The attention weights in the model are interpretable for understanding both scene classification and RGB-D fusion. • State-of-the-art results are achieved on both challenging SUN RGB-D Dataset and NYU Depth V2 Dataset. Abstract RGB-D indoor scene classification is an essential and challenging task. Although convolutional neural network (CNN) achieves excellent results on RGB-D object recognition, it has several limitations when extended towards RGB-D indoor scene classification. 1) The semantic cues such as objects of the indoor scene have high spatial variabilities. The spatially rigid global representation from CNN is suboptimal. 2) The cluttered indoor scene has lots of redundant and noisy semantic cues; thus discerning discriminative information among them should not be ignored. 3) Directly concatenating or summing global RGB and Depth information as presented in popular methods cannot fully exploit the complementarity between two modalities for complicated indoor scenarios. To address the above problems, we propose a novel unified framework named Multi-modal Attentive Pooling Network (MAPNet) in this paper. Two orderless attentive pooling blocks are constructed in MAPNet to aggregate semantic cues within and between modalities meanwhile maintain the spatial invariance. The Intra-modality Attentive Pooling (IAP) block aims to mine and pool discriminative semantic cues in each modality. The Cross-modality Attentive Pooling (CAP) block is extended to learn different contributions across two modalities, which further guides the pooling of the selected discriminative semantic cues of each modality. We further show that the proposed model is interpretable, which helps to understand mechanisms of both scene classification and multi-modal fusion in MAPNet. Extensive experiments and analysis on SUN RGB-D Dataset and NYU Depth Dataset V2 show the superiority of MAPNet over current state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Highlights • A simple but effective coherent dropout is proposed to improve the ability of preventing overfitting in VQA model. • A siamese dropout mechanism is proposed to explicitly decrease the output variance of VQA model during training. • We further develop a deeper and wider encoding module called Multi-path Stacked Residual RNNs to enhance the representation ability of question encoder. • The proposed methods can bring clear improvements to the state-of-the-art VQA models on VQA-v1 and VQA-v2 datasets. Abstract Using dropout in Visual Question Answering (VQA) is a common practice to prevent overfitting. However, the current way to use dropout in multi-path networks may cause two problems: the co-adaptations of neurons and the explosion of output variance. In this paper, we propose coherent dropout and siamese dropout mechanism to solve the two problems, respectively. Specifically, in coherent dropout, the relevant dropout layers in multiple paths are forced to work coherently to maximize the ability of preventing neuron co-adaptations. We show that the coherent dropout is simple in implementation but very effective to overcome overfitting. As for the explosion of output variance, we develop a siamese dropout mechanism to explicitly minimize the difference between the two output vectors produced from the same input data during training phase. Such mechanism can reduce the gap between training and inference phases and make the VQA model more robust. With the help of the two techniques, we further design an enhanced question encoder called Multi-path Stacked Residual RNNs which is deeper and wider and more powerful than current shallow question encoder. Extensive experiments are conducted to verify the effectiveness of coherent dropout, siamese dropout and the enhanced question encoder. And the results show that our methods can bring clear improvements to the state-of-the-art VQA models on VQA-v1 and VQA-v2 datasets. [ABSTRACT FROM AUTHOR]

Highlights • To our knowledge, DS-CAE is the first convolution-based deep learning method for unsupervised network representation learning.Similar to the convolution for image processing, a subgraph-based convolution operation is proposed to scan a tree which is extracted from various graph structures. • In contrast to most existing models for unsupervised learning on graph-structured data, DS-CAE can capture highly non-linear network structure by simultaneously integrating raw node information and network structure into network representation learning. • Due to the deep representation, DS-CAE can map graphs to highly nonlinear deeply learned spaces to effectively preserve both the local and global information in original space. Abstract Network representation learning (NRL) aims to map vertices of a network into a low-dimensional space which preserves the network structure and its inherent properties. Most existing methods for network representation adopt shallow models which have relatively limited capacity to capture highly non-linear network structures, resulting in sub-optimal network representations. Therefore, it is nontrivial to explore how to effectively capture highly non-linear network structure and preserve the global and local structure in NRL. To solve this problem, in this paper we propose a new graph convolutional autoencoder architecture based on a depth-based representation of graph structure, referred to as the depth-based subgraph convolutional autoencoder (DS-CAE), which integrates both the global topological and local connectivity structures within a graph. Our idea is to first decompose a graph into a family of K -layer expansion subgraphs rooted at each vertex aimed at better capturing long-range vertex inter-dependencies. Then a set of convolution filters slide over the entire sets of subgraphs of a vertex to extract the local structural connectivity information. This is analogous to the standard convolution operation on grid data. In contrast to most existing models for unsupervised learning on graph-structured data, our model can capture highly non-linear structure by simultaneously integrating node features and network structure into network representation learning. This significantly improves the predictive performance on a number of benchmark datasets. [ABSTRACT FROM AUTHOR]

Highlights • We further develop the unravelled view of ResNets, which helps us better understand their behaviours. We demonstrate this in the context of a training process, which is the key difference from the original version 1. • We propose a group of relatively shallow convolutional networks based on our new understanding. Some of them perform comparably with the state-of-the-art approaches on the ImageNet classification dataset 2. • We evaluate the impact of using different networks on the performance of semantic image segmentation, and show these networks, as pre-trained features, can boost existing algorithms a lot. Abstract The community has been going deeper and deeper in designing one cutting edge network after another, yet some works are there suggesting that we may have gone too far in this dimension. Some researchers unravelled a residual network into an exponentially wider one, and assorted the success of residual networks to fusing a large amount of relatively shallow models. Since some of their early claims are still not settled, we in this paper dig more on this topic, i.e., the unravelled view of residual networks. Based on that, we try to find a good compromise between the depth and width. Afterwards, we walk through a typical pipeline of developing a deep-learning-based algorithm. We start from a group of relatively shallow networks, which perform as well or even better than the current (much deeper) state-of-the-art models on the ImageNet classification dataset. Then, we initialize fully convolutional networks (FCNs) using our pre-trained models, and tune them for semantic image segmentation. Results show that the proposed networks, as pre-trained features, can boost existing methods a lot. Even without exhausting the sophistical techniques to improve the classic FCN model, we achieve comparable results with the best performers on four widely-used datasets, i.e., Cityscapes, PASCAL VOC, ADE20k and PASCAL-Context. The code and pre-trained models are released for public access 1 1 https://github.com/itijyou/ademxapp. [ABSTRACT FROM AUTHOR]

Highlights • A new model for collaborative filtering in context-aware recommendation, integrating joint interaction model with contextual operating tensor. • One layer to capture user/item interactions via a tensor and another layer to capture contextual information via another tensor. • Extensive experiments on four datasets and novel studies on varying contextual influence and time split recommendation. Abstract In recommender systems, the classical matrix factorization model for collaborative filtering only considers joint interactions between users and items. In contrast, context-aware recommender systems (CARS) use contexts to improve recommendation performance. Some early CARS models treat user, item and context equally, unable to capture contextual impact accurately. More recent models perform context operations on users and items separately, leading to "double-counting" of contextual information. This paper proposes a new model, Joint Interaction with Context Operation (JICO), to integrate the joint interaction model with the context operation model, via two layers. The joint interaction layer models interactions between users and items via an interaction tensor. The context operation layer captures contextual information via a contextual operating tensor. We evaluate JICO on four datasets and conduct novel studies, including varying contextual influence and time split recommendation. JICO consistently outperforms competing methods, while providing many useful insights to assist further analysis. [ABSTRACT FROM AUTHOR]