Search

This paper concerns a new incremental approach to mining data streams. It is known that patterns in a data stream may evolve over time. In many cases, we need to track and analyze the nature of these changes. In the paper, the probabilistic neural networks are considered as basic models for tracking changes in data streams. We present globally convergent stream data mining algorithms applied to problems of regression, classification, and density estimation in a time-varying (drifting) environment. The algorithms are derived from the Parzen kernel-based probabilistic neural networks working in the online mode. For each problem, a theorem is presented ensuring the L 2 convergence of the algorithm designed for tracking drifting regression, density, or discriminant functions. Illustrative examples explain in detail how to choose the bandwidth of the Parzen kernel and the learning rate of the online algorithm. The performance of all algorithms is shown in exemplary simulations. It should be noted that this paper is one of very few, in the existing literature, presenting mathematically justified stream data mining algorithms. • The incremental version of the Generalized Regression Neural Network (IGRNN) able to track drifting regression functions. • The incremental version of the Probabilistic Neural Network (IPNN) working in non-stationary environments. • Application of IPNN for tracking drifting discriminant functions. • Mathematical proofs of the L 2 convergence of all the proposed estimators. [ABSTRACT FROM AUTHOR]

Artificial neural networks, celebrated for their human-like cognitive learning abilities, often encounter the well-known catastrophic forgetting (CF) problem, where the neural networks lose the proficiency in previously acquired knowledge. Despite numerous efforts to mitigate CF, it remains the significant challenge particularly in complex changing environments. This challenge is even more pronounced in cross-domain adaptation following the continual learning (CL) setting, which is a more challenging and realistic scenario that is under-explored. To this end, this article proposes a cross-domain CL approach making possible to deploy a single model in such environments without additional labelling costs. Our approach, namely continual learning approach for many processes (CLAMP), integrates a class-aware adversarial domain adaptation strategy to align a source domain and a target domain. An assessor-guided learning process is put forward to navigate the learning process of a base model assigning a set of weights to every sample controlling the influence of every sample and the interactions of each loss function in such a way to balance the stability and plasticity dilemma thus preventing the CF problem. The first assessor focuses on the negative transfer problem rejecting irrelevant samples of the source domain while the second assessor prevents noisy pseudo labels of the target domain. Both assessors are trained in the meta-learning approach using random transformation techniques and similar samples of the source domain. Theoretical analysis and extensive numerical validations demonstrate that CLAMP significantly outperforms established baseline algorithms across all experiments by at least 10% margin. • This paper presents a scarcely addressed problem, cross-domain continual learning. • This paper proposes a new algorithm, continual learning approach for many processes (CLAMP), for cross-domain continual learning. • Extensive numerical validations and theoretical studies are performed to guarantee the advantage of CLAMP. [ABSTRACT FROM AUTHOR]

• Local and global self-attention mechanisms are used for character embedding. • CNN with multi-size filters are used to extract character information for NER. • A cross-attention method that fuses character and word embedding for NER is proposed • A modified Mogrifier LSTM is presented to improve the performance of NER. • Proposed methods integrated with a transformer-based model achieve good performance. Clinical and biomedical concept extraction is critical in medical analysis using clinical and biomedical documents from professional literature, EHRs and PHRs. Named entity recognition (NER) accurately marks essential information in the literature based on the characteristics of the target entity, providing a method for extracting clinical and biomedical concepts. The performance of NER is heavily embedding-dependent, so recent studies have proposed the method of generating word embedding from character-level information, which can strengthen the representation ability for word embedding. In this paper, we present a novel neural network model including an attention mechanism network and a convolutional neural network (CNN) to further improve character-level embedding. First, an attention mechanism is applied simultaneously to the local and global character embedding. Then, a CNN with multi-size filters is used to extract more information from the character level, which can capture more meaningful features from words with various lengths. In addition, a cross-attention method is used to leverage the interaction between word embedding and character embedding to generate the final word representation. Finally, we modified Mogrifier LSTM to make it suitable for NER tasks and integrated it into our model. Experimental results show that our method is effective and that the model performs better than the baseline models. We also apply our methods proposed in this paper to the transformer-based model and obtain a 90.36 F1-score on NCBI-Disease. [ABSTRACT FROM AUTHOR]

Multimodal fusion, a machine learning technique, significantly enhances decision-making by leveraging complementary information extracted from different data modalities. The success of multimodal fusion relies heavily on the design of the fusion scheme. However, this process traditionally depends on manual expertise and exhaustive trials. To tackle this challenge, researchers have undertaken studies on DARTS-based Neural Architecture Search (NAS) variants to automate the search of fusion schemes. In this paper, we present theoretical and empirical evidence that highlights the presence of catastrophic search bias in DARTS-based multimodal fusion methods. This bias traps the search into a deceptive optimal childnet, rendering the entire search process ineffective. To circumvent this phenomenon, we introduce a novel NAS framework for multimodal fusion, featuring a robust search strategy and a meticulously designed multi-scale fusion search space. Significantly, the proposed framework is capable of capturing modality-specific information across multiple scales while achieving an automatic balance between intra-modal and inter-modal information. We conduct extensive experiments on three commonly used multimodal classification tasks from different domains and compare the proposed framework against state-of-the-art approaches. The experimental results demonstrate the superior robustness and high efficiency of the proposed framework. • This paper presents the first theoretical and empirical evidence demonstrating that DARTS suffers from catastrophic search bias in multimodal fusion, rendering the entire search ineffective. We term this phenomenon as the "Matthew Effect" and give a profound analysis. • A novel NAS framework for multimodal fusion is proposed, wihch features a robust search strategy and a multi-scale fusion search space. The framework leverages the single-path one-shot NAS algorithm as the search strategy, fully circumventing the occurrence of the "Matthew Effect". • The extensive experimental results effectively showcase the exceptional robustness and high efficiency of the proposed framework. [ABSTRACT FROM AUTHOR]

This paper develops a novel insight and procedure that includes a variety of algorithms for finding the best solution in a structured multi-agent system with internal communications and a global purpose. In other words, it finds the optimal communication structure among agents and the optimal policy in this structure. First, a unique reinforcement learning algorithm is proposed to find the optimal policy of each agent in a fixed structure with non-linear function approximators like artificial neural networks (ANN) and with eligibility traces. Secondly, a mechanism is presented to perform self-organization based on the information of the learned policy. Finally, an algorithm that can discover an appropriate inter-structure mapping and then can transfer the previous knowledge to the new structure is developed, which increases the speed of the learning in this new environment after self-organization. This paper is one of the first works that analyzes the problem fully theoretically and devises some algorithms to find the best solution. We use a simplified version of the distributed task allocation problem (DTAP) as our case study. The experimental results verify the stability of our approach and show the high speed of finding the optimal solution as a result of using transfer learning. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) are vulnerable to adversarial examples generated by adding subtle perturbations to benign inputs. While these perturbations are somewhat small due to the L p norm constraint, they are still easily spotted by human eyes. This paper proposes Perceptual Sensitive Attack (PS Attack) to address this flaw with a perceptually adaptive scheme. We add Just Noticeable Difference (JND) as prior information into adversarial attacks, making image changes in areas that are insensitive to the human eyes. By integrating the JND matrix into the L p norm, PS Attack projects perturbations onto the JND space around clean data, resulting in more imperceivable adversarial perturbations. PS Attack also mitigates the trade-off between the imperceptibility and transferability of adversarial images by adjusting a visual coefficient. Extensive experiments manifest that combining PS attacks with state-of-the-art black-box approaches can significantly promote the naturalness of adversarial examples while maintaining their attack ability. Compared to the state-of-the-art transferable attacks, our attacks reduce LPIPS by 8% on average when attacking typically-trained and defense models. [ABSTRACT FROM AUTHOR]

Recent researches show that deep learning model is susceptible to backdoor attacks. Many defenses against backdoor attacks have been proposed. However, existing defense works require high computational overhead or backdoor attack information such as the trigger size. In this paper, we propose a novel backdoor detection method based on intentional adversarial perturbations. The proposed method leverages intentional adversarial perturbation to detect whether an image contains a trigger, which can be applied in both the training stage and the inference stage (sanitize the training set in training stage or detect the backdoor instances in inference stage). Specifically, given an untrusted image, the adversarial perturbation is added to the image intentionally. If the prediction of the model on the perturbed image is consistent with that on the unperturbed image, the input image will be considered as a backdoor instance. Compared with most existing defense works, the proposed method is faster and introduces less computational overhead during backdoor detection process. Moreover, the proposed method maintains the visual quality of the image (as the ℓ 2 norm of the added perturbation is as low as 2.8715, 3.0513 and 2.4362 on Fashion-MNIST, CIFAR-10 and GTSRB datasets, respectively). Experimental results show that, for general backdoor attack, the backdoor detection rate of the proposed defense method is 99.63%, 99.76% and 99.91% on Fashion-MNIST, CIFAR-10 and GTSRB datasets, respectively. For invisible backdoor attack, the backdoor detection rate of the proposed defense method is 99.75% against blended backdoor attack and 98.00% against sample-specific backdoor attack. It is also demonstrated that the proposed method can achieve high defense performance against backdoor attacks under different attack settings (trigger transparency, trigger size and trigger pattern). In addition, the experimental comparison with related work demonstrates that the proposed method has better detection performance and higher detection efficiency. [ABSTRACT FROM AUTHOR]

To further enhance the reliability of Machine Learning (ML) systems, considerable efforts have been dedicated to developing privacy protection techniques. Recently, membership privacy has gained increasing attention, with a focus on determining whether a specific data point is present in the confidential training set of an ML model. However, most current attacks only prioritize attack accuracy and fail to extend their range to the evidence that contributes to the member/non-member classification. This limitation greatly reduces the practicality of Membership Inference Attacks (MIA), as real-world data typically includes multiple features, making it challenging to identify which features are involved in the sensitive training set. Therefore, this paper targets one of the fundamental challenges in membership inference attack: measuring the distance between an attack sample and a member sample. Specifically, we propose a novel threat model called Membership Reconstruction Attack (MRA), which aims to reconstruct the exact distribution of the target training set. MRA achieves this by marking each input dimension (e.g., pixels) according to its similarity to the target dataset in feature space. Our attack demonstrates its effectiveness across various settings, including different major datasets (MNIST, CIFAR-10, CIFAR-100) and different model architectures (AlexNet, ResNet, DenseNet, and generative models). Additionally, we evaluate MRA from the defenders' perspective and test several defense approaches against our attack. [ABSTRACT FROM AUTHOR]

Although recurrent neural networks (RNNs) perfectly solve many difficult problems, their computational complexity significantly increases training time. Therefore, the primary problem with applying RNNs is to shorten the time needed to train and operate a network. An effective solution to this problem is to use parallel processing. In the paper, a particular approach for the Jordan network will be shown, however, the presented idea is applicable to other RNN structures. This type of network is characterized by natural parallelism, and in the paper, this feature is used to significantly accelerate the learning process. High-performance learning has been achieved using a novel parallel three-dimensional architecture. The presented solutions can be implemented in digital hardware. [ABSTRACT FROM AUTHOR]

Deep neural networks easily suffer from weak generalization caused by overfitting on biased data. One popular remedy to alleviate this issue is sample reweighting methods that adaptively adjust the importance of biased samples. Separate from the effort to reduce bias, recent works show that the generalization power can be improved by auxiliary tasks. Inspired by the two lines of works, we extend the sample reweighting methods to auxiliary tasks. In this paper, we propose a novel auxiliary learning framework that improves the primary task by adaptively adjusting the weights of samples from multiple tasks rather than samples from a single task using a weighting function. The weighting function is optimized by meta-learning along the gradient of the loss for meta-data, which is a small unbiased validation data. We also present a task-activation score that indicates the correlation between the learning tendency of the training samples and meta-data samples. This score is utilized as a regularizer for meta-learning objective. Our framework can obtain powerful representations for the primary task on biased data by automatically identifying effective combinations of tasks. Our experiments demonstrate that our proposed method consistently outperforms all baselines and state-of-the-art methods on both corrupted labels and class imbalance settings. [ABSTRACT FROM AUTHOR]

The successful application of modern deep neural network models is deeply influenced by the choice of hyperparameters. As one of the most important hyperparameters, the Learning Rate (LR) needs to be fine-tuned. In order to train models that can perform better, finding a suitable LR schedule quickly has become a pressing problem. Current research often relies on manual use of parallel search methods such as grid search, or serial methods such as Bayesian. However, the variety and number of models and datasets result in a very large LR space, and these methods consume significant resources to find only sub-optimal static LR schedules within a given time frame. It was found that in the currently widely used Distributed Data Parallel (DDP) deep learning, where multiple nodes themselves constitute the population, it is possible to use population algorithms to optimise the searched LR schedule dynamically throughout the training process with little additional training time. To maximise the potential of both the searched LR schedule and the population optimisation of the participating multi-nodes, this paper proposes a Leader Population Learning Rate Schedule (LPLRS) for the DDP deep learning environment, which continuously explores for better learning rates in the neighbourhood of the searched LR schedule and guides the subsequent training process. LPLRS achieves higher model classification accuracy on the Cifar 10&100 test datasets on the state-of-the-art Wide Residual Network with Sharpness-Aware Minimization (WRN(SAM)) model compared to the latest SGDR, CLR, and StepLR learning rate schedules. [ABSTRACT FROM AUTHOR]

• A 14-layer model with high predictive performance is proposed. • The model fuses three types of multi-source and multi-frequency features. • The model addresses three limitations of LSTM. • The model is validated through four comparison experiments on four stock indices. • Each component of the model improves the predictive performance. The most widely used model in stock price forecasting is the long short-term memory network (LSTM). However, LSTM has its limitations, as it does not recognize and extract features well and has a representational bottleneck. Furthermore, the factors affecting stock prices are multi-source and multi-frequency information, making neural network models difficult to handle. In this paper, we introduce a feature fusion residual LSTM (FFRL) model to answer these two questions – how to compensate for the three limitations of LSTM and how to fuse the multi-source and multi-frequency information. FFRL consists of three modules to improve the three limitations of LSTM, namely the feature selection module, feature extraction module, and residual module. To learn features from multi-source and multi-frequency information, FFRL applies the feature selection module to emphasize important features and the feature extraction module to extract deeper features. We demonstrate significant performance improvements of FFRL over comparison models, ablation networks, and visualization methods on a variety of Chinese stocks. [ABSTRACT FROM AUTHOR]

It has been a long-standing goal in natural language processing (NLP) to learn a general linguistic intelligence model that can perform well on many different NLP tasks continually evolving over time while avoiding revisiting all previous data at each stage. Most existing deep neural networks suffer from catastrophic forgetting when dealing with sequential tasks in an incremental way, leading to dramatic performance degradation due to the missing training data of old tasks. In this paper, we propose a Lifelong language method with Adaptive Uncertainty Regularization (LAUR), which can adapt a single BERT model to work with continuously arriving text examples from different NLP tasks. Specifically, LAUR is built on the Bayesian online learning framework, and three uncertainty regularization terms are devised to collaboratively control the parameters so as to resolve the stability-plasticity dilemma in lifelong language learning. The previous posterior constrain parameters that strongly determine the output results, preventing these parameters from changing drastically, while other parameters are encouraged to be updated over time. In addition, we propose a task-specific residual adaptation module in parallel to each layer of BERT to endow LAUR with the capacity to learn better task-specific knowledge. This configuration makes LAUR less prone to losing the knowledge stored in the base BERT network when learning a new task. Experimental results show that LAUR outperforms state-of-the-art lifelong learning models on a variety of NLP tasks. For reproducibility, we submit the code and data at: https://github.com/kiujhytgtrfd2021/LAUR. [ABSTRACT FROM AUTHOR]

It has been a long-standing goal in natural language processing (NLP) to learn a general linguistic intelligence model that can perform well on many different NLP tasks continually evolving over time while avoiding revisiting all previous data at each stage. Most existing deep neural networks suffer from catastrophic forgetting when dealing with sequential tasks in an incremental way, leading to dramatic performance degradation due to the missing training data of old tasks. In this paper, we propose a Lifelong language method with Adaptive Uncertainty Regularization (LAUR), which can adapt a single BERT model to work with continuously arriving text examples from different NLP tasks. Specifically, LAUR is built on the Bayesian online learning framework, and three uncertainty regularization terms are devised to collaboratively control the parameters so as to resolve the stability-plasticity dilemma in lifelong language learning. The previous posterior constrain parameters that strongly determine the output results, preventing these parameters from changing drastically, while other parameters are encouraged to be updated over time. In addition, we propose a task-specific residual adaptation module in parallel to each layer of BERT to endow LAUR with the capacity to learn better task-specific knowledge. This configuration makes LAUR less prone to losing the knowledge stored in the base BERT network when learning a new task. Experimental results show that LAUR outperforms state-of-the-art lifelong learning models on a variety of NLP tasks. For reproducibility, we submit the code and data at: https://github.com/kiujhytgtrfd2021/LAUR. [ABSTRACT FROM AUTHOR]

• A 14-layer model with high predictive performance is proposed. • The model fuses three types of multi-source and multi-frequency features. • The model addresses three limitations of LSTM. • The model is validated through four comparison experiments on four stock indices. • Each component of the model improves the predictive performance. The most widely used model in stock price forecasting is the long short-term memory network (LSTM). However, LSTM has its limitations, as it does not recognize and extract features well and has a representational bottleneck. Furthermore, the factors affecting stock prices are multi-source and multi-frequency information, making neural network models difficult to handle. In this paper, we introduce a feature fusion residual LSTM (FFRL) model to answer these two questions – how to compensate for the three limitations of LSTM and how to fuse the multi-source and multi-frequency information. FFRL consists of three modules to improve the three limitations of LSTM, namely the feature selection module, feature extraction module, and residual module. To learn features from multi-source and multi-frequency information, FFRL applies the feature selection module to emphasize important features and the feature extraction module to extract deeper features. We demonstrate significant performance improvements of FFRL over comparison models, ablation networks, and visualization methods on a variety of Chinese stocks. [ABSTRACT FROM AUTHOR]

In a high-dimensional sample space, the distribution characteristics of samples are complex. It is difficult to accurately describe the local distribution characteristics of samples directly in the original sample space. This paper proposes a multiple discriminant preserving support subspace radial basis function neural networks with graph similarity learning (DPSS-RBFNN) to describe the distribution characteristics of samples in the original high-dimensional space by multiple low-dimensional and simple discriminant preserving subspaces. DPSS-RBFNN includes the discriminant preserving support subspace (DPSS) learning module and the subspace distribution feature extraction (SDFE) module. In the DPSS learning module, the discriminativeness of each attribute and the joint discriminant between attributes are first considered to construct multiple subspaces. The discriminativeness of the features in these subspaces is not lower than that of the original samples. The graph similarity learning is used to measure the similarity of sample distributions between subspaces with different dimensions. Then multiple DPSSs are obtained. In SDFE module, the distribution characteristics of the samples are described by combining the local responses of the feature space extracted by the sub-RBFNN in each DPSS. The experimental results show that the proposed DPSS-RBFNN with few kernels can achieve higher accuracy in the recognition task than state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) aims to relieve traffic congestion. Some multi-agent deep reinforcement learning (MADRL) algorithms have been proposed for ITLC, and most of them use deep neural networks to make decisions. However, the abundant parameters of deep structure lead to the time-consuming training process of MADRL. Recently, a broad reinforcement learning (BRL) approach has been proposed to improve the efficiency of training for an agent. Unlike MADRL algorithms that use deep architecture, BRL utilizes a broad architecture. In this paper, we propose a multi-agent broad reinforcement learning (MABRL) algorithm for ITLC. The MABRL algorithm adopts the broad network to process the joint information and updates the parameters using ridge regression. To increase the effectiveness of interaction among agents, we design a dynamic interaction mechanism (DIM) based on the attention mechanism. The DIM enables agents to aggregate information on particular intersections at appropriate moments. We conduct experiments on three different datasets. The results demonstrate that the effectiveness of MABRL outperforms several state-of-the-art algorithms in alleviating traffic congestion with shorter training time. [ABSTRACT FROM AUTHOR]

In this paper, we present the problem formulation and methodology framework of Super Resolution Perception (SRP) on smart meter data. With the widespread use of smart meters, a massive amount of electricity consumption data can be obtained. Smart meter data is the basis of automated billing and pricing, appliance identification, demand response, etc. However, the provision of high-quality data may be expensive in many cases. In this paper, we propose a novel problem - the SRP problem as reconstructing high-quality data from unsatisfactory data in smart grids. Advanced generative models are then proposed to solve the problem. This technology makes it possible for empowering existing facilities without upgrading existing meters or deploying additional meters. We first mathematically formulate the SRP problem under the Maximum a Posteriori (MAP) estimation framework. The dataset namely Super Resolution Perception Dataset (SRPD) is designed for this problem and released. A case study is then presented, which performs SRP on smart meter data. A network namely Super Resolution Perception Convolutional Neural Network (SRPCNN) is proposed to generate high-frequency load data from low-frequency data. Experiments demonstrate that our SRP models can reconstruct high-frequency data effectively. Moreover, the reconstructed high-frequency data can lead to better appliance identification results. [ABSTRACT FROM AUTHOR]

• The paper proposes a novel dynamic graph recurrent convolutional neural network model, named Dynamic-GRCNN, to deeply capture the spatio-temporal traffic flow features for more accurately predicting urban passenger traffic flows. • The paper presents incidence dynamic graph structures based on historically passenger traffic flows to model traffic station relationships. Different from existing traffic transportation network topological structures based graph relationships between stations, the incidence dynamic graph structures firstly model the traffic relationships from historical passenger flows. • For real urban passenger traffic flows, the paper demonstrates that dynamic spatial-temporal incidence graphs are more suitable to model external changes and influences. • The paper compares Dynamic-GRCNN with state-of-the-art deep learning approaches on three benchmark datasets which contain different types of passenger traffic flows for evaluation. The results show that Dynamic-GRCNN significantly outperforms all the baselines in both effectiveness and efficiency in urban passenger traffic flows prediction. Accurate and real-time traffic passenger flows forecasting at transportation hubs, such as subway/bus stations, is a practical application and of great significance for urban traffic planning, control, guidance, etc. Recently deep learning based methods are promised to learn the spatial-temporal features from high non-linearity and complexity of traffic flows. However, it is still very challenging to handle so much complex factors including the urban transportation network topological structures and the laws of traffic flows with spatial and temporal dependencies. Considering both the static hybrid urban transportation network structures and dynamic spatial-temporal relationships among stations from historical traffic passenger flows, a more effective and fine-grained spatial-temporal features learning framework is necessary. In this paper, we propose a novel spatial-temporal incidence dynamic graph neural networks framework for urban traffic passenger flows prediction. We first model dynamic traffic station relationships over time as spatial-temporal incidence dynamic graph structures based on historically traffic passenger flows. Then we design a novel dynamic graph recurrent convolutional neural network, namely Dynamic-GRCNN, to learn the spatial-temporal features representation for urban transportation network topological structures and transportation hubs. To fully utilize the historical passenger flows, we sample the short-term, medium-term and long-term historical traffic data in training, which can capture the periodicity and trend of the traffic passenger flows at different stations. We conduct extensive experiments on different types of traffic passenger flows datasets including subway, taxi and bus flows in Beijing. The results show that the proposed Dynamic-GRCNN effectively captures comprehensive spatial-temporal correlations significantly and outperforms both traditional and deep learning based urban traffic passenger flows prediction methods. [ABSTRACT FROM AUTHOR]

• A novel CNN-based SISR method with cascaded modules named Lightweight Group Convolution Network (LGCN) is proposed. • The paper develops a Memory Group Convolutional Network (MGCN), which uses group convolution as the convolutional layer to reduce the network parameters. • The paper develops a novel channel attention method for SISR in MGCN, which replaces the fully connection layer with 1 × 1 convolutional layer based on the idea of squeeze and excitation net. • Extensive experiments on benchmark datasets show our LGCN model achieves competitive results against state-of-the-art methods on both accuracy and speed. Recently, deep learning methods have demonstrated significant reconstruction performance on Single Image Super-Resolution (SISR). However, most of them demand a huge amount of computational and memory consumption and hard to be applied to real-world applications. To this end, we propose a fast Lightweight Group Convolution Network (LGCN) model for SISR to alleviate this problem. Specifically, we develop a cascaded memory group convolutional network for SISR, which cascades several Memory Group Convolutional Networks (MGCN). There are two main merits on MGCN. One is that it consists of several group convolutional layers and 1 × 1 convolutional layers with densely connected structure. The group convolution is utilized to reduce the parameters of LGCN, and the 1 × 1 convolution is not only used to create a linear combination of the output of group convolutional layer, but also to gather local information progressively. The other one is that it utilizes channel attention unit to model channel-wise relationships to improve performance. Experimental results on four popular datasets show that the proposed LGCN not only outperforms the state-of-the-art SISR methods, but also achieves faster speed. [ABSTRACT FROM AUTHOR]

• The motion uncertainty problem in target tracking can be addressed with one DNN. • Better estimation accuracy can be obtained by the proposed DNN-based methods. • Bayesian filtering idea is not necessary for designing a tracking neural network. • Real-time tracking with the proposed methods is feasible. • Computation of a DNN can be simplified at the cost of slightly worse performance. Target tracking is a difficult estimation problem due to target motion uncertainty and measurement origin uncertainty. In this paper, we consider the target tracking problem in the presence of only target motion uncertainty. The traditional approaches to address this uncertainty, such as multiple model approaches, can suffer performance degradation when there is a model mismatch. The statistical accuracy of conventional model-based methods is also usually limited because of the measurement errors and insufficient data for the estimation. In this paper, deep neural network-based methods are proposed to handle target motion uncertainty due to their strong capability of fitting any mapping as long as there are sufficient training data. Specifically, a recurrent neural network-based structure is proposed to estimate the true states that is consistent with the sequential manner of target tracking. In addition, it is expected that better performance will be achieved due to access to true states during the training of the networks. We propose two networks that are based on different principles and are capable of real-time tracking. An approach to further reduce the computational load is also introduced. Simulation results show that the proposed methods can handle the target motion uncertainty well and provide better estimation accuracy. [ABSTRACT FROM AUTHOR]

In recent years, hyperparameter tuning has become increasingly essential, especially for neural network models, where models with good performance generally require a long time to train. In contrast, fast-trained models often result in poor model performance. Thus, it is essential to determine the optimal hyperparameter configuration for a neural network model that can produce good model performance and fast training time. This paper presents a multiobjective optimization for Long-Short Term Memory (LSTM) model using Bayesian optimization-Hyperband (BOHB) with a weighted metric scalarization method. The objective functions used are the F1-score (Macro) and model training time, where both of the objective functions are considered equally important. The data used in this paper are the SmSA and EmoT datasets taken from the IndoNLU benchmark. The results are optimized model using multiobjective BOHB optimization with weighted metric scalarization method produced higher F1-score and faster model training time in both datasets compared to the baseline model and optimized model using single objective BOHB. • This paper implements the weighted metric scalarization for the multiobjective BOHB optimization method. • We tested our proposed method to optimize the LSTM model hyperparameter for sentiment analysis. • The data used in this paper are the SmSA and EmoT datasets taken from the IndoNLU benchmark. • We compare our proposed method against the original Bayesian optimization, Hyperband, and BOHB. • The optimized LSTM model using our proposed method produced higher F1-score and faster training time, simultaneously. [ABSTRACT FROM AUTHOR]

• We proposed a social influence diffusion (SID) module to model the social networks by graph neural networks to enhance the representation learning of individual users. • We proposed an interest evolution learning (IEL) module to learn users' dynamic and evolving preferences. This is the first group recommender system that learns users' interest evolution to the best of our knowledge. • A novel group recommendation framework, called DASIIE, is designed to dynamically aggregate SID and IEL through a two-way attention mechanism (TWA), considering both group members' roles and expertise simultaneously. • We conduct extensive experiments on two real-world datasets, and the results show the better performance of our model compared to the state-of-the-art baseline models. The rapid development of social networks has made group activities part of people's daily lives. It is in high demand to recommend items for a group of users that meet their common preferences, known as group recommendations (GRs). Recently, some studies have focused on GRs and adopted deep neural networks to model the process of group decision-making. These approaches can be further divided into two phases: the individual preference learning phase and the group decision phase, both of which face several challenges although recent research has some achievements. In the first individual preference learning phase, the challenges are how to effectively incorporate social networks to overcome the sparsity issue of user interaction data and how to capture the dynamic evolutions of users' preferences over time. The challenge for the second group decision phase is how to adjust group members' preference weights dynamically since each group member may carry different deciding power in different situations, depending on their roles in the group and domain expertise. In this paper, to address these challenges, we propose a novel group recommendation framework that d ynamically a ggregates the s ocial i nfluence diffusion and i nterest e volution (DASIIE). Specifically, DASIIE is composed of three main modules: 1) a social influence diffusion (SID) module which stacks multi-layer graph neural networks to simulate the influence propagation in social networks; 2) an interest evolution learning (IEL) component which learns individuals' interest evolutions over time from their behavior sequences; 3) a two-way attention (TWA) module which learns group members' role weights and expertise weights during the decision-making process simultaneously. We conduct extensive experiments on two real-world datasets, and the results show the better performance of our model compared to the state-of-the-art baseline models. [ABSTRACT FROM AUTHOR]

Over the years, there has been a sharp increase in employing artificial neural networks in modeling and simulation of complicated real-world problems. The main aim of the present paper is to apply an extension of artificial neural networks approach for approximating solution of a two-dimensional fractional-order linear Volterra-type integro-differential initial value problem. For this aim, a suitable four-layered feed forward neural architecture is designed to approximate the solution function with the help of two distinct famous learning algorithms, namely the steepest descent and the quasi-Newton rules, to any desired degree of accuracy. Finally, the applicability and validity of preset techniques are described and tested through some numerical examples. Numerical and simulation results obtained here are also compared with the results obtained by the optimal Homotopy asymptotic method. This practice determined that the quasi-Newton rule provides far more accurate solutions than the other two mentioned above. [ABSTRACT FROM AUTHOR]

As the training process of deep neural networks involves expensive computational cost, speeding up the convergence is of great importance. Nesterov's accelerated gradient (NAG) is one of the most popular accelerated optimizers in the deep learning community, which often exhibits improved convergence performance over gradient descent (GD) in practice. However, theoretical investigations of NAG mainly focus on the convex setting. Since the optimization landscape of the neural network is non-convex, little is known about the convergence and acceleration of NAG. Nowadays, some works make progress towards understanding the convergence of NAG in training over-parameterized neural networks, where the number of the parameters exceeds that of the training instances. Nonetheless, previous studies are limited to the two-layer neural network, which are far from explaining the remarkable success of NAG in optimizing deep neural networks. In this paper, we investigate the convergence of NAG in training two architectures of deep linear networks: deep fully-connected linear neural networks and deep linear ResNets. Based on the over-parameterization regime, we first analyze the residual dynamics induced by the training trajectory of NAG for a deep fully-connected linear neural network under random Gaussian initialization. Our results show that NAG can converge to the global minimum at a (1 - O (1 / κ)) t rate when the width is near-linear in the depth of the network, where t is the number of iterations and κ > 1 is a constant depending on the condition number of the feature matrix. Compared to the (1 - O (1 / κ)) t rate of GD, NAG achieves an acceleration over GD. For deep linear ResNets, we utilize the same analytical approach and obtain a similar convergence result, while the width requirement is independent of the depth. To the best of our knowledge, these are the first theoretical guarantees for the convergence and acceleration of NAG in training deep neural networks. Numerical results show the acceleration of NAG compared to GD in terms of iterations. In addition, we conduct experiments to evaluate the effect of the depth on the convergence rate of NAG, which validate our derived conditions of the width. We hope our results may shed light on understanding the optimization behavior of NAG for modern deep neural networks. [ABSTRACT FROM AUTHOR]

• We propose two algorithms to optimize adversarial perturbation and patch. • We discover that CAEs can quickly decrease the classification accuracy. • We find that two weak attack patterns can be combined to perform stronger attack. • We demonstrate the effectiveness and robustness of the proposed method. Although deep learning has made great progress in many fields, they are still vulnerable to adversarial examples. Many methods for generating adversarial examples have been proposed, which either contain adversarial perturbation or patch. In this paper, we explore the method that creates compound adversarial examples including both perturbation and patch. We show that fusing two weak attack modes can produce more powerful adversarial examples, where the patch covers only 1 % of the pixels at random location in the image, and the perturbation changes only by 2/255 in the original pixel value (scale to 0–1). For both targeted attack and untargeted attack, compound attack can improve the generative efficiency of adversarial examples, and can attain higher attack success rate with fewer iteration steps. The compound adversarial examples successfully attack the models with defensive mechanisms that previously can defend perturbation attack or patch attack. Furthermore, the compound adversarial examples show good transferability on normal trained classifiers and adversarial trained classifiers. Experimental results on a series of widely used classifiers and defense models show that the proposed compound adversarial examples have strong robustness, high effectiveness, and good transferability. [ABSTRACT FROM AUTHOR]

Adversarial examples are well-designed input samples that include perturbations that are imperceptible to the human eye but easily mislead the output of deep neural networks (DNNs). Existing studies synthesize adversarial examples by leveraging simple metrics to penalize perturbations that lack sufficient consideration of the human visual system (HVS), producing noticeable artifacts. To explore why these perturbations are visible,four primary factors that affect the perceptibility of human eyes are summarized in this paper. Based on this investigation, we design a multi-factor metric MulFactorLoss for measuring the perceptual loss between benign examples and adversarial examples. To test the imperceptibility of the multi-factor metric, we propose a novel black-box adversarial attack that is referred to as GreedyFool. GreedyFool applies differential evolution to evaluate the effects of perturbed pixels on the confidence of a target DNN and introduces greedy approximation to automatically generate adversarial perturbations. We conduct extensive experiments on the ImageNet and CIFRA-10 datasets and a comprehensive user study with 60 participants. The experimental results demonstrate that MulFactorLoss is a more imperceptible metric than the existing pixelwise metrics, and GreedyFool achieves a 100% success rate in a black-box manner. [ABSTRACT FROM AUTHOR]

Recent works on residual network and its variants help us to advance our understanding of deep neural networks as differential equations. Specifically, when taking convolution operator as the combinations of differential operators, deep convolutional neural network (DCNN) can be regarded as partial differential equation (PDE). Learn such PDEs have become popular in image restoration especially for image denoising. While current methods for learning the denoising PDEs face two critical issues. On the one hand, the learned PDEs are not guaranteed to behave as progressive denoising as traditional denoising PDEs. On the other hand, the low level feature may diminish with time increases in PDE. To resolve these issues, this paper designs to learn a specialized PDE network (PDNet) which depicts the information propagation in the network as a reaction-diffusion–advection process. Specifically, a reaction-term is introduced to prevent the low-level features from diminishing in the PDE network, and a decreasing residual gain strategy with stochastic supervision is adopted to facilitate the progressive denoising property. The progressive denoising property not only reveals the semantic of the network path but also regularizes the network training. Compared to vanilla DCNNs, the PDNet has clear physical meaning. Meanwhile PDNet simplifies the ResNet in that batch normalization is removed from the residual blocks and there is only one nonlinear unit in a residual block (2 for ResNet). Extensive experimental results on benchmark datasets verified that PDNet is not only efficient but also effective. [ABSTRACT FROM AUTHOR]

Automated emotion recognition using brain electroencephalogram (EEG) signals is predominantly used for the accurate assessment of human actions as compared to facial expression or speech signals. Various signal processing methods have been used for extracting representative features from EEG signals for emotion recognition. However, the EEG signals are non-stationary and vary across the subjects as well as in different sessions of the same subject; hence it exhibits poor generalizability and low classification accuracy for an emotion classification of cross subjects. In this paper, EEG signals-based automated cross-subject emotion recognition framework is proposed using the Fourier-Bessel series expansion-based empirical wavelet transform (FBSE-EWT) method. This method is used to decompose the EEG signals from each channel into four sub-band signals. Manually ten channels are selected from the frontal lobe, from which entropy and energy features are extracted from each sub-band signal. The subject variability is reduced using an average moving filter method on each channel to obtain the smoothened feature vector of size 80. The three feature selection techniques, such as neighborhood component analysis (NCA), relief-F, and mRMR, are used to obtain an optimal feature vector. The machine learning models, such as artificial neural network (ANN), k -nearest neighborhood (k -NN) with two (fine and weighted) functions, and ensemble bagged tree classifiers are trained by the obtained feature vectors. The experiments are performed on two publicly accessible databases, named SJTU emotion EEG dataset (SEED) and dataset for emotion analysis using physiological signals (DEAP). The training and testing of the models have been performed using 10-fold cross-validation and leave-one-subject-out-cross-validation (LOSOCV). The proposed framework based on FBSE-EWT and NCA feature selection approach shows superior results for classifying human emotions compared to other state-of-art emotion classification models. [ABSTRACT FROM AUTHOR]

Recent works on residual network and its variants help us to advance our understanding of deep neural networks as differential equations. Specifically, when taking convolution operator as the combinations of differential operators, deep convolutional neural network (DCNN) can be regarded as partial differential equation (PDE). Learn such PDEs have become popular in image restoration especially for image denoising. While current methods for learning the denoising PDEs face two critical issues. On the one hand, the learned PDEs are not guaranteed to behave as progressive denoising as traditional denoising PDEs. On the other hand, the low level feature may diminish with time increases in PDE. To resolve these issues, this paper designs to learn a specialized PDE network (PDNet) which depicts the information propagation in the network as a reaction-diffusion–advection process. Specifically, a reaction-term is introduced to prevent the low-level features from diminishing in the PDE network, and a decreasing residual gain strategy with stochastic supervision is adopted to facilitate the progressive denoising property. The progressive denoising property not only reveals the semantic of the network path but also regularizes the network training. Compared to vanilla DCNNs, the PDNet has clear physical meaning. Meanwhile PDNet simplifies the ResNet in that batch normalization is removed from the residual blocks and there is only one nonlinear unit in a residual block (2 for ResNet). Extensive experimental results on benchmark datasets verified that PDNet is not only efficient but also effective. [ABSTRACT FROM AUTHOR]

Automated emotion recognition using brain electroencephalogram (EEG) signals is predominantly used for the accurate assessment of human actions as compared to facial expression or speech signals. Various signal processing methods have been used for extracting representative features from EEG signals for emotion recognition. However, the EEG signals are non-stationary and vary across the subjects as well as in different sessions of the same subject; hence it exhibits poor generalizability and low classification accuracy for an emotion classification of cross subjects. In this paper, EEG signals-based automated cross-subject emotion recognition framework is proposed using the Fourier-Bessel series expansion-based empirical wavelet transform (FBSE-EWT) method. This method is used to decompose the EEG signals from each channel into four sub-band signals. Manually ten channels are selected from the frontal lobe, from which entropy and energy features are extracted from each sub-band signal. The subject variability is reduced using an average moving filter method on each channel to obtain the smoothened feature vector of size 80. The three feature selection techniques, such as neighborhood component analysis (NCA), relief-F, and mRMR, are used to obtain an optimal feature vector. The machine learning models, such as artificial neural network (ANN), k -nearest neighborhood (k -NN) with two (fine and weighted) functions, and ensemble bagged tree classifiers are trained by the obtained feature vectors. The experiments are performed on two publicly accessible databases, named SJTU emotion EEG dataset (SEED) and dataset for emotion analysis using physiological signals (DEAP). The training and testing of the models have been performed using 10-fold cross-validation and leave-one-subject-out-cross-validation (LOSOCV). The proposed framework based on FBSE-EWT and NCA feature selection approach shows superior results for classifying human emotions compared to other state-of-art emotion classification models. [ABSTRACT FROM AUTHOR]

In recent years, the number of people who endanger their lives has been increasing rapidly due to the mental burden of depression. The online social network (OSN) provides researchers with another perspective for detecting individuals suffering from depression. However, existing machine learning-based depression detection studies still leave relatively low classification performance, suggesting that there is significant improvement potential in their feature engineering. In this paper, we manually build and publish a large dataset on Sina Weibo (a leading OSN with the largest number of active users in the Chinese community), namely the Weibo User Depression Detection Dataset (WU3D). It includes more than 20,000 normal users and more than 10,000 depressed users, both of which are labeled and rechecked following the DSM-5 official medical and psychological depression document by professionals. Then, we conclude and propose ten statistical features by analyzing the user's text, social behavior, and posted pictures. In the meantime, text-based word features are extracted using the popular pretrained model XLNet. Moreover, we fuse these features from heterogeneous modalities and implement a multitask learning scheme to train our proposed deep neural network classification model, i.e. FusionNet (FN). The experimental results show that FN has excellent to recognize depressed users on the OSN, achieving the highest F1-Score of 0.9772 on the test set. Compared to existing studies, the proposed method has better classification performance and robustness for unbalanced training samples, as well as reasonable training and inference time. Our work provides a method to fuse multimodal information to detect individual-level depression and has reference significance for similar studies on other OSNs. [ABSTRACT FROM AUTHOR]

Solving the multi-objective vehicle routing problem with stochastic demand (MO-VRPSD) is challenging due to its non-deterministic property and conflicting objectives. Most multi-objective evolutionary algorithm dealing with this problem update current population without any guidance from previous searching experience. In this paper, a multi-objective evolutionary algorithm based on artificial neural networks is proposed to tackle the MO-VRPSD. Particularly, during the evolutionary process, a radial basis function network (RBFN) is exploited to learn the potential knowledge of individuals, generate hypothesis and instantiate hypothesis. The RBFN evaluates individuals with different scores and generates new individuals with higher quality while taking into account the non-dominated relationship between individuals. Moreover, integrated with a specific non-dominated sorting strategy, i.e., ENS-SS, along with several effective heuristic operations, the proposed algorithm performs favorably for solving the MO-VRPSD. The experimental results based on the modified Solomon benchmark instances verified the effectiveness of the respective components, and the superiority to other multi-objective evolutionary algorithms. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation aims at alleviating distribution discrepancy when transferring knowledge from a large-scale labeled source domain to a new unlabeled target domain. A prevailing method is adversarial feature adaptation, which generates transferable features via aligning feature distributions across domains to bound domain discrepancy. However, such approach renders suboptimal performances and vulnerable to negative transfer when target domain has less number of classes compared to source domain as in partial domain adaptation. To address this issue, previous methods adopt a transferable feature norm strategy to adapting feature norms of two domains achieving significant transfer gains. Nonetheless, these works do not consider the intrinsic limitation of the architecture design of deep neural networks which greatly influences the loss function of transferability. In this paper, we propose Transferable Regularization and Normalization (TRN), which simultaneously avoids negative transfer via adapting feature norms of both domains and facilitates positive transfer via replacing the existing normalization techniques in mainstream deep backbones. As a general approach, TRN can be simply embedded into deep transfer learning approaches. After a thorough evaluation of proposed method utilizing several benchmark datasets (Office-31, ImageCLEF-DA, Office-Home and VisDA-2017). TRN yielded the state-of-the-art results and outperformed other approaches (IFAN, PADA) by a large margin (0.5%, 7.74%) for various domain adaptation tasks. [ABSTRACT FROM AUTHOR]

Lip reading is a visual alternative to enhance the intelligence of traditional speech recognition, which can benefit from retina-like event cameras that focus on dynamic movements. Spiking Neural Networks (SNNs) are inherently well-suited to cooperate with event cameras. However, employing SNN for event-based lip reading presents significant challenges, particularly in effectively extracting spatio-temporal features from input events and distinguishing between phonetically similar elements. To address these challenges, this paper proposes a novel event-based lip-reading model that leverages the SNN framework, enriched by a designed Spatial-Temporal Attention Block (STAB) and a triplet loss. Specifically, STAB comprises both spatial and temporal attention branches, dynamically emphasizing those spatial and temporal characteristics most reflective of lip movements. STAB also includes a fusion mechanism to integrate these spatial and temporal insights, providing a comprehensive and focused representation of lip movements. In addition, we enhance the model by incorporating triplet loss into the SNN training process, further improving the model's ability to distinguish between visually similar words. Experimental results show the superior performance of our SNN and validate the effect of STAB and triplet loss. We also conduct the energy consumption analysis to affirm the model's energy efficiency. [ABSTRACT FROM AUTHOR]

[Display omitted] The architecture of our proposed method SSGCAE for semi-supervised overlapping community detection in attributed graph. • An end-to-end method SSGCAE for overlapping community detection is proposed. • SSGCAE is based on graph convolutional autoencoder driven by community detection. • SSGCAE is comprehensively evaluated on synthetic and real attributed graphs. • SSGCAE outperforms state-of-the-art baselines. Community detection in attributed graph is of great application value and many related methods have been continually presented. However, existing methods for community detection in attributed graph still cannot well solve three key problems simultaneously: link information and attribute information fusion, prior information integration and overlapping community detection. Aiming at these problems, in this paper we devise a semi-supervised overlapping community detection method named SSGCAE which is based on graph neural networks. This method is composed of three modules: graph convolutional autoencoder (GCAE), semi-supervision and modularity maximization, which are respectively utilized to fuse link information and attribute information, integrate prior information and detect overlapping communities. We treat GCAE as the backbone framework and train it by using the unified loss from these three modules. Through this way, these three modules are jointly correlated via the community membership representation, which is very beneficial to improve the overall performance. SSGCAE is comprehensively evaluated on synthetic and real attributed graphs, and experiment results show that it is very effective and outperforms state-of-the-art baseline approaches. [ABSTRACT FROM AUTHOR]

Long-tailed classifications make it very challenging to deal with class-imbalanced problems using deep convolutional neural networks (CNNs). Existing solutions based on re-balancing methods perform well and use single-task CNNs to train each fine-grained class independently. However, classification tasks are multiplex and involve a coarse-to-fine hierarchical relation. In this paper, we propose a coarse-to-fine knowledge transfer based multi-task CNN, which utilizes the coarse-to-fine structure to promote long-tailed learning. First, we construct a tail hierarchical structure in a coarse-to-fine way to pay greater attention to tail classes than head classes. Second, a multi-task CNN is adopted to simultaneously train coarse- and fine-grained tasks to extract a more generalized knowledge representation than the single-task CNN. Third, we design a coarse-to-fine knowledge transfer strategy to adaptively adjust the task weights to improve fine-grained performance. Extensive experiments on benchmark datasets show that our model achieves better gains than the re-balancing methods. In particular, the proposed model is 3.25% more accurate than the second-best method on the long-tailed tieredImageNet dataset. [ABSTRACT FROM AUTHOR]

• A deep metric learning method is presented for facial expression recognition. • Chi-squared distance is modified for deep metric learning in histogram space. • A new fully histogram based Convolutional Neural Network is proposed. • A triplet loss based on modified chi-squared distance is proposed to learn the CNN. Facial expression recognition is a challenging problem in machine learning. There is much research has been conducted in this field; however, the accuracy of facial expression recognition, especially in uncontrolled conditions, needs much improvement. In this paper, a deep histogram metric learning in a Convolutional Neural Network (CNN) is presented for facial expression recognition. The proposed CNN utilizes a histogram calculation layer to provide statistical description of feature maps at the output of the convolutional layers. To train the proposed CNN in histogram space, a learnable matrix (equivalent to the fully connected layer) is introduced in chi-squared distance equation. Then, the modified equation is used in the loss function. The recognition rates of the proposed CNN for seven-class facial expression recognition on four well-known databases including CK+, MMI, SFEW, and RAF-DB are 98.47%, 83.41%, 61.01%, and 89.28%, respectively. The results show superiority of the proposed CNN compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The traffic density estimation algorithm usually runs on an embedded platform, and its efficiency is very important to the implementation of applications. However, most of the existing methods rely on heavy backbone networks, require high-performance hardware platform support and take a long time to calculate, which are difficult prerequisites to apply in practice. In consideration of the aforementioned problems, this paper proposes a lightweight traffic density estimation method based on structured knowledge transfer (Le-SKT). This method consists of a teacher network and student network. In the teacher network, in order to solve the problem of variable vehicle scales in video images, a multiscale fusion structure is designed, which can fully fuse the low-level and high-level features of an image. To solve the problem of background interference in video images, a fusion module based on an attention mechanism is constructed to enhance the useful traffic flow feature information and suppress the background information. At the same time, the fused traffic flow feature information is dilated convoluted to expand the receptive field and obtain richer contextual vehicle feature information, which generates a trained teacher network. Finally, a lightweight student network is generated by using the structured knowledge of the teacher network, which solves the problems of long computing times and high hardware requirements. The MAE and GAME of this method are reduced to 3.92 and 5.69, respectively in the TRANCOS dataset and 3.74 and 5.57, respectively in the VisDrone2019 vehicle dataset, respectively; The parameter quantity of the student network is only 0.65% of that of the teacher network, which greatly reduces the scale of the neural network model and is suitable for an embedded computing platform with low performance. [ABSTRACT FROM AUTHOR]

The backpropagation (BP) neural network has been widely used in many fields. However, it is still a great challenge to design the architecture and obtain optimal parameters for BP neural networks. For improving the generalization performance, regularization is the most popular technique to train the BP neural networks. In this paper, we propose a novel BP algorithm with graph regularization (BPGR) to obtain optimal parameters, by imposing the graph regularization term to the error function. The essential idea is to force the latent features of hidden layer to be more concentrated, which enhances the generalization performance. Besides, the proposed modified graph regularization facilitates the calculation of gradient and is more capable to penalize the extreme values of weights. Furthermore, the graph regularization can also be integrated with deep neural networks to improve their generalization performance. In addition, we provide the convergence analysis of our method BPGR under some regularity conditions. By comparison on several datasets with five activation functions, experimental results validate the theoretical analysis and demonstrate outstanding performance of BPGR. [ABSTRACT FROM AUTHOR]

Deceptive reviews detection has attracted extensive attentions from the business and research communities in recent years. Existing work mainly uses traditional discrete models with rich features from the viewpoint of linguistics and psycholinguistics. The drawback is that these models fail to capture the global semantic information of a sentence or discourse. Recently, neural network models provide new solutions for this task, and can be used to learn global representation of a review text, achieving competitive performance. We observe that a review text usually contains two types of information. Some words or sentences describe the user's preferences, while others indicate the characteristics of the product. Based on this observation, this paper explores a hierarchical neural network model with attention mechanism, which can learn a global review representation from the viewpoint of user and product, to identify deceptive reviews. Experimental results show that the proposed neural model achieves 91.7% accuracy on the Yelp datasets, outperforming traditional discrete models and neural baseline systems by a large margin. [ABSTRACT FROM AUTHOR]

In recent years, attributed networks are increasingly available for us. How to leverage attribute information to gain a better performance of community detection has attracted growing attention. Most established approaches focus on exploiting attribute information based on the observed homophily assumption that nodes with similar attributes are more likely to be connected. Since not all attributes are truly relevant for the formation of communities, such an assumption may not hold in real-world attributed networks. In many cases, only a subset of attributes are the key factors to make nodes connected to form communities and they may differ largely for different communities. In this paper, we propose a new community detection approach in attributed networks, called SOA, by examining the Subspaces Of Attributes. The basic idea is to regard the community detection as an optimization problem, which aims at learning a new similarity matrix to maximize the homophily. By employing an efficient optimization strategy, the high-quality communities as well as their corresponding relevant attributes (i.e., attribute subspaces) are automatically identified. Extensive experimental results on both synthetic and real-world networks have demonstrated the effectiveness of SOA, and shown its superiority over state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNN) are mainly black boxes, generally suffering from bad interpretability of their behavior and the results obtained. Hence, a human can not easily derive the relations modeled by the network. A reasonable way to provide interpretability for humans are logical rules. In this paper we propose neural logic rule layers (NLRL), which are able to represent arbitrary logic rules in terms of their conjunctive and disjunctive normal forms. Stacking various layers, we are theoretically able to represent arbitrary complex rules by the resulting neural network architecture. The NLRL are end-to-end trainable allowing to learn logic rules directly on the given data without needing any background information about the origin. We show in experiments, that NLRL -enhanced neural networks can model arithmetic and logical operations over the input values. Furthermore, we apply NLRL to image classification tasks and show that interpretability is provided without sacrificing classification performance by exchanging the fully-connected head of the network. We also apply NLRL to a real world industrial control problem where the task is to model the discrete control behaviour of a programmable logic controller (PLC), following a basic step sequence. [ABSTRACT FROM AUTHOR]

Brain extraction (skull stripping) is a challenging problem in neuroimaging. It is due to the variability in conditions from data acquisition or abnormalities in images, making brain morphology and intensity characteristics changeable and complicated. In this paper, we propose an algorithm for skull stripping in Magnetic Resonance Imaging (MRI) scans, namely ASMCNN, by combining the Active Shape Model (ASM) and Convolutional Neural Network (CNN) for taking full of their advantages to achieve remarkable results. Instead of working with 3D structures, we process 2D image sequences in the sagittal plane. First, we divide images into different groups such that, in each group, shapes and structures of brain boundaries have similar appearances. Second, a modified version of ASM is used to detect brain boundaries by utilizing prior knowledge of each group. Finally, CNN and post-processing methods, including Conditional Random Field (CRF), Gaussian processes, and several special rules are applied to refine the segmentation contours. Experimental results show that our proposed method outperforms current state-of-the-art algorithms by a significant margin in all experiments. [ABSTRACT FROM AUTHOR]

• We propose an alternative modeling for the input feature space for the CF problem. • We investigate the how the complexity of input feature space impacts the performance. • Our proposal presents competitive results against the state-of-the-art CF methods. Collaborative Filtering (CF) has been extensively studied over the last decade. Among the most successful methods, neural-based methods have changed the research landscape by bringing cutting-edge artificial neural networks (ANN) and big data to offer the best personalization users have ever experienced. However, a significant issue arose from these neural-based methods is that the increase of the model complexity directly impacts the computational cost. In general, this drawback is a consequence of the high dimensionality and sparsity of the input feature space. We believe that reducing sparsity and dimensionality of the input features is essencial to enhance accuracy while keeping computational costs low. This paper investigates an alternative modeling for the CF setting to represent users or items by using landmarks. This modeling drastically decreases the input feature space and eliminate sparsity while maintaining the underlying information needed to achieve great accuracy. Based on that, we propose a novel neural-based CF method via landmark modeling. Experiments on six real-world benchmark CF datasets were conducted by comparing the proposed method to well-known and widely-used CF methods of the state-of-the-art. The results show that the Neural Landmark method outperforms the other methods in both accuracy and computational performance. [ABSTRACT FROM AUTHOR]

Railway emergencies have the characteristics of unobvious precursors and complex secondary derivatives, which is difficult for decision-makers to make effective emergency response solutions. This paper develops a scenario-response method to construct and deduce the state and severity of railway emergencies. Firstly, the scenario framework is constructed and the key scenario elements are extracted. Secondly, three scenario deduction models based on dynamic Bayesian network, fuzzy neural network, and convolutional neural network are proposed to deduce the process of an emergency. Finally, the "7.23" Yongwen train collision accident as a case study is presented and discussed. The results show that convolutional neural network model with multi time-sequence inputs has an advantage of high accuracy in comparisons with the dynamic Bayesian network model and fuzzy neural network model. [ABSTRACT FROM AUTHOR]

In recent years, the study of linguistic steganalysis has been focused on altering network structure, such as replacing the basic neural units or increasing the model depth, which inevitably increases computational overhead and restricts further improvement in resource-constrained scenarios. In this paper, instead of relying on complex neural networks, we propose an alternative linguistic steganalysis framework based on hierarchical mutual learning to achieve higher detection accuracy with less inference time and model size. In the proposed method, networks with either identical or different structures are trained cooperatively to learn distinct text features from each other. To this end, in addition to the supervised learning loss function, we construct three mimicry loss functions at different feature extraction stages, which can integrate steganalytic features from various levels. Finally, we illustrate how the steganalysis framework can be extended from two networks to multiple networks. Four representative steganalysis networks with different structures are employed to verify the effectiveness of our framework. The experimental results show that the proposed framework can effectively assist networks with fewer resources to perform better in model size, inference time, and detection accuracy than state-of-the-art steganalysis algorithms. [ABSTRACT FROM AUTHOR]

• Fuzzy granulation of interval-valued time series(ITS) is investigated in interval number space. • Two kinds of fuzzy information granules and their designing methods are proposed for a set of interval-valued temporal data. • Two kinds of fuzzy granulation methods are proposed for ITS with respect to the two proposed kinds of fuzzy information granules. • A linear dynamic fuzzy information granule based long-term forecasting model of ITS is proposed. The studies on modeling and analysis of time series based on fuzzy granulation have shown that fuzzy granulation is an effective approach in data mining of time series. However, the investigation of fuzzy granulation of interval-valued time series (ITS) and its applications has just begun in recent years with appearance of few research results. Different from the existing studies, this paper carried out the investigation of fuzzy granulation of ITS in interval number space instead of real number space. Two distinguished concepts, namely static fuzzy information granules and dynamic fuzzy information granules of ITS, are proposed firstly. Then the approaches for constructing a static fuzzy information granule and a dynamic fuzzy information granule are designed respectively. After that, two fuzzy granulation methods for ITS based on the above two approaches are presented in the framework of interval analysis under the guidance of the principle of justifiable granularity. Based on the specific proposed fuzzy granule which is called linear dynamic fuzzy granule, a long-term forecasting model for ITS was developed with the aid of artificial neural network. Experiments conducted on several ITS from stock markets with different dynamic characteristics showed the outperformance of the proposed long-term forecasting model. [ABSTRACT FROM AUTHOR]

Deep neural networks have demonstrated remarkable efficacy in diverse classification tasks. In this paper, we specifically focus on the predictive performance in deep binary classification problems with the sigmoid loss. Given that sigmoid loss is categorized as a non-convex and bounded loss function, it exhibits potential resilience against the disruptive impact of outlier noises. We first derive the convergence rate of the excess misclassification risk for deep ReLU neural networks with the sigmoid loss, a result that attains minimax optimality. To the best of our acknowledge, we are the first to derive the convergence rate for the sigmoid loss. Moreover, we extend our analysis to derive a faster convergence rate under margin assumptions. This achievement renders our findings comparable to those of commonly employed convex loss functions operating under analogous assumptions. Lastly, we undertake a comprehensive validation of the robustness inherent in the sigmoid loss across diverse datasets. [ABSTRACT FROM AUTHOR]

In Unsupervised Domain Adaptation(UDA), the hard samples cause serious negative transfer to model performance. However, the existing UDA methods don't fully pay attention to the fine-grained processing of hard samples in target and source domains. To address these problems, the UDA framework with Hard-sample Dividing and Processing Strategy (HDPS) is proposed in this paper. In HDPS, we define sample division criteria in target domain and source domain respectively and divide the samples into easy samples, hard samples with low confidence and hard samples with high confidence. We further design systematic processing strategy of the three types of samples. Specifically, we firstly define geometric metrics based on the class prototype to divide the hard samples and eliminate them in source domain. We further design two weight allocation strategies based on sample classification entropy in target domain. Finally, we build a teacher-student guidance mechanism to learn the discriminative features of hard samples. The HDPS framework can be easily loaded to other methods as a plug-in, can promote the diversity feature learning of samples, and significantly improve the discriminant performance of the model. Extensive experiments on six benchmark datasets verify that HDPS is effective and superior to most existing UDA methods. [ABSTRACT FROM AUTHOR]