Search

Anomaly detection is one of the most active research areas in various critical domains, such as healthcare, fintech, and public security. However, little attention has been paid to scholarly data, that is, anomaly detection in a citation network. Citation is considered as one of the most crucial metrics to evaluate the impact of scientific research, which may be gamed in multiple ways. Therefore, anomaly detection in citation networks is of significant importance to identify manipulation and inflation of citations. To address this open issue, we propose a novel deep graph learning model, namely graph learning for anomaly detection (GLAD), to identify anomalies in citation networks. GLAD incorporates text semantic mining to network representation learning by adding both node attributes and link attributes via graph neural networks (GNNs). It exploits not only the relevance of citation contents, but also hidden relationships between papers. Within the GLAD framework, we propose an algorithm called Citation PUrpose (CPU) to discover the purpose of citation based on citation context. The performance of GLAD is validated through a simulated anomalous citation dataset. Experimental results demonstrate the effectiveness of GLAD on the anomalous citation detection task. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM AUTHOR]

This paper presents the design of an optimal controller for solving tracking problems subject to unmeasurable disturbances and unknown system dynamics using reinforcement learning (RL). Many existing RL control methods take disturbance into account by directly measuring it and manipulating it for exploration during the learning process, thereby preventing any disturbance induced bias in the control estimates. However, in most practical scenarios, disturbance is neither measurable nor manipulable. The main contribution of this article is the introduction of a combination of a bias compensation mechanism and the integral action in the Q-learning framework to remove the need to measure or manipulate the disturbance, while preventing disturbance induced bias in the optimal control estimates. A bias compensated Q-learning scheme is presented that learns the disturbance induced bias terms separately from the optimal control parameters and ensures the convergence of the control parameters to the optimal solution even in the presence of unmeasurable disturbances. Both state feedback and output feedback algorithms are developed based on policy iteration (PI) and value iteration (VI) that guarantee the convergence of the tracking error to zero. The feasibility of the design is validated on a practical optimal control application of a heating, ventilating, and air conditioning (HVAC) zone controller. [ABSTRACT FROM AUTHOR]

In this paper, an adaptive admittance control scheme is developed for robots to interact with time-varying environments. Admittance control is adopted to achieve a compliant physical robot–environment interaction, and the uncertain environment with time-varying dynamics is defined as a linear system. A critic learning method is used to obtain the desired admittance parameters based on the cost function composed of interaction force and trajectory tracking without the knowledge of the environmental dynamics. To deal with dynamic uncertainties in the control system, a neural-network (NN)-based adaptive controller with a dynamic learning framework is developed to guarantee the trajectory tracking performance. Experiments are conducted and the results have verified the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures. [ABSTRACT FROM AUTHOR]

Generative adversarial networks (GANs) for (generalized) zero-shot learning (ZSL) aim to generate unseen image features when conditioned on unseen class embeddings, each of which corresponds to one unique category. Most existing works on GANs for ZSL generate features by merely feeding the seen image feature/class embedding (combined with random Gaussian noise) pairs into the generator/discriminator for a two-player minimax game. However, the structure consistency of the distributions among the real/fake image features, which may shift the generated features away from their real distribution to some extent, is seldom considered. In this paper, to align the weights of the generator for better structure consistency between real/fake features, we propose a novel multigraph adaptive GAN (MGA-GAN). Specifically, a Wasserstein GAN equipped with a classification loss is trained to generate discriminative features with structure consistency. MGA-GAN leverages the multigraph similarity structures between sliced seen real/fake feature samples to assist in updating the generator weights in the local feature manifold. Moreover, correlation graphs for the whole real/fake features are adopted to guarantee structure correlation in the global feature manifold. Extensive evaluations on four benchmarks demonstrate well the superiority of MGA-GAN over its state-of-the-art counterparts. [ABSTRACT FROM AUTHOR]

A nomaly detection aims at identifying data points which are rare or significantly different from the majority of data points. Many techniques are explored to build highly efficient and effective anomaly detection systems, but they are confronted with many difficulties when dealing with complex data, such as failing to capture intricate feature interactions or extract good feature representations. Deep-learning techniques have shown very promising performance in tackling different types of complex data in a broad range of tasks/problems, including anomaly detection. To address this new trend, we organized this Special Issue on Deep Learning for Anomaly Detection to cover the latest advancements of developing deep-learning techniques specially designed for anomaly detection. This editorial note provides an overview of the paper submissions to the Special Issue, and briefly introduces each of the accepted articles. [ABSTRACT FROM AUTHOR]

Network representation learning (NRL) has shown its effectiveness in many tasks, such as vertex classification, link prediction, and community detection. In many applications, vertices of social networks contain textual information, e.g., citation networks, which form a text corpus and can be applied to the typical representation learning methods. The global context in the text corpus can be utilized by topic models to discover the topic structures of vertices. Nevertheless, most existing NRL approaches focus on learning representations from the local neighbors of vertices and ignore the global structure of the associated textual information in networks. In this article, we propose a unified model based on matrix factorization (MF), named collaborative representation learning (CRL), which: 1) considers complementary global and local information simultaneously and 2) models topics and learns network embeddings collaboratively. Moreover, we incorporate the Fletcher–Reeves (FR) MF, a conjugate gradient method, to optimize the embedding matrices in an alternative mode. We call this parameter learning method as AFR in our work that can achieve convergence after a few numbers of iterations. Also, by evaluating CRL on topic coherence and vertex classification using several real-world data sets, our experimental study shows that this collaborative model not only can improve the performance of topic discovery over the baseline topic models but also can learn better network representations than the state-of-the-art context-aware NRL models. [ABSTRACT FROM AUTHOR]

The learning automaton (LA) that simulates the interaction between an intelligent agent and a stochastic environment to learn the optimal action is an important tool in reinforcement learning. Being confronted with an unknown environment, most learning automata have more than one parameters to be tuned during a pretraining process in which the LA interacts with the environment. Only after the parameters are tuned properly, an LA can act most properly during the training procedure to obtain the optimal behavior. The cost of parameter tuning can be enormous, e.g., possibly millions of interactions are required to seek the best parameter configuration. Therefore, the parameter-free LA that uses identical parameters for every environment and saves further tuning has become the hot spot of this research. This article proposes an efficient parameter-free learning automaton (EPFLA) that depends on a separating function (SF). Taking advantage of both frequentist inference and Bayesian inference, the SF plays a dual role in the proposed scheme: 1) evaluating the difference in performance between actions in the environment and 2) exploring actions by coining an action selection strategy. A proof is provided to ensure the $\epsilon $ -optimality of EPFLA. Comprehensive comparisons verify the privileges of EPFLA over both parameter-based schemes and existing parameter-free schemes. [ABSTRACT FROM AUTHOR]

In traditional graph neural networks (GNNs), graph convolutional learning is carried out through topology-driven recursive node content aggregation for network representation learning. In reality, network topology and node content each provide unique and important information, and they are not always consistent because of noise, irrelevance, or missing links between nodes. A pure topology-driven feature aggregation approach between unaligned neighborhoods may deteriorate learning from nodes with poor structure-content consistency, due to the propagation of incorrect messages over the whole network. Alternatively, in this brief, we advocate a co-alignment graph convolutional learning (CoGL) paradigm, by aligning topology and content networks to maximize consistency. Our theme is to enforce the learning from the topology network to be consistent with the content network while simultaneously optimizing the content network to comply with the topology for optimized representation learning. Given a network, CoGL first reconstructs a content network from node features then co-aligns the content network and the original network through a unified optimization goal with: 1) minimized content loss; 2) minimized classification loss; and 3) minimized adversarial loss. Experiments on six benchmarks demonstrate that CoGL achieves comparable and even better performance compared with existing state-of-the-art GNN models. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) pansharpening is of great importance in improving the spatial resolution of HS images for remote sensing tasks. HS image comprises abundant spectral contents, whereas panchromatic (PAN) image provides spatial information. HS pansharpening constitutes the possibility for providing the pansharpened image with both high spatial and spectral resolution. This article develops a specific pansharpening framework based on a generative dual-adversarial network (called PS-GDANet). Specifically, the pansharpening problem is formulated as a dual task that can be solved by a generative adversarial network (GAN) with two discriminators. The spatial discriminator forces the intensity component of the pansharpened image to be as consistent as possible with the PAN image, and the spectral discriminator helps to preserve spectral information of the original HS image. Instead of designing a deep network, PS-GDANet extends GANs to two discriminators and provides a high-resolution pansharpened image in a fraction of iterations. The experimental results demonstrate that PS-GDANet outperforms several widely accepted state-of-the-art pansharpening methods in terms of qualitative and quantitative assessment. [ABSTRACT FROM AUTHOR]

Most modern learning problems are highly overparameterized, i.e., have many more model parameters than the number of training data points. As a result, the training loss may have infinitely many global minima (parameter vectors that perfectly “interpolate” the training data). It is therefore imperative to understand which interpolating solutions we converge to, how they depend on the initialization and learning algorithm, and whether they yield different test errors. In this article, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which stochastic gradient descent (SGD) is a special case. Recently, it has been shown that for overparameterized linear models, SMD converges to the closest global minimum to the initialization point, where closeness is in terms of the Bregman divergence corresponding to the potential function of the mirror descent. With appropriate initialization, this yields convergence to the minimum-potential interpolating solution, a phenomenon referred to as implicit regularization. On the theory side, we show that for sufficiently-overparameterized nonlinear models, SMD with a (small enough) fixed step size converges to a global minimum that is “very close” (in Bregman divergence) to the minimum-potential interpolating solution, thus attaining approximate implicit regularization. On the empirical side, our experiments on the MNIST and CIFAR-10 datasets consistently confirm that the above phenomenon occurs in practical scenarios. They further indicate a clear difference in the generalization performances of different SMD algorithms: experiments on the CIFAR-10 dataset with different regularizers, $\ell _{1}$ to encourage sparsity, $\ell _{2}$ (SGD) to encourage small Euclidean norm, and $\ell _{\infty }$ to discourage large components, surprisingly show that the $\ell _{\infty }$ norm consistently yields better generalization performance than SGD, which in turn generalizes better than the $\ell _{1}$ norm. [ABSTRACT FROM AUTHOR]

We review the current literature concerned with information plane (IP) analyses of neural network (NN) classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis of how the respective information quantities were estimated. Our survey suggests that compression visualized in IPs is not necessarily information-theoretic but is rather often compatible with geometric compression of the latent representations. This insight gives the IP a renewed justification. Aside from this, we shed light on the problem of estimating mutual information in deterministic NNs and its consequences. Specifically, we argue that, even in feedforward NNs, the data processing inequality needs not to hold for estimates of mutual information. Similarly, while a fitting phase, in which the mutual information is between the latent representation and the target increases, is necessary (but not sufficient) for good classification performance, depending on the specifics of mutual information estimation, such a fitting phase needs to not be visible in the IP. [ABSTRACT FROM AUTHOR]

Reinforcement learning algorithms, such as hindsight experience replay (HER) and hindsight goal generation (HGG), have been able to solve challenging robotic manipulation tasks in multigoal settings with sparse rewards. HER achieves its training success through hindsight replays of past experience with heuristic goals but underperforms in challenging tasks in which goals are difficult to explore. HGG enhances HER by selecting intermediate goals that are easy to achieve in the short term and promising to lead to target goals in the long term. This guided exploration makes HGG applicable to tasks in which target goals are far away from the object’s initial position. However, the vanilla HGG is not applicable to manipulation tasks with obstacles because the Euclidean metric used for HGG is not an accurate distance metric in such an environment. Although, with the guidance of a handcrafted distance grid, grid-based HGG can solve manipulation tasks with obstacles, a more feasible method that can solve such tasks automatically is still in demand. In this article, we propose graph-based hindsight goal generation (G-HGG), an extension of HGG selecting hindsight goals based on shortest distances in an obstacle-avoiding graph, which is a discrete representation of the environment. We evaluated G-HGG on four challenging manipulation tasks with obstacles, where significant enhancements in both sample efficiency and overall success rate are shown over HGG and HER. Videos can be viewed at https://videoviewsite.wixsite.com/ghgg. [ABSTRACT FROM AUTHOR]

The well-known backpropagation learning algorithm is probably the most popular learning algorithm in artificial neural networks. It has been widely used in various applications of deep learning. The backpropagation algorithm requires a separate feedback network to back propagate errors. This feedback network must have the same topology and connection strengths (weights) as the feed-forward network. In this article, we propose a new learning algorithm that is mathematically equivalent to the backpropagation algorithm but does not require a feedback network. The elimination of the feedback network makes the implementation of the new algorithm much simpler. The elimination of the feedback network also significantly increases biological plausibility for biological neural networks to learn using the new algorithm by means of some retrograde regulatory mechanisms that may exist in neurons. This new algorithm also eliminates the need for two-phase adaptation (feed-forward phase and feedback phase). Hence, neurons can adapt asynchronously and concurrently in a way analogous to that of biological neurons. [ABSTRACT FROM AUTHOR]

Fusing low dynamic range (LDR) for high dynamic range (HDR) images has gained a lot of attention, especially to achieve real-world application significance when the hardware resources are limited to capture images with different exposure times. However, existing HDR image generation by picking the best parts from each LDR image often yields unsatisfactory results due to either the lack of input images or well-exposed contents. To overcome this limitation, we model the HDR image generation process in two-exposure fusion as a deep reinforcement learning problem and learn an online compensating representation to fuse with LDR inputs for HDR image generation. Moreover, we build a two-exposure dataset with reference HDR images from a public multiexposure dataset that has not yet been normalized to train and evaluate the proposed model. By assessing the built dataset, we show that our reinforcement HDR image generation significantly outperforms other competing methods under different challenging scenarios, even with limited well-exposed contents. More experimental results on a no-reference multiexposure image dataset demonstrate the generality and effectiveness of the proposed model. To the best of our knowledge, this is the first work to use a reinforcement-learning-based framework for an online compensating representation in two-exposure image fusion. [ABSTRACT FROM AUTHOR]

Model compression is crucial for the deployment of neural networks on devices with limited computational and memory resources. Many different methods show comparable accuracy of the compressed model and similar compression rates. However, the majority of the compression methods are based on heuristics and offer no worst case guarantees on the tradeoff between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient structured pruning algorithm with a provable tradeoff between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, and it approximates the output of a layer of neurons/filters by a coreset of neurons/filters in the previous layer and discards the rest. We apply this framework in a layer-by-layer fashion from the bottom to the top. Unlike previous works, our coreset is data-independent, meaning that it provably guarantees the accuracy of the function for any input $x\in \mathbb {R}^{d}$ , including an adversarial one. [ABSTRACT FROM AUTHOR]

Scene graph generation (SGGen) is a challenging task due to a complex visual context of an image. Intuitively, the human visual system can volitionally focus on attended regions by salient stimuli associated with visual cues. For example, to infer the relationship between man and horse, the interaction between human leg and horseback can provide strong visual evidence to predict the predicate ride. Besides, the attended region face can also help to determine the object man. Till now, most of the existing works studied the SGGen by extracting coarse-grained bounding box features while understanding fine-grained visual regions received limited attention. To mitigate the drawback, this article proposes a region-aware attention learning method. The key idea is to explicitly construct the attention space to explore salient regions with the object and predicate inferences. First, we extract a set of regions in an image with the standard detection pipeline. Each region regresses to an object. Second, we propose the object-wise attention graph neural network (GNN), which incorporates attention modules into the graph structure to discover attended regions for object inference. Third, we build the predicate-wise co-attention GNN to jointly highlight subject’s and object’s attended regions for predicate inference. Particularly, each subject-object pair is connected with one of the latent predicates to construct one triplet. The proposed intra-triplet and inter-triplet learning mechanism can help discover the pair-wise attended regions to infer predicates. Extensive experiments on two popular benchmarks demonstrate the superiority of the proposed method. Additional ablation studies and visualization further validate its effectiveness. [ABSTRACT FROM AUTHOR]

It is challenging to bridge the performance gap between binary convolutional neural network (BCNN) and floating-point CNN (FCNN). This performance gap is mainly caused by the inferior modeling capability and training strategy of BCNN, which leads to substantial residuals in intermediate feature maps between BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This intuition leads to a more effective training strategy for BCNN, i.e., optimizing each binary convolutional block with blockwise distillation loss derived from FCNN. The goal of minimizing the residuals in intermediate feature maps also motivates us to update the binary convolutional block architecture to facilitate the optimization of blockwise distillation loss. Specifically, a lightweight shortcut branch is inserted into each binary convolutional block to complement residuals at each block. Benefited from its squeeze-and-interaction (SI) structure, this shortcut branch introduces a fraction of parameters, e.g., less than 10% overheads, but effectively boosts the modeling capability of binary convolution blocks in BCNN. Extensive experiments on ImageNet demonstrate the superior performance of our method in both classification efficiency and accuracy, e.g., BCNN trained with our methods achieves the accuracy of 60.45% on ImageNet, better than many state-of-the-art ones. [ABSTRACT FROM AUTHOR]

In online learning, the dynamic regret metric chooses the reference oracle that may change over time, while the typical (static) regret metric assumes the reference solution to be constant over the whole time horizon. The dynamic regret metric is particularly interesting for applications, such as online recommendation (since the customers’ preference always evolves over time). While the online gradient (OG) method has been shown to be optimal for the static regret metric, the optimal algorithm for the dynamic regret remains unknown. In this article, we show that proximal OG (a general version of OG) is optimum to the dynamic regret by showing that the proved lower bound matches the upper bound. It is highlighted that we provide a new and general lower bound of dynamic regret. It provides new understanding about the difficulty to follow the dynamics in the online setting. [ABSTRACT FROM AUTHOR]

This article investigates the nonnegative consensus tracking problem for networked systems with a distributed static output-feedback (SOF) control protocol. The distributed SOF controller design for networked systems presents a more challenging issue compared with the distributed state-feedback controller design. The agents are described by multi-input multi-output (MIMO) positive dynamic systems which may contain uncertain parameters, and the interconnection among the followers is modeled using an undirected connected communication graph. By employing positive systems theory, a series of necessary and sufficient conditions governing the consensus of the nominal, as well as uncertain, networked positive systems, is developed. Semidefinite programming consensus design approaches are proposed for the convergence rate optimization of MIMO agents. In addition, by exploiting the positivity characteristic of the systems, a linear-programming-based design approach is also proposed for the convergence rate optimization of single-input multi-output (SIMO) agents. The proposed approaches and the corresponding theoretical results are validated by case studies. [ABSTRACT FROM AUTHOR]

For solving dynamic generalized Lyapunov equation, two robust finite-time zeroing neural network (RFTZNN) models with stationary and nonstationary parameters are generated through the usage of an improved sign-bi-power (SBP) activation function (AF). Taking differential errors and model implementation errors into account, two corresponding perturbed RFTZNN models are derived to facilitate the analyses of robustness on the two RFTZNN models. Theoretical analysis gives the quantitatively estimated upper bounds for the convergence time (UBs-CT) of the two derived models, implying a superiority of the convergence that varying parameter RFTZNN (VP-RFTZNN) possesses over the fixed parameter RFTZNN (FP-RFTZNN). When the coefficient matrices and perturbation matrices are uniformly bounded, residual error of FP-RFTZNN is bounded, whereas that of VP-RFTZNN monotonically decreases at a super-exponential rate after a finite time, and eventually converges to 0. When these matrices are bounded but not uniform, residual error of FP-RFTZNN is no longer bounded, but that of VP-RFTZNN still converges. These superiorities of VP-RFTZNN are illustrated by abundant comparative experiments, and its application value is further proved by an application to robot. [ABSTRACT FROM AUTHOR]

In unsupervised domain adaptation (UDA), a classifier for the target domain is trained with massive true-label data from the source domain and unlabeled data from the target domain. However, collecting true-label data in the source domain can be expensive and sometimes impractical. Compared to the true label (TL), a complementary label (CL) specifies a class that a pattern does not belong to, and hence, collecting CLs would be less laborious than collecting TLs. In this article, we propose a novel setting where the source domain is composed of complementary-label data, and a theoretical bound of this setting is provided. We consider two cases of this setting: one is that the source domain only contains complementary-label data [completely complementary UDA (CC-UDA)] and the other is that the source domain has plenty of complementary-label data and a small amount of true-label data [partly complementary UDA (PC-UDA)]. To this end, a complementary label adversarial network (CLARINET) is proposed to solve CC-UDA and PC-UDA problems. CLARINET maintains two deep networks simultaneously, with one focusing on classifying the complementary-label source data and the other taking care of the source-to-target distributional adaptation. Experiments show that CLARINET significantly outperforms a series of competent baselines on handwritten digit-recognition and object-recognition tasks. [ABSTRACT FROM AUTHOR]

Group activity recognition (GAR) aiming at understanding the behavior of a group of people in a video clip has received increasing attention recently. Nevertheless, most of the existing solutions ignore that not all the persons contribute to the group activity of the scene equally. That is to say, the contribution from different individual behaviors to group activity is different; meanwhile, the contribution from people with different spatial positions is also different. To this end, we propose a novel Position-aware Participation-Contributed Temporal Dynamic Model (P2CTDM), in which two types of the key actor are constructed and learned. Specifically, we focus on the behaviors of key actors, who maintain steady motions (long moving time, called long motions) or display remarkable motions (but closely related to other people and the group activity, called flash motions) at a certain moment. For capturing long motions, we rank individual motions according to their intensity measured by stacking optical flows. For capturing flash motions that are closely related to other people, we design a position-aware interaction module (PIM) that simultaneously considers the feature similarity and position information. Beyond that, for capturing flash motions that are highly related to the group activity, we also present an aggregation long short-term memory (Agg-LSTM) to fuse the outputs from PIM by time-varying trainable attention factors. Four widely used benchmarks are adopted to evaluate the performance of the proposed P2CTDM compared to the state of the art. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) are widely used for online regression due to their ability to generalize nonlinear temporal dependencies. As an RNN model, long short-term memory networks (LSTMs) are commonly preferred in practice, as these networks are capable of learning long-term dependencies while avoiding the vanishing gradient problem. However, due to their large number of parameters, training LSTMs requires considerably longer training time compared to simple RNNs (SRNNs). In this article, we achieve the online regression performance of LSTMs with SRNNs efficiently. To this end, we introduce a first-order training algorithm with a linear time complexity in the number of parameters. We show that when SRNNs are trained with our algorithm, they provide very similar regression performance with the LSTMs in two to three times shorter training time. We provide strong theoretical analysis to support our experimental results by providing regret bounds on the convergence rate of our algorithm. Through an extensive set of experiments, we verify our theoretical work and demonstrate significant performance improvements of our algorithm with respect to LSTMs and the other state-of-the-art learning models. [ABSTRACT FROM AUTHOR]

Soft sensors have been extensively developed and applied in the process industry. One of the main challenges of the data-driven soft sensors is the lack of labeled data and the need to absorb the knowledge from a related source operating condition to enhance the soft sensing performance on the target application. This article introduces deep transfer learning to soft sensor modeling and proposes a deep probabilistic transfer regression (DPTR) framework. In DPTR, a deep generative regression model is first developed to learn Gaussian latent feature representations and model the regression relationship under the stochastic gradient variational Bayes framework. Then, a probabilistic latent space transfer strategy is designed to reduce the discrepancy between the source and target latent features such that the knowledge from the source data can be explored and transferred to enhance the target soft sensor performance. Besides, considering the missing values in the process data in the target operating condition, the DPTR is further extended to handle the missing data problem utilizing the strong generation and reconstruction capability of the deep generative model. The effectiveness of the proposed method is validated through an industrial multiphase flow process. [ABSTRACT FROM AUTHOR]

We propose a novel network pruning approach by information preserving of pretrained network weights (filters). Network pruning with the information preserving is formulated as a matrix sketch problem, which is efficiently solved by the off-the-shelf frequent direction method. Our approach, referred to as FilterSketch, encodes the second-order information of pretrained weights, which enables the representation capacity of pruned networks to be recovered with a simple fine-tuning procedure. FilterSketch requires neither training from scratch nor data-driven iterative optimization, leading to a several-orders-of-magnitude reduction of time cost in the optimization of pruning. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of floating-point operations (FLOPs) and prunes 59.9% of network parameters with negligible accuracy cost for ResNet-110. On ILSVRC-2012, it reduces 45.5% of FLOPs and removes 43.0% of parameters with only 0.69% accuracy drop for ResNet-50. Our code and pruned models can be found at https://github.com/lmbxmu/FilterSketch. [ABSTRACT FROM AUTHOR]

In this article, the finite-time synchronization (FTSYN) of a class of quaternion-valued neural networks (QVNNs) with discrete and distributed time delays is studied. Furthermore, the FTSYN and fixed-time synchronization (FIXSYN) of the QVNNs without time delay are investigated. Different from the existing results, which used decomposition techniques, by introducing an improved one-norm, we use a direct analytical method to study the synchronization problems. Incidentally, several properties of one-norm of the quaternion are analyzed, and then, three effective controllers are proposed to synchronize the drive and response QVNNs within a finite time or fixed time. Moreover, efficient criteria are proposed to guarantee that the synchronization of QVNNs with or without mixed time delays can be realized within a finite and fixed time interval, respectively. In addition, the settling times are reckoned. Compared with the existing work, our advantages are mainly reflected in the simpler Lyapunov analytical process and more general activation function. Finally, the validity and practicability of the conclusions are illustrated via four numerical examples. [ABSTRACT FROM AUTHOR]

Modulation classification is one of the key tasks for communications systems monitoring, management, and control for addressing technical issues, including spectrum awareness, adaptive transmissions, and interference avoidance. Recently, deep learning (DL)-based modulation classification has attracted significant attention due to its superiority in feature extraction and classification accuracy. In DL-based modulation classification, one major challenge is to preprocess a received signal and represent it in a proper format before feeding the signal into deep neural networks. This article provides a comprehensive survey of the state-of-the-art DL-based modulation classification algorithms, especially the techniques of signal representation and data preprocessing utilized in these algorithms. Since a received signal can be represented by either features, images, sequences, or a combination of them, existing algorithms of DL-based modulation classification can be categorized into four groups and are reviewed accordingly in this article. Furthermore, the advantages as well as disadvantages of each signal representation method are summarized and discussed. [ABSTRACT FROM AUTHOR]

Image clustering has recently attracted significant attention due to the increased availability of unlabeled datasets. The efficiency of traditional clustering algorithms heavily depends on the distance functions used and the dimensionality of the features. Therefore, performance degradation is often observed when tackling either unprocessed images or high-dimensional features extracted from processed images. To deal with these challenges, we propose a deep clustering framework consisting of a modified generative adversarial network (GAN) and an auxiliary classifier. The modification employs Sobel operations prior to the discriminator of the GAN to enhance the separability of the learned features. The discriminator is then leveraged to generate representations as to the input to an auxiliary classifier. An objective function is utilized to train the auxiliary classifier by maximizing the mutual information between the representations obtained via the discriminator model and the same representations perturbed via adversarial training. We further improve the robustness of the auxiliary classifier by introducing a penalty term into the objective function. This minimizes the divergence across multiple transformed representations generated by the discriminator model with a low dropout rate. The auxiliary classifier is implemented with a group of multiple cluster-heads, where a tolerance hyper-parameter is used to tackle imbalanced data. Our results indicate that the proposed method achieves competitive results compared with state-of-the-art clustering methods on a wide range of benchmark datasets including CIFAR-10, CIFAR-100/20, and STL10. [ABSTRACT FROM AUTHOR]

Neuromorphic computing is a promising technology that realizes computation based on event-based spiking neural networks (SNNs). However, fault-tolerant on-chip learning remains a challenge in neuromorphic systems. This study presents the first scalable neuromorphic fault-tolerant context-dependent learning (FCL) hardware framework. We show how this system can learn associations between stimulation and response in two context-dependent learning tasks from experimental neuroscience, despite possible faults in the hardware nodes. Furthermore, we demonstrate how our novel fault-tolerant neuromorphic spike routing scheme can avoid multiple fault nodes successfully and can enhance the maximum throughput of the neuromorphic network by 0.9%–16.1% in comparison with previous studies. By utilizing the real-time computational capabilities and multiple-fault-tolerant property of the proposed system, the neuronal mechanisms underlying the spiking activities of neuromorphic networks can be readily explored. In addition, the proposed system can be applied in real-time learning and decision-making applications, brain–machine integration, and the investigation of brain cognition during learning. [ABSTRACT FROM AUTHOR]

A single clustering refers to the partitioning of data such that the similar data are assigned into the same group, whereas the dissimilar data are separated into different groups. Recently, multiview clustering has received significant attention in recent years. However, most existing works tackle the single-clustering scenario, which only use single clustering to partition the data. In practice, nevertheless, the real-world data are complex and can be clustered in multiple ways depending on different interpretations of the data. Unlike these methods, in this article, we apply dual clustering to multiview subspace clustering. We propose a multiview dual-clustering method to simultaneously explore consensus representation and dual-clustering structure in a unified framework. First, multiview features are integrated into a latent embedding representation through a multiview learning process. Second, the dual-clustering segmentation is incorporated into the subspace clustering framework. Finally, the learned dual representations are assigned to the corresponding clusterings. The proposed approach is efficiently solved using an alternating optimization scheme. Extensive experiments demonstrate the superiority of our method on real-world multiview dual- and single-clustering datasets. [ABSTRACT FROM AUTHOR]

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI’s scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html. [ABSTRACT FROM AUTHOR]

How to generate more revenues is crucial to cloud providers. Evidences from the Amazon cloud system indicate that “dynamic pricing” would be more profitable than “static pricing.” The challenges are: How to set the price in real-time so to maximize revenues? How to estimate the price dependent demand so to optimize the pricing decision? We first design a discrete-time based dynamic pricing scheme and formulate a Markov decision process to characterize the evolving dynamics of the price-dependent demand. We formulate a revenue maximization framework to determine the optimal price and theoretically characterize the “structure” of the optimal revenue and optimal price. We apply the $Q$ -learning to infer the optimal price from historical transaction data and derive sufficient conditions on the model to guarantee its convergence to the optimal price, but it converges slowly. To speed up the convergence, we incorporate the structure of the optimal revenue that we obtained earlier, leading to the VpQ-learning ($Q$ -learning with value projection) algorithm. We derive sufficient conditions, under which the VpQ-learning algorithm converges to the optimal policy. Experiments on a real-world dataset show that the VpQ-learning algorithm outperforms a variety of baselines, i.e., improves the revenue by as high as 50% over the $Q$ -learning, speedy $Q$ -learning, and adaptive real-time dynamic programming (ARTDP), and by as high as 20% over the fixed pricing scheme. [ABSTRACT FROM AUTHOR]

Support vector machine (SVM) is a particularly powerful and flexible supervised learning model that analyzes data for both classification and regression, whose usual algorithm complexity scales polynomially with the dimension of data space and the number of data points. To tackle the big data challenge, a quantum SVM algorithm was proposed, which is claimed to achieve exponential speedup for least squares SVM (LS-SVM). Here, inspired by the quantum SVM algorithm, we present a quantum-inspired classical algorithm for LS-SVM. In our approach, an improved fast sampling technique, namely indirect sampling, is proposed for sampling the kernel matrix and classifying. We first consider the LS-SVM with a linear kernel, and then discuss the generalization of our method to nonlinear kernels. Theoretical analysis shows our algorithm can make classification with arbitrary success probability in logarithmic runtime of both the dimension of data space and the number of data points for low rank, low condition number, and high dimensional data matrix, matching the runtime of the quantum SVM. [ABSTRACT FROM AUTHOR]

A convolutional neural network (CNN) is one of the most significant networks in the deep learning field. Since CNN made impressive achievements in many areas, including but not limited to computer vision and natural language processing, it attracted much attention from both industry and academia in the past few years. The existing reviews mainly focus on CNN’s applications in different scenarios without considering CNN from a general perspective, and some novel ideas proposed recently are not covered. In this review, we aim to provide some novel ideas and prospects in this fast-growing field. Besides, not only 2-D convolution but also 1-D and multidimensional ones are involved. First, this review introduces the history of CNN. Second, we provide an overview of various convolutions. Third, some classic and advanced CNN models are introduced; especially those key points making them reach state-of-the-art results. Fourth, through experimental analysis, we draw some conclusions and provide several rules of thumb for functions and hyperparameter selection. Fifth, the applications of 1-D, 2-D, and multidimensional convolution are covered. Finally, some open issues and promising directions for CNN are discussed as guidelines for future work. [ABSTRACT FROM AUTHOR]