Search

Pattern recognition systems always misclassify anomalies, which can be dangerous for uninformed users. Therefore, anomalies must be filtered out from each classification. The main challenge for the anomaly filter design is the huge number of possible anomaly samples compared with the number of samples in the training set. Tailoring the filter for the given classifier is just the first step in this reduction. Paper tests the hypothesis that the filter trained in avoiding "near" anomalies will also refuse the "far" anomalies, and the anomaly detector is then just a classifier distinguishing between "far real" and "near anomaly" samples. As a "far real" samples generator was used, a Generative Adversarial Network (GAN) fake generator that transforms normally distributed random seeds into fakes similar to the training samples. The paper proves the assumption that seeds unused in fake training will generate anomalies. These seeds are distinguished according to their Chebyshev norms. While the fakes have seeds within the hypersphere with a given radius, the near anomalies have seeds within the sphere near cover. Experiments with various anomaly test sets have shown that GAN-based anomaly detectors create a reliable anti-anomaly shield using the abovementioned assumptions. The proposed anomaly detector is tailored to the given classifier, but its limitation is due to the need for the availability of the database on which the classifier was trained. [ABSTRACT FROM AUTHOR]

Lithium-ion batteries are widely used in electric vehicles, energy storage power stations, and many other applications. Accurate and reliable monitoring of battery health status and remaining capacity is the key to establish a lithium-ion cell management system. In this paper, based on a Bayesian optimization algorithm, a deep neural network is structured to evaluate the whole charging curve of the battery using partial charging curve data as input. A 0.74 Ah battery is used for experiments, and the effect of different input data lengths is also investigated to check the high flexibility of the approach. The consequences show that using only 20 points of partial charging data as input, the whole charging profile of a cell can be exactly predicted with a root-mean-square error (RMSE) of less than 19.16 mAh (2.59% of the nominal capacity of 0.74 Ah), and its mean absolute percentage error (MAPE) is less than 1.84%. In addition, critical information including battery state-of-charge (SOC) and state-of-health (SOH) can be extracted in this way to provide a basis for safe and long-lasting battery operation. [ABSTRACT FROM AUTHOR]

The application of deep neural network has achieved remarkable success in object detection. However, the network structures should be still evolved consistently and tuned finely to acquire better performance. This gears to the continuous demands on high performance in those complex scenes, where multi-scale objects to be detected are located here and there. To this end, this paper proposes a network structure called Multi-Scale Coupled Attention (MSCA) under the framework of self-attention learning with methodologies of importance assessment. Architecturally, it consists of a Multi-Scale Coupled Channel Attention (MSCCA) module, and a Multi-Scale Coupled Spatial Attention (MSCSA) module. Specifically, the MSCCA module is developed to achieve the goal of self-attention learning linearly on the multi-scale channels. In parallel, the MSCSA module is constructed to achieve this goal nonlinearly on the multi-scale spatial grids. The MSCCA and MSSCA modules can be connected together into a sequence, which can be used as a plugin to develop end-to-end learning models for object detection. Finally, our proposed network is compared on two public datasets with 13 classical or state-of-the-art models, including the Faster R-CNN, Cascade R-CNN, RetinaNet, SSD, PP-YOLO, YOLO v3, YOLO v5, YOLO v7, YOLOX, DETR, conditional DETR, UP-DETR and FP-DETR. Comparative experimental results with numerical scores, the ablation study, and the performance behaviour all demonstrate the effectiveness of our proposed model. [ABSTRACT FROM AUTHOR]

In this paper, we study the post-hoc calibration of modern neural networks, a problem that has drawn a lot of attention in recent years. Despite the plethora of calibration methods proposed, there is no consensus yet on the inherent complexity of the task and, while some authors claim that simple functions solve the problem, others suggest that more expressive models are needed to capture misscalibration. As a first approach, we focus on the task of confidence scaling, specifically on post-hoc methods that generalize Temperature Scaling, which we refer to as the Adaptive Temperature Scaling family. We begin by demonstrating that while complex models like neural networks provide an advantage when there is ample data, they fail in scenarios where it is limited, notably common in fields like medical diagnosis. We then show how under this ideal data conditions the more expressive methods learn a relationship between the entropy of a prediction and its level of overconfidence, and based on this observation, we propose Entropy-based Temperature Scaling, a simple method that scales the confidence of a prediction according to this relationship. Results show that our method obtains state-of-the-art performance and is robust against data scarcity. Moreover, our proposed model enables a deeper understanding of the calibration process by the interpretation of the entropy as a measure of uncertainty in the network outputs. [ABSTRACT FROM AUTHOR]

Controlled de-orbiting is crucial for a low earth orbiting (LEO) satellite or capsule intending to land in a desired location and to prevent damage to people and property on the ground caused by debris. Drag modulation is one possible mechanism to control de-orbiting, exploiting atmospheric drag variation to reduce the necessary orbital energy from the initial conditions to the re-entry interface. In this context, this paper proposes a novel targeted de-orbiting Artificial Neural Network (ANN) and drag-based guidance algorithm for a LEO artificial satellite. It relies on the combination of empirical and analytical relations with Deep Neural Networks (DNNs) to estimate the main parameters of the optimal control law in real time. The training set is composed of several optimal control solutions generated with a previously developed optimal control algorithm, exploiting the formulation in equinoctial modified orbital parameters to manage the large time scale of the problem and the computational cost. An innovative procedure for the training set generation led to the achievement of general results with a limited number of samples. The successful outcome of the guidance algorithm on over 1000 cases demonstrates its robustness and generalization of results. To conclude, a Linear Quadratic Regulator (LQR) feedback control is applied to one case to deal with a more realistic and uncertain density model. • Analytical optimal guidance for a satellite de-orbiting from Low Earth Orbit. • A novel approach based on Deep Neural Networks. • Prediction of the key parameters of the optimal control law in real time. • A Monte Carlo analysis showed the algorithm's robustness and generalization. • Successful feedback control thanks to a Linear Quadratic Regulator (LQR). [ABSTRACT FROM AUTHOR]

Deformable linear objects (DLOs), such as rods, cables, and ropes, play important roles in daily life. However, manipulation of DLOs is challenging as large geometrically nonlinear deformations may occur during the manipulation process. This problem is made even more difficult as the different deformation modes (e.g., stretching, bending, and twisting) may result in elastic instabilities during manipulation. In this paper, we formulate a physics-guided data-driven method to solve a challenging manipulation task—accurately deploying a DLO (an elastic rod) onto a rigid substrate along various prescribed patterns. Our framework combines machine learning, scaling analysis, and physical simulations to develop a physics-based neural controller for deployment. We explore the complex interplay between the gravitational and elastic energies of the manipulated DLO and obtain a control method for DLO deployment that is robust against friction and material properties. Out of the numerous geometrical and material properties of the rod and substrate, we show that only three non-dimensional parameters are needed to describe the deployment process with physical analysis. Therefore, the essence of the controlling law for the manipulation task can be constructed with a low-dimensional model, drastically increasing the computation speed. The effectiveness of our optimal control scheme is shown through a comprehensive robotic case study comparing against a heuristic control method for deploying rods for a wide variety of patterns. In addition to this, we also showcase the practicality of our control scheme by having a robot accomplish challenging high-level tasks such as mimicking human handwriting, cable placement, and tying knots. [ABSTRACT FROM AUTHOR]

The object of the research is the line segmentation task. To recognize the handwritten text from the documents in image format offline handwriting recognition technology is used. The text recognizer module accepts input as separate lines, so one of the important preprocessing steps is the detection and splitting of all handwritten text into distinct lines. In this paper, the handwritten text line segmentation task, its requirements, problems, and challenges are examined. Two main approaches for this task that are used in modern recognition systems are reviewed. These approaches are statistical projection-based methods and neural-based methods. Multiple works and research papers for each type of approach are reviewed analyzing their strengths and weaknesses considering the described tasks, constraints, and input data peculiarities. Overall acquired results are formed in a single table for comparison. Based on the latest works that utilize deep neural networks the new possibilities of using these methods in recognition systems are described that were unavailable with traditional statistical segmentation approaches. The constructive conclusions are made based on the review, describing the main pros and cons of these two approaches for the line segmentation task. These results can be further used for the correct selection of suitable methods in handwriting recognition systems to improve their performance and quality, and for further research in this area. [ABSTRACT FROM AUTHOR]

Shadow removal for document images is an essential task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large, diverse dataset for document shadow removal takes time and effort. Thus, only small real datasets are available. Graphic renderers have been used to synthesize shadows to create relatively large datasets. However, the limited number of unique documents and the limited lighting environments adversely affect the network performance. This paper presents a large-scale, diverse dataset called the Synthetic Document with Diverse Shadows (SynDocDS) dataset. The SynDocDS comprises rendered images with diverse shadows augmented by a physics-based illumination model, which can be utilized to obtain a more robust and high-performance deep shadow removal network. In this paper, we further propose a Dual Shadow Fusion Network (DSFN). Unlike natural images, document images often have constant background colors requiring a high understanding of global color features for training a deep shadow removal network. The DSFN has a high global color comprehension and understanding of shadow regions and merges shadow attentions and features efficiently. We conduct experiments on three publicly available datasets, the OSR, Kligler's, and Jung's datasets, to validate our proposed method's effectiveness. In comparison to training on existing synthetic datasets, our model training on the SynDocDS dataset achieves an enhancement in the PSNR and SSIM, increasing them from 23.00 dB to 25.70 dB and 0.959 to 0.971 on average. In addition, the experiments demonstrated that our DSFN clearly outperformed other networks across multiple metrics, including the PSNR, the SSIM, and its impact on OCR performance. [ABSTRACT FROM AUTHOR]

Flettner rotors are wind propulsion systems using the Magnus effect to generate thrust, thereby reduce fuel consumption and carbon emissions in the ships. However, rotor unbalance can cause excessive vibrations and energy loss, affecting the performance and stability of the system. There is a need to have a system onboard, which can predict the vibrations. The paper proposes a deep learning approach to predict the vibrations and unbalanced forces of a Flettner rotor based on the data of ECO Flettner rotor onboard the vessel MV Fehn pollux. The paper develops two methods to estimate the direction and magnitude of the unbalanced forces using the reading values of the strain gauges. The work also compares two recurrent neural network models, namely Long-short term memory and Gated Recurrent Unit, for vibration prediction and evaluates their performance using Mean Absolute Error and Root Mean Squared Error metrics. The results show that Longshort term memory model outperforms Gated Recurrent Unit model in prediction accuracy and can be implemented on the system onboard to monitor and prevent rotor unbalance. The paper also suggests some possible solutions for automatic self-balancing of the rotor and identifies some areas for future work. [ABSTRACT FROM AUTHOR]

To achieve successful prevention of fire incidents originating from human activities, it is imperative to possess a thorough understanding. This paper introduces a machine learning approach, specifically utilizing deep neural networks (DNN), to develop predictive models for fire occurrence in Keelung City, Taiwan. It investigates ten factors across demographic, architectural, and economic domains through spatial analysis and thematic maps generated from geographic information system data. These factors are then integrated as inputs for the DNN model. Through 50 iterations, performance indices including the coefficient of determination (R2), root mean square error (RMSE), variance accounted for (VAF), prediction interval (PI), mean absolute error (MAE), weighted index (WI), weighted mean absolute percentage error (WMAPE), Nash–Sutcliffe efficiency (NS), and the ratio of performance to deviation (RPD) are computed, with average values of 0.89, 7.30 × 10−2, 89.21, 1.63, 4.90 × 10−2, 0.97, 2.92 × 10−1, 0.88, and 4.84, respectively. The model's predictions, compared with historical data, demonstrate its efficacy. Additionally, this study explores the impact of various urban renewal strategies using the DNN model, highlighting the significant influence of economic factors on fire incidence. This underscores the importance of economic factors in mitigating fire incidents and emphasizes their consideration in urban renewal planning. [ABSTRACT FROM AUTHOR]

To improve convergence speed of deep neural network trained by gradient descent methods (GDMs), this paper proposed a novel fractional order gradient descent optimizer (FOAdam) based on Caputo operator and Adam algorithm, and analyzed the impact of fractional order on convergence performance using optimization theory and simulation. Furthermore, to address the trade-off between convergence speed and precision under fixed-order conditions, a fractional order scheduler (FOS) was designed based on the Connections Cloud Model (CCM). Numerical experiments conducted on MNIST and CIFAR-10 datasets showed that FOS-FOAdam outperformed other mainstream GDMs in terms of convergence speed and precision. Finally, FOS-FOAdam was applied in recognizing and classifying image datasets of cuttings and cores at the logging site, demonstrating good prospects for practical engineering applications. [ABSTRACT FROM AUTHOR]

The advanced development of communication technologies has made covert communications quite challenging to be recognized. By altering an entity's timing behavior during overt network communication, a covert timing channel (CTC) provides a way to leak sensitive data. This is becoming a serious threat due to the ability to transmit hidden messages without being detected by traditional security systems such as proxies and firewalls. In this paper, we explore the automatic classification and identification of covert timing channels using deep neural networks, namely, Long Short-Term Memory (LSTM), 1D-Convolutional Neural networks (1D-CNN), and a hybrid of LSTM and 1D-CNN. These classifiers have been trained and tested using sequence real inter-arrival times datasets labeled with covert or legitimate. The stream length differs for each dataset; (32, 64, 128, 256, and 512 bytes). Traditional machine-learning models have also been developed for comparisons and evaluation. The evaluation results showed that the hybrid of LSTM and CNN model outperforms other models either developed by deep neural networks or traditional machine learning with an accuracy of %97.5. [ABSTRACT FROM AUTHOR]

• A graph neural network is proposed for learning multi-satellite scheduling policy. • It analyzes the differential impact of various constraints in scheduling. • It models the multi-satellite scheduling problem as a heterogeneous graph. • It designs a variant transformer to learn various relations within graph. • It designs a two-stage model to learn a multi-satellite scheduling policy. In this paper, we investigate the multi-satellite scheduling (MSS) problem, a combinatorial optimization problem (COP) that aims to enhance satellite utilization. Numerous scholars have conducted in-depth studies on this matter. Handcrafted policies are commonly employed in various algorithms. Despite the use of graphical representations, designing policies manually becomes increasingly difficult, particularly as the number of satellites and tasks grows. Consequently, we propose a novel approach for automatically summarizing graph structures and learning policies for MSS problem. The proposed approach is developed based on graph neural network, Transformer, and sequence to sequence framework. First, MSS problem is modeled as a heterogeneous graph following various constraints. Next, the encoder, which is based on GNN and Transformer, is designed to summarize crucial information from the aforementioned graph. Finally, a two-stage decoder is proposed taking into account dynamic information in stepwise solution generation. Since obtaining optimal solutions in practical applications is challenging, we train the model using reinforcement learning. The results of the experiment demonstrate that our proposed method performs effectively in various scenarios. The quality of the solutions generated through this method transcends that of conventional construction methods. Additionally, the method generates solutions quickly, which are comparable in quality to those produced by meta-heuristic algorithms. [ABSTRACT FROM AUTHOR]

Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory has shaped deep neural networks, particularly the information bottleneck principle. This principle optimizes the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the self-supervised information-theoretic learning problem. This framework includes multiple encoders and decoders, suggesting that all existing work on self-supervised learning can be seen as specific instances. We aim to unify these approaches to understand their underlying principles better and address the main challenge: many works present different frameworks with differing theories that may seem contradictory. By weaving existing research into a cohesive narrative, we delve into contemporary self-supervised methodologies, spotlight potential research areas, and highlight inherent challenges. Moreover, we discuss how to estimate information-theoretic quantities and their associated empirical problems. Overall, this paper provides a comprehensive review of the intersection of information theory, self-supervised learning, and deep neural networks, aiming for a better understanding through our proposed unified approach. [ABSTRACT FROM AUTHOR]

Domain generalization (DG) techniques strive to attain the ability to generalize to an unfamiliar target domain solely based on training data originating from the source domains. Despite the increasing attention given to learning from multiple training domains through the application of various forms of invariance across those domains, the enhancements observed in comparison to ERM are nearly insignificant under specified evaluation rules. In this paper, we demonstrate that the disentanglement of spurious and invariant features is a challenging task in conventional training since ERM simply minimizes the loss and does not exploit invariance among domains. To address this issue, we introduce an effective method called specific domain training (SDT), which detects the spurious features and makes them more discernible. By exploiting a masking strategy and weight averaging, it decreases their harmful effects. We provide theoretical and experimental evidence to show the effectiveness of SDT for out-of-distribution generalization. Notably, SDT achieves comparable results to SWAD, the state of the art in DomainBed benchmarks. [ABSTRACT FROM AUTHOR]

Connections between circular RNAs (circRNAs) and microRNAs (miRNAs) assume a pivotal position in the onset, evolution, diagnosis and treatment of diseases and tumors. Selecting the most potential circRNA-related miRNAs and taking advantage of them as the biological markers or drug targets could be conducive to dealing with complex human diseases through preventive strategies, diagnostic procedures and therapeutic approaches. Compared to traditional biological experiments, leveraging computational models to integrate diverse biological data in order to infer potential associations proves to be a more efficient and cost-effective approach. This paper developed a model of Convolutional Autoencoder for CircRNA–MiRNA Associations (CA-CMA) prediction. Initially, this model merged the natural language characteristics of the circRNA and miRNA sequence with the features of circRNA–miRNA interactions. Subsequently, it utilized all circRNA–miRNA pairs to construct a molecular association network, which was then fine-tuned by labeled samples to optimize the network parameters. Finally, the prediction outcome is obtained by utilizing the deep neural networks classifier. This model innovatively combines the likelihood objective that preserves the neighborhood through optimization, to learn the continuous feature representation of words and preserve the spatial information of two-dimensional signals. During the process of 5-fold cross-validation, CA-CMA exhibited exceptional performance compared to numerous prior computational approaches, as evidenced by its mean area under the receiver operating characteristic curve of 0.9138 and a minimal SD of 0.0024. Furthermore, recent literature has confirmed the accuracy of 25 out of the top 30 circRNA–miRNA pairs identified with the highest CA-CMA scores during case studies. The results of these experiments highlight the robustness and versatility of our model. [ABSTRACT FROM AUTHOR]

LiDAR has high accuracy and resolution and is widely used in various fields. In particular, phase-modulated continuous-wave (PhMCW) LiDAR has merits such as low power, high precision, and no need for laser frequency modulation. However, with decreasing signal-to-noise ratio (SNR), the noise on the signal waveform becomes so severe that the current methods to extract the time-of-flight are no longer feasible. In this paper, a novel method that uses deep neural networks to measure the pulse width is proposed. The effects of distance resolution and SNR on the performance are explored. Recognition accuracy reaches 81.4% at a 0.1 m distance resolution and the SNR is as low as 2. We simulate a scene that contains a vehicle, a tree, a house, and a background located up to 6 m away. The reconstructed point cloud has good fidelity, the object contours are clear, and the features are restored. More precisely, the three distances are 4.73 cm, 6.00 cm, and 7.19 cm, respectively, showing that the performance of the proposed method is excellent. To the best of our knowledge, this is the first work that employs a neural network to directly process LiDAR signals and to extract their time-of-flight. [ABSTRACT FROM AUTHOR]

Recently, leveraging deep neural networks for automated colorectal polyp segmentation has emerged as a hot topic due to the favored advantages in evading the limitations of visual inspection, e.g., overwork and subjectivity. However, most existing methods do not pay enough attention to the uncertain areas of colonoscopy images and often provide unsatisfactory segmentation performance. In this paper, we propose a novel boundary uncertainty aware network (BUNet) for precise and robust colorectal polyp segmentation. Specifically, considering that polyps vary greatly in size and shape, we first adopt a pyramid vision transformer encoder to learn multi-scale feature representations. Then, a simple yet effective boundary exploration module (BEM) is proposed to explore boundary cues from the low-level features. To make the network focus on the ambiguous area where the prediction score is biased to neither the foreground nor the background, we further introduce a boundary uncertainty aware module (BUM) that explores error-prone regions from the high-level features with the assistance of boundary cues provided by the BEM. Through the top-down hybrid deep supervision, our BUNet implements coarse-to-fine polyp segmentation and finally localizes polyp regions precisely. Extensive experiments on five public datasets show that BUNet is superior to thirteen competing methods in terms of both effectiveness and generalization ability. • A boundary uncertainty aware network is proposed for accurate polyp segmentation. • A boundary exploration module is proposed to explore boundary cues of polyps. • A boundary uncertainty aware module is proposed to seek error-prone regions. • The proposed network achieves high performance on six public datasets. [ABSTRACT FROM AUTHOR]

Neural network ensembling is a common and robust way to increase model efficiency. In this paper, we propose a new neural network ensemble algorithm based on Audibert's empirical star algorithm. We provide optimal theoretical minimax bound on the excess squared risk. Additionally, we empirically study this algorithm on regression and classification tasks and compare it to most popular ensembling methods. [ABSTRACT FROM AUTHOR]

Since the discovery of the quasicrystal, approximately 100 stable quasicrystals are identified. To date, the existence of quasicrystals is verified using transmission electron microscopy; however, this technique requires significantly more elaboration than rapid and automatic powder X‐ray diffraction. Therefore, to facilitate the search for novel quasicrystals, developing a rapid technique for phase‐identification from powder diffraction patterns is desirable. This paper reports the identification of a new Al–Si–Ru quasicrystal using deep learning technologies from multiphase powder patterns, from which it is difficult to discriminate the presence of quasicrystalline phases even for well‐trained human experts. Deep neural networks trained with artificially generated multiphase powder patterns determine the presence of quasicrystals with an accuracy >92% from actual powder patterns. Specifically, 440 powder patterns are screened using the trained classifier, from which the Al–Si–Ru quasicrystal is identified. This study demonstrates an excellent potential of deep learning to identify an unknown phase of a targeted structure from powder patterns even when existing in a multiphase sample. [ABSTRACT FROM AUTHOR]

The optimal experimental design (OED) provides informative experimental resources and reduces the inherent uncertainty of the experiment. The OED is a framework to maximize performed experiments by controlling design variables. This paper proposes the OED framework combined with deep neural network (DNN) surrogate model to find the optimal design over large and complex design space. The OED has the optimization process to maximize the conditional mutual information (CMI) between hidden properties and composite structural performances. Approximate coordinate exchange (ACE) algorithm was applied to find the optimal design over a high and complicated design space. For fast optimization, surrogate DNN model was used instead of finite element analysis. Two examples were executed for the demonstration of the OED frameworks. From the results of each demonstration, OED provided the results that are consistent with heuristic knowledge. [ABSTRACT FROM AUTHOR]

Soybean (Glycine max (L.) Merr.) is a major source of oil and protein for human food and animal feed; however, soybean crops face diverse factors causing damage, including pathogen infections, environmental shifts, poor fertilization, and incorrect pesticide use, leading to reduced yields. Identifying the level of leaf damage aids yield projections, pesticide, and fertilizer decisions. Deep learning models (DLMs) and neural networks mastering tasks from abundant data have been used for binary healthy/unhealthy leaf classification. However, no DLM predicts and categorizes soybean leaf damage severity (five levels) for tailored pesticide use and yield forecasts. This paper introduces a novel DLM for accurate damage prediction and classification, trained on 2930 near-field soybean leaf images. The model quantifies damage severity, distinguishing healthy/unhealthy leaves and offering a comprehensive solution. Performance metrics include accuracy, precision, recall, and F1-score. This research presents a robust DLM for soybean damage assessment, supporting informed agricultural decisions based on specific damage levels and enhancing crop management and productivity. [ABSTRACT FROM AUTHOR]

Deep neural networks have revolutionized many fields, including image processing, inverse problems, text mining and more recently, give very promising results in systems and control. Neural networks with hidden layers have a strong potential as an approximation framework of predictive control laws as they usually yield better approximation quality and smaller memory requirements than existing explicit (multi-parametric) approaches. In this paper, we first show that neural networks with HardTanh activation functions can exactly represent predictive control laws of linear time-invariant systems. We derive theoretical bounds on the minimum number of hidden layers and neurons that a HardTanh neural network should have to exactly represent a given predictive control law. The choice of HardTanh deep neural networks is particularly suited for linear predictive control laws as they usually require less hidden layers and neurons than deep neural networks with ReLU units for representing exactly continuous piecewise affine (or equivalently min–max) maps. In the second part of the paper we bring the physics of the model and standard optimization techniques into the architecture design, in order to eliminate the disadvantages of the black-box HardTanh learning. More specifically, we design trainable unfolded HardTanh deep architectures for learning linear predictive control laws based on two standard iterative optimization algorithms, i.e., projected gradient descent and accelerated projected gradient descent. We also study the performance of the proposed HardTanh type deep neural networks on a linear model predictive control application. [ABSTRACT FROM AUTHOR]

Recently, it was found that deep neural networks (DNNs) are susceptible to adversarial input perturbations. Most defense strategies adopt the denoising method based on preprocessing, which mitigates the impacts of adversarial perturbations on DNNs by learning the distributions of nonadversarial datasets and projecting adversarial inputs into the learned nonadversarial manifolds. However, existing defense strategies commonly focus on reconstructing clean images while ignoring the role of adversarial perturbations, which results in the reconstructed images failing to achieve the visual quality and classification accuracy of the original clean images, and the induced adversarial robustness improvement is limited. This paper proposes a feature decoupling-interaction network (FDIN), which introduces the concepts of clean features and adversarial features to separate the two kinds of features from the input adversarial examples (AEs) in a feature decoupling-interaction manner. The clean features are used to reconstruct the input image so that it is infinitely close to the original clean image, and the adversarial features are used to reconstruct the adversarial perturbations. Adversarial perturbations are removed from the adversarial examples across multiple cross cycles to improve further the reconstructed image's visual quality and classification accuracy. The features of the original clean image are used as prior knowledge to guide the network to learn the clean features of the adversarial examples and improve the classification accuracy of the model on the clean examples. In addition, a classification loss function based on the Carlini & Wagner (CW) attack algorithm is used instead of the conventional cross-entropy loss function to improve the adversarial robustness of the FDIN. The experimental results show that the proposed method achieves better defense performance than the current state-of-the-art methods on both standard tests and various attack tests and even exceeds the test accuracy of the target classifier on the original test set. • This paper proposes a feature decoupling-interaction network (FDIN). • We introduces the concepts of clean features and adversarial features. • The FDIN separates the clean features and adversarial features. • We use the CW loss instead of the CE loss to improve the adversarial robustness. • The experimental results show that FDIN achieves better defense performance. [ABSTRACT FROM AUTHOR]

Marine ecosystems and their fish habitats are becoming increasingly important due to their integral role in providing a valuable food source and conservation outcomes. Due to their remote and difficult to access nature, marine environments and fish habitats are often monitored using underwater cameras to record videos and images for understanding fish life and ecology, as well as for preserve the environment. There are currently many permanent underwater camera systems deployed at different places around the globe. In addition, there exists numerous studies that use temporary cameras to survey fish habitats. These cameras generate a massive volume of digital data, which cannot be efficiently analysed by current manual processing methods, which involve a human observer. Deep Learning (DL) is a cutting-edge Artificial Intelligence (AI) technology that has demonstrated unprecedented performance in analysing visual data. Despite its application to a myriad of domains, its use in underwater fish habitat monitoring remains under explored. In this paper, we provide a tutorial that covers the key concepts of DL, which help the reader grasp a high-level understanding of how DL works. The tutorial also explains a step-by-step procedure on how DL algorithms should be developed for challenging applications such as underwater fish monitoring. In addition, we provide a comprehensive survey of key deep learning techniques for fish habitat monitoring including classification, counting, localisation, and segmentation. Furthermore, we survey publicly available underwater fish datasets, and compare various DL techniques in the underwater fish monitoring domains. We also discuss some challenges and opportunities in the emerging field of deep learning for fish habitat processing. This paper is written to serve as a tutorial for marine scientists who would like to grasp a high-level understanding of DL, develop it for their applications by following our step-by-step tutorial, and see how it is evolving to facilitate their research efforts. At the same time, it is suitable for computer scientists who would like to survey state-of-the-art DL-based methodologies for fish habitat monitoring. • Marine environments and fish habitats are widely surveyed using underwater cameras. • Deep learning (DL) offers an effective solution for processing underwater videos. • We write a tutorial on the key concepts and steps of DL for underwater monitoring. • We provide a survey of key DL-based visual fish monitoring methods/datasets. • We shed light on challenges/opportunities of DL for underwater fish monitoring. [ABSTRACT FROM AUTHOR]

In this paper, we are interested in the acceleration of numerical simulations. We focus on a hypersonic planetary reentry problem whose simulation involves coupling fluid dynamics and chemical reactions. Simulating chemical reactions takes most of the computational time but, on the other hand, cannot be avoided to obtain accurate predictions. We face a trade-off between cost-efficiency and accuracy: the numerical scheme has to be sufficiently efficient to be used in an operational context but accurate enough to predict the phenomenon faithfully. To tackle this trade-off, we design a hybrid numerical scheme coupling a traditional fluid dynamic solver with a neural network approximating the chemical reactions. We rely on their power in terms of accuracy and dimension reduction when applied in a big data context and on their efficiency stemming from their matrix-vector structure to achieve important acceleration factors (×10 to ×18.6). This paper aims to explain how we design such cost-effective hybrid numerical schemes in practice. Above all, we describe methodologies to ensure accuracy guarantees, allowing us to go beyond traditional surrogate modeling and to use these schemes as references. • Deep Learning-based hybridization speeds up numerical schemes of atmospheric reentry while maintaining high accuracy. • Initializing a scheme with a hybrid code's prediction reduces the convergence time and keeps the exact same guarantees. • Uncertainty analysis provides statistical guarantees concerning approximation errors when using hybridization code. • Neural network approximation error is statistically lower than many other sources of error inherent to numerical simulations. [ABSTRACT FROM AUTHOR]

Due to the inevitable light absorption and scattering, underwater images always suffer from severe quality degradation, leading to significant performance decline for various maritime engineering-related applications. In recent years, deep neural networks (DNNs) have been proven to achieve high-quality enhancement of underwater images, which aims to extract and learn abstract features to exhibit superior performance. However, existing solutions often stack multilayer processing units to enrich features, which not only significantly burdens computational load but also encounters difficulties in fully utilizing and interacting between features at different depths. To address the above problems, this paper presents a cost-effective underwater image enhancement network via cascaded feature extraction, termed CFENet. Specifically, we develop a cost-effective cascaded structure for sufficient feature extraction and multiscale establishment of pixel-level long-range dependencies. In addition, we construct a dual-branch structure for effective feature fusion, so that the detailed texture and semantic information in the image can be simultaneously enhanced. Extensive experiments reveal the superiority of our proposed CFENet in both underwater image enhancement effects and computational complexity. Sufficient ablation study is conducted to demonstrate the effectiveness of each component in the network. [ABSTRACT FROM AUTHOR]

Digital image watermarking technology has made a significant contribution to the copyright protection of digital images. In recent years, researchers have focused on designing various watermarking algorithms to enhance resistance against different forms of attacks. However, the evolution of watermarking attack technology has been sluggish, thus impeding possible advancements in digital copyright protection. Existing watermarking attack methods exhibit a notable drawback, causing substantial deterioration in visual quality and undermining the practical utility of attacked images. In this paper, we propose a high imperceptibility watermarking attack network, named HIWANet, based on deep neural networks. To enhance the watermarking attack ability, a feature extraction module (FEM) is aimed to better capture watermark information features, and a watermarking attack module (WAM) is constructed to learn high-level abstract features of the images. In addition, to ensure the imperceptibility of the watermarking attack, an asymmetric loss function is designed to maintain the quality of the attacked watermarked image. In the experiments, we randomly select 2000 color images from the PASCAL VOC2012 database as the dataset, with the training and test sets containing 1000 distinct images. Compared to traditional watermarking attack methods, our HIWANet achieves a significant increase in bit error rate (improved by 242%), indicating a higher attack ability. Meanwhile, it brings more than 26% improvement in the attack imperceptibility. Furthermore, our HIWANet also offers significant advantages compared to deep learning-based watermarking attack methods. [ABSTRACT FROM AUTHOR]

Segmentation is an important task in a wide range of computer vision applications, including medical image analysis. Recent years have seen an increase in the complexity of medical image segmentation approaches based on sophisticated convolutional neural network architectures. This progress has led to incremental enhancements in performance on widely recognised benchmark datasets. However, most of the existing approaches are computationally demanding, which limits their practical applicability. This paper presents an expand-squeeze dual multiscale residual network (ESDMR-Net), which is a full y convolutional network that is particularly well-suited for resource-constrained computing hardware such as mobile devices. ESDMR-Net focusses on extracting multiscale features, enabling the learning of contextual dependencies among semantically distinct features. The ESDMR-Net architecture allows dual-stream information flow within encoder–decoder pairs. The expansion operation (depthwise separable convolution) makes all of the rich features with multiscale information available to the squeeze operation (bottleneck layer), which then extracts the necessary information for the segmentation task. The Expand-Squeeze (ES) block helps the network pay more attention to under-represented classes, which contributes to improved segmentation accuracy. To enhance the flow of information across multiple resolutions or scales, we integrated dual multiscale residual (DMR) blocks into the skip connection. This integration enables the decoder to access features from various levels of abstraction, ultimately resulting in more comprehensive feature representations. We present experiments on seven datasets from five distinct examples of applications: segmentation of retinal vessels (2 ×), skin lesions (2 ×), digestive tract polyps, lung regions, and cells. Our model demonstrates strong performance, with an F1 score of 0.8287%, 0.8211%, 0.9034%, 0.9451%, 0.9543%, 0.9840%, and 0.8424% on the DRIVE, CHASE, ISIC2017, ISIC2016, CVC-ClinicDB, MC and MoNuSeg datasets, respectively. Remarkably, our model achieves these results despite having significantly fewer trainable parameters, with a reduction of two or even three orders of magnitude. • Multiscale feature extraction to understand contextual dependencies. • The expand-squeeze operation facilitates multiscale information extraction. • Dual-stream information flow within encoder–decoder pairs. • The use of DMR blocks in skip connections improves the information flow. • The deep structure of the network allows it to learn more features. [ABSTRACT FROM AUTHOR]

In weakly coupled few mode fibers (FMFs), used in optical communication systems to increase data capacity, the difference between the effective refractive indices of successive spatial modes is a critical parameter. In this paper, we demonstrate that nanostructured FMFs composed of two different types of subwavelength glass rods can be used to effectively control the propagation properties of individual spatial modes. However, the design and optimization of such fibers proves challenging due to the large number of parameters associated with their free-form refractive index distribution. To address this problem, we use an optimization method based on generative inverse design networks (GIDNs) modified to account for the optical properties of weakly coupled FMFs. Our results show that a nanostructured FMF with a core containing multiple angularly modulated concentric rings can achieve differences between the effective refractive indices of 10 spatial modes exceeding 1.2 × 10−3, surpassing other forms of weakly coupled FMFs. Importantly, the parameters of the structure ensure the feasibility of fabricating the designed fibers. Compared to traditional FMF optimization using a deep neural network with a large, randomly generated data set, our method achieves comparable accuracy in predicting refractive index differences, while achieving better spatial mode separation with a six times smaller data set. By using our proposed approach, which could potentially be applied to the design of other types of optical fibers characterized by a large number of parameters, we can significantly reduce the number of training examples required and the time needed to achieve high-quality results. [ABSTRACT FROM AUTHOR]

This paper proposes a mesh-free computational framework and machine learning theory for solving elliptic PDEs on unknown manifolds, identified with point clouds, based on diffusion maps (DM) and deep learning. The PDE solver is formulated as a supervised learning task to solve a least-squares regression problem that imposes an algebraic equation approximating a PDE (and boundary conditions if applicable). This algebraic equation involves a graph-Laplacian type matrix obtained via DM asymptotic expansion, which is a consistent estimator of second-order elliptic differential operators. The resulting numerical method is to solve a highly non-convex empirical risk minimization problem subjected to a solution from a hypothesis space of neural networks (NNs). In a well-posed elliptic PDE setting, when the hypothesis space consists of neural networks with either infinite width or depth, we show that the global minimizer of the empirical loss function is a consistent solution in the limit of large training data. When the hypothesis space is a two-layer neural network, we show that for a sufficiently large width, gradient descent can identify a global minimizer of the empirical loss function. Supporting numerical examples demonstrate the convergence of the solutions, ranging from simple manifolds with low and high co-dimensions, to rough surfaces with and without boundaries. We also show that the proposed NN solver can robustly generalize the PDE solution on new data points with generalization errors that are almost identical to the training errors, superseding a Nyström-based interpolation method. [ABSTRACT FROM AUTHOR]

Deep Neural Network (DNN) models offer an attractive alternative to existing acoustic source imaging techniques, such as acoustic beamforming, due to their ever-growing potential with increasing computational power. Source resolution of acoustic beamforming methods is limited at lower frequencies and their source maps may possess sidelobes at higher frequencies. However, acoustic beamforming methods are typically robust over a wide range of simulation and experimental conditions, such as (i) the number of sources present, (ii) source frequency and (ii) extraneous noise sources. The performance of DNN models, when these conditions are varied from their specific design criteria, is yet to be investigated and much work is needed in this area before DNN models can be utilized in experiments, such as wind tunnel tests. Furthermore, few studies have been conducted on experimental validation of DNN models, predominately due to the difficulty of large sets of experimentally obtained data needed for DNN model training and the sensitivity of DNN model performance when any of the aforementioned experimental conditions are varied. In this paper, a series of studies on the robustness of DNN models based on numerical data and experimental data are presented. Numerical DNN (NDNN) models are trained using in-phase and random-phase pressure data generated from six sources over design frequencies from 500 Hz to 20,000 Hz. The robustness of the NDNN models is tested via (1) inclusion of extraneous Gaussian white noise, (2) inclusion of extraneous tonal noise near the design frequency, (3) using source frequencies that slightly differ from the design frequencies and (4) using a number of sources that differs from the design source number. DNN model performance metrics are introduced that present a promising framework for future DNN model studies and bridging the gap between NDNN and experimentally trained DNN models. A preliminary experimental validation was conducted using a single speaker that was systematically placed over a speaker grid to generate training data via acoustic superposition, from which an experimentally trained DNN (EDNN) model was produced. The EDNN model yields exceptional noise source localization capability of the DNN model, revealing a promising start for a more sophisticated EDNN model. • Robustness of a numerical DNN model is analyzed for conditions away from its design criteria. • Various conditions are tested, e.g., source number, source frequency, extraneous noise sources. • The numerical DNN model is trained using both in-phase and random phase sources. • Experimental validation is conducted through an experimentally trained DNN model. • The experimental DNN model yields exceptional noise source localization capability. [ABSTRACT FROM AUTHOR]

Active learning alleviates labeling costs by selecting and labeling the most informative examples from an unlabeled pool. However, most existing active learning approaches estimate informativeness with uncalibrated confidence, resulting in unreliable informativeness estimation. These approaches generally ignored two significant issues caused by uncalibrated confidence methods. Firstly, the average uncalibrated confidence generated by modern neural networks is usually higher than the accuracy. Secondly, examples located near the decision boundaries are unstable during prediction when the target model updates parameters in the last several epochs, even throughout the training process. This phenomenon, caused by the forgetting characteristic of neural networks, has a significant impact on some specific models that estimate the informativeness by predicted probability vectors or pseudo labels. To address these issues, in this paper, we propose a novel active learning approach to reliably estimate informativeness with calibrated confidence. Specifically, we integrate the intermediate predictions for each unlabeled example, generated by the target model during the training process, to generate calibrated confidence. The calibrated confidence can capture a tendentious label from an indecisive subset of the class space. We show that the calibrated confidence with tendentiousness can maintain the ability of correct predictions. The empirical results demonstrate that our approach outperforms the state-of-the-art active learning methods on image classification tasks. • The active learning algorithm selects unstable examples by calibrated confidence.. • The confidence calibration method only requires training a single model. • Extensive experiments show the competitive performance of the proposed approach. [ABSTRACT FROM AUTHOR]

Accurate diagnosis of Lithium-ion batteries (Li-ion batteries) degradation plays a critical role in improving the maintenance of energy storage technology. This paper presents a method based on a novel deep network model combined with a data transformation technique to diagnose Li-ion battery degradation modes. Different from conventional studies based on specific experimental and numerical methods to estimate and predict the degradation, the proposed method is based on data-driven approach, by leveraging datasets consisting of voltage/capacity curves, these were converted into incremental capacity (IC) curves and then transformed into images using the gramian angular summation field (GASF) technique. The study adopted two models: Inception-v3 and the proposed model, both underwent fine-tuning and a subsequent transfer learning process. Degradation modes, namely loss of lithium inventory (LLI) and the loss of active materials in both the positive (LAMPE) and negative electrodes (LAMNE), were diagnosed in relation to IC curves. Finally, the model was tested using two different datasets, and the results showed that the proposed method achieved high performance, especially across three Li-ion batteries, three degradation modes, three cells, and various cycles (totaling 378 cases) the proposed method outperformed in 233 cases, thereby outperforming other methods in comparison. Our method provides a flexible data-driven approach that accurately predicts various degradation modes across different cell chemistries throughout their lifespan. [ABSTRACT FROM AUTHOR]

• The proposed method had fused multi-level information compensation and U-Net. • The module designed for multi-level information compensation and compression. • The method had fused compressed features and correlation features. Recently, Deep Convolutional Neural Networks have demonstrated high-quality reconstruction in image super-resolution procedure. In this paper, we propose improved image super-resolution reconstruction via multi-level information compensation and U-Net network to address the problem that the image super-resolution reconstruction algorithm based on deep neural networks tends to lose feature information in the feature extraction process, resulting in the lack of texture and edge details in the reconstructed image. Firstly, we design the U-net like network for image super-resolution reconstruction, which performs multi-level feature extraction and channel compression for the input features through the down-channel branch. It fuses the compressed features and extracts the correlation features of different channels through the bottom module, and performs multi-level feature extraction and channel recovery for the compressed correlation features through the up-channel branch. The multi-level information compensation model is then designed to compensate for the information lost in the channel compression process and the information that is difficult to recover in the channel recovery process of U-net like network. The experimental results can show that the proposed algorithm achieves a significant improvement in Peak Signal-to-Noise Ratio and Structure Similarity Index and visual effect compared with the state-of-arts algorithms. The average experimental results of PSNR from proposed method had improved by 1.63 dB, 1.53 dB, 0.97 dB and 0.94 dB compared to SRCNN, HAT, DAT and CARN, respectively. [ABSTRACT FROM AUTHOR]

We present a numerical framework for learning unknown stochastic dynamical systems using measurement data. Termed stochastic flow map learning (sFML), the new framework is an extension of flow map learning (FML) that was developed for learning deterministic dynamical systems. For learning stochastic systems, we define a stochastic flow map that is a superposition of two sub-flow maps: a deterministic sub-map and a stochastic sub-map. The stochastic training data are used to construct the deterministic sub-map first, followed by the stochastic sub-map. The deterministic sub-map takes the form of residual network (ResNet), similar to the work of FML for deterministic systems. For the stochastic sub-map, we employ a generative model, particularly generative adversarial networks (GANs) in this paper. The final constructed stochastic flow map then defines a stochastic evolution model that is a weak approximation, in term of distribution, of the unknown stochastic system. A comprehensive set of numerical examples are presented to demonstrate the flexibility and effectiveness of the proposed sFML method for various types of stochastic systems. • Proposed a new numerical method for modeling SDEs with observation data. • Established a mathematical framework that utilizes stochastic flow map for effective modeling. • Developed novel stochastic flow map modeling and its DNN structure. [ABSTRACT FROM AUTHOR]

Adversarial training has become the mainstream method to boost adversarial robustness of deep models. However, it often suffers from the trade-off dilemma, where the use of adversarial examples hurts the standard generalization of models on natural data. To study this phenomenon, we investigate it from the perspective of spatial attention. In brief, standard training typically encourages a model to conduct a comprehensive check to input space. But adversarial training often causes a model to overly concentrate on sparse spatial regions. This reduced tendency is beneficial to avoid adversarial accumulation but easily makes the model ignore abundant discriminative information, thereby resulting in weak generalization. To address this issue, this paper introduces an Attention-Enhanced Learning Framework (AELF) for robustness training. The main idea is to enable the model to inherit the attention pattern of standard pre-trained model through an embedding-level regularization. To be specific, given a teacher model built on natural examples, the embedding distribution of teacher model is used as a static constraint to regulate the embedding outputs of the objective model. This design is mainly supported with that the embedding feature of standard model is usually recognized as a rich semantic integration of input. For implementation, we present a simplified AELFs that can achieve the regularization with single cross entropy loss via the parameter initialization and parameter update strategy. This avoids the extra consistency comparison operation between embedding vectors. Experimental observations verify the rationality of our argument, and experimental results demonstrate that it can achieve remarkable improvements in generalization under the high-level robustness. [ABSTRACT FROM AUTHOR]

The rapid development of Deep Neural Networks (DNNs) lays solid foundations for Internet of Things systems. However, mobile devices with limited processing capacity and short battery life confront the difficulties of executing complex DNNs. To satisfy different Quality of Service requirements, a feasible solution is offloading DNN layers to edge nodes and the cloud. The energy-efficient offloading problem for DNN-based applications with the deadline and budget constraints in the edge-cloud environment is still an open and challenging issue. To this end, this paper proposes a Hybrid Chaotic Evolutionary Algorithm (HCEA) incorporating diversification and intensification strategies and a DVFS-enabled version of it (HCEA-DVFS). The Archimedes Optimization Algorithm-based diversification strategy exploits global and local guiding information to improve population diversity during the updating process and employs Metropolis acceptance rule of Simulated Annealing to avoid premature convergence. The Genetic Algorithm-based chaotic intensification strategy is designed to enhance the local search capability of HCEA. Moreover, the Dynamic Voltage Frequency Scaling-enabled adjustment strategies can be embedded into HCEA to further reduce energy consumption by resetting frequency levels and reallocating DNN layers. Experimental results over four DNN-based applications demonstrate that HCEA-DVFS can reduce more energy consumption under different deadlines, budgets, and workloads on average by 7.93, 9.68, 11.02, 11.84, and 19.38 percent in comparison with HCEA, PSO-GA, MCEA, AOA, and Greedy, respectively. • An AOA-based diversification strategy is designed to exploit global and local guiding information for population updating. • A chaotic GA-based intensification strategy is developed to enhance local search capability and introduce randomness. • A DVFS-enabled adjustment strategy is presented to save energy by resetting frequency levels and reallocating DNN layers. • HCEA and HCEA-DVFS outperform baselines in energy consumption optimization under budget and deadline constraints. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU. [ABSTRACT FROM AUTHOR]

The data fusion systems between multiple modals have attracted a wide range of concerns about their adversarial robustness. Recent image captioning attacks lack cross-structure transferability against models with diverse structures. Therefore, the attackers demonstrate satisfactory performance only when they gain full or partial knowledge of target image captioning models. So far, the cross-structure transferability of multi-modal adversarial examples has not been thoroughly investigated. In this paper, a theorem is proposed to analyze the upper bound captioning adversarial transferability. We design a new transfer-based adversarial attack method (Cascade & Allocate) against image captioning models. Our method can be divided into two steps. The first step randomly selects a set of candidate models with diverse structures, and we use a momentum-like strategy to generate perturbations. This 'model cascade' step narrows the gap of the gradient directions between different image captioning models, therefore enhancing the cross-model transferability and generating model-agnostic adversarial perturbations. The second step utilizes the upper bound transferability theorem to allocate the weight of perturbations per model. This 'weight allocate' step enhances the weight of model with the most consistent gradient in all candidate models, which generates more transferable adversarial examples. In the experiments, we compare our approach with other captioning and ensemble-based attacks against five black-box models with different structures on MS COCO and Flickr-30k datasets. The adversarial examples exhibit state-of-the-art adversarial transferability against five black-box models with different structures. The results demonstrate that our approach is practical and generalizes well to a wide range of captioning models. • A cross-structure black-box adversarial attack against image caption models. • A theorem to measure the upper bound of captioning adversarial transferability. • A weight allocation strategy to enhance adversarial transferability. [ABSTRACT FROM AUTHOR]

The rapid growth of online music education has led to increased security risks from cyber intrusions. This paper proposes public cloud networks oriented deep neural networks for effective intrusion detection in online music education environments. Specifically, a novel intrusion detection framework is developed, comprising fuzzy logic based feature selection, chronological salp swarm algorithm optimized deep belief networks, and gated recurrent unit integrated convolutional neural networks. Detailed methodologies are presented for the fuzzy logic system, chronological salp optimization, deep belief network architecture, and convolutional neural networks. Comprehensive experiments are conducted on the NSL-KDD and CICIDS2017 datasets. Various deep neural networks are evaluated and compared, including multi-layer perceptrons, convolutional neural networks, deep belief networks, recurrent neural networks, and the proposed models. Experimental results demonstrate that the proposed fuzzy feature selection and chronological salp swarm algorithm optimized deep belief network achieves a test accuracy of 97.33 %, outperforming other peer models. The gated recurrent unit integrated convolutional neural network obtains a test accuracy of 98.46 %, superior to state-of-the-art methods. While the experiments on the newly created dataset for intrusion detection in cloud-based online music education demonstrate that the proposed models outperform the benchmarks. The experiments verify the effectiveness of the proposed deep learning frameworks for intrusion detection in online music education cloud networks. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) are vulnerable to the attacks of adversarial examples, which bring serious security risks to the learning systems. In this paper, we propose a new defense method to improve the adversarial robustness of DNNs based on stochastic neural networks (SNNs), termed as Margin-SNN. The proposed Margin-SNN mainly includes two modules, i.e., feature uncertainty learning module and label embedding module. The first module introduces uncertainty to the latent feature space by giving each sample a distributional representation rather than a fixed point representation, and leverages the advantages of variational information bottleneck method in achieving good intra-class compactness in latent space. The second module develops a label embedding mechanism to take advantage of the semantic information underlying the labels, which maps the labels into the same latent space with the features, in order to capture the similarity between sample and its class centroid, where a penalty term is equipped to elegantly enlarge the margin between different classes for better inter-class separability. Since no adversarial information is introduced, the proposed model can be learned in standard training to improve adversarial robustness, which is much more efficient than adversarial training. Extensive experiments on data sets MNIST, FASHION MNIST, CIFAR10, CIFAR100 and SVHN demonstrate superior defensive ability of the proposed method. Our code is available at https://github.com/humeng24/Margin-SNN. [ABSTRACT FROM AUTHOR]

Detecting ship wakes is essential for locating moving vessels at sea. Of the various wake types, Kelvin wakes are particularly intriguing because of the vital information they convey about ships. However, identifying Kelvin wakes is challenging due to their expansive planar distributions and their variable brightness and forms. This paper introduces a deep neural network-based technique specifically tailored for detecting Kelvin wakes in large-scale, high-resolution optical images. After distinguishing between land and water, the entire water region of the image was segmented into overlapping sub-images. GoogLeNet was then employed to differentiate between Kelvin wakes and natural sea surfaces within each sub-image. Regions exhibiting Kelvin wakes were subsequently identified by combining the wake-classified sub-images. Given the limited diversity of available Kelvin wake samples, the training dataset merged true and simulated Kelvin wake images, which acted as positive samples for the deep neural network. The proposed method, when applied to high-resolution optical images, showcased outstanding Kelvin wake detection capabilities, achieving a recall rate of 94.0% and a precision of 70.8%. When detection was limited to the vicinity of ship hulls, the recall, precision, overall accuracy, and specificity achieved remarkable rates of 94.0%, 70.8%, 92.3%, and 94.1% respectively. Furthermore, this research delved into the influence of training samples and input channels on the detection accuracy of wakes. Various DNNs, were utilized for comparison, including AlexNet, VGGNet with 16 layers, GoogLeNet, ResNet with 18 layers, DarkNet with 53 layers, DenseNet with 201 layers. GoogLeNet stood out due to its Inception module, which integrates multi-size convolution kernels, achieving a recall rate of 94.0% and a precision of 70.8%. When detection was limited to the vicinity of ship hulls, the recall, precision, overall accuracy, and specificity achieved remarkable rates of 94.0%, 70.8%, 92.3%, and 94.1% respectively. [Display omitted] •Kelvin wake detection method for large-scale imagery by deep learning was proposed. •The method training and running efficiency was improved with high accuracy. •Simulated Kelvin wakes effectively made up for the lack of diversity wake samples. •Recall, precision, overall accuracy, and specificity were all more than 94.0%. [ABSTRACT FROM AUTHOR]

Hardware Trojans (HT) are the most malicious components attacking modern integrated circuits (ICs). Information leakage, incorrect functionality, and overheating are the major effects of HT insertion. Due to the outsourcing of multi-stage digital circuit design and fabrication, HTs can be inserted by rogues at every stage. The types of HTs and insertion methods evolve continuously and their detection becomes more complex. Consequently, designing integrated circuits robust to HT insertion seems efficient. A prerequisite for such a strategy is the evaluation of integrated circuit vulnerability to HT insertion. There are several factors involved in the vulnerability of fabricated integrated circuits such as the distribution of white spaces in the IC layout, the ratio of unutilized routing resources, the amount of activity of the circuit's nodes, the geometrical properties of the circuit's gates, the delay of various paths, and etc. The interrelation of these diverse factors makes IC vulnerability modeling a challenging problem. In this paper, a deep neural network (DNN) based approach is developed to combine all the effective factors in IC vulnerability analysis. Firstly, we generate a comprehensive dataset containing various types and sizes of hardware Trojans inserted into benchmark circuits. The benchmark circuits used in this study are ISCAS 85, ISCAS 89, and ITC 99. Then using the transfer learning concept seven state-of-the-art deep convolutional neural networks are trained and verified using the constructed dataset to classify every integrated circuit into three vulnerability levels (low, moderate, and highly vulnerable classes). Simulation results show that our proposed method achieves almost 95 % accuracy in the most successful case (VGG16). The lowest accuracy belongs to Inception V3 (∼87.65 %) which is much better than the most successful previous studies (72 %). This is mainly because the DNN-based method handles shortcomings (such as incomplete modeling of vulnerability-related factors) in the previous state-of-the-art methods efficiently. [ABSTRACT FROM AUTHOR]

Neural network pruning is a critical field aimed at reducing the infrastructure costs of neural networks by removing parameters. Traditional methods follow a fixed paradigm including pretraining, pruning, and fine-tuning. Despite the close relationship among these three stages, most pruning methods treat them as independent processes. In this paper, we propose a novel two-stage pruning method, which includes pretraining a network that is instructive for subsequent pruning, and a unified optimization model that integrates pruning and fine-tuning. Specifically, in the first stage, we design a group sparse regularized model for pretraining. This model not only safeguards the network from irreversible damage but also offers valuable insights for the pruning process. In the second stage, we introduce an element-wise sparse regularization into pruning model. This model enables us to pinpoint sparse weights more precisely than pretrained network. It automatically derives effective pruning criteria, and omits the step of fine-tuning. To implement the two-stage process in practice, we utilize stochastic gradient algorithm for the pretraining and design a threshold algorithm for pruning stage. Extensive experiments confirm the competitive performance of our proposed method in terms of both accuracy and memory cost when compared to various benchmarks. Furthermore, ablation experiments validate the effectiveness of the proposed pretraining model's guidance for the pruning process. • We propose SOGP to enhance the connection between pretraining and pruning. • We propose a novel non-convex group sparse regularization GT l 1 . • We construct a unified optimization model to integrate pruning and fine-tuning. • Extensive experiments validate the good pruning and accuracy performance of SOGP. [ABSTRACT FROM AUTHOR]

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L 1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%. [ABSTRACT FROM AUTHOR]

The metasurface-based splitters have demonstrated great potential in high-resolution, dark-scene, and ultra-fast imaging due to their excellent capabilities of preserving light intensity, attracting increasingly more research interests. This paper investigates the splitter pattern design, which determines the theoretical performance limit of metasurface-based color imaging systems, to guide their further development. We first establish a unified mathematical model for the splitter-based imaging process. This model reveals that the raw image detected by the splitters is degraded by noise, incomplete sampling, and blurring. Based on this, we develop a deep neural network capable of simultaneously tackling multiple image processing tasks to achieve color image reconstruction. We then conduct evaluation experiments for several existing patterns, through which we establish three design principles to guide the optimization of a splitter pattern. Furthermore, by jointly learning the parameters of our proposed imaging model and reconstruction network, we achieve an automatic optimization of a splitter pattern. The experiments demonstrate that the optimized pattern can achieve better quantitative and visual results than the original pattern, validating the effectiveness of this pattern optimization method. • Establish a unified mathematical model for metasurface-based color imaging systems. • Identify three principles for designing metasureface-based color splitter patterns. • Develop a method for jointly optimizing splitter pattern and image reconstruction. [ABSTRACT FROM AUTHOR]

Robustness has become an important consideration in deep learning. With the help of explainable AI, mismatches between an explained model's decision strategy and the user's domain knowledge (e.g. Clever Hans effects) have been identified as a starting point for improving faulty models. However, it is less clear what to do when the user and the explanation agree. In this paper, we demonstrate that acceptance of explanations by the user is not a guarantee for a machine learning model to be robust against Clever Hans effects, which may remain undetected. Such hidden flaws of the model can nevertheless be mitigated, and we demonstrate this by contributing a new method, Explanation-Guided Exposure Minimization (EGEM), that preemptively prunes variations in the ML model that have not been the subject of positive explanation feedback. Experiments demonstrate that our approach leads to models that strongly reduce their reliance on hidden Clever Hans strategies, and consequently achieve higher accuracy on new data. • Explainable AI may not detect all Clever Hans effects in practical ML scenarios. • Hidden Clever Hans effects can be mitigated using Exposure Minimization. • We propose a solution in the form of a soft-pruning scheme on network activations. • We demonstrate the practical benefits of our approach on image and text data. [ABSTRACT FROM AUTHOR]

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput. • Develop an AI-based signal control framework to collaborate multiple intersections. • The counterfactual baseline can well assign the credit among multiple intersections. • The scheduler module helps agents better understand the entire environment. [ABSTRACT FROM AUTHOR]