Search

The application of deep neural network has achieved remarkable success in object detection. However, the network structures should be still evolved consistently and tuned finely to acquire better performance. This gears to the continuous demands on high performance in those complex scenes, where multi-scale objects to be detected are located here and there. To this end, this paper proposes a network structure called Multi-Scale Coupled Attention (MSCA) under the framework of self-attention learning with methodologies of importance assessment. Architecturally, it consists of a Multi-Scale Coupled Channel Attention (MSCCA) module, and a Multi-Scale Coupled Spatial Attention (MSCSA) module. Specifically, the MSCCA module is developed to achieve the goal of self-attention learning linearly on the multi-scale channels. In parallel, the MSCSA module is constructed to achieve this goal nonlinearly on the multi-scale spatial grids. The MSCCA and MSSCA modules can be connected together into a sequence, which can be used as a plugin to develop end-to-end learning models for object detection. Finally, our proposed network is compared on two public datasets with 13 classical or state-of-the-art models, including the Faster R-CNN, Cascade R-CNN, RetinaNet, SSD, PP-YOLO, YOLO v3, YOLO v5, YOLO v7, YOLOX, DETR, conditional DETR, UP-DETR and FP-DETR. Comparative experimental results with numerical scores, the ablation study, and the performance behaviour all demonstrate the effectiveness of our proposed model. [ABSTRACT FROM AUTHOR]

Water pollution is one of the most challenging environmental issues. A powerful tool for measuring the suitability of water for drinking is required. The Water Quality Index (WQI) is a widely used parameter for the assessment of water quality through mathematical formulas. In this paper, a Deep Neural Network (DNN) model is developed to forecast WQI based on parameters selected for the dry and wet seasons throughout the year. Statistical modeling and unsupervised machine learning techniques are used. These modelings include the Principal Component Analysis/Factor Analysis (PCA/FA) which is used to interpret seasonal changes and the sources of springs under study. The other modeling technique utilized in this study is the Hierarchical Cluster Analysis (HCA). The results of this study reveal that the developed DNN model has achieved a high accuracy of ***. The goodness of fit of the developed model using R-Squared (R2) is 0.98 which is deemed high. The Mean Square Error metric is close to zero. Furthermore, the PCA/FA revealed five major parameters that impact water quality which together account for 92% of the total variance of water quality in summer and 96% in winter. Moreover, results show that the average of the WQI for all springs is of poor water quality at 46.75% during the dry season and medium water quality at 55.5% during the wet season. [ABSTRACT FROM AUTHOR]

The paper describes a novel approach to automated accent identification by training a speech recogniser to distinguish between different versions of the phonemes that make up the language. In this approach, a standard speech recogniser is trained with data where the critical phonemes (vowels and a small set of consonants) are marked as coming in several varieties, one per accent. This is followed by inspection of the output of the recogniser to determine which versions predominate in a given utterance. Put simply, if a speaker produces phonemes that match Levantine versions of those phonemes they are characterised as speaking with a Levantine accent. Similarly, if they produce phonemes that match the Egyptian versions they should be characterised as speaking with an Egyptian accent, and so on. The accuracy of this approach to classifying speakers' accents varies from 79 to 86% when tested on speakers from the five main Arabic accent groups (Gulf, Iraqi, Egyptian, Levantine, Maghrebi), depending on a range of conditions discussed in the paper. These results are an improvement on the state of the art for accent recognition for Arabic, i.e. for classifying spoken, rather than written, material on the basis of the speaker's geographical origin. [ABSTRACT FROM AUTHOR]

In recent years, a number of backdoor attacks against deep neural networks (DNN) have been proposed. In this paper, we reveal that backdoor attacks are vulnerable to image compressions, as backdoor instances used to trigger backdoor attacks are usually compressed by image compression methods during data transmission. When backdoor instances are compressed, the feature of backdoor trigger will be destroyed, which could result in significant performance degradation for backdoor attacks. As a countermeasure, we propose the first compression-resistant backdoor attack method based on feature consistency training. Specifically, both backdoor images and their compressed versions are used for training, and the feature difference between backdoor images and their compressed versions are minimized through feature consistency training. As a result, the DNN treats the feature of compressed images as the feature of backdoor images in feature space. After training, the backdoor attack will be robust to image compressions. Furthermore, we consider three different image compressions (i.e., JPEG, JPEG2000, WEBP) during the feature consistency training, so that the backdoor attack can be robust to multiple image compression algorithms. Experimental results demonstrate that when the backdoor instances are compressed, the attack success rate of common backdoor attack is 6.63% (JPEG), 6.20% (JPEG2000) and 3.97% (WEBP) respectively, while the attack success rate of the proposed compression-resistant backdoor attack is 98.77% (JPEG), 97.69% (JPEG2000), and 98.93% (WEBP) respectively. The compression-resistant attack is robust under various parameters settings. In addition, extensive experiments have demonstrated that even if only one image compression method is used in the feature consistency training process, the proposed compression-resistant backdoor attack has the generalization ability to resist multiple unseen image compression methods. [ABSTRACT FROM AUTHOR]

Neural architecture search (NAS) is an extremely complex optimization task. Recently, population-based optimization algorithms, such as evolutionary algorithm, have been adopted as search strategies for designing neural networks automatically. Various population-based NAS methods are promising in searching for high-performance neural architectures. The explosion gravitation field algorithm (EGFA) inspired by the formation process of planets is a novel population-based optimization algorithm with excellent global optimization capability and remarkable efficiency, compared with the classical population-based algorithms, such as GA and PSO. Thus, this paper attempts to develop a more efficient NAS method, called EGFA-NAS, by utilizing the work mechanisms of EGFA, which relaxes the search discrete space to a continuous one and then utilizes EGFA and gradient descent to optimize the weights of the candidate architectures in conjunction. To reduce the computational cost, a training strategy by utilizing the population mechanism of EGFA-NAS is proposed. In addition, a weight inheritance strategy for the new generated dust individuals is proposed during the explosion operation to improve performance and efficiency. The performance of EGFA-NAS is investigated in two typical micro search spaces: NAS-Bench-201 and DARTS, and compared with various kinds of state-of-the-art NAS competitors. The experimental results demonstrate that EGFA-NAS is able to match or outperform the state-of-the-art NAS methods on image classification tasks with remarkable efficiency improvement. [ABSTRACT FROM AUTHOR]

Legal text retrieval serves as a key component in a wide range of legal text processing tasks such as legal question answering, legal case entailment, and statute law retrieval. The performance of legal text retrieval depends, to a large extent, on the representation of text, both query and legal documents. Based on good representations, a legal text retrieval model can effectively match the query to its relevant documents. Because legal documents often contain long articles and only some parts are relevant to queries, it is quite a challenge for existing models to represent such documents. In this paper, we study the use of attentive neural network-based text representation for statute law document retrieval. We propose a general approach using deep neural networks with attention mechanisms. Based on it, we develop two hierarchical architectures with sparse attention to represent long sentences and articles, and we name them Attentive CNN and Paraformer. The methods are evaluated on datasets of different sizes and characteristics in English, Japanese, and Vietnamese. Experimental results show that: (i) Attentive neural methods substantially outperform non-neural methods in terms of retrieval performance across datasets and languages; (ii) Pretrained transformer-based models achieve better accuracy on small datasets at the cost of high computational complexity while lighter weight Attentive CNN achieves better accuracy on large datasets; and (iii) Our proposed Paraformer outperforms state-of-the-art methods on COLIEE dataset, achieving the highest recall and F2 scores in the top-N retrieval task. [ABSTRACT FROM AUTHOR]

The time series analysis and prediction techniques are highly valued in many application fields, such as economy, medicine and biology, environmental sciences or meteorology, among others. In the last years, there is a growing interest in the sustainable and optimal management of a resource as scarce as essential: the water. Forecasting techniques for water management can be used for different time horizons from the planning of constructions that can respond to long-term needs, to the detection of anomalies in the operation of facilities or the optimization of the operation in the short and medium term. In this paper, a deep neural network is specifically designed to predict water consumption in the short-term. Results are reported using the time series of water consumption for a year and a half measured with 10-min frequency in the city of Murcia, the seventh largest city in Spain by number of inhabitants. The results are compared with K Nearest Neighbors, Random Forest, Extreme Gradient Boosting, Seasonal Autoregressive Integrated Moving Average and two persistence models as naive methods, showing the proposed deep learning model the most accurate results. [ABSTRACT FROM AUTHOR]

Recently, scene text detection based on pixel-level segmentation becomes quite popular in industry, which is suitable for real-time digital meter reading detection. However, detection results of existing methods are usually not satisfactory due to the poor quality of images captured in a complex environment. In this paper, a novel method named Retina Differentiable Binarization (RetinaDB) is proposed for digital meter reading detection, which is robust to low quality images including blur, out-of-focus, shadow and so on. Bottom-up path augmentation and fusion factor are adopted into our feature pyramid network module to enhance the robustness of segmentation network, and an adaptive binarization function is designed for a better approximation to the standard binarization function. The experimental results, obtained on ICDAR 2015 and a digital meter reading dataset, show that the proposed RetinaDB significantly outperforms existing methods in both accuracy and efficiency. In particular, on the digital meter reading dataset, our detector achieves an F1-measure of 97.61%, inferencing at 47.13 FPS when using ResNet-18 as a backbone. When using ResNet-50, our method achieves the best detection results of 98.79%, 99.58% and 98.01% in terms of F1-measure, precision and recall. [ABSTRACT FROM AUTHOR]

Hierarchical Text Classification (HTC) leverages the hierarchical structure of labels to enhance text categorization. Existing methods use a combination of text and structure encoders to generate a composite representation. However, these methods may face challenges when encoding the hierarchy and capturing the label correlations that convey information about the relationships and dependencies among the labels. To address these challenges, we introduce the Hierarchy-Aware Label Correlation (HLC) model in this paper. HLC adopts a customized Graphomer as its structure encoder to learn the hierarchy. Graphormer utilizes self-attention to capture global dependencies and explicit structure encoding mechanisms to model relationships among labels. Additionally, HLC is optimized on Cross-Entropy with Anchor Label (CEAL) loss function, specifically designed to learn the hierarchical label correlations. CEAL introduces an anchor label with a fixed score of zero, distinguishing target labels from non-target ones. This distinctive approach encourages HLC to predict higher scores for true target labels and lower scores for non-target labels compared to the anchor label. We conducted experiments on three benchmark datasets and compared them with existing methods. The results suggest that HLC can be an effective method for HTC. [ABSTRACT FROM AUTHOR]

Traditionally, sensitivity analysis has been utilized to determine the importance of input variables to a deep neural network (DNN). However, the quantification of sensitivity for each neuron in a network presents a significant challenge. In this article, a selective method for calculating neuron sensitivity in layers of neurons concerning network output is proposed. This approach incorporates scaling factors that facilitate the evaluation and comparison of neuron importance. Additionally, a hierarchical multi-scale optimization framework is proposed, where layers with high-importance neurons are selectively optimized. Unlike the traditional backpropagation method that optimizes the whole network at once, this alternative approach focuses on optimizing the more important layers. This paper provides fundamental theoretical analysis and motivating case study results for the proposed neural network treatment. The framework is shown to be effective in network optimization when applied to simulated and UCI Machine Learning Repository datasets. This alternative training generates local minima close to or even better than those obtained with the backpropagation method, utilizing the same starting points for comparative purposes within a multi-start optimization procedure. Moreover, the proposed approach is observed to be more efficient for large-scale DNNs. These results validate the proposed algorithmic framework as a rigorous and robust new optimization methodology for training (fitting) neural networks to input/output data series of any given system. [ABSTRACT FROM AUTHOR]

The cost of collecting and annotating large-scale datasets is expensive, thus the valuable datasets can be considered as the intellectual property (IP) of the dataset creator. To date, all the copyright protection methods for deep learning focus on the copyright protection of the models, while there are no researches on copyright protection of the dataset. Protecting the intellectual property of dataset is a brand new topic which is very challenging. In this paper, we propose an authorization control method to actively protect the dataset from being used to train Deep Neural Network (DNN) models without authorization. To the best of our knowledge, this is the first work on IP protection for dataset. We generate feature space adversarial examples for clean images. Then, we utilize the modified Reversible Image Transformation to hide the clean images into the corresponding feature space adversarial examples to generate the protected images. For the unauthorized users, the model directly trained on the protected dataset will have poor inference accuracy. For the authorized users, the model can be trained on the recovered dataset and will have normal inference accuracy. Experimental results on CIFAR-10 and TinyImageNet datasets demonstrate the effectiveness of the proposed method. It is also demonstrated that the proposed method has an excellent transferability across different models. Moreover, the proposed method is robust to the adaptive attack. [ABSTRACT FROM AUTHOR]

Recently, the intellectual properties (IP) protection of deep neural networks (DNN) has attracted serious concerns. A number of DNN copyright protection methods have been proposed. However, most of the existing DNN watermarking methods can only verify the ownership of the model after the piracy occurs, which cannot actively prevent the occurrence of the piracy and do not support users' identities management, thus can not satisfy the requirements of commercial DNN copyright management. In addition, the query modification attack which was proposed recently can invalidate most of the existing backdoor-based DNN watermarking methods. In this paper, we propose an active intellectual properties protection technique for DNN models via stealthy backdoor and users' identities authentication. For the first time, we use a set of clean images (as the watermark key samples) to embed an additional class into the DNN for ownership verification, and use the image steganography to embed users' identity information into these watermark key images. Each user will be assigned with a unique identity image for identity authentication and authorization control. Since the backdoor instances are clean images outside the dataset, the backdoor trigger is visually imperceptible and concealed. In addition, we embed the watermark by exploiting an additional class outside the main tasks, which establishes a strong connection for watermark key samples and the corresponding label. As a result, the proposed method is concealed, robust, and can resist common attacks and query modification attack. Experimental results demonstrate that, the proposed method can obtain 100% watermark accuracy and 100% fingerprint authentication success rate on Fashion-MNIST and CIFAR-10 datasets. In addition, the proposed method is demonstrated to be robust against the model fine-tuning attack, model pruning attack, and query modification attack. Compared with three existing DNN watermarking methods, the proposed method has better performance on watermark accuracy and robustness against the query modification attack. [ABSTRACT FROM AUTHOR]

In deep neural network (DNN) accelerators, it is expensive to transfer model parameters from the main memory to the processing elements. Data movement accounts for a large number of the inference latency and energy consumption. In this paper, we present three position-based techniques to compress the DNN model parameters. The techniques could lead to significant energy and performance improvement. The three presented compression techniques are lossless. The first technique takes into consideration the regularly repeat property of the DNN weights to compress them. The second technique saves the relative distance between weights instead of the weights to compress the model. The third technique applies Huffman coding on the relative distance based on the second technique. The proposed techniques are assessed on several DNNs. The results show that, the first technique could decrease 38% of latency and 36% energy, respectively. The second technique could decrease 41% of latency and 39% energy, respectively. The third technique could decrease 45% of latency and 43% energy, respectively. Applying Huffman code could achieve additional 7% reduction in both latency and energy based on the second technique. [ABSTRACT FROM AUTHOR]

Deep learning models are easily deceived by adversarial examples, and transferable attacks are crucial because of the inaccessibility of model information. Existing SOTA attack approaches tend to destroy important features of objects to generate adversarial examples. This paper proposes the split grid mask attack (SGMA), which reduces the intensity of model-specific features by split grid mask transformation, effectively highlighting the important features of the input image. Perturbing these important features can guide the development of adversarial examples in a more transferable direction. Specifically, we introduce the split grid mask transformation into the input image. Due to the vulnerability of model-specific features to image transformations, the intensity of model-specific features decreases after aggregation while the intensities of important features remain. The generated adversarial examples guided by destroying important features have excellent transferability. Extensive experimental results demonstrate the effectiveness of the proposed SGMA. Compared to the SOTA attack approaches, our method improves the black-box attack success rates by an average of 6.4% and 8.2% against the normally trained models and the defense ones respectively. [ABSTRACT FROM AUTHOR]

This paper presents HDNN, a proof-of-concept MLIR dialect for cross-platform computing specialized in deep neural networks. As target devices, HDNN supports CPUs, GPUs and TPUs. In this paper, we provide a comprehensive description of the HDNN dialect, outlining how this novel approach aims to solve the P 3 problem of parallel programming (portability, productivity, and performance). An HDNN program is device-agnostic, i.e., only the device specifier has to be changed to run a given workload in one device or another. Moreover, HDNN has been designed to be a domain-specific language, which ultimately helps programming productivity. Finally, HDNN relies on optimized libraries for heavy, performance-critical workloads. HDNN has been evaluated against other state-of-the-art machine learning frameworks on all the hardware platforms achieving excellent performance. We conclude that the ideas and concepts used in HDNN can be crucial for designing future generation compilers and programming languages to overcome the challenges of the forthcoming heterogeneous computing era. [ABSTRACT FROM AUTHOR]

Large-scale and high-quality test samples are extremely scarce in deep neural networks(DNN) testing. Existing test sample optimization methods exhibit the problem of low efficiency and low neuron coverage of optimized test samples, which consistently fail to expose erroneous behaviors of DNNs with corner-case inputs. In this paper, we propose DeepMC, an image classification DNN test sample optimization method jointly guided by misclassification and coverage. Specifically, we select the seed sample from the original test samples according to the misclassification probability. To maximize the misclassification probability and neuron coverage, we construct the joint optimization problem for the seed samples and use the gradient ascent to solve the joint optimization problem. We evaluate this method on two well-known datasets and prevalent image classification DNN models. Compare with DeepXplore, a DL white-box testing framework, DeepMC does not require multiple DNN models with similar functions for cross-referencing, saves 90% time consumption on MNIST, averagely covers 1.87% more neurons, and optimized test samples with more than 69% attack success rate. In addition, the test sample optimized by DeepMC can also be applied to optimize the robustness of the corresponding DNN with an average 3% improvement of the model's accuracy. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNNs) are rapidly gaining popularity in a variety of important domains. Unfortunately, modern DNNs have been shown to be vulnerable to a variety of attacks and buggy behavior. This has motivated recent work in formally analyzing the properties of such DNNs. This paper introduces SyReNN, a tool for understanding and analyzing a DNN by computing its symbolic representation. The key insight is to decompose the DNN into linear functions. Our tool is designed for analyses using low-dimensional subsets of the input space, a unique design point in the space of DNN analysis tools. We describe the tool and the underlying theory, then evaluate its use and performance on three case studies: computing Integrated Gradients, visualizing a DNN's decision boundaries, and repairing buggy DNNs. [ABSTRACT FROM AUTHOR]

It is more than a decade since machine learning and especially its leading subtype deep learning have become one of the most interesting topics in almost all areas of science and industry. In numerous contexts, at least one of the applications of deep learning is utilized or is going to be utilized. Using deep learning for image classification is now very popular and widely used in various use cases. Many types of research in medical sciences have been focused on the advantages of deep learning for image classification problems. Some recent researches show more than 90% accuracy for breast tissue classification which is a breakthrough. A huge number of computations in deep neural networks are considered a big challenge both from software and hardware point of view. From the architectural perspective, this big amount of computing operations will result in high power consumption and computation runtime. This led to the emersion of deep learning accelerators which are designed mainly for improving performance and energy efficiency. Data reuse and localization are two great opportunities for achieving energy-efficient computations with lower runtime. Data flows are mainly designed based on these important parameters. In this paper, DLA-H and BJS, a deep learning accelerator, and its data flow for histopathologic image classification are proposed. The simulation results with the MAESTRO tool showed 756 cycles for total runtime and 3.21 × 10 6 GFLOPS roofline throughput that is an extreme performance improvement in comparison to current general-purpose deep learning accelerators and data flows. [ABSTRACT FROM AUTHOR]

The weak interpretability of the deep reinforcement learning (DRL) model becomes a serious impediment to the application of DRL agents in certain areas requiring high reliability. To interpret the behavior of a DRL agent, researchers use saliency maps to discover important parts of the agent's observation that influence its decision. However, the representations of saliency maps still cannot explicitly present the cause and effect between an agent's actions and its observations. In this paper, we analyze the inference procedure with respect to the DRL architecture and propose embedding interpretable intermediate representations for an agent's policy, the intermediate representations that are compressed and abstracted for explanation. We utilize a conceptual embedding technique to regulate the latent representation space of the deep models that can produce interpretable causal factors aligned with human concepts. Furthermore, the information loss of intermediate representation is analyzed to define the model performance upper bound and to measure the model performance degeneration. Experiments validate the effectiveness of the proposed method and the relationship between the observation information and an agent's performance upper bound. [ABSTRACT FROM AUTHOR]

Deep neural networks are vulnerable to adversarial examples, which are crafted by applying small, human-imperceptible perturbations on the original images, so as to mislead deep neural networks to output inaccurate predictions. Adversarial attacks can thus be an important method to evaluate and select robust models in safety-critical applications. However, under the challenging black-box setting, most existing adversarial attacks often achieve relatively low success rates on normally trained networks and adversarially trained networks. In this paper, we regard the generation process of adversarial examples as an optimization process similar to deep neural network training. From this point of view, we introduce AdaBelief optimizer and crop invariance into the generation of adversarial examples, and propose AdaBelief Iterative Fast Gradient Method (ABI-FGM) and Crop-Invariant attack Method (CIM) to improve the transferability of adversarial examples. By adopting adaptive learning rate into the iterative attacks, ABI-FGM can optimize the convergence process, resulting in more transferable adversarial examples. CIM is based on our discovery on the crop-invariant property of deep neural networks, which we thus leverage to optimize the adversarial perturbations over an ensemble of crop copies so as to avoid overfitting on the white-box model being attacked and improve the transferability of adversarial examples. ABI-FGM and CIM can be readily integrated to build a strong gradient-based attack to further boost the success rates of adversarial examples for black-box attacks. Moreover, our method can also be naturally combined with other gradient-based attack methods to build a more robust attack to generate more transferable adversarial examples against the defense models. Extensive experiments on the ImageNet dataset demonstrate the method's effectiveness. Whether on normally trained networks or adversarially trained networks, our method has higher success rates than state-of-the-art gradient-based attack methods. [ABSTRACT FROM AUTHOR]

Subsea navigation by autonomous underwater vehicles (AUVs) is a demanding task that involves the integration of inertial sensors, gyrocompasses, Doppler velocity loggers, and reference from acoustic beacons. In this paper, we propose to augment this information by providing an external measurement of heading change. We rely on the direction of sand ripples, which are abundant on the seabed near the shore and whose direction is, locally, constant. Thus, any apparent change in their directivity, as detected by the AUV, would reflect as a change in the vehicle's heading. Considering this, we developed a mechanism that detects regions of interest (ROIs) containing sand ripples within a synthetic aperture sonar (SAS) image, segments the ROI into highlight and shadow, and evaluates the angle difference between ROIs within two consecutive SAS images. For detection of sand ripples and estimation of angle difference, we employ two deep neural networks, while for segmentation we formulate a fuzzy-logic clustering. Taking advantage of a transfer learning approach, we trained the deep networks on simulated SAS images and on a large database of 2088 real SAS images, which we share for reproducibility. Results from real SAS images from three different sites show a good trade-off between precision and recall for sand-ripple detection, and an error of a few degrees in the heading change estimation, which well exceeds a geometrical-based benchmark. We also show performance from a real-time experiment for which we implemented our method on an AUV and estimated its heading change on-the-fly. [ABSTRACT FROM AUTHOR]

Neural networks have been very successful in many applications; we often, however, lack a theoretical understanding of what the neural networks are actually learning. This problem emerges when trying to generalise to new data sets. The contribution of this paper is to show that, for the residual neural network model, the deep layer limit coincides with a parameter estimation problem for a nonlinear ordinary differential equation. In particular, whilst it is known that the residual neural network model is a discretisation of an ordinary differential equation, we show convergence in a variational sense. This implies that optimal parameters converge in the deep layer limit. This is a stronger statement than saying for a fixed parameter the residual neural network model converges (the latter does not in general imply the former). Our variational analysis provides a discrete-to-continuum Γ -convergence result for the objective function of the residual neural network training step to a variational problem constrained by a system of ordinary differential equations; this rigorously connects the discrete setting to a continuum problem. [ABSTRACT FROM AUTHOR]

This paper presents a novel speaking-aid system for enhancing esophageal speech (ES). The method adopted in this paper aims to improve the quality of esophageal speech using a combination of a voice conversion technique and a time dilation algorithm. In the proposed system, a Deep Neural Network (DNN) is used as a nonlinear mapping function for vocal tract vector transformation. Then the converted frames are used to determine realistic excitation and phase vectors from the target training space using a frame selection algorithm. Next, in order to preserve speaker identity of the esophageal speakers, we use the source vocal tract features and propose to apply on them a time dilation algorithm to reduce the unpleasant esophageal noises. Finally the converted speech is reconstructed using the dilated source vocal tract frames and the predicted excitation and phase. DNN and Gaussian mixture model (GMM) based voice conversion systems have been evaluated using objective and subjective measures. Such an experimental study has been realized also in order to evaluate the changes in speech quality and intelligibility of the transformed signals. Experimental results demonstrate that the proposed methods provide considerable improvement in intelligibility and naturalness of the converted esophageal speech. [ABSTRACT FROM AUTHOR]

In the era of the Internet of Things (IoT), the volume of data is increasing immensely causing rapid growth in network data communication and data congestion. Computational offloading thus becomes a crucial and imperative action in terms of delay-sensitive task completion and data processing for the resource constraint end-users. Nowadays fog computing, as a complement of cloud computing, has emerged a well-known concept in terms of enhancing data processing capability as well as energy conservation in low-powered networks. In this paper, we consider a heterogeneous fog-cloud network architecture where the data processing is performed on the local or remote computing device by adopting a binary offloading policy. Based on the proposed system model, we calculate the total delay and energy consumption of data processing throughout the network and formulate a mixed-integer optimization problem to jointly optimize the offloading decision and bandwidth allocation. In order to solve such an NP-hard problem, we have proposed a deep-learning-based binary offloading strategy that employs multiple parallel deep neural networks (DNNs) to make offloading decisions. Such offloading decisions are subsequently placed in a relay memory system to train and test all DNNs. Simulation results show a near-optimal performance of the proposed offloading strategy while remarkably maintaining the quality of service by decreasing overall delay and energy consumption. [ABSTRACT FROM AUTHOR]

Most state-of-the-art speech systems use deep neural networks (DNNs). These systems require a large amount of data to be learned. Hence, training state-of-the-art frameworks on under-resourced speech challenges are difficult tasks. As an example, a challenge could be the limited amount of data to model impaired speech. Furthermore, acquiring more data and/or expertise is time-consuming and expensive. In this paper, we focus on the following speech processing tasks: automatic speech recognition, speaker identification, and emotion recognition. To assess the problem of limited data, we firstly investigate state-of-the-art automatic speech recognition systems, as this is the hardest task (due to the wide variability in each language). Next, we provide an overview of techniques and tasks requiring fewer data. In the last section, we investigate few-shot techniques by interpreting under-resourced speech as a few-shot problem. In that sense, we propose an overview of few-shot techniques and the possibility of using such techniques for the speech problems addressed in this survey. It is true that the reviewed techniques are not well adapted for large datasets. Nevertheless, some promising results from the literature encourage the usage of such techniques for speech processing. [ABSTRACT FROM AUTHOR]

In this paper, we propose a bi-attention, a multi-layer attention and an attention mechanism and convolution neural network based text representation and classification model (ACNN). The bi-attention have two attention mechanism to learn two context vectors, forward RNN with attention to learn forward context vector c⃗ and backward RNN with attention to learn backward context vector c⃖, and then concatenation c⃗ and c⃖ to get context vector c. The multi-layer attention is the stack of the bi-attention. In the ACNN, the context vector c is obtained by the bi-attention, then the convolution operation is performed on the context vector c, and the max-pooling operation is used to reduce the dimension. After max-pooling operation the text is converted to low-dimensional sentence vector m. Finally, the Softmax classifier be used for text classification. We test our model on 8 benchmarks text classification datasets, and our model achieved a better or the same performance compare with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The growing trend of solar photovoltaic (PV) adoption has motivated homeowners and independent solar PV plants to assume the role of electricity generators. Utility companies need to address the changes in supply and demand to formulate effective distribution plans. Monitoring the power output of the photovoltaic array is a key task in ensuring the correct estimation of the electricity production of the system. Nevertheless, the literature recognizes the obstacle that expensive on-site monitoring sensors have on the cost of small-scale and medium-scale PV applications. Mathematical models based on the estimation of power output from historical data are the most used techniques to address the need for expensive monitoring systems. Estimating the power output of a PV array depends on the cell temperature, ambient temperature, and solar irradiance. This paper proposes a machine learning model to forecast site-specific ambient temperature and solar irradiance. The results contribute to the generation of low-cost data-driven models to save money by eliminating the need to install on-site sensors. The methodology employed off-site publicly-available weather data from neighboring weather stations as an alternative to on-site measurements. A 5-layer deep neural network was trained using 5 years' worth of historical data from a remote weather station in Green Bay, WI, to predict on-site parameters for a solar array located about 45 miles away in Keshena, WI. The model was found to be suitable for site-specific ambient temperature prediction. The ozone Dobson was a key parameter for solar irradiance prediction, given that the predictive accuracy of the proposed model was limited by the size of the training data. [ABSTRACT FROM AUTHOR]

Learning with Noisy Labels (LNL) methods aim to improve the accuracy of Deep Neural Networks (DNNs) when the training set contains samples with noisy or incorrect labels, and have become popular in recent years. Existing popular LNL methods frequently regard samples with high learning difficulty (high-loss and low prediction probability) as noisy samples; however, irregular feature patterns from hard clean samples can also cause high learning difficulty, which can lead to the misclassification of hard clean samples as noisy samples. To address this insufficiency, we propose the Samples' Learning Risk-based Learning with Noisy Labels (SLRLNL) method. Specifically, we propose to separate noisy samples from hard clean samples using samples' learning risk, which represents samples' influence on DNN's accuracy. We show that samples' learning risk is comprehensively determined by samples' learning difficulty as well as samples' feature similarity to other samples, and thus, compared to existing LNL methods that solely rely on the learning difficulty, our method can better separate hard clean samples from noisy samples, since the former frequently possess irregular feature patterns. Moreover, to extract more useful information from samples with irregular feature patterns (i.e., hard samples), we further propose the Relabeling-based Label Augmentation (RLA) process to prevent the memorization of hard noisy samples and better learn the hard clean samples, thus enhancing the learning for hard samples. Empirical studies show that samples' learning risk can identify noisy samples more accurately, and the RLA process can enhance the learning for hard samples. To evaluate the effectiveness of our method, we compare it with popular existing LNL methods on CIFAR-10, CIFAR-100, Animal-10N, Clothing1M, and Docred. The experimental results indicate that our method outperforms other existing methods. The source code for SLRLNL can be found at https://github.com/yangbo1973/SLRLNL. [ABSTRACT FROM AUTHOR]

Introduction: Inaccurate, untimely diagnoses of fundus diseases leads to vision-threatening complications and even blindness. We built a deep learning platform (DLP) for automatic detection of 30 fundus diseases using ultra-widefield fluorescein angiography (UWFFA) with deep experts aggregation. Methods: This retrospective and cross-sectional database study included a total of 61,609 UWFFA images dating from 2016 to 2021, involving more than 3364 subjects in multiple centers across China. All subjects were divided into 30 different groups. The state-of-the-art convolutional neural network architecture, ConvNeXt, was chosen as the backbone to train and test the receiver operating characteristic curve (ROC) of the proposed system on test data and external test date. We compared the classification performance of the proposed system with that of ophthalmologists, including two retinal specialists. Results: We built a DLP to analyze UWFFA, which can detect up to 30 fundus diseases, with a frequency-weighted average area under the receiver operating characteristic curve (AUC) of 0.940 in the primary test dataset and 0.954 in the external multi-hospital test dataset. The tool shows comparable accuracy with retina specialists in diagnosis and evaluation. Conclusions: This is the first study on a large-scale UWFFA dataset for multi-retina disease classification. We believe that our UWFFA DLP advances the diagnosis by artificial intelligence (AI) in various retinal diseases and would contribute to labor-saving and precision medicine especially in remote areas. [ABSTRACT FROM AUTHOR]

A review of the existing two- and three-phase relative permeability correlations shows a lot of pitfalls and restrictions imposed by (a) their assumptions (b) generalization ability and (c) difficulty with updating in real-time for different reservoirs systems. These increase the uncertainty in its prediction which is crucial owing to the fact that relative permeability is useful for predicting future reservoir performance, effective mobility, ultimate recovery, and injectivity among others. Laboratory experiments can be time-consuming, complex, expensive and done with core samples which in some circumstances may be difficult or impossible to obtain. Deep Neural Networks (DNNs) with their special capability to regularize, generalize and update easily with new data has been used to predict oil–water relative permeability. The details have been presented in this paper. In addition to common parameters influencing relative permeability, Baker and Wyllie parameter combinations were used as input to the network after comparing with other models such as Stones, Corey, Parker, Honapour using Corey and Leverett-Lewis experimental data. The DNN automatically used the best cross validation result (in a five-fold cross validation) for its training until convergence by means of Nesterov-accelerated gradient descent which also minimizes the cost function. Predictions of non-wetting and wetting-phase relative permeability gave good match with field data obtained for both validation and test sets. This technique could be integrated into reservoir simulation studies, save cost, optimize the number of laboratory experiments and further demonstrate machine learning as a promising technique for real-time reservoir parameters prediction. [ABSTRACT FROM AUTHOR]

N-ary error correcting output codes (ECOC) decompose a multi-class problem into simpler multi-class problems by splitting the classes into N subsets (meta-classes) to form an ensemble of N-class classifiers and combine them to make predictions. It is one of the most accurate ensemble learning methods for traditional classification tasks. Deep learning has gained increasing attention in recent years due to its successes on various tasks such as image classification and speech recognition. However, little is known about N-ary ECOC with deep neural networks (DNNs) as base learners, probably due to the long computation time. In this paper, we show by experiments that N-ary ECOC with DNNs as base learners generally exhibits superior performance compared with several state-of-the-art ensemble learning methods. Moreover, our work contributes to a more efficient setting of the two crucial hyperparameters of N-ary ECOC: the value of N and the number of base learners to train. We also explore valuable strategies for further improving the accuracy of N-ary ECOC. [ABSTRACT FROM AUTHOR]

Generalized zero-shot extreme multi-label learning (GZXML) aims to predict relevant labels for unknown instances from a set of seen and unseen labels and is widely used in engineering applications. Since the supervisory information of the instances is incomplete in this task, the existing methods classify such instances based on the semantic relationships between the instances and labels. However, the supervisory information of the seen labels is also crucial for achieving high prediction performance. To bridge this gap, we propose collaborative learning of supervision and correlations for GZXML (CLSC-XML). CLSC-XML leverages both the semantic relationships between instances and labels and the supervisory information of the seen labels to enhance the prediction results for unseen labels. Specifically, CLSC-XML extracts discriminative and representational features, which are then fed into classification and correlation modules for collaborative learning. Furthermore, to enrich the incomplete supervised information, we propose the generation of features for unseen labels (GFUL) algorithm. The classifier is trained alternately with the GFUL algorithm. The classifier provides semantic guidance to the GFUL algorithm, and in turn, the GFUL algorithm helps the classification model enrich the supervised information. Experimental results show that CLSC-XML outperforms the state-of-the-art methods and requires less training time. [ABSTRACT FROM AUTHOR]

Musical instrument sound synthesis (MISS) often utilizes a text-to-speech framework because of its similarity to speech in terms of generating sounds from symbols. Moreover, a plucked string instrument, such as electric bass guitar (EBG), shares acoustical similarities with speech. We propose an attack-sustain (AS) representation of the playing technique to take advantage of this similarity. The AS representation treats the attack segment as an unvoiced consonant and the sustain segment as a voiced vowel. In addition, we propose a MISS framework for an EBG that can control its playing techniques: (1) we constructed a EBG sound database containing a rich set of playing techniques, (2) we developed a dynamic time warping and timbre conversion to align the sounds and AS labels, (3) we extend an existing MISS framework to control playing techniques using AS representation as control symbols. The experimental evaluation suggests that our AS representation effectively controls the playing techniques and improves the naturalness of the synthetic sound. [ABSTRACT FROM AUTHOR]

This article presents a data-driven approach called Intelligent Team Formation and Player Selection (ITFPS), aimed at assisting football coaches in making informed decisions about team formation and player selection. The proposed approach utilizes deep neural networks to evaluate the suitability of each player for different positions on the field. The problem is then formulated as finding the maximum weighted matching in a complete bipartite graph, with the objective of achieving the best possible alignment between team members and the positions designated by the coach. The Hungarian algorithm is employed to facilitate this matching process. Furthermore, the approach allows coaches to select a system of distinct representatives for each position, based on the specific qualities expected from players in a given match. The effectiveness of the approach is demonstrated through tests conducted on four teams from the 2021–2022 English Premier League. The results indicate that ITFPS produces decisions comparable to those made by successful coaches. By optimizing team formations and enabling the utilization of rotating formations, this approach not only enhances team performance but also empowers coaches to make strategic decisions while fully leveraging the potential of their players. ITFPS serves as an intelligent assistant for coaches, complementing their strategic perspectives. [ABSTRACT FROM AUTHOR]

Time series forecasting is an important problem, with many real world applications. Transformer models have been successfully applied to natural language processing tasks, but have received relatively little attention for time series forecasting. Motivated by the differences between classification tasks and forecasting, we propose PI-Transformer, an adaptation of the Transformer architecture designed for time series forecasting, consisting of three parts: First, we propose a novel initialization method called Persistence Initialization, with the goal of increasing training stability of forecasting models by ensuring that the initial outputs of an untrained model are identical to the outputs of a simple baseline model. Second, we use ReZero normalization instead of Layer Normalization, in order to further tackle issues related to training stability. Third, we use Rotary positional encodings to provide a better inductive bias for forecasting. Multiple ablation studies show that the PI-Transformer is more accurate, learns faster, and scales better than regular Transformer models. Finally, PI-Transformer achieves competitive performance on the challenging M4 dataset, both when compared to the current state of the art, and to recently proposed Transformer models for time series forecasting. [ABSTRACT FROM AUTHOR]

Time-frequency (T-F) masking is an effective method for stereo speech source separation. However, reliable estimation of the T-F mask from sound mixtures is a challenging task, especially when room reverberations are present in the mixtures. In this paper, we propose a new stereo speech separation system where deep neural networks are used to generate soft T-F mask for separation. More specifically, the deep neural network, which is composed of two sparse autoencoders and a softmax regression, is used to estimate the orientations of the dominant source at each T-F unit, based on low-level features, such as mixing vector (MV), interaural level, and phase difference (IPD/ILD). The dataset for training the networks was generated by the convolution of binaural room impulse responses (RIRs) and clean speech signals positioned in different angles with respect to the sensors. With the training dataset, we use unsupervised learning to extract high-level features from low-level features and use supervised learning to find the nonlinear functions between high-level features and the orientations of dominant source. By using the trained networks, the probability that each T-F unit belongs to different sources (target and interferers) can be estimated based on the localization cues which is further used to generate the soft mask for source separation. Experiments based on real binaural RIRs and TIMIT dataset are provided to show the performance of the proposed system for reverberant speech mixtures, as compared with a model-based T-F masking technique proposed recently. [ABSTRACT FROM AUTHOR]

This paper presents a sequence transcription approach for the automatic diacritization of Arabic text. A recurrent neural network is trained to transcribe undiacritized Arabic text with fully diacritized sentences. We use a deep bidirectional long short-term memory network that builds high-level linguistic abstractions of text and exploits long-range context in both input directions. This approach differs from previous approaches in that no lexical, morphological, or syntactical analysis is performed on the data before being processed by the net. Nonetheless, when the network is post-processed with our error correction techniques, it achieves state-of-the-art performance, yielding an average diacritic and word error rates of 2.09 and 5.82 %, respectively, on samples from 11 books. For the LDC ATB3 benchmark, this approach reduces the diacritic error rate by 25 %, the word error rate by 20 %, and the last-letter diacritization error rate by 33 % over the best published results. [ABSTRACT FROM AUTHOR]

The focus of this work is on the application of classical Model Order Reduction techniques, such as Active Subspaces and Proper Orthogonal Decomposition, to Deep Neural Networks. We propose a generic methodology to reduce the number of layers in a pre-trained network by combining the aforementioned techniques for dimensionality reduction with input-output mappings, such as Polynomial Chaos Expansion and Feedforward Neural Networks. The motivation behind compressing the architecture of an existing Convolutional Neural Network arises from its usage in embedded systems with specific storage constraints. The conducted numerical tests demonstrate that the resulting reduced networks can achieve a level of accuracy comparable to the original Convolutional Neural Network being examined, while also saving memory allocation. Our primary emphasis lies in the field of image recognition, where we tested our methodology using VGG-16 and ResNet-110 architectures against three different datasets: CIFAR-10, CIFAR-100, and a custom dataset. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have recently showed remarkable performance in various computer vision tasks, including classification and segmentation of medical images. Deep ensembles (an aggregated prediction of multiple DNNs) were shown to improve a DNN's performance in various classification tasks. Here we explore how deep ensembles perform in the image segmentation task, in particular, organ segmentations in CT (Computed Tomography) images. Ensembles of V-Nets were trained to segment multiple organs using several in-house and publicly available clinical studies. The ensembles segmentations were tested on images from a different set of studies, and the effects of ensemble size as well as other ensemble parameters were explored for various organs. Compared to single models, Deep Ensembles significantly improved the average segmentation accuracy, especially for those organs where the accuracy was lower. More importantly, Deep Ensembles strongly reduced occasional "catastrophic" segmentation failures characteristic of single models and variability of the segmentation accuracy from image to image. To quantify this we defined the "high risk images": images for which at least one model produced an outlier metric (performed in the lower 5% percentile). These images comprised about 12% of the test images across all organs. Ensembles performed without outliers for 68%–100% of the "high risk images" depending on the performance metric used. [ABSTRACT FROM AUTHOR]

Meningiomas have the highest incidence rate of all primary intracranial and central nervous system tumors. Accurate preoperative grading of meningiomas is extraordinarily meaningful for treatment strategy and patient prognosis. Magnetic resonance imaging (MRI) is the most common method for meningioma grading. Existing methods are typically two-stage models and require image-level classifications or region of interest (ROI) annotations for assistant diagnosis, thereby adding massive manual annotations and time costs. Meanwhile, most of these methods use only a single MRI slice, which may lose the overall meningioma information and are inconsistent with the actual clinical diagnosis process. To address the above problems, a multi-instance learning (MIL) method based on spatial continuous category representation is proposed for case-level meningioma grading. It considers the MRI case and corresponding slices as a bag and instances, respectively, and requires only a case-level label to diagnose a patient. To make the most of the serialization characteristics of MRI images, this method selects continuous instance-feature sequences under each category that are most suspected to contain tumors and further integrates these sequences into bag-level features for classification. In addition, an end-to-end meningioma grading architecture is designed to support the proposed MIL method, which directly takes the original MRI images of the patient as input and outputs the meningioma grade prediction. To train and evaluate the proposed method, datasets with different slice thicknesses are constructed. The experimental results demonstrate that our method performs satisfactorily compared with other related methods for meningioma grading. [ABSTRACT FROM AUTHOR]

The super power of deep learning in image classification problems have become very popular and applicable in many areas like medical sciences. Some of the medical applications are real-time and may be implemented in embedded devices. In these cases, achieving the highest level of accuracy is not the only concern. Computation runtime and power consumption are also considered as the most important performance indicators. These parameters are mainly evaluated in hardware design phase. In this research, an energy efficient deep learning accelerator for endoscopic images classification (DLA-E) is proposed. This accelerator can be implemented in the future endoscopic imaging equipments for helping medical specialists during endoscopy or colonoscopy in order of making faster and more accurate decisions. The proposed DLA-E consists of 256 processing elements with 1000 bps network on chip bandwidth. Based on the simulation results of this research, the best dataflow for this accelerator based on MobileNet v2 is kcp_ws from the weight stationary (WS) family. Total energy consumption and total runtime of this accelerator on the investigated dataset is 4.56 × 109 MAC (multiplier–accumulator) energy and 1.73 × 107 cycles respectively, which is the best result in comparison to other combinations of CNNs and dataflows. [ABSTRACT FROM AUTHOR]

Developing machine learning models that outperform the existing ones has been defined as the main goal of researchers and industry experts. In this context, the carbon footprint caused by conducting extended experiments on huge datasets for fine tuning the model parameters is usually ignored. Here, we present an efficient hybrid trust-aware incremental recommendation system that is based on factorization of user-item matrices. The method uses trust networks of the customers obtained from social networks. The main advantage of the method is that it is incremental and there is no need to re-train the system from scratch in presence of new customer activities and only newly added users and items are learned. To tackle the sparseness of the matrix, known as cold-start problem, we employed a deep neural network to decrease the user ratings approximation error. Moreover, we evaluated three different initialization methods for matrix factorization algorithm. The method is evaluated on two versions of popular MovieLens dataset from GroupLens research and also Extended Epinions dataset. The experimental results confirm that our method significantly requires less computational resources, while providing comparable prediction error. [ABSTRACT FROM AUTHOR]

In this work, we present the Gaussian Class-Conditional Simplex (GCCS) loss: a novel approach for training deep robust multiclass classifiers that improves over the state-of-the-art in terms of classification accuracy and adversarial robustness, with little extra cost for network training. The proposed method learns a mapping of the input classes onto Gaussian target distributions in a latent space such that a hyperplane can be used as the optimal decision surface. Instead of maximizing the likelihood of target labels for individual samples, our loss function pushes the network to produce feature distributions yielding high inter-class separation and low intra-class separation. The mean values of the learned distributions are centered on the vertices of a simplex such that each class is at the same distance from every other class. We show that the regularization of the latent space based on our approach yields excellent classification accuracy. Moreover, GCCS provides improved robustness against adversarial perturbations, outperforming models trained with conventional adversarial training (AT). In particular, our model learns a decision space that minimizes the presence of short paths toward neighboring decision regions. We provide a comprehensive empirical evaluation that shows how GCCS outperforms state-of-the-art approaches over challenging datasets for targeted and untargeted gradient-based, as well as gradient-free adversarial attacks, both in terms of classification accuracy and adversarial robustness. [ABSTRACT FROM AUTHOR]

Music source separation has traditionally followed the encoder-decoder paradigm (e.g., hourglass, U-Net, DeconvNet, SegNet) to isolate individual music components from mixtures. Such networks, however, result in a loss of location-sensitivity, as low-resolution representation drops the useful harmonic patterns over the temporal dimension. We overcame this problem by performing singing voice separation using a high-resolution representation learning (HRNet) system coupled with a long short-term memory (LSTM) module to retain high-resolution feature map and capture the temporal behavior of the acoustic signal. We called this joint combination of HRNet and LSTM as HR-LSTM. The predicted spectrograms produced by this system are close to ground truth and successfully separate music sources, achieving results superior to those realized by past methods. The proposed network was tested using four datasets (DSD100, MIR-1K, Korean Pansori, and Nepal Idol singing voice). Our experiments confirmed that the proposed HR-LSTM outperforms state-of-the-art networks at singing voice separation when the DSD100 dataset is used, performs comparably to alternative methods when the MIR-1K dataset is used, and separates the voice and accompaniment components well when the Pansori and NISVS datasets are used. In addition to proposing and validating our network, we also developed and shared our Nepal Idol dataset. [ABSTRACT FROM AUTHOR]

This study presents a comprehensive experimental dataset and a novel classification model based on Deep Neural Networks to estimate surface roughness for additive manufacturing. Many problems exist due to the very complex nature of the production process. Some focus on the production planning phase, including the nesting problem under many constraints. However, it is not possible to solve the main function without a clear understanding of the nature of the constraints. The purpose of this research is to present a method to automate the surface roughness estimation process in the production planning phase. The significance of this study is to implement a data-driven model for one of the most critical decision constraints in the nesting process. Solving this problem will automate a key decision constraint, and it might be implemented as an automated constraint module in solving the nesting problem. The proposed model focused on selective laser sintering (SLS) technology based on polyamide 12 powder applications. A comprehensive dataset is designed to simulate the behaviour of an industrial SLS manufacturing process based on a 3D positioning strategy. A set of samples with random positions are also created to test present the model's robustness. The proposed classification model is based on Deep Neural Networks (DNN) with hyper-parameters designed for the problem. The dataset and the model provide a new user interface to estimate the surface roughness depending on the coordinates of a given product surface in an SLS production chamber and the production parameters employed in the production planning phase. The results show that the model can classify sample surfaces as "rough" or "smooth" with a very high percentage (95.8%) for the training set and with 100% for the test set. Benchmark results also show that the model outperforms other machine learning methods in classifying the surface roughness successfully on the test set. [ABSTRACT FROM AUTHOR]

Uncommon observations that significantly vary from the norm are referred to as outliers. Outlier detection, which aims to detect unexpected behavior, is a critical topic that has attracted significant attention in a wide range of research areas and application domains, including video surveillance, network intrusion detection, disease outbreak detection, and others. Deep learning-based techniques for outlier detection have currently outperformed machine learning and shallow approaches on streaming data, which are big and complicated datasets. Despite the fact that deep learning has been successfully applied in a variety of application domains, developing an effective and appropriate model is a difficult task due to the dynamic nature and variations of real-world applications and data. Hence, this research proposes a novel deep learning model based on a deep neural network (DNN) to handle the outlier detection problem in the context of streaming data. The proposed DNN model is developed with multiple hidden layers to improve feature abstraction and capabilities. Extensive experiments performed on four real-world outlier benchmark datasets, available at the UCI repository, and comparisons to state-of-the-art approaches are used to evaluate the proposed model's performance. Experiment results demonstrate that it outperforms both machine learning algorithms and deep learning competitors, resulting in significant performance gains. Particularly, when compared to other algorithms, the evaluation results clearly demonstrated the efficacy of the proposed approach, with much higher accuracy, recall and f1-score rates of 99.63%, 99.014% and 99.437%, respectively. [ABSTRACT FROM AUTHOR]

The main limitation in developing deep neural network (DNN) models to predict bioactivity properties of chemicals is the lack of sufficient assay data to train the network's classification layers. Focusing on feedforward DNNs that use atom- and bond-based structural fingerprints as input, we examined whether layers of a fully trained DNN based on large amounts of data to predict one property could be used to develop DNNs to predict other related or unrelated properties based on limited amounts of data. Hence, we assessed if and under what conditions the dense layers of a pre-trained DNN could be transferred and used for the development of another DNN associated with limited training data. We carried out a quantitative study employing more than 400 pairs of assay datasets, where we used fully trained layers from a large dataset to augment the training of a small dataset. We found that the higher the correlation r between two assay datasets, the more efficient the transfer learning is in reducing prediction errors associated with the smaller dataset DNN predictions. The reduction in mean squared prediction errors ranged from 10 to 20% for every 0.1 increase in r2 between the datasets, with the bulk of the error reductions associated with transfers of the first dense layer. Transfer of other dense layers did not result in additional benefits, suggesting that deeper, dense layers conveyed more specialized and assay-specific information. Importantly, depending on the dataset correlation, training sample size could be reduced by up to tenfold without any loss of prediction accuracy. [ABSTRACT FROM AUTHOR]

Micro-Expressions (MEs) are a kind of short-lived and uncontrollable facial expressions. Therefore, the MEs recognition task poses a great challenge to both the psychological and computer vision research communities. In this study, a new ensemble algorithm is proposed by fusing two different deep learning frameworks: Deep Forest (DF) and Convolutional Neural Networks (CNN) (DFN for short). A modified DF structure is deployed to extract features through the multi-grained scanning technique, along with three different sliding windows to gain diverse motion features. In addition, two shallow CNNs are deployed to extract the features from the three-dimensional optical flow vector and the apex frame. In this way, the fusion of DF and CNNs forms DFN to extract the static and dynamic features for MEs, to generate diverse features with high-level abstraction. Consequently, this heterogeneous ensemble deploys the high diversity in these two models to promote the overall discriminative ability. Comprehensive experiments have confirmed the robustness and effectiveness of the proposed DFN with relatively less computational consumption. Related theoretical analysis has been given to further provide evidence and insights into the proposed method. Our source code is publicly available for non-commercial or research use at https://github.com/MLDMXM2017/DFN_ME. [ABSTRACT FROM AUTHOR]

Tuning and optimising the operations executed in deep learning frameworks is a fundamental task in accelerating the processing of deep neural networks (DNNs). However, this optimisation usually requires extensive manual efforts in order to obtain the best performance for each combination of tensor input size, layer type, and hardware platform. In this work, we present BestOf, a novel online auto-tuner that optimises the training and inference phases of DNNs. BestOf automatically selects at run time, and among the provided alternatives, the best performing implementation in each layer according to gathered profiling data. The evaluation of BestOf is performed on multi-core architectures for different DNNs using PyDTNN, a lightweight library for distributed training and inference. The experimental results reveal that the BestOf auto-tuner delivers the same or higher performance than that achieved using a static selection approach. [ABSTRACT FROM AUTHOR]

Ultrasound imaging is one of the most frequently used diagnostic tools for detecting and analyzing abnormalities of the breast. Recently proposed methods for the automated analysis of breast ultrasound images have shown great success, especially in the classification of breast abnormalities into malignant or benign lesions. In this study, we explore the use of a deep convolutional neural network with a multi-scale module for the automated assessment of BI-RADS category on breast ultrasound images. We propose a multi-input region of interest extraction method to extract the breast region from ultrasound images, with this method avoiding the deformation of region of interest images and providing high classification performance. Moreover, we also propose an order-constrained loss function that fully considers the continuity between BI-RADS categories and shows higher performance than traditional loss functions. A large annotated dataset containing 8246 breast ultrasound images was collected to train and evaluate the proposed methods. Ablation experiments were performed to validate the effectiveness of the proposed methods. Experimental results indicate that our method can be used to mimic experienced radiologists in the assessment of the BI-RADS category of breast ultrasound images and that the automated interpretations could be acceptable in routine clinical breast ultrasound examination reports. [ABSTRACT FROM AUTHOR]