Search

Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combined with a parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exist two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to a decrease in convergence accuracy. In this paper, we combine parallel mechanism with gradient quantization and periodic full-gradient compensation, and propose a new distributed optimization method named CP-SGD, which can hide the overhead of gradient compression, overlap part of the communication and obtain high convergence accuracy. The local update operation in CP-SGD allows the next iteration to be launched quickly without waiting for the completion of gradient compression and the current communication process. Besides, the accuracy loss caused by gradient compression is solved by k-step correction method introduced in CP-SGD, which provides a gradient correction every k iterations. We prove that CP-SGD has a convergence guarantee and it achieves at least O (1 K + 1 K) convergence rate, where K is the number of iterations. We conduct extensive experiments on MXNet to verify the convergence properties and scaling performance of CP-SGD. Experimental results on a 32-GPU cluster show that convergence accuracy of CP-SGD is close to or even slightly better than that of S-SGD, and its end-to-end time is 30% less than 2-bit gradient compression under a 56Gbps bandwidth environment. In addition, we analyze the performance of CP-SGD when training on 8, 16 and 32 GPUs. It is found that CP-SGD is suitable for most compression-supported update algorithms, and its scalability is approximately linear. • CP-SGD: Distributed Stochastic Gradient Descent with Compression and Periodic Compensation. • CP-SGD help distributed deep learning train with high-accuracy and friendly-communication simultaneously. • Combining the advantages of parallel mechanism and gradient compression. • K-step correction technique makes CP-SGD achieve codesign optimizations and improve compression-based methods greatly. • CP-SGD suits most compression-supported update algorithms and gradient compression strategies. [ABSTRACT FROM AUTHOR]

DNA 4 mC plays a crucial role in the genetic expression process of organisms. However, existing deep learning algorithms have shortcomings in the ability to represent DNA sequence features. In this paper, we propose a 4 mC site identification algorithm, DNABert-4mC, based on a fusion of the pruned pre-training DNABert-Pruning model and artificial feature encoding to identify 4 mC sites. The algorithm prunes and compresses the DNABert model, resulting in the pruned pre-training model DNABert-Pruning. This model reduces the number of parameters and removes redundancy from output features, yielding more precise feature representations while upholding accuracy.Simultaneously, the algorithm constructs an artificial feature encoding module to assist the DNABert-Pruning model in feature representation, effectively supplementing the information that is missing from the pre-trained features. The algorithm also introduces the AFF-4mC fusion strategy, which combines artificial feature encoding with the DNABert-Pruning model, to improve the feature representation capability of DNA sequences in multi-semantic spaces and better extract 4 mC sites and the distribution of nucleotide importance within the sequence. In experiments on six independent test sets, the DNABert-4mC algorithm achieved an average AUC value of 93.81%, outperforming seven other advanced algorithms with improvements of 2.05%, 5.02%, 11.32%, 5.90%, 12.02%, 2.42% and 2.34%, respectively. [Display omitted] • Train a new pruning pretraining model on DNA sequences to extract feature information. • The introduction of manually assisted features has expanded the search space for sequence feature. • Attention fusion strategy better promotes the fusion output of features from different dimensions. • The complementary nature of machine features and manual features better fits the target features. [ABSTRACT FROM AUTHOR]

Operators from various industries have been pushing the adoption of wireless sensing nodes for industrial monitoring, and such efforts have produced sizeable condition monitoring datasets that can be used to build diagnosis algorithms capable of warning maintenance engineers of impending failure or identifying current system health conditions. However, single operators may not have sufficiently large fleets of systems or component units to collect sufficient data to develop data-driven algorithms. One potential solution to overcome the challenge of having limited representative datasets is to merge datasets from multiple operators with the same type of assets. However, directly sharing data across the company's borders yields privacy concerns. Federated learning (FL) has emerged as a promising solution to leverage datasets from multiple operators to train a decentralized asset fault diagnosis model while maintaining data confidentiality. However, the performance of traditional FL algorithms degrades when local clients' datasets are heterogeneous. The dataset heterogeneity is particularly prevalent in fault diagnosis applications due to the high diversity of operating conditions and system configurations. To address this challenge, this paper proposes a novel clustering-based FL algorithm where clients are clustered based on their dataset similarity. Estimating dataset similarity between clients without explicitly sharing data is achieved by training probabilistic deep learning models and having each client examine the predictive uncertainty of the other clients' models on its local dataset. Clients are then clustered for FL based on relative prediction accuracy and uncertainty. Experiments on three bearing fault datasets, two publicly available and one newly collected for this work, show that our algorithm significantly outperforms FedAvg and a cosine similarity-based algorithm by 5.1% and 30.7% on average over the three datasets. Further, using a probabilistic classification model has the additional advantage of accurately quantifying its predictive uncertainty, which we show it does exceptionally well. [ABSTRACT FROM AUTHOR]

Synchronous SGD with data parallelism, the most popular parallelization strategy for CNN training, suffers from the expensive communication cost of averaging gradients among all workers. The iterative parameter updates of SGD cause frequent communications and it becomes the performance bottleneck. In this paper, we propose a lazy parameter update algorithm that adaptively adjusts the parameter update frequency to address the expensive communication cost issue. Our algorithm accumulates the gradients if the difference of the accumulated gradients and the latest gradients is sufficiently small. The less frequent parameter updates reduce the per-iteration communication cost while maintaining the model accuracy. Our experimental results demonstrate that the lazy update method remarkably improves the scalability while maintaining the model accuracy. For ResNet50 training on ImageNet, the proposed algorithm achieves a significantly higher speedup (739.6 on 2048 Cori KNL nodes) as compared to the vanilla synchronous SGD (276.6) while the model accuracy is almost not affected (<0.2% difference). • Adaptive parameter update frequency control for scalable parallel deep learning. • Allowing gradient accumulation considering the direction of the gradients. • Reducing per-iteration communication cost at the input side model layers. • Overlapping gradient communications with backpropagation computations. [ABSTRACT FROM AUTHOR]

Object drift determination is a crucial issue in long-term tracking. Most existing object drift determination criteria require manually selecting different thresholds on different datasets to determine whether the object is lost. In this case, choosing the appropriate threshold is a complex problem. An object drift determination network based on dual-template joint decision-making is proposed to address this issue. The proposed object drift determination network not only does not require selection of thresholds on different datasets, and can be used as a plug-and-play module of short-term visual tracking algorithm to achieve long-term visual tracking tasks with good generalization ability. The proposed object drift determination network is applied to four short-term baseline trackers and constructs four long-term visual tracking algorithms. Experimental results verify that all four improved algorithms significantly improve long-term visual tracking performance compared to the original algorithms. In addition, the determined speed of the object drift determination network proposed in this paper reaches 111 FPS, which has little effect on the long-term visual tracking speed. • New drift determination network without threshold selection is proposed. • New template update criterion is proposed to ensure the quality of templates. • Capable of acting as a plug-and-play module for most basic trackers. • The effectiveness of the proposed method is verified on five public datasets. [ABSTRACT FROM AUTHOR]

Most deep learning-based image enhancement algorithms have been developed based on the image-to-image translation approach, in which enhancement processes are difficult to interpret. In this paper, we propose a novel interpretable image enhancement algorithm that estimates multiple transformation functions to describe complex color mapping. First, we develop a histogram-based multiple transformation function estimation network (HMTF-Net) to estimate multiple transformation functions by exploiting both the spatial and statistical information of the input images. Second, we estimate pixel-wise weight maps, which indicate the contribution of each transformation function at each pixel, based on the local structures of the input image and the transformed images obtained by each transformation function. Finally, we obtain the enhanced image as the weighted sum of the transformed images using the estimated weight maps. Extensive experiments confirm the effectiveness of the proposed approach and demonstrate that the proposed algorithm outperforms state-of-the-art image enhancement algorithms for different image enhancement tasks. [ABSTRACT FROM AUTHOR]

• Rethinking the PASCAL-VOC and MS-COCO dataset for small object detection. • There are different labeling errors in these two benchmarks by visual analysis. • Building several specific datasets to analyze small object detection. • Data quality is another factor that affects the performance of small object detection. The data and the algorithm are critical to deep learning-based small object detectors. In this paper, we rethink the PASCAL-VOC and MS-COCO dataset for small object detection. By visual analysis of the original annotations, we find that there are different labeling errors in these two datasets. To solve these problems, we build specific datasets, including SDOD, Mini6K, Mini2022 and Mini6KClean. The experimental results of several typical algorithms (e.g. SSD, YOLOv5, Faster RCNN and Deformable DETR) on the datasets show that data labeling errors (such as missing labels, category label errors, inappropriate labels) are another factor that affects the detection performance of small objects. [ABSTRACT FROM AUTHOR]

Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when". In the early years, speaker diarization algorithms were developed for speech recognition on multispeaker audio recordings to enable speaker adaptive processing. These algorithms also gained their own value as a standalone application over time to provide speaker-specific metainformation for downstream tasks such as audio retrieval. More recently, with the emergence of deep learning technology, which has driven revolutionary changes in research and practices across speech application domains, rapid advancements have been made for speaker diarization. In this paper, we review not only the historical development of speaker diarization technology but also the recent advancements in neural speaker diarization approaches. Furthermore, we discuss how speaker diarization systems have been integrated with speech recognition applications and how the recent surge of deep learning is leading the way of jointly modeling these two components to be complementary to each other. By considering such exciting technical trends, we believe that this paper is a valuable contribution to the community to provide a survey work by consolidating the recent developments with neural methods and thus facilitating further progress toward a more efficient speaker diarization. • The latest trends and approaches to speaker diarization as part of speech interaction applications. • Overview of the development of speaker diarization in the era of deep learning. • Review of diarization techniques belonging to the proposed taxonomy. • Introduction of techniques used in the traditional, modular speaker diarization systems. • Recent advancements in joint training approaches and fully end-to-end models. • A perspective of how speaker diarization has been investigated in the context of ASR. • Review of the challenges, the future of speaker diarization and the applications of speaker diarization. [ABSTRACT FROM AUTHOR]

This paper presents an algorithm for automated detection and classification of multi-cell phytoliths, one of the major components of many archaeological and paleoenvironmental deposits. This identification, based on phytolith wave pattern, is made using a pretrained VGG19 deep learning model. This approach has been tested in three key phytolith genera for the study of agricultural origins in Near East archaeology: Avena , Hordeum and Triticum. Also, this classification has been validated at species-level using Triticum boeoticum and dicoccoides images. Due to the diversity of microscopes, cameras and chemical treatments that can influence images of phytolith slides, three types of data augmentation techniques have been implemented: rotation of the images at 45-degree angles, random colour and brightness jittering, and random blur/sharpen. The implemented workflow has resulted in an overall accuracy of 93.68% for phytolith genera, improving previous attempts. The algorithm has also demonstrated its potential to automatize the classification of phytoliths species with an overall accuracy of 100%. The open code and platforms employed to develop the algorithm assure the method's accessibility, reproducibility and reusability. • Deep learning automates identification and classification of multi-cell phytoliths. • Can be applied to digitalised slides and integrated into digital microscopes. • Can differentiate between phytolith genera but also species. • Presents an overall accuracy of 93% and analyses the whole slide. • Provides complementary metrics with new relevant information on phytoliths. [ABSTRACT FROM AUTHOR]

The real-time, and accurate inference of model parameters is of great importance in many scientific and engineering disciplines that use computational models (such as a digital twin) for the analysis and prediction of complex physical processes. However, fast and accurate inference for processes of complex systems cannot easily be achieved in real-time with state-of-the-art methods under noisy real-world conditions with the requirement of a real-time response. The primary reason is that the inference of model parameters with traditional techniques based on optimization or sampling often suffers from computational and statistical challenges, resulting in a trade-off between accuracy and deployment time. In this paper, we propose a novel framework for inference of model parameters based on reinforcement learning. The proposed methodology is demonstrated and evaluated on two different physics-based models of turbofan engines. The experimental results demonstrate that the proposed methodology outperforms all other tested methods in terms of speed and robustness, with high inference accuracy. • Reinforcement learning enables real-time and end-to-end calibration of physics-based models. • The calibration policy is learnt without any labelled data or demonstration. • The constrained Lyapunov-based actor–critic algorithm provides action stability. • Learnt policies are robust to sensor noise and model bias. • The proposed RL-based method outperforms current state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The deployment of systems experiencing high-rate dynamic events, such as hypersonic vehicles, advanced weaponry, and active blast mitigation systems, require high-rate structural health monitoring (HRSHM) capabilities in the sub-millisecond realm to ensure continuous operations and safety. However, the development of high-rate feedback systems is a complex task because these dynamic systems are uniquely characterized by (1) large uncertainties in their external loads, (2) high levels of non-stationarity and heavy disturbance, and (3) unmodeled dynamics from changes in system configuration. In this paper, we present a deep learning algorithm specifically engineered for HRSHM applications. It consists of an ensemble of recurrent neural networks (RNNs) constructed with long short-term memory (LSTM) cells with transfer learning capabilities to cope with the highly limited availability of training data as it is typical for high-rate systems. The use of an ensemble of RNNs empowers multi-rate sampling capability to capture multi-temporal features of the time series, thus enabling modeling of non-stationarities. Also, because the RNNs use short-sequence LSTMs and are arranged in parallel, the computation time is substantially reduced to the sub-millisecond range. The performance of the algorithm is investigated on experimental high-rate dynamic data produced from accelerated drop tower tests and benchmarked against results from a prior investigation using a purely on-the-edge algorithm termed variable input space observer (VIO) and its hybrid architecture (hybrid VIO) that incorporated physical knowledge and is considered as an upper bound on the algorithm performance. Numerical results show that the ensemble of RNNs, composed of five RNNs, significantly outperformed the VIO, with performance close to that of the hybrid VIO when it comes to the estimation error metrics, with an average computation time of 25 μ s per prediction step. Yet, the ensemble of RNNs exhibits chattering for low-amplitude excitations, likely attributable to the behavior of RNNs sampling at small time delays. An examination of the RNN weights and hidden states confirms that the algorithm can capture multi-temporal features, and an investigation with respect to noise in the training dataset showed that the algorithm is robust for signal-to-noise ratios up to 20 dB. • Machine learning-based architecture for non-stationary time-series analysis. • Transfer learning for accelerated training with limited data. • Ensemble-based system estimation using parallel recurrent neural networks integrated with attention mechanism is proposed. • Sib-millisecond estimation of a high-rate system is achieved though the proposed architecture. [ABSTRACT FROM AUTHOR]

Person re-identification(ReID) is an intelligent video surveillance technology that retrieves the same person from different cameras. This task is extremely challenging due to changes in person poses, different camera views, and occlusion. In recent years, person ReID based on deep learning technology has received widespread attention due to the rapid development and excellent performance of deep learning. In this paper, we first divide person ReID based on deep learning approaches into seven types, i.e., fused hand-crafted features deep model, representation learning model, metric learning model, part-based deep model, video-based model, GAN-based model, unsupervised model. Furthermore, we launched a brief overview of the seven types. Then, we introduce some examples of commonly used datasets, compare the performance of some algorithms on image and video datasets in recent years, and analyze the advantages and disadvantages of various methods. Finally, we summarize the possible future research directions of person ReID technology. • Dividing person ReID based on deep learning approaches into seven types. • Introducing some examples of commonly used datasets. • Comparing the performance of some algorithms on image and video datasets in recent years. • Analyzing the advantages and disadvantages of various methods. • Summarizing the possible future research directions of person ReID technology. [ABSTRACT FROM AUTHOR]

• A neural network enhanced Boltzmann model is proposed. • The mechanical and neural models are unified into a differentiable architecture and the neural- ODE-type training strategy is constructed. • A general numerical scheme is designed to solve the universal Boltzmann equation. • Numerical experiments of homogeneous and inhomogeneous systems are provided to validate the current method. One of the biggest challenges for simulating the Boltzmann equation is the evaluation of fivefold collision integral. Given the recent successes of deep learning and the availability of efficient tools, it is an obvious idea to try to substitute the calculation of the collision operator by the evaluation of a neural network. However, it is unlcear whether this preserves key properties of the Boltzmann equation, such as conservation, invariances, the H-theorem, and fluid-dynamic limits. In this paper, we present an approach that guarantees the conservation properties and the correct fluid dynamic limit at leading order. The concept originates from a recently developed scientific machine learning strategy which has been named "universal differential equations". It proposes a hybridization that fuses the deep physical insights from classical Boltzmann modeling and the desirable computational efficiency from neural network surrogates. The construction of the method and the training strategy are demonstrated in detail. We conduct an asymptotic analysis and illustrate the multi-scale applicability of the method. The numerical algorithm for solving the neural network-enhanced Boltzmann equation is presented as well. Several numerical test cases are investigated. The results of numerical experiments show that the time-series modeling strategy enjoys the training efficiency on this supervised learning task. [ABSTRACT FROM AUTHOR]