Search

The computer-aided diagnosis of focal liver lesions (FLLs) can help improve workflow and enable correct diagnoses; FLL detection is the first step in such a computer-aided diagnosis. Despite the recent success of deep-learning-based approaches in detecting FLLs, current methods are not sufficiently robust for assessing misaligned multiphase data. By introducing an attention-guided multiphase alignment in feature space, this study presents a fully automated, end-to-end learning framework for detecting FLLs from multiphase computed tomography (CT) images. Our method is robust to misaligned multiphase images owing to its complete learning-based approach, which reduces the sensitivity of the model's performance to the quality of registration and enables a standalone deployment of the model in clinical practice. Evaluation on a large-scale dataset with 280 patients confirmed that our method outperformed previous state-of-the-art methods and significantly reduced the performance degradation for detecting FLLs using misaligned multiphase CT images. The robustness of the proposed method can enhance the clinical adoption of the deep-learning-based computer-aided detection system., IEEE TETCI. 14 pages, 8 figures, 5 tables

Learning classifiers with imbalanced data can be strongly biased toward the majority class. To address this issue, several methods have been proposed using generative adversarial networks (GANs). Existing GAN-based methods, however, do not effectively utilize the relationship between a classifier and a generator. This article proposes a novel three-player structure consisting of a discriminator, a generator, and a classifier, along with decision boundary regularization. Our method is distinctive in which the generator is trained in cooperation with the classifier to provide minority samples that gradually expand the minority decision region, improving performance for imbalanced data classification. The proposed method outperforms the existing methods on real data sets as well as synthetic imbalanced data sets.

Obtaining accurate pixel-level localization from class labels is a crucial process in weakly supervised semantic segmentation and object localization. Attribution maps from a trained classifier are widely used to provide pixel-level localization, but their focus tends to be restricted to a small discriminative region of the target object. An AdvCAM is an attribution map of an image that is manipulated to increase the classification score produced by a classifier before the final softmax or sigmoid layer. This manipulation is realized in an anti-adversarial manner, so that the original image is perturbed along pixel gradients in directions opposite to those used in an adversarial attack. This process enhances non-discriminative yet class-relevant features, which make an insufficient contribution to previous attribution maps, so that the resulting AdvCAM identifies more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and the excessive concentration of attributions on a small region of the target object. Our method achieves a new state-of-the-art performance in weakly and semi-supervised semantic segmentation, on both the PASCAL VOC 2012 and MS COCO 2014 datasets. In weakly supervised object localization, it achieves a new state-of-the-art performance on the CUB-200-2011 and ImageNet-1K datasets., IEEE TPAMI, 2022

As industries become automated and connectivity technologies advance, a wide range of systems continues to generate massive amounts of data. Many approaches have been proposed to extract principal indicators from the vast sea of data to represent the entire system state. Detecting anomalies using these indicators on time prevent potential accidents and economic losses. Anomaly detection in multivariate time series data poses a particular challenge because it requires simultaneous consideration of temporal dependencies and relationships between variables. Recent deep learning-based works have made impressive progress in this field. They are highly capable of learning representations of the large-scaled sequences in an unsupervised manner and identifying anomalies from the data. However, most of them are highly specific to the individual use case and thus require domain knowledge for appropriate deployment. This review provides a background on anomaly detection in time-series data and reviews the latest applications in the real world. Also, we comparatively analyze state-of-the-art deep-anomaly-detection models for time series with several benchmark datasets. Finally, we offer guidelines for appropriate model selection and training strategy for deep learning-based time series anomaly detection.

Deep neural networks (DNNs) have achieved impressive results in several image classification tasks. However, these architectures are unstable for adversarial examples (AEs) such as inputs crafted by a hardly perceptible perturbation with the intent of causing neural networks to make errors. AEs must be considered to prevent accidents in areas such as unmanned car driving using visual object detection in Internet of Things (IoT) networks. Gaussian noise with label smoothing or logit squeezing can be used to increase the robustness against AEs in the training of DNNs. However, from a model interpretability aspect, Gaussian noise with label smoothing does not increase the adversarial robustness of the model. To resolve this problem, we tackle the AE instead of measuring the accuracy of the model against AEs. Considering that a robust model shows a small curvature of the loss surface, we propose a metric to measure the strength of the AEs and the robustness of the model. Furthermore, we introduce a method to verify the existence of the obfuscated gradients of the model based on the black-box attack sanity check method. The proposed method enables us to identify a gradient masking problem wherein the model does not provide useful gradients and exploits false defenses. We evaluate our technique against representative adversarially trained models using the CIFAR10, CIFAR100, SVHN, and Restricted ImageNet datasets. Our results show that the performance of some false defense models decreases by up to 32% compared to the previous evaluation metrics. Moreover, our metric reveals that traditional metrics used to measure the robustness of the model may produce false results.

Despite recent developments in deep learning and their success in computer vision, model efficiency is increasingly becoming a vital factor for their deployment in various real-world applications. To provide a more effective form of computational capabilities, bio-inspired spiking neural networks (SNNs) have attracted considerable interest in recent years. SNNs are known as the third-generation of neural networks, which mimic how information is transmitted and processed in the human brain. SNNs enable sparse yet efficient information transmission through spike trains, leading to exceptional computational and energy efficiency. Nevertheless, critical challenges in SNNs to date are two-fold: (a) latency: the number of time steps required to achieve competitive results and (b) synaptic operations: the total number of spikes generated during inference. Many researchers have attempted to address these challenges, but their applications have been mainly limited to image classification. In this work, we present a threshold voltage balancing method for object detection in SNNs, which utilizes Bayesian optimization to find optimal threshold voltages in SNNs. We specifically design Bayesian optimization to consider important characteristics of SNNs such as latency and number of synaptic operations. Furthermore, we introduce two-phase threshold voltages to provide faster and more accurate object detection while providing high energy efficiency. According to experimental results, the proposed methods achieve the state-of-the-art object detection accuracy in SNNs, and converge 2x and 1.85x faster than conventional methods on PASCAL VOC and MS COCO, respectively. Moreover, the total number of synaptic operations is reduced by 40.33% and 45.31% on PASCAL VOC and MS COCO, respectively.

Bridging the exponentially growing gap between the numbers of unlabeled and labeled protein sequences, several studies adopted semi-supervised learning for protein sequence modeling. In these studies, models were pre-trained with a substantial amount of unlabeled data, and the representations were transferred to various downstream tasks. Most pre-training methods solely rely on language modeling and often exhibit limited performance. In this paper, we introduce a novel pre-training scheme called PLUS, which stands for Protein sequence representations Learned Using Structural information. PLUS consists of masked language modeling and a complementary protein-specific pre-training task, namely same-family prediction. PLUS can be used to pre-train various model architectures. In this work, we use PLUS to pre-train a bidirectional recurrent neural network and refer to the resulting model as PLUS-RNN. Our experiment results demonstrate that PLUS-RNN outperforms other models of similar size solely pre-trained with the language modeling in six out of seven widely used protein biology tasks. Furthermore, we present the results from our qualitative interpretation analyses to illustrate the strengths of PLUS-RNN. PLUS provides a novel way to exploit evolutionary relationships among unlabeled proteins and is broadly applicable across a variety of protein biology tasks. We expect that the gap between the numbers of unlabeled and labeled proteins will continue to grow exponentially, and the proposed pre-training method will play a larger role., Comment: Published in IEEE Access 2021 (https://ieeexplore.ieee.org/document/9529198)

There exist relations, which vary with time or by an event, between high dimensional elements in most real-world datasets. A dynamic graph or network has been used as one of the remarkable approaches to represent and analyze them. In spite of the advantages of representing data in the form of graphs, it is difficult to apply representation (deep) learning to graphs. Recently, AlphaFold by DeepMind has shown remarkable results in applying deep learning to graphs. This research is part of the current effort to extend the input domain of deep learning to arbitrarily graphs and their dynamics of variations. In this paper, we propose a method to predict the evolution of graphs by learning spatio-temporal features called dynamics. The method involves two main processes: extracting spatial features from static graphs obtained at different times and learning temporal features from the time-varying connection structure. Instead of predicting the overall changes of a highly complex graph, we detect the dynamic anomaly by predicting the affinity score with respect to a node (e.g., a hub as an important factor) of a dynamics graph. This facilitates the learning dynamics of graphs having sparsity of connections by alleviating the curse of dimensions using the fact that most graphs of real-world problems are scale-free. To justify our approach, we apply our method to real-world problems such as computer networks and public transportation. Experimental results show that our approach is competitive with other existing methods.

Spiking neural networks (SNNs) have emerged as energy-efficient neural networks with temporal information. SNNs have shown a superior efficiency on neuromorphic devices, but the devices are susceptible to noise, which hinders them from being applied in real-world applications. Several studies have increased noise robustness, but most of them considered neither deep SNNs nor temporal information. In this paper, we investigate the effect of noise on deep SNNs with various neural coding methods and present a noise-robust deep SNN with temporal information. With the proposed methods, we have achieved a deep SNN that is efficient and robust to spike deletion and jitter., Comment: Accepted to DAC 2021

Weakly supervised semantic segmentation produces a pixel-level localization from a classifier, but it is likely to restrict its focus to a small discriminative region of the target object. AdvCAM is an attribution map of an image that is manipulated to increase the classification score. This manipulation is realized in an anti-adversarial manner, which perturbs the images along pixel gradients in the opposite direction from those used in an adversarial attack. It forces regions initially considered not to be discriminative to become involved in subsequent classifications, and produces attribution maps that successively identify more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and limits the attributions of the regions that already have high scores. On PASCAL VOC 2012 test images, we achieve mIoUs of 68.0 and 76.9 for weakly and semi-supervised semantic segmentation respectively, which represent a new state-of-the-art., Comment: CVPR 2021

Weakly supervised segmentation methods using bounding box annotations focus on obtaining a pixel-level mask from each box containing an object. Existing methods typically depend on a class-agnostic mask generator, which operates on the low-level information intrinsic to an image. In this work, we utilize higher-level information from the behavior of a trained object detector, by seeking the smallest areas of the image from which the object detector produces almost the same result as it does from the whole image. These areas constitute a bounding-box attribution map (BBAM), which identifies the target object in its bounding box and thus serves as pseudo ground-truth for weakly supervised semantic and instance segmentation. This approach significantly outperforms recent comparable techniques on both the PASCAL VOC and MS COCO benchmarks in weakly supervised semantic and instance segmentation. In addition, we provide a detailed analysis of our method, offering deeper insight into the behavior of the BBAM., CVPR 2021

Models from machine learning (ML) or artificial intelligence (AI) increasingly assist in guiding experimental design and decision making in molecular biology and medicine. Recently, Language Models (LMs) have been adapted from Natural Language Processing (NLP) to encode the implicit language written in protein sequences. Protein LMs show enormous potential in generating descriptive representations (embeddings) for proteins from just their sequences, in a fraction of the time with respect to previous approaches, yet with comparable or improved predictive ability. Researchers have trained a variety of protein LMs that are likely to illuminate different angles of the protein language. By leveraging the bio_embeddings pipeline and modules, simple and reproducible workflows can be laid out to generate protein embeddings and rich visualizations. Embeddings can then be leveraged as input features through machine learning libraries to develop methods predicting particular aspects of protein function and structure. Beyond the workflows included here, embeddings have been leveraged as proxies to traditional homology-based inference and even to align similar protein sequences. A wealth of possibilities remain for researchers to harness through the tools provided in the following protocols. © 2021 The Authors. Current Protocols published by Wiley Periodicals LLC. The following protocols are included in this manuscript: Basic Protocol 1: Generic use of the bio_embeddings pipeline to plot protein sequences and annotations Basic Protocol 2: Generate embeddings from protein sequences using the bio_embeddings pipeline Basic Protocol 3: Overlay sequence annotations onto a protein space visualization Basic Protocol 4: Train a machine learning classifier on protein embeddings Alternate Protocol 1: Generate 3D instead of 2D visualizations Alternate Protocol 2: Visualize protein solubility instead of protein subcellular localization Support Protocol: Join embedding generation and sequence space visualization in a pipeline.

Deep neural networks are widely used and exhibit excellent performance in many areas. However, they are vulnerable to adversarial attacks that compromise networks at inference time by applying elaborately designed perturbations to input data. Although several defense methods have been proposed to address specific attacks, other types of attacks can circumvent these defense mechanisms. Therefore, we propose Purifying Variational AutoEncoder (PuVAE), a method to purify adversarial examples. The proposed method eliminates an adversarial perturbation by projecting an adversarial example on the manifold of each class and determining the closest projection as a purified sample. We experimentally illustrate the robustness of PuVAE against various attack methods without any prior knowledge about the attacks. In our experiments, the proposed method exhibits performances that are competitive with state-of-the-art defense methods, and the inference time is approximately 130 times faster than that of Defense-GAN which is a state-of-the art purifier method.

Sleep staging is an important part of clinical neurology. However, it is still performed manually by technical experts and is labor-intensive and time-consuming. To overcome these obstacles in the manual sleep staging process, a large number of machine learning-based classifiers with hand-engineered features have been proposed. Additionally, combinations of a deep neural network (DNN) have been recently highlighted as the state-of-the-art classifiers in view of their effectiveness for automatic sleep staging. In spite of the existence of a large number of these types of classifiers, to-this-date, no prior DNN-based approach has attempted sleep-stage classification using pediatric electroencephalographic (EEG) signals. In this paper, we propose a novel end-to-end classifier based on a multi-domain hybrid neural network (HNN-multi) approach consisting of a convolutional neural network and bidirectional long short-term memory for automatic sleep staging with pediatric scalp EEG recordings. To find effective temporal, spatial, and domain-specific conditions, we investigated noticeable changes in the classification performance corresponding to: 1) the length of input signals; 2) the number of channels; and 3) the types of input signals in the time and frequency domains. Our HNN-based classifier yielded the best performance metrics using 30-s time series in combination with an instantaneous frequency using a 19-channel, three-stage classification, with overall accuracy, F1 score, and Cohen's Kappa, equal to 92.21%, 0.90, and 0.88, respectively. We suggest that an effective combination of temporal and spatial time-domain clues with time-varying frequency domain information plays a pivotal role in pediatric, automatic sleep staging. Sufficiently reasonable performance of our HNN-based approach coping with the highly complicated pediatric EEG signatures hopefully sheds light on the clinical feasibility of the DNN-based automatic sleep staging for pediatric neurology.

Instantaneous drowsiness ( i.e., lapse or micro-sleep) during various activities such as driving or construction causes enormous socioeconomic losses. Thus, a virtuous cycle system that monitors a subject’s drowsiness can improve work efficiency and safety. We propose a novel framework to detect instantaneous drowsiness with only a two-second length of electroencephalography (EEG). To achieve reliable performance, we use multitaper power spectral density for feature extraction along with extreme gradient boosting as a machine learning classifier. In addition, we introduce a novel phenotype labeling of instantaneous drowsiness by combining both task dependent and independent measures of alertness (psychomotor vigilance task and electrooculography technique, respectively). The results show that our techniques outperform others used in previous studies. We also identified which spectral components ( $\theta $ , $\alpha $ , and $\gamma $ ) and channels (Fp1, Fp2, T3, T4, O1, O2, and electrocardiogram) play important roles in our drowsiness detection framework. To verify the applicability for a mobile environment, we implemented our framework on a wireless EEG, as well as on a wired EEG. We hereby present our successful results.

Cardiovascular disease is the leading cause of death in the world. It is vital to prevent it by rapid diagnosis and appropriate management through periodic blood pressure (BP) measurement. Recently, many studies have been conducted on methods to measure BP without a cuff. One of the most common methods of predicting BP without a cuff is to use the correlation between pulse wave velocity (PWV) and BP. Studies that predict BP through PWV have two problems to overcome: 1) Additional efforts are required to extract PWV features manually from various biomedical signals, such as electrocardiogram (ECG) and photoplethysmogram (PPG); and 2) in predicting BP using biomedical signals from other people, individual periodic calibration is required because the correlation between PWV and BP differs from person to person. In this study, we proposed a cuffless BP prediction method based on a deep convolutional neural network (CNN) that can overcome the problems mentioned above. The proposed CNN method 1) can use raw signals for training without PWV feature extraction; and 2) automatically learns the characteristics of biomedical signals from other people to predict BP accurately without calibration. We propose two schemes: extraction through multiple dilated convolution, and concentration through strided convolution with a large kernel, to process sequential ECG and PPG signals through CNN. BP prediction performance was the best when both ECG and PPG signals were used together. To this end, we conducted extensive experiments on the different settings of the proposed method and constructed an effective learning model. The proposed method achieved excellent performance in predicting both systolic blood pressure and diastolic blood pressure over other known approaches. We also verified that the performance of our method fulfills international standard protocols, AAMI, and BHS.

Background The conventional scores of the neuropsychological batteries are not fully optimized for diagnosing dementia despite their variety and abundance of information. To achieve low-cost high-accuracy diagnose performance for dementia using a neuropsychological battery, a novel framework is proposed using the response profiles of 2666 cognitively normal elderly individuals and 435 dementia patients who have participated in the Korean Longitudinal Study on Cognitive Aging and Dementia (KLOSCAD). Methods The key idea of the proposed framework is to propose a cost-effective and precise two-stage classification procedure that employed Mini Mental Status Examination (MMSE) as a screening test and the KLOSCAD Neuropsychological Assessment Battery as a diagnostic test using deep learning. In addition, an evaluation procedure of redundant variables is introduced to prevent performance degradation. A missing data imputation method is also presented to increase the robustness by recovering information loss. The proposed deep neural networks (DNNs) architecture for the classification is validated through rigorous evaluation in comparison with various classifiers. Results The k-nearest-neighbor imputation has been induced according to the proposed framework, and the proposed DNNs for two stage classification show the best accuracy compared to the other classifiers. Also, 49 redundant variables were removed, which improved diagnostic performance and suggested the potential of simplifying the assessment. Using this two-stage framework, we could get 8.06% higher diagnostic accuracy of dementia than MMSE alone and 64.13% less cost than KLOSCAD-N alone. Conclusion The proposed framework could be applied to general dementia early detection programs to improve robustness, preciseness, and cost-effectiveness.

Automated diagnosis using deep neural networks in chest radiography can help radiologists detect life-threatening diseases. However, existing methods only provide predictions without accurate explanations, undermining the trustworthiness of the diagnostic methods. Here, we present XProtoNet, a globally and locally interpretable diagnosis framework for chest radiography. XProtoNet learns representative patterns of each disease from X-ray images, which are prototypes, and makes a diagnosis on a given X-ray image based on the patterns. It predicts the area where a sign of the disease is likely to appear and compares the features in the predicted area with the prototypes. It can provide a global explanation, the prototype, and a local explanation, how the prototype contributes to the prediction of a single image. Despite the constraint for interpretability, XProtoNet achieves state-of-the-art classification performance on the public NIH chest X-ray dataset., 10 pages, 7 figures. Accepted to CVPR2021

Motivation: MicroRNAs (miRNAs) play pivotal roles in gene expression regulation by binding to target sites of messenger RNAs (mRNAs). While identifying functional targets of miRNAs is of utmost importance, their prediction remains a great challenge. Previous computational algorithms have major limitations. They use conservative candidate target site (CTS) selection criteria mainly focusing on canonical site types, rely on laborious and time-consuming manual feature extraction, and do not fully capitalize on the information underlying miRNA-CTS interactions. Results: In this paper, we introduce TargetNet, a novel deep learning-based algorithm for functional miRNA target prediction. To address the limitations of previous approaches, TargetNet has three key components: (1) relaxed CTS selection criteria accommodating irregularities in the seed region, (2) a novel miRNA-CTS sequence encoding scheme incorporating extended seed region alignments, and (3) a deep residual network-based prediction model. The proposed model was trained with miRNA-CTS pair datasets and evaluated with miRNA-mRNA pair datasets. TargetNet advances the previous state-of-the-art algorithms used in functional miRNA target classification. Furthermore, it demonstrates great potential for distinguishing high-functional miRNA targets., Comment: This article has been accepted for publication in Bioinformatics published by Oxford University Press

In this paper, we address the problem of image anomaly detection and segmentation. Anomaly detection involves making a binary decision as to whether an input image contains an anomaly, and anomaly segmentation aims to locate the anomaly on the pixel level. Support vector data description (SVDD) is a long-standing algorithm used for an anomaly detection, and we extend its deep learning variant to the patch-based method using self-supervised learning. This extension enables anomaly segmentation and improves detection performance. As a result, anomaly detection and segmentation performances measured in AUROC on MVTec AD dataset increased by 9.8% and 7.0%, respectively, compared to the previous state-of-the-art methods. Our results indicate the efficacy of the proposed method and its potential for industrial application. Detailed analysis of the proposed method offers insights regarding its behavior, and the code is available online (https://github.com/nuclearboy95/Anomaly-Detection-PatchSVDD-PyTorch).

Recent algorithmic advances in electrocardiogram (ECG) classification are largely contributed to deep learning. However, these methods are still based on a relatively straightforward application of deep neural networks (DNNs), which leaves incredible room for improvement. In this paper, as part of the PhysioNet / Computing in Cardiology Challenge 2020, we developed an 18-layer residual convolutional neural network to classify clinical cardiac abnormalities from 12-lead ECG recordings. We focused on examining a collection of data pre-processing, model architecture, training, and post-training procedure refinements for DNN-based ECG classification. We showed that by combining these refinements, we can improve the classification performance significantly. Our team, DSAIL_SNU, obtained a 0.695 challenge score using 10-fold cross-validation, and a 0.420 challenge score on the full test data, placing us 6th in the official ranking.

In the oil and gas exploration, well logs, the most convenient and economic data source, usually contain missing values due to various reasons. It is crucial to generate accurate synthetic logs for such missing intervals in terms of precise well log interpretation. In this paper, we propose a workflow of generating synthetic logs using cutting-edge machine learning techniques. Unlike existing methods, we exploit a generative model, which can deal with various missing patterns with a single model, and we combine it with a supervised model. With well log data of various regions, we show that our models accurately generate missing logs and outperforms existing supervised-only models. It is expected that our model is beneficial in the real field because of its performance and simplicity.

Heart disease and stroke are the leading causes of death worldwide. High blood pressure greatly increases the risk of heart disease and stroke. Therefore, it is important to control blood pressure (BP) through regular BP monitoring; as such, it is necessary to develop a method to accurately and conveniently predict BP in a variety of settings. In this paper, we propose a method for predicting BP without feature extraction using fully convolutional neural networks (CNNs). We measured single multi-wave photoplethysmography (PPG) signals using a smartphone. To find an effective wavelength of PPG signals for the generation of accurate BP measurements, we investigated the BP prediction performance by changing the combinations of the input PPG signals. Our CNN-based BP predictor yielded the best performance metrics when a green PPG time signal was used in combination with an instantaneous frequency signal. This combination had an overall mean absolute error (MAE) of 5.28 and 4.92 mmHg for systolic and diastolic BP, respectively. Thus, our CNN-based approach achieved comparable results to other approaches that use a single PPG signal.

Adversarial examples cause neural networks to produce incorrect outputs with high confidence. Although adversarial training is one of the most effective forms of defense against adversarial examples, unfortunately, a large gap exists between test accuracy and training accuracy in adversarial training. In this paper, we identify Adversarial Feature Overfitting (AFO), which may cause poor adversarially robust generalization, and we show that adversarial training can overshoot the optimal point in terms of robust generalization, leading to AFO in our simple Gaussian model. Considering these theoretical results, we present soft labeling as a solution to the AFO problem. Furthermore, we propose Adversarial Vertex mixup (AVmixup), a soft-labeled data augmentation approach for improving adversarially robust generalization. We complement our theoretical analysis with experiments on CIFAR10, CIFAR100, SVHN, and Tiny ImageNet, and show that AVmixup significantly improves the robust generalization performance and that it reduces the trade-off between standard accuracy and adversarial robustness., To appear in CVPR 2020 (Oral)

The interpretation of well logs is crucial for understanding subsurface geological formations and reservoir characterization. Categorizing subsurface into several geological facies types assists in identifying potential locations of hydrocarbon resources. Geologists interpret each well log feature such as gamma ray, sonic velocity, neutron porosity and resistivity, based on logging characteristics and their petrophysical domain knowledge. By integrating with core analysis result, well logs interpretation result becomes accurate, and geologists finally determine rock types of subsurface for the target well. Recently, various studies propose to leverage the power of machine learning algorithms for facies classification to reduce laborious interpretation work. Most works utilize supervised learning algorithms that require input/label paired data for training. In case of facies classification, however, the accurate core analysis for determining facies is an expensive process, and the other methods require hard work and often result in noisy data. In this paper, we develop an unsupervised facies identification model based on deep neural networks. Our model is purely unsupervised, which do not require facies label for input well logs. Because well logs response from the same rock type would have high similarity, we apply an unsupervised clustering algorithm, where data points with similar traits are assigned to the same group, while dissimilar data points are assigned to the different groups. We demonstrate that our unsupervised facies identification model outperforms other unsupervised clustering algorithms for facies identification. Particularly, our clustered result shows highly similar distribution to the true facies distribution, which suggests future development of automated facies identification.

Spiking neural networks (SNNs) have gained considerable interest due to their energy-efficient characteristics, yet lack of a scalable training algorithm has restricted their applicability in practical machine learning problems. The deep neural network-to-SNN conversion approach has been widely studied to broaden the applicability of SNNs. Most previous studies, however, have not fully utilized spatio-temporal aspects of SNNs, which has led to inefficiency in terms of number of spikes and inference latency. In this paper, we present T2FSNN, which introduces the concept of time-to-first-spike coding into deep SNNs using the kernel-based dynamic threshold and dendrite to overcome the aforementioned drawback. In addition, we propose gradient-based optimization and early firing methods to further increase the efficiency of the T2FSNN. According to our results, the proposed methods can reduce inference latency and number of spikes to 22% and less than 1%, compared to those of burst coding, which is the state-of-the-art result on the CIFAR-100., Accepted to DAC 2020

Drug metabolism is determined by the biochemical and physiological properties of the drug molecule. To improve the performance of a drug property prediction model, it is important to extract complex molecular dynamics from limited data. Recent machine learning or deep learning based models have employed the atom- and bond-type information, as well as the structural information to predict drug properties. However, many of these methods can be used only for the graph representations. Message passing neural networks (MPNNs) (Gilmer et al., 2017) is a framework used to learn both local and global features from irregularly formed data, and is invariant to permutations. This network performs an iterative message passing (MP) operation on each object and its neighbors, and obtain the final output from all messages regardless of their order. In this study, we applied the MP-based attention network (Nikolentzos et al., 2019) originally developed for text learning to perform chemical classification tasks. Before training, we tokenized the characters, and obtained embeddings of each molecular sequence. We conducted various experiments to maximize the predictivity of the model. We trained and evaluated our model using various chemical classification benchmark tasks. Our results are comparable to previous state-of-the-art and baseline models or outperform. To the best of our knowledge, this is the first attempt to learn chemical strings using an MP-based algorithm. We will extend our work to more complex tasks such as regression or generation tasks in the future.

We propose an interpretable Capsule Network, \(\textit{iCaps}\), for image classification. A capsule is a group of neurons nested inside each layer, and the one in the last layer is called a class capsule, which is a vector whose norm indicates a predicted probability for the class. Using the class capsule, existing Capsule Networks already provide some level of interpretability. However, there are two limitations which degrade its interpretability: 1) the class capsule also includes classification-irrelevant information, and 2) entities represented by the class capsule overlap. In this work, we address these two limitations using a novel class-supervised disentanglement algorithm and an additional regularizer, respectively. Through quantitative and qualitative evaluations on three datasets, we demonstrate that the resulting classifier, \(\textit{iCaps}\), provides a prediction along with clear rationales behind it with no performance degradation.

To demystify the "black box" property of deep neural networks for natural language processing (NLP), several methods have been proposed to interpret their predictions by measuring the change in prediction probability after erasing each token of an input. Since existing methods replace each token with a predefined value (i.e., zero), the resulting sentence lies out of the training data distribution, yielding misleading interpretations. In this study, we raise the out-of-distribution problem induced by the existing interpretation methods and present a remedy; we propose to marginalize each token out. We interpret various NLP models trained for sentiment analysis and natural language inference using the proposed method., Comment: 10 pages, 5 figures, to be published in the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)

Drugs are classified according to their biological and chemical reactions, and the systems that they target. Thus, an accurate and efficient prediction method for drug class discovery would reveal key properties of candidate drugs, significantly conserving time and resources in drug repositioning and design. Previous approaches, based on data mining or statistics, required complicated feature construction in advance. Knowing that deep learning can identifying patterns in high-dimensional datasets without elaborate feature selection or engineering, we constructed a model for predicting drug classes using deep neural networks - with biological and chemical structure data. Our proposed model outperforms previous learning-based methods in terms of prediction accuracy.

A deep learning–based model predicts activity of a widely used genome editing tool in a highly accurate manner., We evaluated SpCas9 activities at 12,832 target sequences using a high-throughput approach based on a human cell library containing single-guide RNA–encoding and target sequence pairs. Deep learning–based training on this large dataset of SpCas9-induced indel frequencies led to the development of a SpCas9 activity–predicting model named DeepSpCas9. When tested against independently generated datasets (our own and those published by other groups), DeepSpCas9 showed high generalization performance. DeepSpCas9 is available at http://deepcrispr.info/DeepSpCas9.

Background Quantitative structure-activity relationship (QSAR) is a computational modeling method for revealing relationships between structural properties of chemical compounds and biological activities. QSAR modeling is essential for drug discovery, but it has many constraints. Ensemble-based machine learning approaches have been used to overcome constraints and obtain reliable predictions. Ensemble learning builds a set of diversified models and combines them. However, the most prevalent approach random forest and other ensemble approaches in QSAR prediction limit their model diversity to a single subject. Results The proposed ensemble method consistently outperformed thirteen individual models on 19 bioassay datasets and demonstrated superiority over other ensemble approaches that are limited to a single subject. The comprehensive ensemble method is publicly available at http://data.snu.ac.kr/QSAR/. Conclusions We propose a comprehensive ensemble method that builds multi-subject diversified models and combines them through second-level meta-learning. In addition, we propose an end-to-end neural network-based individual classifier that can automatically extract sequential features from a simplified molecular-input line-entry system (SMILES). The proposed individual models did not show impressive results as a single model, but it was considered the most important predictor when combined, according to the interpretation of the meta-learning.

With the recent success of deep learning, the amount of data and computation continues to grow daily. Hence a distributed deep learning system that shares the training workload has been researched extensively. Although a scale-out distributed environment using commodity servers is widely used, not only is there a limit due to synchronous operation and communication traffic but also combining deep neural network (DNN) training with existing clusters often demands additional hardware and migration between different cluster frameworks or libraries, which is highly inefficient. Therefore, we propose TensorLightning which integrates the widely used data pipeline of Apache Spark with powerful deep learning libraries, Caffe and TensorFlow. TensorLightning embraces a brand-new parameter aggregation algorithm and parallel asynchronous parameter managing schemes to relieve communication discrepancies and overhead. We redesign the elastic averaging stochastic gradient descent algorithm with pruned and sparse form parameters. Our approach provides the fast and flexible DNN training with high accessibility. We evaluated our proposed framework with convolutional neural network and recurrent neural network models; the framework reduces network traffic by 67% with faster convergence.

Heat shock proteins (HSPs) play a pivotal role as molecular chaperones against unfavorable conditions. Although HSPs are of great importance, their computational identification remains a significant challenge. Previous studies have two major limitations. First, they relied heavily on amino acid composition features, which inevitably limited their prediction performance. Second, their prediction performance was overestimated because of the independent two-stage evaluations and train-test data redundancy. To overcome these limitations, we introduce two novel deep learning algorithms: (1) time-efficient DeepHSP and (2) high-performance DeeperHSP. We propose a convolutional neural network (CNN)-based DeepHSP that classifies both non-HSPs and six HSP families simultaneously. It outperforms state-of-the-art algorithms, despite taking 14–15 times less time for both training and inference. We further improve the performance of DeepHSP by taking advantage of protein transfer learning. While DeepHSP is trained on raw protein sequences, DeeperHSP is trained on top of pre-trained protein representations. Therefore, DeeperHSP remarkably outperforms state-of-the-art algorithms increasing F1 scores in both cross-validation and independent test experiments by 20% and 10%, respectively. We envision that the proposed algorithms can provide a proteome-wide prediction of HSPs and help in various downstream analyses for pathology and clinical research.

Highlights • The emotional public responses to a nation-wide outbreak of Middle East respiratory syndrome (MERS) in Korea was analyzed. • Massive media outlet data was collected during the outbreak. • An intriguing loop of information transfers between the media and the public was discovered. • This method would be helpful for alleviating the unnecessary fear and overreaction of the public regarding infectious diseases., From May to July 2015, there was a nation-wide outbreak of Middle East respiratory syndrome (MERS) in Korea. MERS is caused by MERS-CoV, an enveloped, positive-sense, single-stranded RNA virus belonging to the family Coronaviridae. Despite expert opinions that the danger of MERS might be exaggerated, there was an overreaction by the public according to the Korean mass media, which led to a noticeable reduction in social and economic activities during the outbreak. To explain this phenomenon, we presumed that machine learning-based analysis of media outlets would be helpful and collected a number of Korean mass media articles and short-text comments produced during the 10-week outbreak. To process and analyze the collected data (over 86 million words in total) effectively, we created a methodology composed of machine-learning and information-theoretic approaches. Our proposal included techniques for extracting emotions from emoticons and Internet slang, which allowed us to significantly (approximately 73%) increase the number of emotion-bearing texts needed for robust sentiment analysis of social media. As a result, we discovered a plausible explanation for the public overreaction to MERS in terms of the interplay between the disease, mass media, and public emotions.

NAND flash-based storage devices (NFSDs) are widely employed owing to their superior characteristics when compared to hard disk drives. However, NAND flash memory (NFM) still exhibits drawbacks, such as a limited lifetime and an erase-before-write requirement. Along with effective software management, the implementation of a cache buffer is one of the most common solutions to overcome these limitations. However, the read/write performance becomes saturated primarily because the eviction overhead caused by limited DRAM capacity significantly impacts overall NFSD performance. This paper therefore proposes a method that hides the eviction overhead and overcomes the saturation of the read/write performance. The proposed method exploits the new intra-request idle time (IRIT) in NFSD and employs a new data management scheme. In addition, the new pre-store eviction scheme stores dirty page data in the cache to NFMs in advance. This reduces the eviction overhead by maintaining a sufficient number of clean pages in the cache. Further, the new pre-load insertion scheme improves the read performance by frequently loading data that needs to be read into the cache in advance. Unlike previous methods with large migration overhead, our scheme does not cause any eviction/insertion overhead because it actually exploits the IRIT to its advantage. We verified the effectiveness of our method, by integrating it into two cache management strategies which were then compared. Our proposed method reduced read latency by 43% in read-intensive traces, reduced write latency by 40% in write-intensive traces, and reduced read/write latency by 21% and 20%, respectively, on average compared to NFSD with a conventional write cache buffer.

When a deep neural network is trained on data with only image-level labeling, the regions activated in each image tend to identify only a small region of the target object. We propose a method of using videos automatically harvested from the web to identify a larger region of the target object by using temporal information, which is not present in the static image. The temporal variations in a video allow different regions of the target object to be activated. We obtain an activated region in each frame of a video, and then aggregate the regions from successive frames into a single image, using a warping technique based on optical flow. The resulting localization maps cover more of the target object, and can then be used as proxy ground-truth to train a segmentation network. This simple approach outperforms existing methods under the same level of supervision, and even approaches relying on extra annotations. Based on VGG-16 and ResNet 101 backbones, our method achieves the mIoU of 65.0 and 67.4, respectively, on PASCAL VOC 2012 test images, which represents a new state-of-the-art., ICCV 2019

The spiking neural networks (SNNs) are considered as one of the most promising artificial neural networks due to their energy efficient computing capability. Recently, conversion of a trained deep neural network to an SNN has improved the accuracy of deep SNNs. However, most of the previous studies have not achieved satisfactory results in terms of inference speed and energy efficiency. In this paper, we propose a fast and energy-efficient information transmission method with burst spikes and hybrid neural coding scheme in deep SNNs. Our experimental results showed the proposed methods can improve inference energy efficiency and shorten the latency., Accepted to DAC 2019

The main obstacle to weakly supervised semantic image segmentation is the difficulty of obtaining pixel-level information from coarse image-level annotations. Most methods based on image-level annotations use localization maps obtained from the classifier, but these only focus on the small discriminative parts of objects and do not capture precise boundaries. FickleNet explores diverse combinations of locations on feature maps created by generic deep neural networks. It selects hidden units randomly and then uses them to obtain activation scores for image classification. FickleNet implicitly learns the coherence of each location in the feature maps, resulting in a localization map which identifies both discriminative and other parts of objects. The ensemble effects are obtained from a single network by selecting random hidden unit pairs, which means that a variety of localization maps are generated from a single image. Our approach does not require any additional training steps and only adds a simple layer to a standard convolutional neural network; nevertheless it outperforms recent comparable techniques on the Pascal VOC 2012 benchmark in both weakly and semi-supervised settings., To appear in CVPR 2019

We evaluated SpCas9 activities at 12,832 target sequences using a high-throughput approach based on a human cell library containing sgRNA-encoding and target sequence pairs. Deep learning-based training on this large data set of SpCas9-induced indel frequencies led to the development of a SpCas9-activity predicting model named DeepSpCas9. When tested against independently generated data sets (our own and those published by other groups), DeepSpCas9 showed unprecedentedly high generalization performance. DeepSpCas9 is available athttp://deepcrispr.info/DeepCas9.

Attacks on embedded devices are likely to occur any time in unexpected manners. Thus, the defense systems based on fixed sets of rules will easily be subverted by such unexpected, unknown attacks. Learning-based anomaly detection may potentially prevent new unknown zero-day attacks by leveraging the capability of machine learning (ML) to learn the intricate true nature of software hidden within raw information. This paper introduces our work to develop an MPSoC, called RTAD, which can efficiently support in hardware various ML models that run to detect anomalous behaviors on embedded devices in a real-time fashion, and thus enable the devices to counteract the anomalies in the field. In the IoT era, the importance of security for embedded devices cannot be exaggerated because they will become an enticing target for adversaries as they are being integrated into everyday life to provide users with various services. The above-mentioned potential of learning-based detection is believed to benefit those deployed devices under attacks occurring any time during their field operations in unexpected manners. We hereby assume that ML models are trained with runtime branch information as their data features since a sequence of branches serves as a record of control flow transfers during program execution. In fact, there have been numerous ML studies that examine various types of branches in order to infer (or detect) anomaly in branch behaviors that may be induced by diverse attacks that can cause deviant control flow in software. Our goal of real-time anomalous branch behavior inference poses two challenges to our development of RTAD. Firstly, RTAD must collect and transfer in a timely fashion a sequence of branches as the input to the ML model. Secondly, RTAD must be able to promptly process the delivered branch data with the ML model. To tackle these challenges, we have implemented in RTAD two core components: an input generation module and a GPU-inspired ML processing engine. According to our experiments, RTAD enables various ML models to infer anomaly instantly after the victim program behaves aberrantly as the result of attacks being injected into the system.

Recent advances in next-generation sequencing technologies have facilitated the use of deoxyribonucleic acid (DNA) as a novel covert channels in steganography. There are various methods that exist in other domains to detect hidden messages in conventional covert channels. However, they have not been applied to DNA steganography. The current most common detection approaches, namely frequency analysis-based methods, often overlook important signals when directly applied to DNA steganography because those methods depend on the distribution of the number of sequence characters. To address this limitation, we propose a general sequence learning-based DNA steganalysis framework. The proposed approach learns the intrinsic distribution of coding and non-coding sequences and detects hidden messages by exploiting distribution variations after hiding these messages. Using deep recurrent neural networks (RNNs), our framework identifies the distribution variations by using the classification score to predict whether a sequence is to be a coding or non-coding sequence. We compare our proposed method to various existing methods and biological sequence analysis methods implemented on top of our framework. According to our experimental results, our approach delivers a robust detection performance compared to other tools.

Neural networks (NNs) are becoming the tool of choice for sharpening blurred images. We discuss and categorize deblurring NNs. Then we evaluate seven NNs for non-blind deblurring (NBD), and seven NNs and four optimization techniques for blind deblurring (BD). To do this we use several current datasets containing pairs of sharp and blurred images, synthesized either by convolving sharp images with blur kernels or by averaging consecutive sharp images, so as to produce both uniform and non-uniform blurs. We also introduce a newly reorganized benchmark dataset in which blurred images have been classified using attributes that depend on the extent of the blur. We use this dataset to compare the effectiveness of single and multi-scale training in coping with large blurs. On NBD, NNs that use regularization with a denoising prior network outperform other denoising NNs; and NNs that use a deep image prior network outperform other deconvolution NNs. On BD, NNs outperform optimizations in signal-difference terms, but not in terms of perceptual fidelity. We found that multi-scale training helps NNs to deal with large blurs, and RNNs outperform CNNs. We also observed that GANs using a perceptual loss function produce artifacts; but also that some form of perceptual fidelity loss is required to get the best results from NNs. We contend that the domain bias of current datasets works against robustness and generality. And we discuss the potential of more sophisticated perceptual loss functions, attention techniques, and unsupervised learning.