Search

Hand gestures are becoming an important part of the communication method between humans and machines in the era of fast-paced urbanization. This paper introduces a new standard dataset for hand gesture recognition, Static HAnd PosturE (SHAPE), with adequate side, variation, and practicality. Compared with the previous datasets, our dataset has more classes, subjects, or scenes than other datasets. In addition, the SHAPE dataset is also one of the first datasets to focus on Asian subjects with Asian hand gestures. The SHAPE dataset contains more than 34,000 images collected from 20 distinct subjects with different clothes and backgrounds. A recognition architecture is also presented to investigate the proposed dataset. The architecture consists of two phases that are the hand detection phase for preprocessing and the classification phase by customized state-of-the-art deep neural network models. This paper investigates not only the high accuracy, but also the lightweight hand gesture recognition models that are suitable for resource-constrained devices such as portable edge devices. The promising application of this study is to create a human–machine interface that solves the problem of insufficient space for a keyboard or a mouse in small devices. Our experiments showed that the proposed architecture could obtain high accuracy with the self-built dataset. Details of our dataset can be seen online at https://users.soict.hust.edu.vn/linhdt/dataset/ [ABSTRACT FROM AUTHOR]

In many public security applications such as anomaly detection, it is important to re-identify a group of pedestrians by other surveillance cameras, which ascribes to the group retrieval problem. Most previous studies focus on single-person re-identification (re-id) and ignore the correlations among group members, and they lack a large and comprehensive group retrieval benchmark to associate these two tasks. To address this issue, this paper focuses on solving the group retrieval problem and uses it to improve re-id. First, the paper build a comprehensive benchmark for both group retrieval and the group-aided re-id task by proposing a novel pedestrian group retrieval dataset named "SYSU-Group" and a corresponding group-associated re-id dataset named "Group-reID", which introduces realistic challenges such as variations of pose, viewpoint, illumination, and intra-group layout. The paper then proposes the Siamese Verification-Identification-based Group Retrieval (SVIGR) method, which combines verification and identification modules in a Siamese network to extract robust person features and follows the principle of minimum distance matching to realize group retrieval. Finally, a group-guided re-id method named group retrieval correlation (GRC) is proposed to improve re-id with additional group information. Experimental results on three various group retrieval benchmarks demonstrate the superiority and effectiveness of our method. [ABSTRACT FROM AUTHOR]

Due to its insensitivity to light intensity and the capability to capture multidimensional information, polarimetric imaging technology has been proven to have advantages over traditional intensity-based imaging techniques for object detection tasks in adverse environmental conditions, particularly in road traffic scenarios. Recently, with the rapid development of artificial intelligence technology, deep learning (DL)-powered object detection techniques can further enhance recognition accuracy and algorithm robustness. This improvement is made possible by the ability of DL technology to leverage large datasets and extract deeper levels of target-specific features. However, constructing large-scale polarimetric datasets poses challenges as obtaining polarimetric information requires multiple captures of intensity images. In other words, the workload is several times higher compared to traditional imaging techniques. To address the current scarcity of polarimetric datasets and evaluate the practical performance of various algorithms on polarimetric datasets, this paper proposes a Polarimetric Object Detection Benchmark (PODB) dataset. The PODB provides an integrated quality evaluation framework for DL-based object detection algorithms in complex road scenes by incorporating polarimetric imaging. Besides, we conducted extensive object detection experiments using the PODB, which allowed for a comprehensive validation and performance evaluation of effective benchmark algorithms. Furthermore, a physics-based multi-scale image fusion cascaded object detection neural network model is proposed. By combining the multidimensional information provided by polarized images with an adaptive learning multi-decision object detection neural network model, the recognition accuracy of complex road scenes in adverse weather conditions has been improved by approximately 10%. Additionally, we anticipate that PODB will serve as an effective platform for evaluating and comparing the performance of object detection algorithms, as well as providing researchers with a baseline for future studies in developing new DL-based methods. • Polarimetric imaging excels in challenging conditions, like road traffic. • Scarce polarimetric road data due to acquisition difficulties. • Proposed Polarimetric Object Detection Benchmark dataset for deep learning-based algorithms. • Introduction of a novel multi-scale image fusion cascaded neural network model. • Achieving a 10% accuracy boost for complex, adverse weather road scenes. [ABSTRACT FROM AUTHOR]

Data splitting is an important step in the artificial neural network (ANN) development process, whereby the available data are divided into training, testing, and validation subsets to ensure good generalization ability of the model. Considering that only one split of the data is typically used when developing ANN models, data splitting has a significant impact on model performance, depending on which data are allocated to the three subsets. Therefore, it is important to find a data splitting method that consistently results in predictive validation errors that are representative of the predictive errors obtained over the full range of the available data. This paper addresses this issue by introducing a benchmarking approach for comparing different data splitting methods in terms of (1) bias, which is the difference between the expected validation performance over the entire data set and that obtained using a particular data splitting method and (2) variability, which is the spread of the validation errors obtained by repeated implementation of that method. The utility of the proposed approach is assessed on a number of well-known data splitting methods in the context of four water resources ANN modelling problems. The results obtained indicate that the proposed approach for comparing data splitting methods is more representative than the previous approach where a value of zero is used as the predictive performance benchmark, as it can avoid the selection of an over-optimistic data splitting method that under-represents extreme data in the validation set. [ABSTRACT FROM AUTHOR]

Large-scale neuromorphic hardware platforms, specialized computer systems for energy efficient simulation of spiking neural networks, are being developed around the world, for example as part of the European Human Brain Project (HBP). Due to conceptual differences, a universal performance analysis of these systems in terms of runtime, accuracy and energy efficiency is non-trivial, yet indispensable for further hard- and software development. In this paper we describe a scalable benchmark based on a spiking neural network implementation of the binary neural associative memory. We treat neuromorphic hardware and software simulators as black-boxes and execute exactly the same network description across all devices. Experiments on the HBP platforms under varying configurations of the associative memory show that the presented method allows to test the quality of the neuron model implementation, and to explain significant deviations from the expected reference output. [ABSTRACT FROM AUTHOR]

GPUs have been broadly used to accelerate big data analytics, scientific computing and machine intelligence. Particularly, matrix multiplication and convolution are two principal operations that use a large proportion of steps in modern data analysis and deep neural networks. These performance-critical operations are often offloaded to the GPU to obtain substantial improvements in end-to-end latency. In addition, multifarious workload characteristics and complicated processing phases in big data demand a customizable yet performant operator library. To this end, GPU vendors, including NVIDIA and AMD, have proposed template and composable GPU operator libraries to conduct specific computations on certain types of low-precision data elements. We formalize a set of benchmarks via CUTLASS, NVIDIA's templated library that provides high-performance and hierarchically designed kernels. The benchmarking results show that, with the necessary fine tuning, hardware-level ASICs like tensor cores could dramatically boost performance in specific operations like GEMM offloading to modern GPUs. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of " what workloads are ideal for SNNs and how to evaluate SNNs makes sense ". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics. [ABSTRACT FROM AUTHOR]

Cardiovascular disease (CVD) accounts for about half of non-communicable diseases. Vessel stenosis in the coronary artery is considered to be the major risk of CVD. Computed tomography angiography (CTA) is one of the widely used noninvasive imaging modalities in coronary artery diagnosis due to its superior image resolution. Clinically, segmentation of coronary arteries is essential for the diagnosis and quantification of coronary artery disease. Recently, a variety of works have been proposed to address this problem. However, on one hand, most works rely on in-house datasets, and only a few works published their datasets to the public which only contain tens of images. On the other hand, their source code have not been published, and most follow-up works have not made comparison with existing works, which makes it difficult to judge the effectiveness of the methods and hinders the further exploration of this challenging yet critical problem in the community. In this paper, we propose a large-scale dataset for coronary artery segmentation on CTA images. In addition, we have implemented a benchmark in which we have tried our best to implement several typical existing methods. Furthermore, we propose a strong baseline method which combines multi-scale patch fusion and two-stage processing to extract the details of vessels. Comprehensive experiments show that the proposed method achieves better performance than existing works on the proposed large-scale dataset. The benchmark and the dataset are published at https://github.com/XiaoweiXu/ImageCAS-A-Large-Scale-Dataset-and-Benchmark-for-Coronary-Artery-Segmentation-based-on-CT. • Tens of works on coronary artery segmentation but without large-scale public datasets. • Propose the first large-scale (1,000 CTs) dataset for coronary artery segmentation. • Propose a benchmark including typical existing works for fair comparison. • A strong baseline method is also proposed with the optimal performance. [ABSTRACT FROM AUTHOR]

A Manhattan search algorithm to minimize artificial neural network error function is outlined in this paper. From an existing position in Cartesian coordinate, a search vector moves in orthogonal directions to locate minimum function value. The search algorithm computes optimized step length for rapid convergence. This step is performed when consecutive search is successful in minimizing function value. The optimized step length identifies favorable descent direction to minimize function value. The search method is suitable for complex error surface where derivative information is difficult to obtain or when the error surface is nearly flat. The rate of change in function value is almost negligible near the flat surface. Most of the derivative based training algorithm faces difficulty in such scenarios. This algorithm avoids derivative information of an error function. Therefore, it is an attractive search method when derivative based algorithm faces difficulty due to complex ridges and flat valleys. In case the algorithm gets into trapped minimum, the search vector takes steps to move out of a local minimum by exploring neighborhood descent search directions. The algorithm differs from the first and second order derivative based training methods. To measure the performance of the algorithm, estimation of electric energy generation model from Fiji Islands and 'L-T' letter recognition problems are solved. Bootstrap analysis shows that the algorithm's predictive and classification abilities are high. The algorithm is reliable when solution to a problem is unknown. Therefore, the algorithm identifies benchmark solution. [ABSTRACT FROM AUTHOR]

Abstract Background Several recent publications have demonstrated the use of convolutional neural networks to classify images of melanoma at par with board-certified dermatologists. However, the non-availability of a public human benchmark restricts the comparability of the performance of these algorithms and thereby the technical progress in this field. Methods An electronic questionnaire was sent to dermatologists at 12 German university hospitals. Each questionnaire comprised 100 dermoscopic and 100 clinical images (80 nevi images and 20 biopsy-verified melanoma images, each), all open-source. The questionnaire recorded factors such as the years of experience in dermatology, performed skin checks, age, sex and the rank within the university hospital or the status as resident physician. For each image, the dermatologists were asked to provide a management decision (treat/biopsy lesion or reassure the patient). Main outcome measures were sensitivity, specificity and the receiver operating characteristics (ROC). Results Total 157 dermatologists assessed all 100 dermoscopic images with an overall sensitivity of 74.1%, specificity of 60.0% and an ROC of 0.67 (range = 0.538–0.769); 145 dermatologists assessed all 100 clinical images with an overall sensitivity of 89.4%, specificity of 64.4% and an ROC of 0.769 (range = 0.613–0.9). Results between test-sets were significantly different (P < 0.05) confirming the need for a standardised benchmark. Conclusions We present the first public melanoma classification benchmark for both non-dermoscopic and dermoscopic images for comparing artificial intelligence algorithms with diagnostic performance of 145 or 157 dermatologists. Melanoma Classification Benchmark should be considered as a reference standard for white-skinned Western populations in the field of binary algorithmic melanoma classification. Highlights • This paper provides the first open access melanoma classification benchmark for both non-dermoscopic and dermoscopic images. • Algorithms can now be easily compared to the performance of dermatologists in terms of sensitivity, specificity and ROC. • The melanoma benchmark allows comparability between algorithms of different publications and provides a new reference standard. [ABSTRACT FROM AUTHOR]

With the development of data centers, network bandwidth has rapidly increased, reaching hundreds of Gbps. However, the network I/O processing performance of CPU improvement has not kept pace with this growth in recent years, which leads to the CPU being increasingly burdened by network applications in data centers. To address this issue, Data Processing Unit (DPU) has emerged as a hardware accelerator designed to offload network applications from the CPU. As a new hardware device, the DPU architecture design is still in the exploration stage. Previous DPU benchmarks are not neutral and comprehensive, making them unsuitable as general benchmarks. To showcase the advantages of their specific architectural features, DPU vendors tend to provide some particular architecture-dependent evaluation programs. Moreover, they fail to provide comprehensive coverage and cannot adequately represent the full range of network applications. To address this gap, we propose an application-driven scalable benchmark suite called DPUBench. DPUBench classifies DPU applications into three typical scenarios — network, storage, and security, and includes a scalable benchmark framework that contains essential Operator Set in these scenarios and End-to-end Evaluation Programs in real data center scenarios. DPUBench can easily incorporate new operators and end-to-end evaluation programs as DPU evolves. We present the results of evaluating the NVIDIA BlueField-2 using DPUBench and provide optimization recommendations. [ABSTRACT FROM AUTHOR]

Background: Recently, deep neural networks have been successfully applied in many biological fields. In 2020, a deep learning model AlphaFold won the protein folding competition with predicted structures within the error tolerance of experimental methods. However, this solution to the most prominent bioinformatic challenge of the past 50 years has been possible only thanks to a carefully curated benchmark of experimentally predicted protein structures. In Genomics, we have similar challenges (annotation of genomes and identification of functional elements) but currently, we lack benchmarks similar to protein folding competition. Results: Here we present a collection of curated and easily accessible sequence classification datasets in the field of genomics. The proposed collection is based on a combination of novel datasets constructed from the mining of publicly available databases and existing datasets obtained from published articles. The collection currently contains nine datasets that focus on regulatory elements (promoters, enhancers, open chromatin region) from three model organisms: human, mouse, and roundworm. A simple convolution neural network is also included in a repository and can be used as a baseline model. Benchmarks and the baseline model are distributed as the Python package 'genomic-benchmarks', and the code is available at https://github.com/ML-Bioinfo-CEITEC/genomic%5fbenchmarks. Conclusions: Deep learning techniques revolutionized many biological fields but mainly thanks to the carefully curated benchmarks. For the field of Genomics, we propose a collection of benchmark datasets for the classification of genomic sequences with an interface for the most commonly used deep learning libraries, implementation of the simple neural network and a training framework that can be used as a starting point for future research. The main aim of this effort is to create a repository for shared datasets that will make machine learning for genomics more comparable and reproducible while reducing the overhead of researchers who want to enter the field, leading to healthy competition and new discoveries. [ABSTRACT FROM AUTHOR]

Super-resolution (SR) is one of the important pre-processing methods to refine the text images quality. Though there are numerous introduced algorithms to increase the spatial resolution for textual images, analysis on SR methods using deep learning is still insufficient. In this paper, we focus on evaluating the performance of various deep SR methods which have already confirmed to perform well in natural images super-resolution. Three evaluation metrics are used to analyze the performance of each method, such as peak signal-to-noise ratio (PSNR), structure similarity index (SSIM), and optical character recognition accuracy (OCRAcc). Experimental results show that deeper networks perform better than shallow networks for super-resolution problem. In overall, deep recursive convolutional network (DRCN) and deep laplacian pyramid network (LapSRN) alternately achieve the best performance. Then, very deep super-resolution network (VDSR) obtains the 3 rd rank following both methods. [ABSTRACT FROM AUTHOR]

The paper systematically studies the impact of a range of recent advances in convolution neural network (CNN) architectures and learning methods on the object categorization (ILSVRC) problem. The evaluation tests the influence of the following choices of the architecture: non-linearity (ReLU, ELU, maxout, compatability with batch normalization), pooling variants (stochastic, max, average, mixed), network width, classifier design (convolutional, fully-connected, SPP), image pre-processing, and of learning parameters: learning rate, batch size, cleanliness of the data, etc. The performance gains of the proposed modifications are first tested individually and then in combination. The sum of individual gains is greater than the observed improvement when all modifications are introduced, but the “deficit” is small suggesting independence of their benefits. We show that the use of 128 × 128 pixel images is sufficient to make qualitative conclusions about optimal network structure that hold for the full size Caffe and VGG nets. The results are obtained an order of magnitude faster than with the standard 224 pixel images. [ABSTRACT FROM AUTHOR]

Image classification is a fundamental task in remote sensing image processing. In recent years, deep convolutional neural networks (DCNNs) have experienced significant breakthroughs in natural image recognition. The remote sensing field, however, is still lacking a large-scale benchmark similar to ImageNet. In this paper, we propose a remote sensing image classification benchmark (RSI-CB) based on massive, scalable, and diverse crowdsourced data. Using crowdsourced data, such as Open Street Map (OSM) data, ground objects in remote sensing images can be annotated effectively using points of interest, vector data from OSM, or other crowdsourced data. These annotated images can, then, be used in remote sensing image classification tasks. Based on this method, we construct a worldwide large-scale benchmark for remote sensing image classification. This benchmark has large-scale geographical distribution and large total image number. It contains six categories with 35 sub-classes of more than 24,000 images of size 256 × 256 pixels. This classification system of ground objects is defined according to the national standard of land-use classification in China and is inspired by the hierarchy mechanism of ImageNet. Finally, we conduct numerous experiments to compare RSI-CB with the SAT-4, SAT-6, and UC-Merced data sets. The experiments show that RSI-CB is more suitable as a benchmark for remote sensing image classification tasks than other benchmarks in the big data era and has many potential applications. [ABSTRACT FROM AUTHOR]

• Present the methodologies and evaluation results for the cardiac segmentation algorithms selected from the submissions to the MS-CMRSeg challenge, in conjunction with MICCAI 2019. • Provide a fair and intuitive comparison between the supervised methods and UDA algorithms for cardiac segmentation. • Provide datasets and evaluation tools for an ongoing development of MS-CMR based cardiac segmentation algorithms. [Display omitted] Accurate computing, analysis and modeling of the ventricles and myocardium from medical images are important, especially in the diagnosis and treatment management for patients suffering from myocardial infarction (MI). Late gadolinium enhancement (LGE) cardiac magnetic resonance (CMR) provides an important protocol to visualize MI. However, compared with the other sequences LGE CMR images with gold standard labels are particularly limited. This paper presents the selective results from the Multi-Sequence Cardiac MR (MS-CMR) Segmentation challenge, in conjunction with MICCAI 2019. The challenge offered a data set of paired MS-CMR images, including auxiliary CMR sequences as well as LGE CMR, from 45 patients who underwent cardiomyopathy. It was aimed to develop new algorithms, as well as benchmark existing ones for LGE CMR segmentation focusing on myocardial wall of the left ventricle and blood cavity of the two ventricles. In addition, the paired MS-CMR images could enable algorithms to combine the complementary information from the other sequences for the ventricle segmentation of LGE CMR. Nine representative works were selected for evaluation and comparisons, among which three methods are unsupervised domain adaptation (UDA) methods and the other six are supervised. The results showed that the average performance of the nine methods was comparable to the inter-observer variations. Particularly, the top-ranking algorithms from both the supervised and UDA methods could generate reliable and robust segmentation results. The success of these methods was mainly attributed to the inclusion of the auxiliary sequences from the MS-CMR images, which provide important label information for the training of deep neural networks. The challenge continues as an ongoing resource, and the gold standard segmentation as well as the MS-CMR images of both the training and test data are available upon registration via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mscmrseg/). [ABSTRACT FROM AUTHOR]

Neural networks have been applied to seismic inversion problems since the 1990s. More recently, many publications have reported the use of Deep Learning (DL) neural networks capable of performing seismic inversion with promising results. However, when solving a seismic inversion problem with DL, each author uses, in addition to different DL models, different datasets and different metrics for performance evaluation, which makes it difficult to compare performances. Depending on the data used for training and the metrics used for evaluation, one model may be better or worse than another. Thus, it is quite challenging to choose the appropriate model to meet the requirements of a new problem. This work aims to review some of the proposed DL methodologies, propose appropriate performance evaluation metrics, compare the performances, and observe the advantages and disadvantages of each model implementation when applied to the chosen datasets. The publication of this benchmark environment will allow fair and uniform evaluations of newly proposed models and comparisons with currently available implementations. [ABSTRACT FROM AUTHOR]

Stock price direction prediction is an important issue in the financial world. Even small improvements in predictive performance can be very profitable. The purpose of this paper is to benchmark ensemble methods (Random Forest, AdaBoost and Kernel Factory) against single classifier models (Neural Networks, Logistic Regression, Support Vector Machines and K-Nearest Neighbor). We gathered data from 5767 publicly listed European companies and used the area under the receiver operating characteristic curve (AUC) as a performance measure. Our predictions are one year ahead. The results indicate that Random Forest is the top algorithm followed by Support Vector Machines, Kernel Factory, AdaBoost, Neural Networks, K-Nearest Neighbors and Logistic Regression. This study contributes to literature in that it is, to the best of our knowledge, the first to make such an extensive benchmark. The results clearly suggest that novel studies in the domain of stock price direction prediction should include ensembles in their sets of algorithms. Our extensive literature review evidently indicates that this is currently not the case. [ABSTRACT FROM AUTHOR]

• Physics-based and data-driven modeling approaches are quantified with the TCLab. • A PID tuning method is proposed to replay disturbances from closed-loop data. • Model predictive control is benchmarked with three standard models. • Real-time results are presented to motivate further benchmark studies. Standard benchmarks are important repositories to establish comparisons between competing model and control methods, especially when a new method is proposed. This paper presents details of an Arduino micro-controller temperature control lab as a benchmark for modeling and control methods. As opposed to simulation studies, a physical benchmark considers real process characteristics such as the requirement to meet a cycle time, discrete sampling intervals, communication overhead with the process, and model mismatch. An example case study of the benchmark is quantifying an optimization approach for a PID controller with 5.4% improved performance. A multivariate example shows the quantified performance improvement by using model predictive control with a physics-based model, an autoregressive time series model, and a Hammerstein model with an artificial neural network to capture the static nonlinearity. These results demonstrate the potential of a hardware benchmark for transient modeling and regulatory or advanced control methods. [ABSTRACT FROM AUTHOR]