Search

The term Text Mining, which is given to the set of techniques used for the extraction, cleaning and processing of the information in texts, has become useful to provide valuable information to other algorithms and widely used with statistical and machine learning methods. By enabling the extraction of useful insights from textual data, Text Mining has become a potent tool in decision-making and knowledge discovery across many areas, including health care, government, education and industry. R is a mature open-source programming environment that has overstepped its initial scope of application for statistical computing and graphics to be used in pretty all the Data Science knowledge Area Groups. The objective of this paper is to present review and benchmarking analysis of packages for text mining techniques with R in computational systems. The paper reviews thirteen different packages comparing them on their execution time and memory used, for which new tests have been specifically designed. The results of this approach have been intended to be used over the most common tasks carried out when analyzing texts, and comparisons included allow R users to know which packages are best for each task and to improve their performance. Text mining package (tm) stands out particularly in Tokenization and Stemming techniques, while fastTextR is the best choice for Topic Modeling and Normalization. Also in the case of the Term Frequency-Inverse Document Frequency (TF-IDF) technique, the textir package is a clear choice. The other packages will depend on whether the technique is applied to a document-term matrix (DTM) or to plain text. In addition, there are packages that perform better in runtime than in memory usage and vice versa, making the choice more difficult. Packages such as udpipe can achieve better results working in parallel. Future works will include the same analysis for parallel computing, hybrid approaches, and novel algorithms.

Python is currently the number one language in the TIOBE index and has been the second most popular language on GitHub for years. But so far, there are only a few bug databases that contain bugs for Python projects and even fewer in which bugs can be reproduced. In this paper, we present a manually curated database of reproducible Python bugs called PyBugHive. The initial version of PyBugHive is a benchmark of 149 real, manually validated bugs from 11 Python projects. Each entry in our database contains the summary of the bug report, the corresponding patch, and the test cases that expose the given bug. PyBugHive features a rich command line interface for accessing both the buggy and fixed versions of the programs and provides the abstraction for executing the corresponding test cases. The interface facilitates highly reproducible empirical research and tool comparisons in fields such as testing, automated program repair, or bug prediction. The usage of our database is demonstrated through a use case involving a large language model, GPT-3.5. First, we evaluated the bug detection capabilities of the model with the help of the bug repository. Using multiple prompts, we found out that GPT-3.5 was able to detect 67 out of 149 bugs (45%). Furthermore, we leveraged the constructed bug dataset in assessing the automatic program repair capabilities of GPT-3.5 by comparing the generated fixes with the real patches contained in the dataset. However, its performance was far worse in this task compared to bug detection, as it was able to fix only one of the detected issues.

Ultra-wideband (UWB) localization has emerged in GPS-denied environments as a crucial facilitator for diverse industries, including logistics, healthcare applications, and societal domains. Despite notable progress, UWB algorithms from the scientific literature are often evaluated in isolation in very specific conditions and hence difficult to compare. This paper introduces a novel benchmark platform designed to assess the performance of 11 prominent UWB accuracy-enhancing algorithms, both independently and in combination. A key feature of the platform is its incorporation of multiple diverse evaluation metrics, including mean average error, latency, and spatial error. We showcase the significance of adopting alternative metrics such as spatial error, which often, depending on the use case, offers greater relevance than the prevalently used mean average error. Furthermore, we show that “more is better” does not hold true when combining multiple accuracy-improving algorithms for UWB systems: combining multiple accuracy-improving algorithms reveals instances of diminishing returns and can even result in overall performance decline. Additionally, we caution against blind reliance on accuracy outcomes reported in the scientific literature when designing UWB systems for use cases that are different in terms of requirements or environment. Finally, we also provide algorithmic recommendations for distinct surroundings, exemplary applications, and usage contexts, assisting in driving efficient design in future UWB research and adoption.

As the importance of data increases, data is continuously collected from diverse sources such as sensors, IoT devices, and edge computing devices. To manage these continuously monitored data, it is often organized chronologically with time which is referred as time-series data. By managing the data using time, data from different streams can be analyzed in a comprehensive manner with an identical index which is time. However, due to the unique characteristics of time-series data, it is essential for the underlying database systems to understand the characteristics of the time-series data. To handle this, time-series database systems, which specially target time-series data, are emerging. These database systems have different performance characteristics due to the unique characteristics of the data which should be investigated to efficiently store and analyze the data. In this paper, we analyze the time-series database from the perspective of I/O using various storage devices from HDD, SATA and NVMe SSD. First, we analyze the I/O characteristics such as runtime, throughput and size of total requests using various storage devices. In addition, we analyze the effect of unique time-series database features such as data chunk interval, compression and number of workers. Our analysis results show that adapting high-performance devices can greatly improve the performance of the database by up to $33.22\times $ .

This paper compares various performance metrics of the conventional direct digital frequency synthesis (DDFS) with DDFS introducing the quarter method, which is most commonly used among ROM-based DDFS. The evaluation is implemented at the register-transfer level (RTL) in the low-power and general-purpose 65 nm CMOS technologies, and 180 nm CMOS technology. We have observed that phase-to-amplitude converter (PAC) becomes a key building block that mainly determines the performance of DDFS. At a 12-bit resolution, the PAC of the conventional DDFS occupies about 97% of the total cell area and accounts for approximately 83% of the total power consumption. On the other hand, the PAC of DDFS with the quarter method applied occupies about 80% of the total cell area and uses about 45% of the total power consumption. In addition, because the quarter method requires additional decoding logic to map the phase to the corresponding amplitude, this method has area and power consumption disadvantages at relatively small resolution targets below 10-bit. However, at relatively high-resolution targets beyond 11-bit, the quarter method has significant benefits in terms of various performance metrics (e.g., area, power consumption, and speed). Then, the quarter method has advantages at resolutions exceeding 7 bits based on figure of merit (FoM). Based on FoM, a similar pattern is evaluated despite changing the CMOS technology. As a result, the quarter method is the crucial structure in DDFS architecture. The designed DDFS is verified in implementation results using a field programmable gate array (FPGA) in 65 nm CMOS technology.

In this paper, we propose a novel benchmark called the StarCraft Multi-Agent Exploration Challenges(SMAC-Exp), where agents learn to perform multi-stage tasks and to use environmental factors without precise reward functions. The previous challenges (SMAC) recognized as a standard benchmark of Multi-Agent Reinforcement Learning are mainly concerned with ensuring that all agents cooperatively eliminate approaching adversaries only through fine manipulation with obvious reward functions. SMAC-Exp, on the other hand, is interested in the exploration capability of MARL algorithms to efficiently learn implicit multi-stage tasks and environmental factors as well as micro-control. This study covers both offensive and defensive scenarios. In the offensive scenarios, agents must learn to first find opponents and then eliminate them. The defensive scenarios require agents to use topographic features. For example, agents need to position themselves behind protective structures to make it harder for enemies to attack. We investigate a total of twelve MARL algorithms under both sequential and parallel episode settings of SMAC-Exp and observe that recent approaches perform well in similar settings to the previous challenge, but we discover that current multi-agent approaches place relatively less emphasis on exploration perspectives. To a limited extent, we observe that an enhanced exploration method has a positive effect on SMAC-Exp, however, there is still a gap that state-of-the-art algorithms cannot resolve the most challenging scenarios of SMAC-Exp. Consequently, we propose a new axis for future research into Multi-Agent Reinforcement Learning studies.

Automatic parallelization of sequential programs combined with auto-tuning is an alternative to manual parallelization. With wider research directions and the increased number of performance tuning tools that have been developed, it has become increasingly difficult to choose a particular tuning tool. This paper reviews the fundamentals of different performance optimization and tuning techniques. It also surveys several tuning frameworks and classifies them into different groups based on their criteria. Developing benchmarks for HPC and validating their accuracy are demanding tasks for computer architects, researchers, and application developers. In addition to providing a survey of performance-tuning tools, we also performed a detailed review of current benchmarks and discussed the requirements for future benchmarks. We performed a detailed comparison of these tuning tools based on other features such as speedup and infrastructure details. We believe that this study will be a very useful resource for parallel computing communities, especially for early-stage parallel computing and performance researchers to gain exposure to existing performance optimization options.

Synchronization is a crucial process in the operation of grid-connected distributed energy sources. Synchronization techniques are classified in closed-loop (CLS) and open-loop synchronization (OLS) techniques. Unlike the CLS technique, OLS techniques are recent and less known. This paper discusses a comparative analysis of the three prominent state-of-the-art OLS techniques. Accordingly, a benchmark model is proposed to systematically evaluate the performance of the OLS techniques under grid voltage disturbances such as harmonic distortions, voltage unbalance, DC offset, and low voltage ride through operation. Given the hardware resource utilization, the computational complexity and the execution time are also presented to judge the efficacy of the proposed model. Thus, engineers/or researchers can better understand the choice of hardware resources and the preferred dynamic performance of a synchronization scheme applied to control and protect a grid-tied power converter. Simulation and experimental results are presented and discussed to demonstrate the purpose of the proposed model.

Nowadays, the use of hardware accelerators to boost the performance of HPC applications is a consolidated practice, and among others, GPUs are by far the most widespread. More recently, some data centers have successfully deployed also FPGA accelerated systems, especially to boost machine learning inference algorithms. Given the growing use of machine learning methods in various computational fields, and the increasing interest towards reconfigurable architectures, we may expect that in the near future FPGA based accelerators will be more common in HPC systems, and that they could be exploited also to accelerate general purpose HPC workloads. In view of this, tools able to benchmark FPGAs in the context of HPC are necessary for code developers to estimate the performance of applications, as well as for computer architects to model that of systems at scale. To fulfill these needs, we have developed FER (FPGA Empirical Roofline), a benchmarking tool able to empirically measure the computing performance of FPGA based accelerators, as well as the bandwidth of their on-chip and off-chip memories. FER measurements enable to draw Roofline plots for FPGAs, allowing for performance comparisons with other processors, such as CPUs and GPUs, and to estimate at the same time the performance upper-bounds that applications could achieve on a target device. In this paper we describe the theoretical model on which FER relies, its implementation details, and the results measured on Xilinx Alveo accelerator cards.

The expansion of autonomous driving operations requires the research and development of accurate and reliable self-localization approaches. These include visual odometry methods, in which accuracy is potentially superior to GNSS-based techniques while also working in signal-denied areas. This paper presents an in-depth review of state-of-the-art visual and point cloud odometry methods, along with a direct performance comparison of some of these techniques in the autonomous driving context. The evaluated methods include camera, LiDAR, and multi-modal approaches, featuring knowledge and learning-based algorithms, which are compared from a common perspective. This set is subject to a series of tests on road driving public datasets, from which the performance of these techniques is benchmarked and quantitatively measured. Furthermore, we closely discuss their effectiveness against challenging conditions such as pronounced lighting variations, open spaces, and the presence of dynamic objects in the scene. The research demonstrates increased accuracy in point cloud-based methods by surpassing visual techniques by roughly 33.14% in trajectory error. This survey also identifies a performance stagnation in state-of-the-art methodologies, especially in complex conditions. We also examine how multi-modal architectures can circumvent individual sensor limitations. This aligns with the benchmarking results, where the multi-modal algorithms exhibit greater consistency across all scenarios, outperforming the best LiDAR method (CT-ICP) by 5.68% in translational drift. Additionally, we address how current AI advances constitute a way to overcome the current development plateau.

Modern computational tasks are often required to not only guarantee predefined accuracy, but get the result fast. Optimizing calculations using floating point numbers, as opposed to integers, is a non-trivial task. For this reason, there is a need to explore new ways to improve such operations. This paper presents analysis and comparison of various floating point formats – float, posit and bfloat. One of the promising areas in which the problem of using such values can be considered to be the most acute is neural networks. That is why we pay special attention to algorithms of linear algebra and artificial intelligence to assess efficiency of new data types in this area. The research results showed that software implementations of posit16 and posit32 have high accuracy, but they are not particularly fast; on the other hand, bfloat16 is not much different from float32 in accuracy, but significantly surpasses it in performance for large amounts of data and complex machine learning algorithms. Thus, posit16 can be used in systems with less stringent performance requirements, as well as in conditions of limited computer memory; and also in cases when bfloat16 cannot provide required accuracy. As for bfloat16, it can speed up systems based on the IEEE 754 standard, but it cannot solve all the problems of conventional floating point arithmetic. Thus, although posits and bfloats are not a full fledged replacement for float, they provide (under certain conditions) advantages that can be useful for implementation of machine learning algorithms.

Metaheuristic algorithms are constructed to solve optimization problems, but they cannot solve all the problems with best solutions. This work proposes a novel self-adaptive metaheuristic optimization algorithm, named Optimal Stochastic Process Optimizer (OSPO), which can solve different kinds of optimization problems with promising performance. Specifically, OSPO regards the procedure of optimization as a realization of stochastic process, and with the help of Subjective Probability Distribution Function (SPDF) and Receding Sampling Strategy proposed in this paper, OSPO can control the exploration-exploitation property online by the adaptive modification of the parameters in SPDF. This adaptive exploration-exploitation property of OSPO contributes to dealing with different kinds of problems; thus, it makes OSPO have the potential to solve at least a vast majority of optimization problems. The proposed algorithm is first benchmarked on uni-modal, multi-modal and composite test functions both in low and high dimensions. The results are verified by comparative studies with seven well-performed metaheuristic algorithms. Then, 21 real-world optimization problems are used to further investigate the effectiveness of OSPO. The winners of CEC2020 Competition on Real-World Single Objective Constrained Optimization, SASS algorithm, sCMAgES algorithm, EnMODE algorithm and COLSHADE algorithm are used as four comparative algorithms in real-world optimization problems. The analysis of simulations demonstrates that OSPO is able to provide very competitive performance compared to the comparative meta-heuristics both in benchmark functions and in real-world optimization problems; thus, the potential of OSPO to solve at least a vast majority of optimization problems is verified. A corresponding MATLAB APP demo is available on https://github.com/JiahongXu123/OSPO-algorithm.git.

The cardiac auscultation using the classical stethoscope (PCG: phonological cardiogram) is known as the most famous method to detect Cardiovascular Disease (CVD). However, this exam requires a qualified cardiologist which relies on the cardiac cycle vibration sound (heart muscle contractions and valves closure) to detect abnormalities in heart during the pumping action. Many research works have been conducted for detecting CVD from PCG signals by using public and private datasets. Due to the lack of CVD recognition benchmark, classification results are very heterogeneous and can not be compared objectively. In this paper, we apply transfer learning to Pascal public dataset and provide an experimental benchmark without any denoising or cleaning steps. The main goal is to generate a set of experimental results which can be used as starting reference for future CVD recognition research based on PCG.

Large-scale point clouds scanned by light detection and ranging (lidar) sensors provide detailed geometric characteristics of scenes due to the provision of 3D structural data. The semantic segmentation of large-scale point clouds is a crucial step for an in-depth understanding of complex scenes. Of late, although a large number of point cloud semantic segmentation algorithms have been proposed, semantic segmentation methods are still far from being satisfactory in terms of precision and accuracy of large-scale point clouds. For machine learning (ML) and deep learning (DL) methodologies, the semantic segmentation is largely influenced by the quality of training sets and methods themselves. Therefore, we construct a new point cloud dataset, namely CSPC-Dataset (Complex Scene Point Cloud Dataset) for large-scale scene semantic segmentation. CSPC-Dataset point clouds are acquired by a wearable laser mobile mapping robot. It covers five complex urban and rural scenes and mainly includes six types of objects, i.e., ground, car, building, vegetation, bridge, and pole. It provides large-scale outdoor scenes with color information, which has advantages such as the scene more complete, point density relatively uniform, diversity and complexity of objects and the high discrepancy between different scenes. Based on the CSPC-Dataset, we construct a new benchmark, which includes approximately 68 million points with explicit semantic labels. To extend the dataset into a wide range of applications, this paper provides the semantic segmentation results and comparative analysis of 7 baseline methods based on CSPC-Dataset. In the experiment part, three groups of experiments are conducted for benchmarking, which offers an effective way to make comparisons with different point-labeling algorithms. The labeling results have shown that the highest Intersection over Union (IoU) of pole, ground, building, car, vegetation, and bridge for all benchmarks is 36.0%, 97.8%, 93.7%, 65.6%, 92.0%, and 69.6%.

Feature selection has gained much consideration from scholars working in the domain of machine learning and data mining in recent years. Feature selection is a popular problem in Machine learning with the goal of finding optimal features with increase accuracy. As a result, several studies have been conducted on multi-objective feature selection through numerous multi-objective techniques and algorithms. The objective of this paper is to present a systematic literature review of the challenges and issues of the multi-objective feature selection problem and critically analyses the proposed techniques used to tackle this problem. The conducted review covered all related studies published since 2012 up to 2019. The outcomes of the reviewed of these studies clearly showed that no perfect solution to the multi-objective feature selection problem yet. The authors believed that the conducted review would serve as the main source of the techniques and methods used to resolve the problem of multi-objective feature selection. Furthermore, current challenges and issues are deliberated to find promising research domains for further study.

Place recognition can be achieved by identifying whether a pair of images (a labeled reference image and a query image) depict the same place, regardless of appearance changes due to different viewpoints or lighting conditions. It is an important component of systems for camera localization and for loop closure detection and a widely studied problem for indoor or urban environments. Recently, the use of robots in agriculture and automatic gardening has created new challenges due to the highly repetitive appearance with prevalent green color and repetitive texture of garden-like scenes. The lack of available data recorded in gardens or plant fields makes difficult to improve localization algorithms for such environments. In this paper, we propose a new data set of garden images for testing algorithms for visual place recognition. It contains images with ground truth camera pose recorded in real gardens at different times, with varying light conditions. We also provide ground truth for all possible pairs of images, indicating whether they depict the same place or not. We also performed a thorough benchmark of several holistic (whole-image) descriptors, and provide the results on the proposed data set. We observed that existing descriptors have difficulties with scenes with repetitive textures and large changes of camera viewpoint.

The Wasserstein metric or earth mover's distance is a useful tool in statistics, computer science and engineering with many applications to biological or medical imaging, among others. Especially in the light of increasingly complex data, the computation of these distances via optimal transport is often the limiting factor. Inspired by this challenge, a variety of new approaches to optimal transport has been proposed in recent years and along with these new methods comes the need for a meaningful comparison. In this paper, we introduce a benchmark for discrete optimal transport, called DOTmark, which is designed to serve as a neutral collection of problems, where discrete optimal transport methods can be tested, compared with one another, and brought to their limits on large-scale instances. It consists of a variety of grayscale images, in various resolutions and classes, such as several types of randomly generated images, classical test images and real data from microscopy. Along with the DOTmark we present a survey and a performance test for a cross section of established methods ranging from more traditional algorithms, such as the transportation simplex, to recently developed approaches, such as the shielding neighborhood method, and including also a comparison with commercial solvers.