Search

We introduce a comprehensive benchmark for local features and robust estimation algorithms, focusing on the downstream task—the accuracy of the reconstructed camera pose—as our primary metric. Our pipeline's modular structure allows easy integration, configuration, and combination of different methods and heuristics. This is demonstrated by embedding dozens of popular algorithms and evaluating them, from seminal works to the cutting edge of machine learning research. We show that with proper settings, classical solutions may still outperform the perceived state of the art. Besides establishing the actual state of the art, the conducted experiments reveal unexpected properties of structure from motion pipelines that can help improve their performance, for both algorithmic and learned methods. Data and code are online (https://github.com/ubc-vision/image-matching-benchmark), providing an easy-to-use and flexible framework for the benchmarking of local features and robust estimation methods, both alongside and against top-performing methods. This work provides a basis for the Image Matching Challenge (https://image-matching-challenge.github.io). [ABSTRACT FROM AUTHOR]

In an idealised vision of science the scientific literature is error-free. Errors reported during peer review are supposed to be corrected prior to publication, as further research establishes new knowledge based on the body of literature. It happens, however, that errors pass through peer review, and a minority of cases errata and retractions follow. Automated screening software can be applied to detect errors in manuscripts and publications. The contribution of this paper is twofold. First, we designed the erroneous reagent checking (ERC) benchmark to assess the accuracy of fact-checkers screening biomedical publications for dubious mentions of nucleotide sequence reagents. It comes with a test collection comprised of 1679 nucleotide sequence reagents that were curated by biomedical experts. Second, we benchmarked our own screening software called Seek&Blastn with three input formats to assess the extent of performance loss when operating on various publication formats. Our findings stress the superiority of markup formats (a 79% detection rate on XML and HTML) over the prominent PDF format (a 69% detection rate at most) regarding an error flagging task. This is the first published baseline on error detection involving reagents reported in biomedical scientific publications. The ERC benchmark is designed to facilitate the development and validation of software bricks to enhance the reliability of the peer review process. [ABSTRACT FROM AUTHOR]

The fluctuation of the bird population reflects the change in the ecosystem, which plays a vital role in ecosystem conservation. However, manual counting is still the mainstream method for bird population counting, which is time-consuming and laborious. One major bottleneck in developing efficient, accurate, and intelligent learning algorithms to counting birds is the lack of large-scale datasets. In this paper, the first large-scale bird population counting dataset, named Bird-Count, with multi-modality morphology annotations is proposed. This paper first evaluates various state-of-the-art (SOTA) models for crowd counting on the Bird-Count and gets poor results. The reason is that the forms, appearances, and postures among different birds are more variant than the crowd. To mitigate these challenges, a simple yet effective plug-and-play framework, called Morphology Prior Knowledge Fusion Network (MPKNet), which can be used on-site to help generate a high-precision bird population density map by incorporating morphological prior knowledge, is proposed. Comprehensive evaluations show that the proposed method can reduce the error rate by 6.02% compared with the current SOTA crowd counting algorithms on average. Moreover, with the above technologies, the intelligent bird population monitoring system is deployed in several important wetland national nature reserves for bird protection. [ABSTRACT FROM AUTHOR]

Traditional methods for object detection typically necessitate a substantial amount of training data, and creating high-quality training data is time-consuming. We propose a novel Few-Shot Object Detection network (FSODv2) in this paper that aims to detect objects from previously unseen categories using only a few annotated examples. Attention RPN, Multi-Relation Detector, and Contrastive Training strategy are central to our method (Fan et al., in: CVPR, 2020), which exploit similarity between few shot support set and query set to detect novel objects while suppressing false detection in the background. We also contribute a new dataset, FSOD-1k, which contains 1000 categories of various objects with high-quality annotations to train our network. To the best of our knowledge, this is one of the first datasets designed for few-shot object detection. This paper improves our FSOD model through well-designed model calibration in three areas: (1) we propose an improved FPN with multi-scale support inputs to calibrate the multi-scale support-query feature matching by exploiting multi-scale features from the same support image with different input scales; (2) we introduce a support classification supervision branch to calibrate the support feature supervision, aligning to the query feature training supervision; (3) we propose backbone calibration to preserve prior knowledge while alleviating backbone bias toward base classes by employing classification dataset to help our model calibration procedure, where such dataset has previously only been used for pre-training in other related works. Besides, we propose a Fast Attention RPN to improve evaluation speed and save computational memory during inference. Once trained, our few-shot network can detect objects from previously unseen categories without further training or fine-tuning, resulting in new state-of-the-art performance on different datasets in the few-shot setting. Our method is general in scope and has numerous potential applications. The dataset link is https://github.com/fanq15/Few-Shot-Object-Detection-Dataset. [ABSTRACT FROM AUTHOR]

This paper introduces the Better Hide Communities (BHC) benchmark dataset aimed at standardizing evaluations in community deception across networks. BHC addresses the need for a common framework to assess the effectiveness of existing and perspective deception strategies by enabling their comparative analyses. BHC serves as a foundation for future work in developing sophisticated algorithms for community deception, enhancing the understanding of algorithmic abilities to employ deceptive measures within communities. Additionally, it offers valuable insights into the varying degrees of resilience that different detection algorithms exhibit against deception strategies. [ABSTRACT FROM AUTHOR]

As Internet of Things (IoT) devices become ubiquitous, they face increasing cybersecurity threats. Unlike standard 1-to-1 communication, the unique challenge posed by n-to-n communication in IoT is that messages must not be encrypted for a single recipient but for a group of recipients. For this reason, using Secure Group Communication (SGC) schemes is necessary to encrypt n-to-n communication efficiently for large group sizes. To this end, the literature presents various SGC schemes with varying features, performance profiles, and architectures, making the selection process challenging. A selection from this multitude of SGC schemes should best be made based on a benchmark that provides an overview of the performance of the schemes. Such a benchmark would make it much easier for developers to select an SGC scheme, but such a benchmark still needs to be created. This paper aims to close this gap by presenting a benchmark for SGC schemes that focus on IoT. Since the design of a benchmark first requires the definition of the underlying business problems, we defined suitable problems for using SGC schemes in the IoT sector as the first step. We identified a common problem for the centralized and decentralized/hybrid SGC schemes, whereas the distributed/contributory SGC schemes required defining an independent business problem. Based on these business problems, we first designed a specification-based benchmark, which we then extended to a hybrid benchmark through corresponding implementations. Finally, we deployed our hybrid benchmark in a typical IoT environment and measured and compared the performance of different SGC schemes. Our findings reveal notable impacts on calculation times and storage requirements without a trusted Central Instance (CI) in distributed/contributory SGC schemes. [ABSTRACT FROM AUTHOR]

This paper addresses errors in mean return estimates in continuous-time asset allocation models. A standard approach postulates that stochastic factors explain expected asset returns. The problem is then to estimate these factors from observed asset prices via filtering. Recent advances have also combined asset prices with expert opinions to improve the estimates. However, these methods have limitations: stocks prices favor momentum strategies, and expert opinions require careful debiasing. To resolve these issues, we propose a jump-diffusion risk-sensitive benchmarked asset management model in which investors estimate the factors from both traditional and alternative data. We show that this model admits a unique C 1 , 2 solution, and we derive the optimal investment policy in quasi-closed form. We find that investors construct their portfolios from a passive core and an active satellite. The passive core adds considerations for jump risk to a simple benchmark replication. The active satellite blends security selection, and factor tilts with event-driven strategies unique to jump-diffusion problems. Thus, our model explains the most popular investment strategies. Furthermore, the improved expert forecast model and the introduction of alternative data provide factor tilters with new tools to sharpen their asset allocation. [ABSTRACT FROM AUTHOR]

Cloud computing platforms have been continuously evolving. Features such as the Elastic Fabric Adapter (EFA) in the Amazon Web Services (AWS) platform have brought yet another revolution in the High Performance Computing (HPC) world, further accelerating the convergence of HPC and cloud computing. Other public clouds also support similar features further fueling this change. In this paper, we show how and why the performance of a large-scale computational fluid dynamics (CFD) HPC application on AWS competes very closely with the one on Beskow—a Cray XC40 supercomputer at the PDC Center for High-Performance Computing - in terms of cost-efficiency with strong scaling up to 2304 processes. We perform an extensive set of micro and macro benchmarks in both environments and conduct a comparative analysis. Until as recently as 2020 these benchmarks have notoriously yielded unsatisfactory results for the cloud platforms compared with on-premise infrastructures. Our aim is to access the HPC capabilities of the cloud, and in general to demonstrate how researchers can scale and evaluate the performance of their application in the cloud. [ABSTRACT FROM AUTHOR]

Hand gestures are becoming an important part of the communication method between humans and machines in the era of fast-paced urbanization. This paper introduces a new standard dataset for hand gesture recognition, Static HAnd PosturE (SHAPE), with adequate side, variation, and practicality. Compared with the previous datasets, our dataset has more classes, subjects, or scenes than other datasets. In addition, the SHAPE dataset is also one of the first datasets to focus on Asian subjects with Asian hand gestures. The SHAPE dataset contains more than 34,000 images collected from 20 distinct subjects with different clothes and backgrounds. A recognition architecture is also presented to investigate the proposed dataset. The architecture consists of two phases that are the hand detection phase for preprocessing and the classification phase by customized state-of-the-art deep neural network models. This paper investigates not only the high accuracy, but also the lightweight hand gesture recognition models that are suitable for resource-constrained devices such as portable edge devices. The promising application of this study is to create a human–machine interface that solves the problem of insufficient space for a keyboard or a mouse in small devices. Our experiments showed that the proposed architecture could obtain high accuracy with the self-built dataset. Details of our dataset can be seen online at https://users.soict.hust.edu.vn/linhdt/dataset/ [ABSTRACT FROM AUTHOR]

A high-quality domain-oriented dataset is crucial for the domain-specific named entity recognition (NER) task. In this study, we introduce a novel education-oriented Chinese NER dataset (EduNER). To provide representative and diverse training data, we collect data from multiple sources, including textbooks, academic papers, and education-related web pages. The collected documents span ten years (2012–2021). A team of domain experts is invited to accomplish the education NER schema definition, and a group of trained annotators is hired to complete the annotation. A collaborative labeling platform is built for accelerating human annotation. The constructed EduNER dataset includes 16 entity types, 11k+ sentences, and 35,731 entities. We conduct a thorough statistical analysis of EduNER and summarize its distinctive characteristics by comparing it with eight open-domain or domain-specific NER datasets. Sixteen state-of-the-art models are further utilized for NER tasks validation. The experimental results can enlighten further exploration. To the best of our knowledge, EduNER is the first publicly available dataset for NER task in the education domain, which may promote the development of education-oriented NER models. [ABSTRACT FROM AUTHOR]

Using multiple moving cameras with different and time-varying views can significantly expand the capability of multiple human tracking in larger areas and with various perspectives. In particular, the use of moving cameras of complementary top and horizontal views can facilitate multi-human detection and tracking from both global and local perspectives. As a new challenging problem that draws more and more attention in recent years, one main issue is the lack of a comprehensive dataset for credible performance evaluation. In this paper, we present such a new dataset consisting of videos synchronously recorded by drone and wearable cameras, with high-quality annotations of the covered subjects and their cross-frame and cross-view associations. We also propose a pertinent baseline algorithm for multi-view multiple human tracking and evaluate it on this new dataset against the annotated ground truths. Experimental results verify the usefulness of the new dataset and the effectiveness of the proposed baseline algorithm. [ABSTRACT FROM AUTHOR]

Research on graph representation learning (a.k.a. embedding) has received great attention in recent years and shows effective results for various types of networks. Nevertheless, few initiatives have been focused on the particular case of embeddings for bipartite graphs. In this paper, we first define the graph embedding problem in the case of bipartite graphs. Next, we propose a taxonomy of approaches used to tackle this problem and draw a description of state-of-the-art methods. Then, we establish their pros and cons with respect to conventional network embeddings. Finally, we provide a description of available resources to lead experiments on the subject. [ABSTRACT FROM AUTHOR]

Human pose estimation (HPE) is a computer vision application that estimates human body joints from images. It gives machines the capability to better understand the interaction between humans and the environment. For this accomplishment, many HPE methods have been deployed in robots, vehicles, and unmanned aerial vehicles (UAVs). This effort raised the challenge of balance between algorithm performance and efficiency, especially in UAVs, where computational resources are limited for saving battery power. Despite the considerable progress in the HPE problem, there are very few methods that are proposed to face this challenge. To highlight the severity of this fact, the proposed paper presents a brief review and an HPE benchmark from the aspect of algorithms performance and efficiency under UAV operation. More specifically, the contribution of HPE methods in the last 22 years is covered, along with the variety of methods that exist. The benchmark consists of 36 pose estimation models in 3 known datasets with metrics that fulfill the paper aspect. From the results, MobileNet-based models achieved competitive performance and the lowest computational cost, in comparison with ResNet-based models. Finally, benchmark results are projected in edge devices hardware specifications to analyze the appropriateness of these algorithms for UAV deployment. [ABSTRACT FROM AUTHOR]

Mobile cloud computing (MCC) integrates mobile computing and cloud computing aiming to extend the capabilities of mobile devices through offloading techniques. In MCC, many controlled experiments have been performed using mobile applications as benchmarks. Usually, these applications are used to validate proposed algorithms, architectures or frameworks. The task of choosing a specific benchmark to evaluate MCC proposals is difficult because there is no standard applications list. This paper presents a systematic mapping study for benchmarks used in MCC research. Taking 5 months of work, we have read 763 papers from MCC field. We catalogued the applications and characterized them considering three facets: category (e.g., games, imaging tools); evaluated resource (e.g., time, energy); and platform (e.g., Android, iPhone). The mapping study evidences research gaps and research trends. Providing a list of downloadable standardized benchmarks, this work can aid better choices to guide more reliable research studies since the same application could be used for different scientific purposes. [ABSTRACT FROM AUTHOR]

Purpose: The production and construction of buildings cause significant environmental impacts besides those arising from their operation. Recently, some European countries have started introducing life cycle assessment as a mandatory calculation method for new buildings, and it is foreseen that by 2030 this will be done in every member state, at first without any legal minimum values. Methods: Extensive databases on the embodied impacts of buildings, which would be needed to support setting the baseline impacts, are still missing. This paper proposes an approach for determining bottom-up reference values. A large building sample is generated describing "technically feasible" new buildings. Instead of analysing a few typical buildings, the main parameters describing a building are determined and the ranges are defined that these parameters typically take. With the variation of these parameters, a large building sample is generated, and the surfaces and built-in material quantities are determined for typical construction solutions to assess environmental performance. Results and discussion: The method is demonstrated by calculating the reference embodied benchmark values for new residential buildings in Hungary. The results show a baseline embodied Global Warming Potential of 9.5–15.5 kg CO2-eq/m2/yr for single-family houses and 9.1–14.3 kg CO2-eq/m2/yr for multi-family houses. Conclusions: This method is suitable for estimating the environmental impact of typical new buildings in countries where a large pool of real building data is not yet available. [ABSTRACT FROM AUTHOR]

Recently, there has been an upsurge of activity in image-based non-photorealistic rendering (NPR), and in particular portrait image stylisation, due to the advent of neural style transfer (NST). However, the state of performance evaluation in this field is poor, especially compared to the norms in the computer vision and machine learning communities. Unfortunately, the task of evaluating image stylisation is thus far not well defined, since it involves subjective, perceptual, and aesthetic aspects. To make progress towards a solution, this paper proposes a new structured, three-level, benchmark dataset for the evaluation of stylised portrait images. Rigorous criteria were used for its construction, and its consistency was validated by user studies. Moreover, a new methodology has been developed for evaluating portrait stylisation algorithms, which makes use of the different benchmark levels as well as annotations provided by user studies regarding the characteristics of the faces. We perform evaluation for a wide variety of image stylisation methods (both portrait-specific and general purpose, and also both traditional NPR approaches and NST) using the new benchmark dataset. [ABSTRACT FROM AUTHOR]

Real-world engineering design problems are widespread in various research disciplines in both industry and industry. Many optimization algorithms have been employed to address these kinds of problems. However, the algorithm's performance substantially reduces with the increase in the scale and difficulty of problems. Various versions of the optimization methods have been proposed to address the engineering design problems in the literature efficiently. In this paper, a comprehensive review of the meta-heuristic optimization methods that have been used to solve engineering design problems is proposed. We use six main keywords in collecting the data (meta-heuristic, optimization, algorithm, engineering, design, and problems). It is worth mentioning that there is no survey or comparative analysis paper on this topic available in the literature to the best of our knowledge. The state-of-the-art methods are presented in detail over several categories, including basic, modified, and hybrid methods. Moreover, we present the results of the state-of-the-art methods in this domain to figure out which version of optimization methods performs better in solving the problems studied. Finally, we provide remarkable future research directions for the potential methods. This work covers the main important topics in the engineering and artificial intelligence domain. It presents a large number of published works in the literature related to the meta-heuristic optimization methods in solving various engineering design problems. Future researches can depend on this review to explore the literature on meta-heuristic optimization methods and engineering design problems. [ABSTRACT FROM AUTHOR]

This paper presents and discusses the results obtained by the participants to the benchmark described in de Hoop et al, Comput. Geosci. (2024). The benchmark uses a model for CO2 geological storage and focuses on the coupling between two-phase flow and geochemistry. Several test cases of various levels of difficulty are proposed, both in one and two spatial dimensions. Six teams participated in the benchmark, each with their own simulation code, though not all teams attempted all the cases. The codes used by the participants are described, and the results obtained on the various test cases are compared, as well as the performance of the codes. It is shown that the results obtained are widely consistent, giving a good level of confidence in the outcome of the benchmark. The general complexity of two-phase flow coupled with chemical reactions altering porous media means that some differences between the codes remain. Besides, from the convergence study, it is clear that the two-dimensional problem has a relatively high sensitivity to a spatial resolution which adds to the complexity. [ABSTRACT FROM AUTHOR]

Affordance grounding aims to locate objects' "action possibilities" regions, an essential step toward embodied intelligence. Due to the diversity of interactive affordance, i.e., the uniqueness of different individual habits leads to diverse interactions, which makes it difficult to establish an explicit link between object parts and affordance labels. Human has the ability that transforms various exocentric interactions into invariant egocentric affordance to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from the exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowledge of the object and transferring it to the egocentric image using only the affordance label as supervision. However, there is some "interaction bias" between personas, mainly regarding different regions and views. To this end, we devise a cross-view affordance knowledge transfer framework that extracts affordance-specific features from exocentric interactions and transfers them to the egocentric view to solve the above problems. Furthermore, the perception of affordance regions is enhanced by preserving affordance co-relations. In addition, an affordance grounding dataset named AGD20K is constructed by collecting and labeling over 20K images from 36 affordance categories. Experimental results demonstrate that our method outperforms the representative models regarding objective metrics and visual quality. The code is available via: github.com/lhc1224/Cross-View-AG. [ABSTRACT FROM AUTHOR]

We conduct an empirical study of machine learning functionalities provided by major cloud service providers, which we call machine learning clouds. Machine learning clouds hold the promise of hiding all the sophistication of running large-scale machine learning: Instead of specifying how to run a machine learning task, users only specify what machine learning task to run and the cloud figures out the rest. Raising the level of abstraction, however, rarely comes free—a performance penalty is possible. How good, then, are current machine learning clouds on real-world machine learning workloads? We study this question by conducting benchmark on the mainstream machine learning clouds. Since these platforms continue to innovate, our benchmark tries to reflect their evolvement. Concretely, this paper consists of two sub-benchmarks—mlbench and automlbench. When we first started this work in 2016, only two cloud platforms provide machine learning services and limited themselves to model training and simple hyper-parameter tuning. We then focus on binary classification problems and present mlbench, a novel benchmark constructed by harvesting datasets from Kaggle competitions. We then compare the performance of the top winning code available from Kaggle with that of running machine learning clouds from both Azure and Amazon on mlbench. In the recent few years, more cloud providers support machine learning and include automatic machine learning (AutoML) techniques in their machine learning clouds. Their AutoML services can ease manual tuning on the whole machine learning pipeline, including but not limited to data preprocessing, feature selection, model selection, hyper-parameter, and model ensemble. To reflect these advancements, we design automlbench to assess the AutoML performance of four machine learning clouds using different kinds of workloads. Our comparative study reveals the strength and weakness of existing machine learning clouds and points out potential future directions for improvement. [ABSTRACT FROM AUTHOR]

This paper introduces a new metaheuristic algorithm called bacteria phototaxis optimizer (BPO). It is designed to solving optimization issues. Inspired by the bacteria phototaxis under the control of photosensory proteins in nature, and based on the basic law of bacterial colony growth and evolution, we have designed the photosensory protein concentration, phototaxis motion and growth operators. These three operators exhibit a highly adaptive and information interaction mechanism. The goal is to simulate the phototaxis process of bacteria and form a complete model of BPO. At the same time, BPO is compared with eight most representative as well as newly generated metaheuristics. Its performance is verified by using 23 well-known benchmark functions with three different types. Additionally, we have conducted several evaluation processes, such as qualitative and quantitative analysis as well as parametric and nonparametric tests. Finally, five classical engineering design problems are used to further test the effectiveness of the algorithm in solving constrained problems. The aforementioned experimental results show that compared with other algorithms, BPO has better accuracy, convergence, and robustness and shows strong competitiveness and optimization performance. [ABSTRACT FROM AUTHOR]

Music inpainting is a sub-task of automated music generation that aims to infill incomplete musical pieces to help musicians in their musical composition process. Many methods have been developed for this task. However, we observe a tendency for each method to be evaluated using different datasets and metrics in the papers where they are presented. This lack of standardization hinders an adequate comparison of these approaches. To tackle these problems, we present MUSIB, a new benchmark for musical score inpainting with standardized conditions for evaluation and reproducibility. MUSIB evaluates four models: Variable Length Piano Infilling (VLI), Music InpaintNet, Music SketchNet, and AnticipationRNN, and over two commonly used datasets: JSB Chorales and IrishFolkSong. We also compile, extend, and propose metrics to adequately quantify note attributes such as pitch and rhythm with Note Metrics, but also higher-level musical properties with the introduction of Divergence Metrics, which operate by comparing the distance between distributions of musical features. Our evaluation shows that VLI, a model based on Transformer architecture, is the best performer on a larger dataset, while VAE-based models surpass this Transformer-based model on a relatively small dataset. With MUSIB, we aim at inspiring the community towards better reproducibility in music generation research, setting an example for strongly founded comparisons among SOTA methods. [ABSTRACT FROM AUTHOR]

With the development of computer vision technology, many advanced computer vision methods have been successfully applied to animal detection, tracking, recognition and behavior analysis, which is of great help to ecological protection, biodiversity conservation and environmental protection. As existing datasets applied to target tracking contain various kinds of common objects, but rarely focus on wild animals, this paper proposes the first benchmark, named Wild Animal Tracking Benchmark (WATB), to encourage further progress of research and applications of visual object tracking. WATB contains more than 203,000 frames and 206 video sequences, and covers different kinds of animals from land, sea and sky. The average length of the videos is over 980 frames. Each video is manually labelled with thirteen challenge attributes including illumination variation, rotation, deformation, and so on. In the dataset, all frames are annotated with axis-aligned bounding boxes. To reveal the performance of these existing tracking algorithms and provide baseline results for future research on wild animal tracking, we benchmark a total of 38 state-of-the-art trackers and rank them according to tracking accuracy. Evaluation results demonstrate that the trackers based on deep networks perform much better than other trackers like correlation filters. Another finding on the basis of the evaluation results is that wild animals tracking is still a big challenge in computer vision community. The benchmark WATB and evaluation results are released on the project website https://w-1995.github.io/. [ABSTRACT FROM AUTHOR]

In standard machine learning implementations, training and inference takes place on servers located remotely from where data is gathered. With the advent of the Internet of Things (IoT), the groundwork is laid to shift half of that computing burden (inferencing) closer to where data is gathered. This paradigm shift to edge computing can significantly decrease the latency and cost of these tasks. Many small, powerful devices have been developed in recent years with the potential to fulfill that goal. In this paper, we analyze two such devices, the NVIDIA Jetson AGX Xavier Developer Kit and the Microsoft Azure Stack Edge Pro (2 GPUs). In addition, the NVIDIA DGX-1 system containerized in a ruggedized case is also taken for running inference model at the Edge. For comparison, the performance of these devices is compared to more common inferencing devices, including a laptop, desktop, and high performance computing (HPC) system. The inferencing model used for testing is the Enhanced Super-Resolution Generative Adversarial Networks (ESRGANs), which was developed using techniques borrowed primarily from other GAN designs, most notably SRGANs and Relativistic average GANs (RaGANs), along with some novel techniques. Metrics chosen for benchmarking were inferencing time, GPU power consumption, and GPU temperature. We found that inferencing using ESRGANs was approximately 10 to 20 times slower on the Jetson edge device, but used approximately 100 to 300 times less power, and was approximately 2 times cooler than any of the other devices tested. The inferencing using ESRGANs performed very similarly on the Azure device as on the more traditional methods. The Azure device performed with slightly slower speeds and equivalent temperatures to the other devices, but with slightly less power consumption. [ABSTRACT FROM AUTHOR]

Incremental model transformation (IMT) tools have been proposed to improve performances of model transformations by updating only the parts of a model that need to be changed when another model on which it depends has been changed. Yet, the question is how these tools are suitable for modeling large and complex systems with rich modeling languages as used in industry. In this paper, we report the results of a benchmark of the most mature IMT tools. Particularly, we benchmark MoTE, eMoflon, VIATRA and YAMTL to evaluate their usability, maintainability and runtime performances. Our benchmark is based on a model transformation of an industrial case study using the standard architecture description language AADL. We propose a reusable evaluation framework, available for tool developers and users. Besides the capability to process large models, our benchmark also assesses IMT tool performances according to the different kinds of complex structures that typically exist in models of rich languages, as well as the complexity of the transformation specifications. Our results show the promising potential of some tools to specify sophisticated specifications and transform large models with good performance, but their use still requires the help of their developers, especially to solve serious problems with their runtime performance. [ABSTRACT FROM AUTHOR]

Bus passenger re-identification is a special case of person re-identification, which aims to establish identity correspondence between the front door camera and the back door camera. In bus environment,it is hard to capture the full body of the passengers. So this paper proposes a bus passenger re-identification dataset,which contains 97,136 head images of 1,720 passengers obtained from hundreds of thousands of video frames with different lighting and perspectives. We also provide a evaluation applied to the dataset based on deep learning and triplet loss. After data augmentation,using ResNet with trihard loss as benchmark network and pre-training on pedestrian re-identification dataset Market-1501, we achieve mAP accuracy of 55.79% and Rank-1 accuracy of 67.91% on passenger re-identification dataset. [ABSTRACT FROM AUTHOR]

This paper proposes a benchmark study for reactive multiphase multicomponent flow in porous media. Modeling such problem leads to a highly nonlinear coupled system of partial differential equations, ordinary differential equations and algebraic constraints, which requires special numerical treatment. The benchmark consists of five test problems in total (both in 1D and in 2D), with varying degrees of difficulty, designed to verify the algorithms and the codes dedicated to simulating coupled isothermal Hydro-Chemical processes during injection and storage of CO2 in the subsurface. It is intended to be used as a basis for comparing codes in order to better understand different couplings such as chemical reactions with two-phase flow, phase behavior with equilibrium reactions, dissolution and precipitation. [ABSTRACT FROM AUTHOR]

The Portia spider, a notable member of the jumping spider family (Salticidae), is widely recognized for its intricate hunting strategies and remarkable problem-solving prowess. Several species fall under the “Portia” genus, with habitats spanning regions in Africa, Asia, and Australia. Demonstrating the ability to tackle new challenges, these spiders can learn and adapt their strategies based on prior experiences. This study introduces the Portia Spider Algorithm (PSA), a swarm-based technique inspired by the unique predatory strategies of the Portia spider. We conducted rigorous assessments of PSA performance against 23 classical test functions, 29 CEC2017 test cases, and 5 engineering optimization tasks. To demonstrate the effectiveness of the PSA, outcomes were juxtaposed with those of renowned algorithms. This paper explores the mechanics, advantages, and potential applications of PSA within the vast domain of computational optimization. [ABSTRACT FROM AUTHOR]

In this paper, we introduce an image dataset for fine-grained classification of dog breeds: the Tsinghua Dogs Dataset. It is currently the largest dataset for fine-grained classification of dogs, including 130 dog breeds and 70,428 real-world images. It has only one dog in each image and provides annotated bounding boxes for the whole body and head. In comparison to previous similar datasets, it contains more breeds and more carefully chosen images for each breed. The diversity within each breed is greater, with between 200 and 7000+ images for each breed. Annotation of the whole body and head makes the dataset not only suitable for the improvement of finegrained image classification models based on overall features, but also for those locating local informative parts. We show that dataset provides a tough challenge by benchmarking several state-of-the-art deep neural models. The dataset is available for academic purposes at https://cg.cs.tsinghua.edu.cn/ThuDogs/. [ABSTRACT FROM AUTHOR]

This paper introduces a new image-based handwritten historical digit dataset named Arkiv Digital Sweden (ARDIS). The images in ARDIS dataset are extracted from 15,000 Swedish church records which were written by different priests with various handwriting styles in the nineteenth and twentieth centuries. The constructed dataset consists of three single-digit datasets and one-digit string dataset. The digit string dataset includes 10,000 samples in red–green–blue color space, whereas the other datasets contain 7600 single-digit images in different color spaces. An extensive analysis of machine learning methods on several digit datasets is carried out. Additionally, correlation between ARDIS and existing digit datasets Modified National Institute of Standards and Technology (MNIST) and US Postal Service (USPS) is investigated. Experimental results show that machine learning algorithms, including deep learning methods, provide low recognition accuracy as they face difficulties when trained on existing datasets and tested on ARDIS dataset. Accordingly, convolutional neural network trained on MNIST and USPS and tested on ARDIS provide the highest accuracies 58.80 % and 35.44 % , respectively. Consequently, the results reveal that machine learning methods trained on existing datasets can have difficulties to recognize digits effectively on our dataset which proves that ARDIS dataset has unique characteristics. This dataset is publicly available for the research community to further advance handwritten digit recognition algorithms. [ABSTRACT FROM AUTHOR]

Visual localization, i.e., camera pose estimation in a known scene, is a core component of technologies such as autonomous driving and augmented reality. State-of-the-art localization approaches often rely on image retrieval techniques for one of two purposes: (1) provide an approximate pose estimate or (2) determine which parts of the scene are potentially visible in a given query image. It is common practice to use state-of-the-art image retrieval algorithms for both of them. These algorithms are often trained for the goal of retrieving the same landmark under a large range of viewpoint changes which often differs from the requirements of visual localization. In order to investigate the consequences for visual localization, this paper focuses on understanding the role of image retrieval for multiple visual localization paradigms. First, we introduce a novel benchmark setup and compare state-of-the-art retrieval representations on multiple datasets using localization performance as metric. Second, we investigate several definitions of "ground truth" for image retrieval. Using these definitions as upper bounds for the visual localization paradigms, we show that there is still significant room for improvement. Third, using these tools and in-depth analysis, we show that retrieval performance on classical landmark retrieval or place recognition tasks correlates only for some but not all paradigms to localization performance. Finally, we analyze the effects of blur and dynamic scenes in the images. We conclude that there is a need for retrieval approaches specifically designed for localization paradigms. Our benchmark and evaluation protocols are available at https://github.com/naver/kapture-localization. [ABSTRACT FROM AUTHOR]

The EEDI is one measure to identify the energy efficiency of a vessel. Besides EEDI there are many other aspects relevant when considering the energy efficiency quality of a vessel, most based on design insights. In this paper, a set of ten relevant indicators are identified to establish the performance of the vessel and pinpoint the source of an energy efficiency improvement. These indicators are applied to the dry bulk vessels delivered between 2010 and 2016. Finally, for all indicators with a reasonable fit, the significance of the differences between trend lines is investigated. This results in groupings with equal performance, showing a clear split between vessels delivered between 2014 and 2016 and before. This is in line with the EEDI enforcement starting 2013. However, for the vessels below 125.000 DWT, the effect so far is traced back to be an increase in contract speed only. No other aspects changed significantly enough. Above this size, a reduction in speed and power can be observed, as predicted by many scholars. [ABSTRACT FROM AUTHOR]

Object-based cloud storage system has an important role in handling big data. All available cloud storage systems deal with scalability, reliability or durability issues. However, there is lack of work addressing data variety. In a previous paper, a basic architecture of an object-based schema oriented data storage system has been proposed which stores data in an encapsulated way. The system comprises account layer, container layer, object layer, database layer and schema layer. In this paper, the architecture proposed in our previous paper has been elaborated. For example, the communication protocols of the proposed system are explained. Moreover, this architecture is realized to test its effectiveness on health data in terms of query execution performance and flexibility on the basis of four different queries of database computation (e.g., append, read, aggregate and delete). The result set are collected on three types of datasets (table, document, file) taken from healthcare scenario. Each type of dataset consists of four different sets of data records. The performance is compared with Amazon S3 (i.e., bucket oriented object-based data storage system) and Microsoft Azure (i.e., account-container oriented object-based data storage system). Flexibility property is also analyzed with respect to these three database operations (i.e., READ, WRITE and DELETE) on three types of experimental datasets (table, document, file) with Amazon S3. [ABSTRACT FROM AUTHOR]

Latest studies on few-shot object detection (FSOD) mainly focuses on achieving better performance in novel class through few-shot fine-tuning. This approach is based on the large amount of annotated data in base class to pre-train models. However, in many scenarios it is difficult or impossible to gather substantial data for the base class of the same domain as the novel class. This leads to the cross-domain few-shot object detection problem. Existing research on scenarios of cross-domain few-shot object detection is only limited to generalization where the base and novel class are from highly similar domains of natural images (i.e., from COCO to VOC). In this respect, we propose the Broader Study of Cross-Domain Few-Shot Object Detection (BSCD-FSOD) benchmark. The generalization is thus possible both from a natural image domain to another, for instance from COCO to ExDark (low-light images) and from a natural image domain to a dissimilar one, for instance from COCO to Clipar1k (artistic media images), to NEU-DET (industrial defect images) or to DOTA (satellite images). We have performed extensive experiments on the proposed benchmark to evaluate state-of-art transfer learning approaches. Research results show that, previous state-of-art methods are no longer state-of-art. Pretraining models on COCO instead of VOC can help them learn a wider feature distribution. The less the domains are similar to those of natural images, the more trainable network components fine-tuning thus requires. The performance of all methods is correlated to both the target domain and the degree of similarity between source and target domain. The results we get show the diversity of data distribution and validate the value of our benchmark, which will offer guidance for future researches. [ABSTRACT FROM AUTHOR]

Games as a service is similar to software as a service, which provides players with game content on a continuous monetization model. Game revenue forecast is vital to game developers to make the right business decisions, such as determining the marketing budget, controlling the development cost, and setting up benchmarks for evaluating game publishing performance. How to make the revenue forecast and integrate it with the game publishing process is hard for small and medium-sized independent (indie) game developers. This includes all steps of the process, from forecasting to decision-making based on the results. This paper provides a data-driven method that uses the mobile game revenue forecast based on different time-series prediction models to drive the game publishing. We demonstrate how to use the data-driven method to guide an indie game studio to forecast revenue and then set the revenue forecast as the internal benchmark to drive game publishing. In practice, we involve a real game project from an indie game studio and provide guidance for one of their casual game projects. Then, based on the revenue forecast, we discuss how to set the revenue forecast as an internal benchmark and drive the actions for mobile game publishing. Finally, we make a conclusion on how our data-driven method can be used to drive mobile game publishing and also discuss future research work. [ABSTRACT FROM AUTHOR]

This paper presents the comparison of the results of modal and nonlinear analyses carried out on a 2-story masonry building with rigid diaphragms, inspired by the Pizzoli's town hall (AQ, Italy). The case study is one of the Benchmark Structures (labeled BS6) in the "URM nonlinear modelling–Benchmark project" funded by the Italian Department of Civil Protection (DPC) within the framework of the ReLUIS projects. The building has been instrumented since 2009 with a permanent monitoring system by the Osservatorio Sismico delle Strutture (OSS) of the DPC and was hit by the 2016/2017 Central Italy earthquake sequence. In the research first phase, modal and nonlinear static analyses were carried out in a blind prediction, without any preliminary calibration of the models, but referring only to commonly made assumptions on materials and modelling. Five computer programs based on the Equivalent Frame Model (EFM) approach were used. Four different structural configurations were considered: with weak spandrels (A), with tie rods coupled to spandrels (B), with RC ring beams coupled to spandrels (C) and with "shear type" idealization (D). In the research second phase, two of the developed EFMs were calibrated in the elastic range using the results of available Ambient Vibration Tests (AVTs). The models were then validated in the nonlinear range by simulating the dynamic response of the structure recorded during the mainshocks of the 2016/2017 Central Italy earthquake. Recorded and numerical results were compared at both the global and local scale. [ABSTRACT FROM AUTHOR]

Since the OTB100 benchmark dataset is released, it has been widely used in a large number of researches on object tracking for performance evaluation. However, the existing datasets are insufficient to evaluate trackers in handling different challenging factors. In this paper, we present the first dataset and benchmark for tracking objects with abrupt motion (AMTSet). The dataset consists of 50 videos of special scenes from our real life, such as camera switching, sudden dynamic change, low frame rate video, etc., which are quite challenging in object tracking. Boundary boxes over 10K frames are marked manually, and all of them are manually labelled for common attributes of object tracking, such as scale variation, illumination variation, occlusion, motion blur, etc. We benchmark the dataset on 36 representative trackers and rank them according to the tracking conditions and results. Furthermore, we propose an evaluation measure for object tracking to better highlight the performances of the trackers against abrupt motion. Our goal is to supplement the existing baseline datasets and provide researchers with more perfect baseline data in order to better evaluate the performance of different trackers. [ABSTRACT FROM AUTHOR]

Due to the lack of established real-world benchmark suites for static taint analyses of Android applications, evaluations of these analyses are often restricted and hard to compare. Even in evaluations that do use real-world apps, details about the ground truth in those apps are rarely documented, which makes it difficult to compare and reproduce the results. To push Android taint analysis research forward, this paper thus recommends criteria for constructing real-world benchmark suites for this specific domain, and presents TaintBench, the first real-world malware benchmark suite with documented taint flows. TaintBench benchmark apps include taint flows with complex structures, and addresses static challenges that are commonly agreed on by the community. Together with the TaintBench suite, we introduce the TaintBench framework, whose goal is to simplify real-world benchmarking of Android taint analyses. First, a usability test shows that the framework improves experts’ performance and perceived usability when documenting and inspecting taint flows. Second, experiments using TaintBench reveal new insights for the taint analysis tools Amandroid and FlowDroid: (i) They are less effective on real-world malware apps than on synthetic benchmark apps. (ii) Predefined lists of sources and sinks heavily impact the tools’ accuracy. (iii) Surprisingly, up-to-date versions of both tools are less accurate than their predecessors. [ABSTRACT FROM AUTHOR]

Software-intensive systems have been growing in both size and complexity. Consequently, developers need better support for measuring and controlling the software quality. In this context, software metrics aim at quantifying different software quality aspects. However, the effectiveness of measurement depends on the definition of reliable metric thresholds, i.e., numbers that characterize a metric value as critical given a quality aspect. In fact, without proper metric thresholds, it might be difficult for developers to indicate problematic software components for correction, for instance. Based on a literature review, we have found several existing methods for deriving metric thresholds and observed their evolution. Such evolution motivated us to propose a new method that incorporates the best of the existing methods. In this paper, we propose a novel benchmark-based method for deriving metric thresholds. We assess our method, called Vale's method, using a set of metric thresholds derived with the support of our method, aimed at composing detection strategies for two well-known code smells, namely god class and lazy class. For this purpose, we analyze three benchmarks composed of multiple software product lines. In addition, we demonstrate our method in practice by applying it to a benchmark composed of 103 Java open-source software systems. In the evaluation, we compare Vale's method to two state-of-the-practice threshold derivation methods selected as a baseline, which are Lanza's method and Alves' method. Our results suggest that the proposed method provides more realistic and reliable thresholds, with better recall and precision in the code smell detection, when compared to both baseline methods. [ABSTRACT FROM AUTHOR]

This paper has two contributions. First, we introduce a clustering basic benchmark. Second, we study the performance of k-means using this benchmark. Specifically, we measure how the performance depends on four factors: (1) overlap of clusters, (2) number of clusters, (3) dimensionality, and (4) unbalance of cluster sizes. The results show that overlap is critical, and that k-means starts to work effectively when the overlap reaches 4% level. [ABSTRACT FROM AUTHOR]

The thermal issue is of great importance during the layout design of heat source components in systems engineering, especially for high functional-density products. Thermal analysis requires complex simulation, which leads to an unaffordable computational burden to layout optimization as it iteratively evaluates different schemes. Surrogate modeling is an effective method for alleviating computation complexity. However, the temperature field prediction (TFP) with complex heat source layout (HSL) input is an ultra-high dimensional nonlinear regression problem, which brings great difficulty to traditional regression models. The deep neural network (DNN) regression method is a feasible way for its good approximation performance. However, it faces great challenges in data preparation for sample diversity and uniformity in the layout space with physical constraints and proper DNN model selection and training for good generality, which necessitates the efforts of layout designers and DNN experts. To advance this cross-domain research, this paper proposes a DNN-based HSL-TFP surrogate modeling task benchmark. With consideration for engineering applicability, sample generation, dataset evaluation, DNN model, and surrogate performance metrics are thoroughly investigated. Experiments are conducted with ten representative state-of-the-art DNN models. A detailed discussion on baseline results is provided, and future prospects are analyzed for DNN-based HSL-TFP tasks. [ABSTRACT FROM AUTHOR]

This paper compares two novel motion capture systems using several biomechanical measurements during walking operations under various conditions. The first one is CATRASYS ('Cassino tracking system') that was initially used as motion capture system applied to Robotics since 1994 at the Cassino University. The second one is the commercial Kinect™ system, which consists of a depth sensor and specific software, and can detect and track human forms. This paper studies problems and experiences in using both measuring systems through a comparison of results obtained from their use. It also enables characterization of walking operations by means of a model based on the paths of the ankle and knee points. [ABSTRACT FROM AUTHOR]

This paper details the steps followed to benchmark a cash management platform of an investment bank using a generic benchmarking solution called BABEL. We highlight the modular design of BABEL, and present an evaluation methodology and best practices for its application on real world systems. The performance results collected with BABEL for the cash management use case enables to define the right tradeoffs in terms of consistency and availability, in a way that respects the service level agreements defined by the clients. On the other hand, we show that the overhead caused by BABEL's integration with the platform at runtime is very negligible. [ABSTRACT FROM AUTHOR]

This paper revises data envelopment analysis (DEA) estimates of efficiency for decision making units (DMU). This is done by smoothing the DEA efficiency frontier by training an artificial neural network (ANN) to fit a closed functional form for input or output prediction of multiple DMUs. By using neural network we overcome the regression model shortcoming of a single dependent variable. DEA is known to be sensitive to the operational efficiency of Decision Making Units (DMU). In the absence of high-performing DMUs, DEA reports falsely elevated efficiency levels. The problem of non-smooth, piece-wise linear efficiency frontier poses a handicap in the measurement of relative efficiency. In this paper we describe a procedure that uses two-stage DEA, a data generation procedure (DGP) and ANN training and prediction to overcome this data-based shortcoming. We also provide two empirical illustrations of this procedure and conclude by explaining the differences in the revised efficiencies of the DMUs. Though we have used constant returns to scale for the empirical illustrations, any other return-to-scale formulation would work just as well. [ABSTRACT FROM AUTHOR]

The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008-2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community's progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, ) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges. [ABSTRACT FROM AUTHOR]

We surveyed the relationship between the scale of space weather events and their occurrence rate in Japan, and we discussed the social impact of these phenomena during the Project for Solar–Terrestrial Environment Prediction (PSTEP) in 2015–2019. The information was compiled for domestic users of space weather forecasts for appropriate preparedness against space weather disasters. This paper gives a comprehensive summary of the survey, focusing on the fields of electricity, satellite operations, communication and broadcasting, satellite positioning usage, aviation, human space activity, and daily life on the Earth's surface, using the cutting-edge knowledge of space weather. Quantitative estimations of the economic impact of space weather events on electricity supply and aviation are also given. Some topics requiring future research, which were identified during the survey are also described. [ABSTRACT FROM AUTHOR]

In this paper, we present a systematic review and evaluation of existing single-image low-light enhancement algorithms. Besides the commonly used low-level vision oriented evaluations, we additionally consider measuring machine vision performance in the low-light condition via face detection task to explore the potential of joint optimization of high-level and low-level vision enhancement. To this end, we first propose a large-scale low-light image dataset serving both low/high-level vision with diversified scenes and contents as well as complex degradation in real scenarios, called Vision Enhancement in the LOw-Light condition (VE-LOL). Beyond paired low/normal-light images without annotations, we additionally include the analysis resource related to human, i.e. face images in the low-light condition with annotated face bounding boxes. Then, efforts are made on benchmarking from the perspective of both human and machine visions. A rich variety of criteria is used for the low-level vision evaluation, including full-reference, no-reference, and semantic similarity metrics. We also measure the effects of the low-light enhancement on face detection in the low-light condition. State-of-the-art face detection methods are used in the evaluation. Furthermore, with the rich material of VE-LOL, we explore the novel problem of joint low-light enhancement and face detection. We develop an enhanced face detector to apply low-light enhancement and face detection jointly. The features extracted by the enhancement module are fed to the successive layer with the same resolution of the detection module. Thus, these features are intertwined together to unitedly learn useful information across two phases, i.e. enhancement and detection. Experiments on VE-LOL provide a comparison of state-of-the-art low-light enhancement algorithms, point out their limitations, and suggest promising future directions. Our dataset has supported the Track "Face Detection in Low Light Conditions" of CVPR UG2+ Challenge (2019–2020) (http://cvpr2020.ug2challenge.org/). [ABSTRACT FROM AUTHOR]

Recent advances in pension product development seem to favour alternatives to the risk free asset often used in the financial theory as a performance standard for measuring the value generated by an investment or a reference point for determining the value of a financial instrument. To this end, in this paper, we apply the simplest machine learning technique, namely, a fully nonparametric smoother with the covariates and the smoothing parameter chosen by cross-validation to forecast stock returns in excess of different benchmarks, including the short-term interest rate, long-term interest rate, earnings-by-price ratio, and the inflation. We find that, net-of-inflation, the combined earnings-by-price and long-short rate spread form our best-performing two-dimensional set of predictors for future annual stock returns. This is a crucial conclusion for actuarial applications that aim to provide real-income forecasts for pensioners. [ABSTRACT FROM AUTHOR]

Purpose: Life cycle assessment (LCA) is considered a robust method to analyse the environmental impacts of products and is used in public and private market applications such as Green Public Procurement (GPP) and Environmental Management Systems (EMS). Despite the usefulness of the methodology, difficulties exist with the interpretation of LCA results. The use of benchmarks can facilitate this process, but there is yet little research on the definition of environmental benchmarks. The aim of this paper is to analyse the distribution of environmental performance used for the definition of the benchmark and how it effects the use in selected product categories. Method: LCA results from 54 Environmental Product Declarations (EPDs) for insulation materials and 49 EPDs for bakery products are tested for their distribution. The outcome from the statistical analysis is used to compare and evaluate three calculation methods for a benchmark. Results and discussion: The results of the study show that distributions and mid- and end-points of environmental performances of the studied indicators differ significantly for the two product categories. While some indicators for bakery products were closer to a normal distribution, most of the indicators are not normally distributed. This is reflected in the comparison of the chosen calculation methods for a benchmark, which showed that the distribution of the data affects the classification of the benchmark as well as the position of values on the benchmark. Conclusion: The results emphasise that analysis of further product groups and the associated distribution of the environmental performance is needed to understand the implications of calculation methods on a benchmark. The availability of comparatively large datasets in a common structure is crucial for these analyses and can be facilitated through the digitalisation of LCA- and EPD-information. Furthermore, more research is needed on the communication formats for different benchmarking options, which must be applied for the different intended audiences to be effective. [ABSTRACT FROM AUTHOR]

This paper discusses a novel approach to evaluate the real-time performance of computing platforms embedded in robotic systems. The motivation behind this work is the need for the robotic systems to meet their timing constraints, thus requiring time predictable real-time systems. We proposed a benchmark, named the Robotstone, which is an adaptation of a traditional real-time performance benchmark widely known in the real-time systems community. Thus, the presented work makes it easier for a robotic engineer to apply the benchmark in the modern robotics context by filtering out issues that do not matter on the robotics applications and adapting the benchmark’s relevant portions. The Robotstone has a set of experiments related to time-constrained application scenarios usually found in robotic systems. Each experiment defines an application-specific parameter that increases at every iteration until the system overloads. The real-time performance is then evaluated through the breakdown point, i.e., the system configuration in the application scenario when any functional or timing constraint is not met. The proposed toolset has been evaluated on two distinct platforms representing a class of embedded computing systems usually employed in robotic systems. Obtained results demonstrate the applicability of the Robotstone benchmark for a quick assessment of computing systems’ real-time performance often required on initial development stages of a robotic system. [ABSTRACT FROM AUTHOR]