Search

The significance of earth observation data spans diverse fields and domains, driving the need for efficient management. Nevertheless, the exponential increase in data volume brings new challenges that complicate processing and storing data. This article proposes an optimized multi-modular service for earth observation data management in response to these challenges. The suggested approach focuses on choosing the optimal configurations for the storage and processing layers to improve the performance and cost-effectiveness of managing data. By employing the recommended optimized strategies, earth observation data can be managed more effectively, resulting in fast data processing and reduced costs. [ABSTRACT FROM AUTHOR]

Air pollution significantly affects human health and the environment. It is caused by the emission of diverse pollutants into the atmosphere. Most of the measurements for air quality are done through on-ground sensors that are single points and do not cover well a given territory. Currently, there are space missions that are aiming to complement these on-the-ground systems with more synoptic views. The management and processing of remote sensing and on-ground sensor data have become increasingly complex, requiring temporal and spatial analysis to extract meaningful information and identify patterns and trends of vast amounts of data from various sources. Data Cubes framework helps overcome processing challenges, effectively managing and analyzing large volumes of data in multiple dimensions, such as spatial, spectral, and temporal. The article presents a scalable EO processing and visualization service designed for Data Cubes to explore and analyze multidimensional array data obtained from the Sentinel 5P satellite by performing shared-memory parallel simulations. The service provides a comprehensive understanding of the studied region over a specific period through statistical analysis and visualization. A case study was conducted over the territory of Armenia from September 2018 to August 2019 to evaluate the performance and capabilities of the service. [ABSTRACT FROM AUTHOR]

Earth observation technology has become increasingly crucial for monitoring various aspects of our planet through systematic data collection. However, processing the large volume of satellite data generated can be challenging, requiring high-performance computing solutions. The Dask framework has gained popularity for its flexibility and efficiency in processing large amounts of data in a distributed manner. Nevertheless, determining the optimal Dask cluster configuration remains challenging, as it requires balancing performance and cost objectives. A novel multi-objective optimization service is proposed to address this challenge that enhances the performance and cost efficiency of earth observation data processing workflows. Our approach is to generate a set of Pareto-optimal solutions, allowing users to make informed decisions regarding the optimal trade-offs between performance and cost. The effectiveness is demonstrated using real-world earth observation datasets and outperforms existing performance and cost-efficiency solutions. [ABSTRACT FROM AUTHOR]

Earth observation (EO) satellite data is essential to environmental monitoring. At a national and regional level, the open data cubes harness the power of satellite data by providing application programming interfaces and services to the end-users. The volume and the complexity of satellite observations are increasing, demanding novel approaches for data storing, managing, and processing. High-performance computing (HPC) and cloud platforms may improve Big EO data processing performance. However, it is necessary to consider several vital aspects for efficient and flexible EO data processing, such as the interoperability from cloud-HPC and EO data repositories, automatic provisioning and scaling of cloud-HPC resources, cost-effectiveness, support of new EO data formats and open-source packages, or linkage with data cube platforms. The article proposes a scalable EO data processing platform interoperable from cloud-HPC and EO data repositories. The platform enables linking any data repository supporting web coverage service or SpatioTemporal Asset Catalog Application Programming Interfaces (STAC-API), and any cloud or HPC resource supporting scheduling system API for providing access to the cluster backends. [ABSTRACT FROM AUTHOR]

The MapReduce framework manages Big Data sets by splitting the large datasets into a set of distributed blocks and processes them in parallel. Data compression and in-memory file systems are widely used methods in Big Data processing to reduce resource-intensive I/O operations and improve I/O rate correspondingly. The article presents a performance-efficient modular and configurable decision-making robust service relying on data compression and in-memory data storage indicators. The service consists of Recommendation and Prediction modules, predicts the execution time of a given job based on metrics, and recommends the best configuration parameters to improve Hadoop and Spark frameworks' performance. Several CPU and dataintensive applications and micro-benchmarks have been evaluated to improve the performance, including Log Analyzer, WordCount, and K-Means. [ABSTRACT FROM AUTHOR]

The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes. [ABSTRACT FROM AUTHOR]