Back to Search
Start Over
PERFORMANCE-EFFICIENT RECOMMENDATION AND PREDICTION SERVICE FOR BIG DATA FRAMEWORKS FOCUSING ON DATA COMPRESSION AND IN-MEMORY DATA STORAGE INDICATORS.
- Source :
- Scalable Computing: Practice & Experience; Dec2021, Vol. 22 Issue 4, p401-411, 11p
- Publication Year :
- 2021
-
Abstract
- The MapReduce framework manages Big Data sets by splitting the large datasets into a set of distributed blocks and processes them in parallel. Data compression and in-memory file systems are widely used methods in Big Data processing to reduce resource-intensive I/O operations and improve I/O rate correspondingly. The article presents a performance-efficient modular and configurable decision-making robust service relying on data compression and in-memory data storage indicators. The service consists of Recommendation and Prediction modules, predicts the execution time of a given job based on metrics, and recommends the best configuration parameters to improve Hadoop and Spark frameworks' performance. Several CPU and dataintensive applications and micro-benchmarks have been evaluated to improve the performance, including Log Analyzer, WordCount, and K-Means. [ABSTRACT FROM AUTHOR]
Details
- Language :
- English
- ISSN :
- 18951767
- Volume :
- 22
- Issue :
- 4
- Database :
- Complementary Index
- Journal :
- Scalable Computing: Practice & Experience
- Publication Type :
- Academic Journal
- Accession number :
- 154259505
- Full Text :
- https://doi.org/10.12694/scpe.v22i4.1945