1. A Big Data Platform for heterogeneous data collection and analysis in large-scale data centres
- Author
-
Lucia Morganti, Elisabetta Ronchieri, B Martelli, Matteo Galletti, Doina Cristina Duma, Simone Rossi Tisbeni, Jacopo Gasparetto, Francesco Minarini, Arianna Carbone, Daniele Cesini, Claudia Cavallaro, Diego Michelotto, Antonio Falabella, Elisabetta Furlan, Giusy Sergi, Rossi Tisbeni, Simone, CESINI, Daniele, Martelli, Barbara, Carbone, Arianna, Cavallaro, Claudia, Duma, Doina Cristina, Falabella, Antonio, Galletti, Matteo, Gasparetto, Jacopo, Furlan, Elisabetta, Michelotto, Diego, Minarini, Francesco, Morganti, Lucia, Ronchieri, Elisabetta, and Sergi, Giusy
- Subjects
Data collection ,Exploit ,business.industry ,End user ,Computer science ,Big data ,Cloud computing ,Data science ,JSON ,Big Data Platform, Data Mining, Log analysis, Data ingestion, INFN, CNAF ,Data center ,Worldwide LHC Computing Grid ,business ,computer ,computer.programming_language - Abstract
The INFN-CNAF data centre hosts the Italian Tier~1 site for the Worldwide LHC Computing Grid (WLCG), while also serving several other research and technological transfer programs. The challenges posed by the upcoming runs of LHC, together with the opportunity of moving the data centre itself to a bigger site, require a thorough redesign of its monitoring system. The large but heterogeneous amount of logging data and metrics produced daily are fundamental for monitoring activities and, once harmonised, can also be used to build Predictive Maintenance models based on Big Data techniques. In this work we describe the Big Data Platform, a new monitoring infrastructure under development at CNAF. The Big Data Platform relies on a modular, highly scalable architecture based on open source technologies and able to exploit modern frameworks such as containerisation and cloud support. It is capable of collecting data from heterogeneous data sources, clean and harmonise them, and store them as JSON files on different solutions, based on the needs of the end user. Data can then be visualised using Kibana, or analysed through a platform based on Jupyter Notebooks.
- Published
- 2021