Search

The framework for clustering of error messages, ClusterLogs, was developed as a flexible and modular tool for the needs of large-scale distributed computing infrastructures. Various types of failures are being constantly registered during the execution of millions of computing operations daily. Monitoring systems are faced with a challenging task of the analysis of considerable amount of multi-sourced error messages. It is critical to present the information about errors to human experts in a way that makes them able to analyze it. ClusterLogs is a pipeline of methods that allows the discovering of common patterns in error messages and group similar messages for further analysis. This pipeline is able to cluster heterogeneous messages without any prior knowledge about textual pattern and can be executed in a fully automated way. In this work we describe a visual analytics web application that provides the analyst with the visual means of setting up clustering pipeline parameters and facilitates the interpretation and validation of the clustering results.

Large-scale distributed computing infrastructures ensure the operation and maintenance of scientific experiments at the LHC: more than 160 computing centers all over the world execute tens of millions of computing jobs per day. ATLAS — the largest experiment at the LHC — creates an enormous flow of data which has to be recorded and analyzed by a complex heterogeneous and distributed computing environment. Statistically, about 10–12% of computing jobs end with a failure: network faults, service failures, authorization failures, and other error conditions trigger error messages which provide detailed information about the issue, which can be used for diagnosis and proactive fault handling. However, this analysis is complicated by the sheer scale of textual log data, and often exacerbated by the lack of a well-defined structure: human experts have to interpret the detected messages and create parsing rules manually, which is time-consuming and does not allow identifying previously unknown error conditions without further human intervention. This paper is dedicated to the description of a pipeline of methods for the unsupervised clustering of multi-source error messages. The pipeline is data-driven, based on machine learning algorithms, and executed fully automatically, allowing categorizing error messages according to textual patterns and meaning.