An empirical evaluation of the use of models to improve the understanding of safety compliance needs

Context Critical systems in application domains such as automotive, railway, aerospace, and healthcare are required to comply with safety standards. The understanding of the safety compliance needs specified in these standards can be difficult from their text. A possible solution is to use models. Objective We aim to evaluate the use of models to understand safety compliance needs. Method We have studied the effectiveness, efficiency, and perceived benefits in understanding these needs, with models and with the text of safety standards, by means of an experiment. The standards considered are DO-178C and EN 50128. We use SPEM-like diagrams to graphically represent the models. Results The mean effectiveness of 20 undergraduate students in understanding the needs and the mean efficiency were higher with models (22% and 38%, respectively), and the difference is statistically significant (p-value ≤ 0.02). Most of the students agreed upon the ease of understanding the structure of safety compliance needs with models when compared to the text, but on average, the students were undecided about whether the models are easy to understand or easier to understand than the text. Conclusions The results allow us to claim that the use of models can improve the understanding of safety compliance needs. Nonetheless, there seems to be room for improvement in relation to the perceived benefits. It must be noted that our conclusions may differ if the subjects were experienced practitioners.