Search

The architecture of complex software systems is a collection of decisions that are very expensive to change. This makes effective software architecture evaluation methods essential in today’s system development for mission critical systems. We have previously developed MEMS for evaluating middleware architectures, which provides an effective assessment of important quality attributes and their characterizations. To provide additional quantitative assessments on the overall system performance using actual runtime data, we employed a set of statistical procedures in this work. Our proposed assessment procedures comprises a standard sensitivity analysis procedure that utilizes leverage statistics to identify and remove influential data points, and an estimator for evaluating system stability and a metric for evaluating system load capacity. Experiments were conducted using real runtime datasets. Results show that our procedures effectively identified and isolated abnormal data points, and provided valuable statistics to show system stability. Our approach thus provides a sound statistical basis to support software architecture evaluation.

Architecture evaluation methods provide general guidelines to assess quality attributes of systems, which are not necessarily straightforward to practice with. With COTS middleware based systems, this assessment process is further complicated by the complexity of middleware technology and a number of design and deployment options. Efficient assessment is key to produce accurate evaluation results for stakeholders to ensure good decisions are made on system acquisition. In this paper, a systematic evaluation method called MEMS is developed to provide some structure to this assessment process. MEMS produces the evaluation plan with thorough design of experiments, definition of metrics and development of techniques for measurement. This paper presents MEMS and its application to a mission critical middleware system.