Evaluation in the Design of Complex Systems

We identify literature that argues the process of creating knowledge-based system is often imbalanced. In most knowledge-based systems, development is often technology-driven instead of requirement-driven. Therefore, we argue designers must recognize that evaluation is a critical link in the application of requirement-driven development models because it provides the information that keeps the iterative developmental process on track. In this study we took a closer look at the decision-making in the design of a complex performance system in a non-profit organization. The purpose was to gain insights into the roles and functions of evaluation during the process of building a performance support system. This study adopted an exploratory case study as its inquiry method. Findings suggested that effective communication is as important a function to evaluation as is decision-making, and that the implementation of evaluation seldom follows the textbook guidelines. That is, evaluation practice is a patchwork history of the designers' knowledge and experience. Further, neither complexity nor the informality of the evaluation protocols seems to influence the quality of decision-making.