Search

Modern window-based user interface systems generate user interface events as natural products of their normal operation. Because such events can be automatically captured and because they indicate user behavior with respect to an application's user interface, they have long been regarded as a potentially fruitful source of information regarding application usage and usability. However, because user interface events are typically voluminous and rich in detail, automated support is generally required to extract information at a level of abstraction that is useful to investigators interested in analyzing application usage or evaluating usability.This survey examines computer-aided techniques used by HCI practitioners and researchers to extract usability-related information from user interface events. A framework is presented to help HCI practitioners and researchers categorize and compare the approaches that have been, or might fruitfully be, applied to this problem. Because many of the techniques in the research literature have not been evaluated in practice, this survey provides a conceptual evaluation to help identify some of the relative merits and drawbacks of the various classes of approaches. Ideas for future research in this area are also presented.This survey addresses the following questions: How might user interface events be used in evaluating usability? How are user interface events related to other forms of usability data? What are the key challenges faced by investigators wishing to exploit this data? What approaches have been brought to bear on this problem and how do they compare to one another? What are some of the important open research questions in this area?

The two most commonly used techniques for evaluating the fit between application design and use - namely, usability testing and beta testing with user feedback - suffer from a number of limitations that restrict evaluation scale (in the case of usability tests) and data quality (in the case of beta tests). They also fail to provide developers with an adequate basis for: (1) assessing the impact of suspected problems (and proposed solutions) on users at large, and (2) deciding where to focus scarce development and evaluation resources to maximize the benefit for users at large. This article describes an approach to usage data and user feedback collection that addresses these limitations to provide developers with a complementary source of usage - and usability-related information. This research has been subjected to a number of evaluative activities including: (1) the development of three research prototypes at NYNEX Corporation, the University of Colorado, and the University of California, (2) the incorporation of one prototype by independent third party developers as part of an integrated demonstration scenario performed by Lockheed Martin Corporation, abd (3) observation and participation in two industrial development projects, at NYNEX and Microsoft Corporations, in which developers sought to improve the application development process based on usage data and user feedback. The proposed approach involves a devlopment platform for creating software agents that are deployed over the Internet to observe application use and report usage data and user feedback to developers to help improve the fit between design and use. The data can be used to illuminate how applications are used, to uncover mismatches in actual versus expected use, and to increase user involvement in the evolution of interactive systems. This research is aimed at helping developers make more informed design, impact assessment, and effort allocation decisions, ultimately leading to more cost-effective development of software that is better suited to user needs.

Software architectures evolve as the result of numerous, interrelated design decisions. Existing approaches to analysis, however, tend to provide feedback only after numerous design decisions have been made. As a result, theydo not directly support the evolutionary nature of the architecture design process or the software architect's decision-making process. In this paper we present an approach to architectural analysis stemming from previous work in domain oriented design environments that is based on critics and criticism control mechanisms. This approach more closely supports evolution and the needs of architects by providing feedback as individual design decisions are beingconsidered. We discuss the theoretical motivations for the critic-based approach, the implementation and management of critics, support for diverse and extensible groups of critics, and the combined use of critics and existing analysis techniques.

Involving end users in the development of interactive systems increases the likelihood those systems will be useful and usable. User involvement, however, is both time and resource intensive. Intemet-and World-Wide-Web-based software release models have magnified these problems. At the same time, these practices have begun to blur the distinction between development and use and, in so doing, have provided developers with unprecedented, and currently underutilized, opportunities for increasing user involvement. We propose an approach — based on expectation-driven event monitoring and expectation agents — that leverages these opportunities to support ongoing user involvement in the software development process.

Empirical evaluation of software systems in actual usage situations is critical in software engineering. Prototyping, beta testing, and usability testing are widely used to refine system requirements, detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Web-based deployment. This paper presents an approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility. A general framework for comparing software monitoring systems is presented and used to compare the proposed approach to existing techniques.

Empirical evaluation of software systems in actual use is critical in software engineering. Prototyping, beta testing, and usability testing are widely employed to refine system requirements, to detect anomalous or unexpected system and user behavior, and to evaluate software usefulness and usability. The World Wide Web enables cheap, rapid, and large-scale distribution of software for evaluation purposes. However, current techniques for collecting usage data have not kept pace with the opportunities presented by Web-based deployment. This paper presents an agent-based approach and prototype system that makes large-scale collection of usage data over the Internet a practical possibility.