Search

In this article, we reflect on our experience applying a framework for evaluating systems change to an evaluation of a statewide West Virginia alliance funded by the National Science Foundation (NSF) to improve the early persistence of rural, first-generation, and other underrepresented minority science, technology, engineering, and mathematics (STEM) students in their programs of study. We begin with a description of the project and then discuss the two pillars around which we have built our evaluation of this project. Next, we present the challenge we confronted (despite the utility of our two pillars) in identifying and analyzing systems change, as well as the literature we consulted as we considered how to address this difficulty. Finally, we describe the framework we applied and examine how it helped us and where we still faced quandaries. Ultimately, this reflection serves two key purposes: (1) to consider a few of the challenges of measuring changes in systems, and (2) to discuss our experience applying one framework to address these issues. [ABSTRACT FROM AUTHOR]

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program. Aligned with the 5LQDA method for ATLAS.ti, I first contextualize the strategies used to develop the analytical framework (Levels 1 & 2) and then deconstruct the translation process (Level 3) of pairing conceptual strategies with software tools. I then demonstrate the potential of the software in terms of selected and constructed tools (Levels 4 & 5) and discuss the importance of choosing tools aligned with the project's conceptual strategies. This paper acts as a resource to convey the value of integrating 5LQDA for ATLAS.ti throughout qualitative evaluation by engaging in an iterative process that creates opportunity for engagement. [ABSTRACT FROM AUTHOR]

The purpose of this study was to develop triangulation coding methods for a large-scale action research and evaluation project and to examine how practitioners and policy makers interpreted both convergent and divergent data. We created a color-coded system that evaluated the extent of triangulation across methodologies (qualitative and quantitative), data collection methods (observations, interviews, and archival records), and stakeholder groups (five distinct disciplines/organizations). Triangulation was assessed for both specific data points (e.g., a piece of historical/contextual information or qualitative theme) and substantive findings that emanated from further analysis of those data points (e.g., a statistical model or a mechanistic qualitative assertion that links themes). We present five case study examples that explore the complexities of interpreting triangulation data and determining whether data are deemed credible and actionable if not convergent. [ABSTRACT FROM AUTHOR]

Researcher–community partnerships are a necessary but not sufficient facet of effective research and evaluation with community-based projects and in clinical settings. This article describes two approaches that we have integrated into a multiyear, multiphase research and evaluation initiative supporting the health and well-being of vulnerable families. Specifically, we adopted a relational approach, intentionally and consistently focusing on building relationships, and a trauma-informed approach, highlighting safety across all levels. These innovative approaches have facilitated success in conducting safe, meaningful research and evaluation with community partners. Based on these approaches, we outline specific strategies and key considerations used in the context of the initiative, with the goal of encouraging others to adopt relational and trauma-informed methodological approaches and use these frameworks in research and evaluation efforts in applied settings. [ABSTRACT FROM AUTHOR]

Project schedules are logic models that focus on the timing of program activities. Value derives from the fact that schedule changes are not random. Why they occur, and how long they last, can reveal information that would not be easily revealed with other approaches to evaluation. Also, using project schedules as logic models forges a strong link between evaluation and program management. This article addresses three questions: How do models function in research? What makes project schedules a form of logic model? and How can evaluation data be gleaned from schedules? When using implementation schedules as a logic model, two guidelines must be observed. First, data collection must take place at regular intervals rather than be synchronized to program activities. Second, contact with stakeholders must respect their time commitments. The methodology will be illustrated with a case example from evaluating a "Peer to Peer" program to minimize unsafe electronic device use. [ABSTRACT FROM AUTHOR]

This article presents a discourse analytic study of how the concept of impartiality is socially constructed by members of the development aid community through an examination of linguistic traits and patterns within (a) inter- and intraorganizational interactions and (b) relevant aid evaluation policy documents. A qualitative analysis of unstructured and semistructured interviews with development professionals in Japan and a content analysis of relevant evaluation policies and documents have revealed observable evidence of distinct institutional practices of impartiality via project evaluations. Most notably, in contrast to a notion of evaluator impartiality that is strengthened through the principle of independence, development professionals in Japan perceive evaluations (‘hyouka’) to be a fundamentally hierarchical construct in which evaluator impartiality can be strengthened or legitimized through authoritative or hierarchical means. [ABSTRACT FROM AUTHOR]

Use of evaluation findings is a valued outcome for most evaluators. However, to optimize use, the findings need to be disseminated to potential users in formats that facilitate use of the information. This reflective case narrative uses a national evaluation of a multisite National Science Foundation (NSF) program as the setting for describing the evolution of, and strategic planning for, effectively disseminating evaluation findings and recommendations. The program being evaluated is NSF's Advanced Technological Education (ATE) program. The dissemination to more than 200 active ATE projects and other audiences forced the evaluation project to be creative in its approaches. The dissemination strategies are related to two very common types of evaluation information: surveys and site visits. Approaches used include Web postings of various reports, brochures, interactive data displays, guidebooks, and videoconferencing. Advantages and disadvantages in terms of use of the techniques by various audiences are discussed. [ABSTRACT FROM AUTHOR]

A multistakeholder evaluation procedure is presented to address the many challenges in evaluating the performance of condominial sewer projects in Brazil. Condominial sewerage is a promising appropriate technology that is coproduced by users and public agencies, but little is known about project performance. This article shows that multistakeholder evaluation is more appropriate to the research setting than evaluations based on a single stakeholder’s perspective, is more reliable than the use of expert judgments alone, and provides more information than recommended sewer performance indicators. The author argues that the perspectives of multiple stake-holders should be included in project evaluations. The multistakeholder approach presented is a workable solution to the condominial sewer evaluation problem. [ABSTRACT FROM AUTHOR]