Search

In the coming years, public programs will capture even more and richer data than they do now, including data from web-based tools used by participants in employment services, from tablet-based educational curricula, and from electronic health records for Medicaid beneficiaries. Program evaluators seeking to take full advantage of these data streams will require novel statistical methods, such as Bayesian approach. A Bayesian approach to randomized program evaluations efficiently identifies what works for whom. The Bayesian approach design adapts to accumulating evidence: Over the course of an evaluation, more study subjects are allocated to treatment arms that are more promising, given the specific subgroup from which each subject comes. We identify conditions under which there is more than a 90% chance that inference from the Bayesian adaptive design is superior to inference from a standard design, using less than one third the sample size. [ABSTRACT FROM AUTHOR]

Treatment effect estimates from a regression discontinuity design (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the evaluation population of interest suggests that RDD evaluations lack external validity. New methodological research offer strategies for studying and sometimes improving external validity in RDDs. This article examines four techniques: comparative RDD, covariate matching RDD, treatment effect derivatives, and statistical tests for local selection bias. The goal of the article is to help evaluators understand the logic, assumptions, data requirements, and reach of the new methods. [ABSTRACT FROM AUTHOR]

George Grob presented the fifth and final Eleanor Chelimsky Forum address at the 2017 annual meeting of the Eastern Evaluation Research Society. In this commentary, I respond to several points that George raises in the American Journal of Evaluation paper based on that address. An overarching theme of my comments involves the potential for strengthening the linkages between evaluation theory and evaluation practice. Research on evaluation should also be added to this pairing. [ABSTRACT FROM AUTHOR]

This article is the fifth in a series on the relationship of evaluation theory and practice. It was commissioned by the Eastern Evaluation Research Society and the American Evaluation Association as part of The Chelimsky Forum, launched in 2013 to honor Eleanor Chelimsky, one of the most influential and respected program evaluators of our era, and the former director of the Program Evaluation and Methodology Division of the U.S. Government Accountability Office. Supported by the Robert Wood Johnson Foundation, the forum is aimed at discussing challenges in the fit between theory and practice and strategies for addressing them. This article focuses on the relationships between evaluators and their users, especially policy makers. It tries to see things through the eyes of these users, recognizing their values and concepts related to evidence, proof, and “the truth” as they see it. It therefore tries to come to grips with the common core of evaluation thinking among a host of such clients, aiming to speak to them in their own language, but without sacrificing evaluator independence. It also recognizes the modern world of electronic reports and the possibility of both widespread and targeted dissemination of evaluation results. Finally, it offers some hints about evaluator development in this modern world, especially in the volatile world of policy and politics. [ABSTRACT FROM AUTHOR]

Efforts to promote professionalism in evaluation, whether through certification, credentialing, or other path, are not new, but there is a new push to adopt sets of essential evaluator competencies, both in the United States and globally, that are intended to advance professionalism of the field. This emphasis on professionalism is relevant to the American Journal of Evaluation Section on Professional Values and Ethics in that this section focuses on how values are to influence evaluation practice and on how our understanding of valuing affects our view of the ethics that should guide evaluators. In his invited contribution to this issue’s section, Tom Schwandt provides an account of professionalism based on a view of evaluators serving a moral purpose in serving society. This account, a democratic professionalism in which citizens are engaged as co-owners of evaluation, emphasizes a professional ethos of evaluation that complements the focus of those working to advance the more technical aspects of professionalism. To introduce Schwandt’s essay, we review some of the recognized promises and challenges for professionalism in evaluation and highlight the promise of Schwandt’s approach for managing the tension in professions between serving professionals and serving society. [ABSTRACT FROM AUTHOR]

Information about the 2016 Annual Innovative Methods Meeting hosted by the Office of Planning, Research, and Evaluation in the Administration for Children and Families of the U.S. Department of Health and Human Services is presented. Topics discussed include the efforts of the Social Innovation Fund (SIF), the alternatives to randomized-controlled trials (RCTs) for assessing program impacts and the social service delivery. The meeting features researchers from academia and evaluation firms.

A field experiment was conducted to test the effectiveness of norm-based persuasive messages for increasing response rates in online survey research. Participants in an interdisciplinary conference were asked to complete two successive postconference surveys and randomly assigned to one of two groups at each time point. The experimental group received a message asking them to complete an online survey that highlighted a descriptive social norm indicating typical response rates among attendees of prior similar conferences. The control group received a generic request to complete the online survey without any norm-based appeals. As expected, participants receiving a message highlighting the descriptive social norm, when asked to complete an online survey were more likely to comply with the request as compared to those who received the generic message without normative information. Participants receiving a double dose of the descriptive norm treatment at both time points were more likely to complete the second online survey than those who did not receive the descriptive norm intervention at either time point. Implications for future research on evaluation are discussed. [ABSTRACT FROM PUBLISHER]

Drawing on an evaluative framework inspired by the capability approach, the article assesses the nature and implications of company responses to the British legislation transposing Directive 2002/14/EC on information and consultation rights of employees. Evidence from five case studies in the business services and the financial sectors suggests that the introduction of the Information and Consultation of Employees Regulations 2004 drove the spread of voluntary arrangements. However, the legislation has not promoted so far an effective framework for the development of deliberative procedures between management and labour with the aim of advancing a 'capability for voice'. This is attributed to its institutional design and the limited degree to which extra-legal 'conversion factors' are available in the British industrial relations system. [ABSTRACT FROM AUTHOR]

Imagine a world where evaluation is a social epidemic. where individuals, groups, organizations, and communities are constantly learning about and from evaluations. The author believes that we are well on our way to creating a "global cascade" of evaluative thinking and practice. Evidence of this phenomenon can be seen in the field's increasing commitment to evaluation capacity building and participatory evaluation approaches and the expanding applications of technology for accessing and disseminating information. If the old saying is "The second act is the best" then we might ask ourselves, "How can evaluation realize its potential for making a difference in our increasingly complex and changing world?" The author argues that addressing today's critical evaluation theory, practice, and policy issues requires us to understand the ways in which evaluation facilitates learning, how evaluators can maximize learning from and about evaluation, and what it would mean if the evaluation profession was a sustainable learning community. [ABSTRACT FROM AUTHOR]

This action research study addresses the link between building evaluation capacity and enhancing evaluation use for learning. The author shares her experiences and reports on the evidence collected from a set of self-evaluation capacity-building interventions that she implemented. Using a mixed-method approach, the author first determined that the organization lacked the prerequisites to learn from evaluation. She then implemented self-evaluation capacity-building interventions, accompanied by a real-time study of how the participants changed attitudinally, cognitively, and behaviorally. Results indicate the potential of these interventions to facilitate an individual's ability to learn from evaluation and underscore the need for supportive organizational contexts, structures, and processes if evaluation use for learning is to occur throughout the organization. [ABSTRACT FROM AUTHOR]

Focuses on the issues affecting the field of evaluation research. Problem on the scale of quantitative studies of evaluation; Concept of causation; Effect of fact-value dichotomy on the development of the field of evaluation.

Focuses on the tracking and probing performance of public program service delivery for evaluation professionals in the U.S. Method for improving the understanding of the operations and results of public programs; Potential roles of evaluation professionals; Effects of performance measurement on the supply of evaluation expertise.

Comments on the factors related to the projections for the field of evaluation research. Competing pressures in projecting the field of evaluation; Effect of high-probability predictions on plausibility recommendations; Discussion on ideal-future option for evaluation.

Presents a preview on the projections in the field of evaluation research in the U.S. Developments occurring in the field of evaluation; Demand for evaluator expertise; Factors affecting the field of evaluation.

Examines the role of practitioner skill on evaluation program teaching in the United States. Description of the professor-student involvement on evaluation separate from coursework; Discussion on directed evaluation experience; Implication of challenges on teaching program evaluation.

Presents an interview with University of Illinois professor Jennifer C. Greene on the evaluation of Natural Resources Leadership Program in the United States. Illustration of evaluation methods; Basis of the evaluation; Purpose of the evaluation.

Examines an evaluation framework based on the perceived needs of grant-giving foundations in the United States. Role of evaluation in grant-giving foundations; Experience of creating and managing evaluation offices; Challenges faced by organization regarding evaluation.

Focuses on the development of collection capacity of the evaluation community in the United States. Impact of collective capacity building on quality of life among evaluators; Factors affecting the relationship of evaluators; Recommendations for improving collective capacity relationship.

Focuses on ethical issues faced by program evaluators. Classic example of a stakeholder fearing evaluation due to apprehensions about the consequences of potentially unfavorable results; First in a series of institutional evaluations of the university's administrative services authorized by the state legislature; Dilemma about whether to avoid certain areas of inquiry in the study.

Deals with ethical challenges faced by program evaluators. Conduct of evaluation within a political system with multiple layers of authority and influence; Importance of inclusion; Conflict between a school director's intentions and evaluator's need to conduct a competent evaluation.

Presents a second set of response to comments made by the author of an article dealing with the use of meta-analysis in evaluation practice. Clarification of nature and depth of disagreements with Professor Mark Lipsey; Examination of how the application of scientific method to contemporary life can lead to the deformation of praxis; Social scientific evaluation practice suggested by Lipsey.

Comments on an article dealing with the use of meta-analysis in evaluation practice. Arguments by Professor Mark Lipsey; Efficacy of social interventions; Insight into the design of evaluation research on interventions; Meta-theoretical investigation of the technique of meta-analysis.

Suggests the use of meta-analysis in developing taxonomies of programs. Systematic summaries of evaluation studies; General characteristics of programs; Choice of method and procedure; Continuous improvement program for evaluation practice.

Highlights the value of Planned Variation evaluations to teachers and program operators. Collaboration between an evaluator and one or more program operators to evaluate the effectiveness of variations of an educational or social program; Establishment of university-based centers for evaluation; Utilization in the Planned Variation Cross-Validation Model.

Analyzes data variability to address relationships among performance measures. Increased emphasis on the use of performance measures; Analysis strategies for examining relationships; Hypothesized connections among program delivery, outputs, and desired outcomes; Costs and caveats.

Focuses on international development program evaluation in the United States and developing countries. Definition of development programs; Lack of longitudinal databases as a difficulty in evaluation research; Differences in the role of international donor agencies; Suggestion of a beneficial experience exchange between international practitioners.

Examines the application of empowerment evaluation approach in the context of comprehensive communities initiatives. Discussion on the role of evaluators and comparison to other approaches; Requirement of creativity for the project; Factors contributing to the effectiveness of the approach; Suggestion of stakeholder and evaluator involvement.

Focuses on the planned variation approach for improving social and educational programs in the United States. Implications for the needs of program operators; Provision of methodological advantages and tests program theory; Consideration of issues for implementation; Comparison of the approach to other types of theory-based evaluation.

Comments on the role of the human resources staff in assisting employee assistance program evaluations. Reason for hiring an external consultant in conducting formative evaluation; Need for evaluators to analyze the context and political implications of evaluation; Provisions under the Program Evaluation Standard; Tips in avoiding ethical problems.

Comments on the employee assistance program evaluation of the fictional Gee-Gaws R Us company. Identification of the corporate problem instigated by Human Resources; Purpose of the formative evaluation to the Employee Assistance Program; Significance of ethical evaluation in assessing corporate situations.

Focuses on the significance of the certification process for evaluators in the field of evaluation. Development of an evaluator certification system for the American Evaluation Association (AEA); Challenges in implementing the certification process; Steps in determining the sufficiency of definitional and procedural processes.

Presents the metaevaluation of the St. Pierre, Layzer, Goodson, and Bernstein (1999) evaluation study. Use of the Program Evaluation Standards and the American Evaluation Association's Guiding Principles for Evaluators in standardizing evaluations; Purpose of the metaevaluation; Implication of metaevaluation on evaluation practices.

Describes a voluntary system for credentialing evaluators in the field of evaluation. Discussion on credentialing issues by the American Evaluation Association during the 1998 Chicago session; Inclusion of an unabridged and adapted version of the table used for certification purposes; Rationale of a voluntary certification system.

Comments on the article of Professor John Stanfield on race awareness in the United States. Recapitulation on American educational and social structures; Significance of creating institutional change; Problems of educating educationally and socially disadvantaged children; Value of rethinking methods for minority evaluation programs.

Examines the combination of programmatic theory and structural equation modeling (SEM) in designing and evaluating behavioral interventions. Implication of program theory to evaluation practices; Case study of a randomized experiment on sexual risk taking reduction; Suggestions for improving and assessing SEM programs.

Focuses on the significance of advancing the application and development of theory-based evaluation research in public health practices. Use of guides in defining program and problem theories; Implementation of strong reasoning in justifying theory-based evaluations; Reasons in examining program theory.

Comments on the impact of management revisions on evaluation practices. Suggestion for evaluators in conducting proper deliberation and progress reports; Importance of organizational skills in performing evaluative works; Enumeration of the Program Evaluation Standards.

Presents the Reader-Focused Writing Project evaluation by CIRCE in Illinois. Demonstration of a close-to-deal evaluation case regarding benefits of veterans; Performance rating of the CIRCE study in managing the program for the Veterans Benefits Administration; Verdict of meta-evaluators on the CIRCE program evaluation.

Provides an abbreviated summary of a completed evaluation study critiqued by competent evaluators in Chicago, Illinois. Evaluation of the Reader Focused Writing program for the Veterans Benefits Administration (VBA); Identification of VBA training goals; Evaluation design as an interpretive data; Excerpts from the data collection and analysis section.

Part II. Provides a systematic meta-evaluation of the Veterans Benefits Administration's Reader Focused Writing program evaluation in the United States. Implications for the improvement of professional practice; Use of a systematic inquiry in assessing evaluative standards; Importance of evaluator competence and ethical professionalism in work performance.

Comments on the implications of human disability and diversity for evaluation practice in the United States. Impact of human differences on social researches; Challenge of including disabled people in cultural and ethical value assessments; Description of the disability social model.

Focuses on the relevance of physical disability in evaluation in the United States. Disability as a dimension of human diversity; Prevalence of neglect of disability in evaluation; Comparison between the moral and medical model of disability; Importance of core values to evaluation practices; Issues concerning evaluation of disability programs.

Presents key aspects of an integrative framework for social evaluation practices in the United States. Description of an innovative scheme for categorizing evaluation methods within four inquiry modes; Relevance of a realist philosophy to the framework; Schism due to the paradigm war between qualitative and quantitative evaluation methodologies.

Assesses the effectiveness of a strategy in obtaining active written parental consent for outcome evaluation of an alcohol, tobacco and drug abuse prevention program in the United States. Implementation of the study in school districts; Use of the multiple case study approach and a performance adequacy analysis; Cost effective analysis of the program.

Presents a conceptual framework for practicing a theory-driven approach to program evaluation. Role of theory in defining the problem and target population of a program design; Identification of factors affecting treatment process through theory; Incorporation of variables reflecting theoretical concepts; Three categories of the framework variables.

Presents some non-randomized constructed comparison groups for evaluating age-related outcomes of intervention programs for young children in the United States. Impact of treatment on age-dependent variables; Effect of economic conditions on the developmental process of children; Correlation between the length of treatment time and the different age spans.

Describes the experience of a social evaluator in constructing a World Wide Web site to collect evaluation data of an educational professional development program. Lessons on Web site development; Comparison of data quality between electronic and paper-and-pencil methods; Advantages of Web-based data collection.

Comments on the importance of professional ethics in the management of evaluation standards by social evaluators in the United States. Role of practice codes in stating principles and aspirations in general terms; Need to prioritize standards according to the hierarchy of values; Illustration of ethical conflicts concerning social standards.