Search

In 2015, The Wellcome Trust launched a new strategic approach to its work and created a framework to measure success. This article focuses on a particular dimension of this framework: research impact linked to the Trust's pool of research award holders. This article describes why contribution tracing was chosen as an evaluation approach. It also describes how it was used in innovative ways to handle a large number of cases with potential for impact and develop an organisation-wide evaluation plan that makes the most out of data science techniques while increasing understanding of evaluation and facilitating collaboration and joint planning with colleagues across the organisation. [ABSTRACT FROM AUTHOR]

Evaluation is essential to understand whether and how policies and other interventions work, why they sometimes fail, and whether they represent a good use of resources. Evaluability assessment (EA) is a means of collaboratively planning and designing evaluations, seeking to ensure they generate relevant and robust evidence that supports decision-making and contributes to the wider evidence base. This article reports on the context, the process undertaken and evidence from participants in an EA facilitated with public service workers involved in implementing a complex, area-based community improvement initiative. This is a novel context in which to conduct an EA. We show how the process allows practitioners at all levels to identify activities for evaluation and co-produce the theory of change developed through the EA. This enables evaluation recommendations to be developed that are relevant to the implementation of the programme, and which take account of available data and resources for evaluation. [ABSTRACT FROM AUTHOR]

The article examines the national evaluation culture of Romania from the perspective of shared values by contributors to the establishment of evaluation capacity. The author discusses differences in the values shared by the evaluation managers and by the evaluators, as well as the values that are and are not shared by the majority.

The current debate around the emergence of participatory approaches in evaluation practice suggests that participatory evaluation may be considered an organizational learning praxis, one which facilitates the development of a holistic process of intentional change. Through critical reflection on how participatory evaluation has been conceptualized, this article offers an overview of some of the contextual challenges encountered when using participatory evaluation to enable the creation of learning environments. Given the pluralistic nature of modern organizations and some contextual constraints, evaluators appear to have largely developed a more instrumental type of learning, which may, paradoxically, result in a significant source of resistance to intentional change. This article proposes a process of capacity building for evaluative research (CBER). This process offers a collaborative way of overcoming unforeseen resistance to intentional change by overcoming the challenges found in the relationship between participatory evaluation and organizational learning. The article concludes by suggesting some epistemological and organizational issues that evaluators should take into account when enabling the implementation of a process of CBER in pluralistic organizations. [ABSTRACT FROM AUTHOR]

The objective was to assess the impact of new guidance on the conduct of narrative synthesis in systematic reviews of effectiveness, by means of a blinded comparison of guidance-led narrative synthesis against a meta-analysis of the same study data. The conclusions of the two syntheses were broadly similar. However, differences between the approaches meant that conclusions about the impact of moderators of effect appeared stronger when derived from the meta-analysis, whereas implications for future research appeared more extensive when derived from the narrative synthesis. These findings emphasize that a rigorously conducted narrative synthesis can add meaning and value to the findings of meta-analysis. The guidance framework provided a useful vehicle for structuring a narrative synthesis and increasing transparency and rigour of the process. While there may be risks with overinterpretation of study data, the framework, tools and techniques described in the guidance appear to increase the transparency and reproducibility of narrative synthesis. [ABSTRACT FROM AUTHOR]

Recent developments in innovation theory and policy have led policy makers to assign particular importance to supporting networks of cooperation among heterogeneous economic actors, especially in production systems composed of small and medium enterprises. Such innovative policies call for parallel innovations in policy analysis, monitoring and assessment. Our analysis of a policy experiment aimed at supporting innovation networks in the Italian region of Tuscany intends to address some issues connected with the design, monitoring and evaluation of such interventions. Combining tools from ethnographic research and social networks analysis, the article explores the structural elements of the policy programme, its macroscopic impact on the regional innovation system and the success of individual networks in attaining their specific objectives. This innovative approach allows us to derive some general methodological suggestions for the design and evaluation of similar programmes. [ABSTRACT FROM AUTHOR]

This article provides some reflective thoughts on current research on regulatory impact assessment (RIA) in Europe. The narrative draws on the author's fieldwork diaries. It starts from the methodological problems of identifying and analysing regulatory impact assessment in the UK, Denmark, the Netherlands and Sweden. This leads to more fundamental theoretical questions about research design, the relationship between supply and demand of research and, ultimately, the aims of academic research in this area. The conclusions draw lessons for the emerging research agenda on better regulation. The main problem is not whether 'real' RIAs exist or not, since this would be equivalent to trying to fit the amoeba-like RIA 'substance' into our abstract 'forms'. The interesting problems are: what are the functions of different approaches to assessment, how are they used and how can they improve governance? [ABSTRACT FROM AUTHOR]

Embedded in the critical realist tradition, this article aims to explore the potentialities of the case study for evaluation purposes when complexity and specificity are moderate. Three objectives are pursued. First, it is stated that the focus of the evaluation effort is not necessarily the programme itself, but can be one premise upon which the programme is built. Second, taking into account Donald Campbell's contributions, it is asserted that, under favourable conditions, not only deterministic propositions, but also probabilistic ones can be rejected as highly improbable via case studies. Third, making use of the Context-Mechanism-Outcome model proposed by Pawson and Tilley, this article demonstrates the capacity of a critical case study to sustain theory building. [ABSTRACT FROM AUTHOR]

This article argues that complexity theory has the potential to bring important insights in reframing of the role and practice of evaluation but that the utility of complexity theory needs to be developed to support its application in evaluation research. This article focuses on the implications of a reformed relationship between theory and the empirical setting for what we can learn in evaluating policy. It suggests that while complexity theory provides a new way of looking at causal relationships and how we should approach them, we need to develop complexity-consistent approaches to understanding the interplay of agency and structure at the local level, the level at which explanation is deemed possible in complexity theory. It is proposed that the theory of negotiated order offers a basis to fill that gap in evaluation practice and to provide promise for theory-informed policy. [ABSTRACT FROM AUTHOR]

The article discusses the connections of evaluation. This study focuses on the issue facing evaluators today in looking for the critical connection of for whom are evaluations made and to what are they connected. Complex evaluation systems are now embedded in the system which makes the necessary connection to host policy communities, civil society actors, commissioners, professionals, and many other entities. Decisive evaluation continually challenge the evaluator's practice. Change is felt where it is essential.

This article describes an evaluation conducted in a region of northern Italy, based on the realistic evaluation approach. This case study is related to the application at regional level of national legislative measures against personal drug consumption, the so-called Article 75, through the implementation of a pilot programme. Article 75 DPR 309/90, is a 'sanctions-based' intervention mainly based on the mechanism 'threat-dissuasion', such that people who are found in possession of illicit drugs for personal use can be temporarily detained by the police, have a mandatory interview and abstinence test and sometimes have an administrative penalty or a mandatory treatment. First, the article describes the sequence of mechanisms anticipated by decision-makers (legislator) and the mechanisms that actually emerged from the programme evaluation; second, the outcomes resulting from the interaction between the mechanisms and the specific contexts are analysed. Finally, a few suggestions are proposed and the advantages of the realistic evaluation approach are discussed. [ABSTRACT FROM AUTHOR]

This article investigates the use of collaborative action research in both the development and evaluation of a project designed to develop inclusive practice in Early Years and childcare settings. The purpose of the evaluation was to ascertain how practitioners understand the term 'inclusion', how those understandings were translated into practice and how changes in that practice might be conceptualized and carried out. The article explores the use of facilitated collaborative action research as a core element in developmental evaluation. Reflecting on other approaches such as theory of change, it considers whether the use of action research supported the critical examination and development of a project, or whether, by being so similar in design, it constricted the evaluation in terms of reliability of data, accountability and providing a framework for planning and development. [ABSTRACT FROM AUTHOR]

This article presents some of the results from a study in progress, focusing on the influence of the institutional distance between evaluators and evaluees on the utilization of evaluations. The basis for the results presented here is an analysis of ten case studies from Switzerland. These cases involve evaluations that were carried out in different institutional contexts, with widely varying institutional distances between evaluators and evaluees. 'Qualitative Comparative Analysis' (QCA) has been used to interpret the cases, in order to allow a combination of case- and variable-centred comparisons. The analysis indicates that, under certain conditions, the institutional distance between evaluators and evaluees has no influence on the use of evaluations. In particular, formative objectives can be achieved quite independently of distance. When interpreting the results, however, one should not neglect the fact that they are solely based on a systematic evaluation of ten case studies with QCA. Generalization is not possible on this basis, nor is this the aim of the present article. On the contrary, the objective is to continue developing the debate about the influence of the institutional distance between evaluators and evaluees on the utilization of evaluations. [ABSTRACT FROM AUTHOR]

Integration of process with outcome evaluations has been recommended as a way of addressing some of the criticisms of Randomized Controlled Trials (RCTs) for evaluating social interventions. This is the second of two articles (the first was published in issue 10(4) of this journal) discussing the design, implementation and analysis of a process evaluation as an integral part of the RIPPLE (Randomized Intervention of Pupil Peer-Led sex Education) Study. The methods for integrating process and outcome data are outlined and findings are presented to illustrate how integration of process and outcome data enabled exploration of: 1) variation between schools in the extent and quality of the intervention implemented, and any impact of this on outcomes; 2) the processes by which the intervention might affect outcomes; and 3) the extent to which different subgroups of students or schools may have responded differently to the intervention. The article concludes with a discussion of the methodological issues arising from this attempt at integrating process and outcome evaluations within the design of an RCT. [ABSTRACT FROM AUTHOR]

Theory-based evaluation approaches are becoming increasingly popular in the evaluation of comprehensive community initiatives. Such an approach, Theory of Change (ToC), has been used in the external evaluations of two of four Scottish Health Demonstration projects. This article provides an empirically based analysis of the issues raised in the process of articulating a project's ToC. It then considers the value of the approach in relation to sharpening planning, providing formative feedback, improving performance management, guiding internal and external evaluation, judging impact and reducing problems of attribution. A conclusion is reached that, whilst such approaches are by no means a panacea, they offer one useful framework within which to negotiate a range of evaluation practice decisions. [ABSTRACT FROM AUTHOR]

The growing interest in evaluation as a new form of environmental governance stems from two developments: the movement towards New Public Management, and the search for new policy instruments for managing complexity; uncertainty and plurality of values in the pursuit of sustainable development. The former holds increasing accountability as the main purpose of evaluation, while the latter stresses the importance of learning. These approaches are often considered as mutually exclusive, but recent literature has underlined their complementary roles in policy making. This article examines to what extent the OECD Environmental Performance Review (EPR) programme has succeeded in combining the objectives of learning and accountability within a single evaluation framework. The EPRs have been relatively successful in avoiding the negative side-effects often associated with traditional performance measurement, but have usually failed to generate broad debate. Focusing on enhancing accountability through social learning would probably contribute to overcoming this problem. [ABSTRACT FROM AUTHOR]

Because of the global scale and diversity of their work, international aid agencies face major problems when attempting to represent their plans and evaluate their achievements. In this second of two articles looking at types of change processes, the focus is on complex processes of change that include mutual influence, parallel processes and feedback loops. Four practically oriented arguments are put forward for using a network perspective to represent these processes: the broad applicability of a network framework, its scalability, the range of measurement and descriptive tools available and the multidisciplinary body of theory and research available to inform agencies' theories of change. Networks are then contrasted with hierarchies as background metaphors, and implications are identified for the monitoring and evaluation of development projects. In this article relevant examples have been drawn from the author's consultancy experience with development aid programmes in Bangladesh and Ghana. [ABSTRACT FROM AUTHOR]

This article discusses some of the issues facing young people experiencing disadvantage in the labour market and associated implications for evaluation of programmes to address disadvantage. Drawing on the experiences of an evaluation study undertaken in an English county, the article explores alternative approaches to evaluation and the need for additional indicators to accompany quantitative outputs such as numbers into training or employment. The article is structured as follows: introduction to issues facing young people in the labour market; comment on evaluation of social programmes; a case study example of a local evaluation study; and discussion of evaluation issues arising from the study. [ABSTRACT FROM AUTHOR]

Focuses on a methodological approach Project Cycle management (PCM) that aims to improve the effectiveness of projects, programs and policies related to social sciences. Fundamental principles of the approach; Important aspects of PCM; Operational applicability of PCM; Limitations of PCM.

This article proposes a theoretical framework and a grid to unify models and to describe programme evaluation. It identifies three components in the evaluation process: the information-collecting process; the value-judgement product; and the use of the latter in decision making. It defines three differentiation axes (or dimensions): the social enunciation, the external or internal position of actors, and the formal or informal character of the evaluation components. The relevance for the practitioner and the researcher is discussed and briefly illustrated with two examples from the health promotion sector: a project selection process and a retrospective study of programme-evaluation practices. [ABSTRACT FROM AUTHOR]

All evaluations have a cost but not necessarily a value. Their value does not depend on their cost but on their use, and this article discusses factors affecting the use of evaluations. These factors could be taken into account in order to increase and improve the use made of evaluations and, consequently, their value. Two key issues (lags and the attribution problem) for the evaluation of the use of evaluations are discussed and a 'possibilist' approach to evaluation use is presented. [ABSTRACT FROM AUTHOR]

This article describes the 'mutual catalytic model' of programme design and formative evaluation. The model presumes two conditions under which it can be useful. Firstly, those responsible for a programme at the point of service delivery should participate equally in a programme's design and in its evaluation. This collaboration will: 1) increase the clarity of programme objectives and the level of consensus regarding them; 2) improve the evaluation design and process by connecting the evaluation with the realities at the point of service delivery; and 3) increase the likelihood that staff will use the results of the evaluation to improve programme design. Secondly, evaluators are required relatively frequently to facilitate the review of programme objectives and procedures, since reviews of this kind rarely take place without an extra stimulus and sometimes need a practical hand to guide them. In the mutual catalytic model of formative evaluation, local practitioners help to specify programme goals and improve evaluation, while evaluators help to elaborate programme goals and evaluation needs. This model is exemplified using experiences with homeless-child care centres in Seville in Spain. [ABSTRACT FROM AUTHOR]

Presents a reply to Saville Kushner's comments on the author's article on evaluation research, published in the April 2002 issue of the journal 'Evaluation.' Concept of methodology in relation to evaluation; Types of methodologies used in social and educational program evaluation; Need for mixed-method evaluation.

Comments on an article on evaluation research by Jennifer Greene et al. Criticism of the reasons cited for mixing various methods of evaluation; Assessment of mixed methods of evaluation in relation to social organization, pluralism, power and politics.

Additionality is a recurring issue when evaluating the impact of public support for research and development (R&D). To what degree does public support spur additional R&D efforts and output? This article discusses the additionality concept and its measurement. Verbal reports from recipients of public support are a widely used method for measuring the additionality of ongoing R&D programmes. However, it is often argued that these reports cannot be trusted since the recipients might have an incentive to answer strategically to maintain funding. Based on the verbal reports of project additionality in Norwegian evaluations over the last two decades, involving 2624 observations, higher additionality is reported in projects further from the market involving higher risks and uncertainty than in projects closer to market launch. This is in line with a priori expectations. It will be argued that the problem of strategic answering might be over-rated, and that verbal reports can provide important evidence. This is exemplified by results from verbal reports of additionality obtained by interviewing the customers of 21 different Norwegian research institutes.[SUP1] They show that public support is more important for the realization of projects within small- and medium-sized enterprises than with larger firms. [ABSTRACT FROM AUTHOR]

In the UK a great deal of attention is currently focused on the potential of the 'theories of change' approach to evaluating complex public policy interventions. However, there is still relatively little empirical material describing its application. This article discusses the use of 'theories of change' in the national evaluation of English Health Action Zones (HAZs). It locates 'theories of change' within the wider context of evaluation approaches and assesses its strengths and weaknesses as an evaluation framework. The article then focuses on a key aspect of complex public policy interventions - cross-sector collaboration. Drawing on data about cross-sector partnerships and community involvement from the English HAZ evaluation, the article explores the contribution of 'theories of change' towards examining the building of collaborative capacity in HAZs. The article also describes the 'co-research' approach being employed within the national HAZ evaluation. It discusses how this approach can complement the use of 'theories of change', contribute to managing change within organizations and communities and facilitate more effective use of evaluation within a local health context. [ABSTRACT FROM AUTHOR]

What we expect from public programs determines how the programs will be judged. This article presents a systematic values inquiry used to examine how much importance citizens and stakeholders attached to specific indicators in determining the success of a public preschool program. Four groups were surveyed: teachers, administrators, parents and the public. All four groups showed significant differences with respect to the importance of 29 possible indicators. The four groups agreed on the importance of high-quality services and about the lack of importance associated with economic benefits for families and certain educational outcomes for the preschoolers. Using confirmatory factor analysis, the opinions of the groups were shown to have similar underlying structures; however, teachers made greater distinctions between the groups of potential outcomes and the public distinguished between outcomes the least. Systematic values inquiry can be an important tool in designing evaluations that will produce information that can influence the judgments of citizens and stakeholders about a program's value. [ABSTRACT FROM AUTHOR]

Evaluation research is tortured by time constraints. The policy cycle revolves quicker than the research cycle, with the result that 'real time' evaluations often have little influence on policy making. As a result, the quest for evidence-based policy has turned increasingly to systematic reviews of the results of previous inquiries in the relevant policy domain. However, this shifting of the temporal frame for evaluation is in itself no guarantee of success. Evidence, whether new or old, never speaks for itself. Accordingly, there is debate about the best strategy of marshalling bygone research results into the policy process. This article joins the imbroglio by examining the logic of the two main strategies of systematic review: 'meta-analysis' and 'narrative review'. Whilst they are often presented as diametrically opposed perspectives, this article argues that they share common limitations in their understanding of how to provide a template for impending policy decisions. This review provides the background for Part II of the article (to be published in the next issue, Evaluation 8[3]), which considers the merits of a new model for evidence-based policy, namely 'realist synthesis'. [ABSTRACT FROM AUTHOR]

Performance measurement systems are cast as significant responses to contemporary public demands for government accountability for social spending and social programs. Performance measurements are then critiqued as inadequate measures of social program quality, as inadequate representations of program quality, and as at odds with evaluative processes that advance the ideals of deliberative democracy. [ABSTRACT FROM AUTHOR]

The burgeoning performance management movement, with its emphasis on social program 'results' measured typically by a limited set of quantitative indicators, has developed a life of its own largely apart from the evaluation research movement. Reflecting the differences in the professional history, interests and training underlying the two movements, the relationship between these disparate approaches to establishing public accountability has lacked coordination and defied integration. This article discusses the basic concepts guiding the evolution of these movements in the context of the goals of information production, and explores the major conceptual, measurement and methodological problems resulting from the lack of accommodation between them. It also provides suggestions about how these two important approaches can be better integrated, both professionally and organizationally, for the purpose of enhancing the reliability and validity of social program assessments, and therefore for improving policy development and program management. [ABSTRACT FROM AUTHOR]

An obituary for evaluation researcher Carol H. Weiss is presented.

Editorial. Introduces a series of articles related to the evaluation of social action programs. Role of evaluators in terms of program design as well as formative evaluation; Evaluation of a local program addressing the needs of young people disadvantaged in the labor market; Analysis of program-evaluation practices.

Editorial. Comments on the evaluation of a community development programs. Change on the development program; Benefits of the mixing methods on evaluation of development program evaluation; Demands of development programs.

Presents an index of articles in the fifth volume, 1999 issue of the 'Evaluation' journal. 'Evaluation and Performance Management in Government,' by Ian C. Davies; 'The Inequality of Performance Measurements,' by Jennifer C. Greene; 'Evaluation and the Social Construction of Impacts,' by F.B. van der Meer.

Editorial. Addresses issues concerning performance management and evaluation. Debate surrounding evaluation methods around the world; Comparison between performance management and evaluation research.

A calendar of events related to evaluation research conferences is presented which include the UKES Annual Conference from April 17 to 18, 2013, the Mixed Methods International Conference form June 25 to 27, 2012, and the American Evaluation Association Annual Conference from October 14 to 9, 2013.

The article reports on the German Evaluation Society's initiative for training commissioners. The German Evaluation Society (DeGEval) has established a group of researchers that would compile orienting materials to commissioners and evaluators. This is to give them training in evaluation including important information for evaluation tender or terms of reference and in selecting the best tender in quality, finances, and in monitoring the evaluation process.

Focuses on fourth annual conference of the French Evaluation Society. Venue of the conference; Theme of the conference; Topics on which debate took place in the conference.

Focuses on the International Development Association which is established to promote evaluation capacity in international development. Date of the establishment of the association; Supporters of the association; Activities of the association.

Introduces articles to the special 1999 issue of the 'Evaluation' journal. Application of evaluation in international development, urban regeneration and training and labor markets; Concern with balancing stakeholder and enlightenment objectives with the social construction of impacts; Evaluation and risk management.