Search

Percorsi is a matched savings program aimed at helping students from low-income households attend college. Previous experimental research has demonstrated the positive effects of the program on college enrollment and persistence. This paper investigates the extent to which these experimental results are attributable to the applied targeting strategy: students were admitted to the study through a targeting mechanism based on household income and on a predictive model of university enrollment probability. Applicants with a very high predicted university enrollment probability and those who revealed no intention of going to university were excluded from the program. The authors assess the performance of this approach by comparing cost-effectiveness estimates against simulated scenarios, showing that both the implemented and a simplified targeting yield a cost-effectiveness gain over a no targeting scenario. The study makes the case for an increased use of prior knowledge and collaboration between researchers and program managers to enhance programs' cost-effectiveness. [ABSTRACT FROM AUTHOR]

This paper addresses non-academic training in evaluation with a twofold perspective. The first one will focus on the reconstruction of a map of the existing training offer in Italy. This map, updated at September 2022, will propose an overview of the existing courses in terms of both main features (e.g.: duration, structure, providers) and macro-contents. The second perspective will match the results of this analysis with the profile, overall and by competences, of the Expert in Evaluation of Policies, Programmes, Projects and Institutions as defined by the Italian Evaluation Society (A.I.V.). This match intends to explore the correspondence between the courses and the A.I.V. proposal. In such a way, an attempt will be made, on the one hand, to contextualize the courses with respect to A.I.V. evaluator's profile and competences and, on the other hand, to do the same with A.I.V. profile in relation to evaluation courses. Finally, some recommendations will be proposed on the basis of the evidence produced by the analysis. [ABSTRACT FROM AUTHOR]

The main objective of the paper is to address and explore a much-debated issue within the evaluation community: the idea of "democratic evaluation". What exactly is meant by this term? When can an evaluation be said to be democratic? Through the adoption of what behaviours should this ideal translate into concrete working practice? The essay aims to contribute to broadening the community's debate on these questions, which are central at a historical-political moment such as the post-emergence from Covid-19 and in the midst of an international crisis, that between Russia and Ukraine, which calls democratic evaluation into question in our country. On a general level, in fact, the situation of uncertainty due to the unpredictable effects of the pandemic, the social and economic crisis in the country and the consequent growing distrust of the citizens towards the administration seem to have increased the demand for accountability, leading to talk, as in the rest of Europe as well, of a "democratic deficit". On the other hand, recent administrative re-centralization, even assisted by current evaluative practices, only aggravates the inefficiency of public administration. Accountability, freedom and democracy are never given, they need to be defended, always regained: evaluation assumes a strategic role to this end; it can help bring citizens closer to public administration. The crisis between Russia and Ukraine is changing geopolitical balances; the effects of the war economy, inflation, the energy crisis, and the possible escalation of the conflict add to the uncertain situation due to the pandemic. New inequalities and an increase in poverty threaten to generate deep rifts in the country's socioeconomic fabric. Systemic shocks add to chronic problems: a demographic collapse, stagnation, job insecurity, and a lag in digitization. At the same time, the political system fails to represent and stabilize consensus over time. Governments also suffer from this instability, with the inability to design and implement structural reforms for the country, including that of the bureaucracy. Social complexity, chronic problems and systemic shocks, require the ability to apply effective public policies. State intervention and the ability of the bureaucracy to be efficient and responsive to social change are central issues for the health of democracy. Public policies move within a framework of very high complexity, in which the role of evaluation emerges, which should tend to the development of democratic accountability, illuminating decision-making choices through its methodological rigor, analytical capacity and scientific knowledge. The essay thus aims to address the potential of a democratic evaluation that can broaden the number of stakeholders, the field of values that are taken into account, the repertoire of research methods, the field of interest of policies, and the very definition of the objects of evaluations. [ABSTRACT FROM AUTHOR]

The literature on evaluation capacity building in educational-school contexts underlines the need to adopt training models tailored to schools, enhancing aspects such as collaboration among teachers, school culture, attitudes towards evaluation. This paper presents the results emerging from three training paths on school self-evaluation (North, Centre, and South Italian macro-areas). The project involved a judgment sample of 42 Italian schools. At the end of the training, a Satisfaction Questionnaire was administered to teachers (N. 243). The Questionnaire aimed of investigating the participants satisfaction about more dialogic and relational aspects of evaluation capacity training. As the results show, some of the pathways were able to capture trainees' needs to a greater extent, inspiring confidence and supporting the development of their evaluative skills. [ABSTRACT FROM AUTHOR]

The paper aims to highlight the importance of source triangulation (Denzin, 2009; Grion et al., 2021) in school innovation evaluation processes. This contribution is part of a broader investigation aimed at evaluating the spillover of innovation processes within schools that have joined the Educational Vanguards Movement: in particular, the perceptions of School Principals, teachers and students regarding some aspects considered peculiar to the evaluation of school transformations from an innovation perspective are compared here. The results suggest a greater agreement between teachers and School Principals with respect to the levels of collaboration and sharing of innovative practices, but the opinions among the three actors involved on the use of innovative teaching methodologies and evaluation methods are discordant. Triangulation thus appears to be an added value that can provide an overall picture on the crucial aspects of organizational evaluation. [ABSTRACT FROM AUTHOR]

Internationally and nationally, the need for a "culture of evaluation" has progressively grown in schools, considering internal and external evaluation processes as tools for implementing an integrated model of inclusive, equitable and sustainable education. Evaluation can become an agent of change balancing modes of accountability and improvement at the micro-individual and macro-systemic levels when it is exercised with formative intentions by mindful as well as reflective professionals who interact their expertise through effective, distributed, expert and collaborative leadership. The paper is intended to investigate the collaborative dimension in relation to evaluative practices as a trait of teaching professionalism to nurture Evaluation Capacity Building for School Improvement. A questionnaire addressed to a sample of university students revealed that future teachers' expectations of collaboratively acted evaluative practices are higher than the perception found in a group of in-service teachers. An initial descriptive data analysis suggests that capacity-building processes should also be activated in evaluation for the purpose of improvement, for which it is not enough to trigger the initial training of future teachers, but it is also necessary to work on the training of in-service teachers because training needs to be co-constructed and strengthened. [ABSTRACT FROM AUTHOR]

The Covid-19 pandemic has required new reflections on the evaluation of multi-dimensional programs aimed at multiple stakeholders. The traditional approach, based on the blueprint approach, resulted unsuitable for grasping the complexity of the new scenario, due to its focus on objectives achieved. This paper aims to describe the evaluation of two projects for the inclusion of marginalized young people, through the Positive Thinking lenses. In fact, it turned out to be the theoretical approach most capable of responding to the complexities that emerged in this peculiar historical period. The evaluation becomes useful for triggering improvement actions and for capturing the efforts of the organizations. The contribution intends to open the theoretical reflection on the complexity of the programs, and on the role of the evaluator in accompanying the organizations by supporting them, in order to develop effective actions for the beneficiaries. [ABSTRACT FROM AUTHOR]

The paper proposes a case study about integration of focus group and survey to listen teenagers and preteens in the participatory evaluation of a local educational project. The contribution discusses strengths and weaknesses of the focus group as a participatory instrument in the evaluation process and the integration with the survey. Furthermore, the authors reflect about possible ways to enhance the approach, starting from the Youth Participatory Evaluation model and focusing on the role of teenagers in different evaluation stages. [ABSTRACT FROM AUTHOR]

This paper proposes a reflection on the connection between social innovation-driven social design and social impact evaluation (SIE). The author argues for a cognitive and learning function of SIE that can be conducted through the choice of methods, approaches and tools that can capture and enhance the innovations introduced by the programs. To this end, the construction path of the research design of the "Ri-belli" project, funded by the "Educational Poverty Alleviation Fund", is presented. The intervention is representative of the renewed system of second welfare, marked by the involvement of the private social sector in the policy making process and the desire to disseminate evaluative practice in the social sphere. [ABSTRACT FROM AUTHOR]

This paper aims to analyze the reflexivity issue in scientific research practice. Starting from a theoretical reflection on the epistemological and methodological reasons listing reflexivity among the main practical issues of social and evaluative research, the contribution seeks to highlight how the sus-tainability of reflective research practices actually swings between adequacy and challenges. With particular reference to pragmatic approaches to knowledge, it is believed that the main elements that can undermine reflex-ivity are the contexts in which the investigation takes place and the language's indexicality. Finally, particular attention is paid to the transactional nature of the evaluative research in which contexts and languages constitute a mix of indisputable relevance, for its effectiveness and quality. [ABSTRACT FROM AUTHOR]

Since the Theory of Change became a widespread tool, both in programs and in evaluation, evaluators have been more concerned about what a theory looks like rather than about the nature of change. Change can take place in different ways: either according to planned directions or in unexpected, even surprising, ways. This article proposes to take a different perspective: beyond an evaluation demand for the explanation/verification of the intended working of a program it is suggested to demand how change can happen, possibly even in a surprising way, that might challenge rigid hypotheses and models. To this purpose, it suggests to grasp the elaboration of "possibilism", originated by Colorni, theorized by Hirschman and practiced by Tendler. Such an approach may suit those evaluators who advocate that evaluation challenge a ToC's assumptions, and that new hypotheses, based on the experience of an intervention implementation, be developed in an iterative way. [ABSTRACT FROM AUTHOR]

To answer some evaluation questions, the Independent Evaluation process of the Veneto Region's Smart Specialisation Strategy (ROP ERDF 2014-2020) will be taken as a case study, considering the subsequent preparation of the new M&E system (2021-2027). Starting from the reference context and the definition of the evaluand, the article lists the methods and analysis techniques used to demonstrate the importance of implementing learning and empowerment-oriented evaluation paths in order to provide policymakers with a basis for the preparation of the next Programming period. [ABSTRACT FROM AUTHOR]

The ability of territorial welfare systems to respond to the needs expressed by different local contexts is under pressure due to the growing complexity of our societies. In this context, the need to innovate territorial welfare models and practices can be realized through experiments and implementation and evaluation strategies that are based on the active participation of users served by the services. Participation and the centrality of beneficiaries are now recognized as important prerequisites for meeting the needs of the community. Similarly, a more in-depth and solid understanding of different contexts can contribute to strengthening the capacity of local welfare systems to develop solutions to existing and emerging issues. Combining these three development directions (research, evaluation and participation) is a promising and relatively unexplored path in the field of territorial welfare services. In this perspective, emancipatory research allows for recognizing and valuing the analytical capabilities of people living in marginalized situations, making them active participants in the research process, capable of gaining awareness and creating knowledge about their living conditions and increasing their level of empowerment. Starting in 2018, the Social Services of the Municipality of Prato experimented with the "Social Board," a specific procedure for activating personalized socio-labour inclusion pathways provided by the Guaranteed Minimum Income (REI). Within this context, an emancipatory research process was promoted with some service users. The goal has been, on one hand, to create new knowledge about the dynamics of marginalization and the interaction between users and social services, and on the other hand, to promote an empowerment process for the involved actors. The article shows that the effective involvement of service beneficiaries in the research allowed both users and social workers to reflect on the implementation of social policies, identifying innovative welfare strategies aimed at the actual empowerment of disadvantaged individuals. [ABSTRACT FROM AUTHOR]

In this article, we will address an aspect many have considered "exotic" in the evolving evaluation landscape. We are referring to the diverse world of Big Data (BD) and Artificial Intelligence (AI), which arguably represent one of the significant innovations in the field of evaluation and will likely continue to grow and evolve in the decades to come. We will discuss three specific aspects: 1) the opportunities and main uses that BD and AI offer to evaluation; 2) the challenges and risks that evaluation research and practice are facing concerning the 'new' availability of data and analysis methods not classically referable to the world of evaluators; 3) The state of diffusion in Italy of BD and AI in evaluation research and practice. [ABSTRACT FROM AUTHOR]

The European Digital Agenda represented a priority of the Europe 2020 strategy for smart growth of the European Union, towards the objectives of Europe 2030. The article outlines the regulatory framework on the digital strategy in Europe and Italy, proposing a theoretical-methodological discussion on the evaluation of policies on digitalization. The national results are presented in relation to the DESI INDEX, the main European tool for the observation of socio-cultural changes resulting from investments in digitalization, in the pre and post COVID period, discussing on the current state of evaluation practice in the implementation of Digital Agenda in Italy. The policy cycle outlined by the EU 2021-2027 programming and by the Recovery Plan on the topic of digital transition leads to a discussion of the main challenges of the evaluation practice, evidence of the structural complexity of the policies on digitization, stressing the need for multifocal and pluralistic evaluative perspectives of programs. [ABSTRACT FROM AUTHOR]

After a brief review of the evaluation of development cooperation at the international level, the essay describes the Italian case, the evolution of the institutional structure dedicated to the evaluation of Italian development cooperation and the evaluation practices of Italian civil society organizations. The essay reports the experience of the Italian Evaluation Association's Thematic Group "Evaluating Development Cooperation". The Thematic Group was born in 2019 to stimulate a debate in Italy on evaluation theories and practices of international cooperation. The essay discusses the role of the GT in encouraging the consolidation of a culture of evaluation in the Italian system of cooperation. [ABSTRACT FROM AUTHOR]

The article reviews the last 30 years of EU multiannual programming. The ultimate objective is to capture how the way of interpreting evaluation has changed during this long journey and how the Italian PA has used this 'tool' to increase the effectiveness of policies and strengthen awareness in planning and implementing public investments co-financed by the European Union. In six programming cycles, from 1989-1993 to 2021-2027, evaluation has had ups and downs, passing from the utmost marginality to a certain centrality. The approaches and tools put in place by the European Commission to make it useful for improving the results of the resources invested have been various, and this article attempts to summarise them, also in order to continue with momentum in the drive to strengthen public action for which evaluation is the vehicle. The retrospective analysis of the different interpretations that the evaluation processes have had in the Structural Funds (mainly ERDF and ESF) and in the rural development policies (EAFRD) allows us to learn some relevant lessons on the most effective strategies and on those that have not proved appropriate. It analyses how evaluation acted at the time of the pandemic, what lessons it has left in the scientific community in this particular stage of our recent history, also by means of a survey of insiders and PA staff involved in evaluation processes. Prospects for the near future are pointed out, providing some hints and suggestions for strengthening evaluation as an opportunity to affect the quality of administrative action in the country system and capable of fostering a cultural change in the management of public affairs. [ABSTRACT FROM AUTHOR]

The Italian public administration is confronted with two challenges peculiar to the current times, which share some common features: on the one hand, the use of ever-changing digital technologies; and, on the other hand, the role in social innovation processes involving multiple actors. In the first case, the emergence of public digital platforms calls for re-organization processes based on greater integration among internal departments, and stable relations between different administrative levels and between public and private actors. In the second case, the social demand for innovation in public services and policies leads to the development of ways of involving different actors in collaborative processes that are configured as innovation ecosystems. The thesis of this contribution underlines how the analysis of these modes of collaborative governance constitutes a relevant field for evaluation research, also because of the opportunity to integrate some significant elaborations developed by policy analysis. [ABSTRACT FROM AUTHOR]

In this contribution, the Author intends to take stock of the situation regarding the main Italian public institutions which have assumed a leading role in the evaluation of public policies, with particular regard to the evaluation of the public administration, the evaluation of the education and training system, the evaluation of universities and research, the analysis and evaluation of labor policies and social policies, the evaluation of development cooperation initiatives, the evaluation of cohesion policies, the evaluation of agricultural policies. For each of these areas the public institutions of reference are identified, the role covered is outlined and the main activities carried out within the framework of the functions assigned to them are described. The purpose of the essay is to provide a general overview: the sources reported can be used for appropriate insights. [ABSTRACT FROM AUTHOR]

Teacher appraisal is a major focus of educational reform, for it is believed that effective appraisal practices contribute to improving students' learning outcomes and principals' ability to manage their schools. This study arises from a secondary analysis of data collected in the 2018 edition of the Teaching and Learning International Survey (TALIS) promoted by the Organization for Economic Co-operation and Development (OECD). More specifically, and firstly, it focuses on school administrators' views concerning the agents who provide formal appraisal to teachers in lower secondary schools, the frequency with which such appraisals occur, the modes of teacher evaluation, the users of such information, and the evaluations' operational repercussions. Secondly, it focuses on teachers' perceptions regarding the agents who provide them with feedback, the modes via which assessments are conducted, and the positive impact that such evaluations generate on teaching behaviour. Since at least some of the information collected among principals and teachers focuses on the same topics, it is also possible to explore the attunement among them, that is, whether educators working in the same school report convergent or conflicting accounts of appraisal practices. Over one-third of schools do not implement any formal assessment. Teacher appraisal and feedback practices tend to be centred on both school administrators, assisted by their management staffs, and other teachers in the same school, whereas external agents tend to play a marginal role. Findings point to a substantial discrepancy, between school leaders and teachers, in how appraisal/feedback practices are carried out, especially but not exclusively as regards the role of school management. This gap in the perception of largely factual circumstances is an issue of concern. This study also develops some critical reflections concerning the way in which TALIS explores these issues. [ABSTRACT FROM AUTHOR]

This article discusses the role of Universities in past and present Italian cooperation activities for the development of poor countries, tracing the origins of Italian university cooperation and the evolution of its approaches until present. In particular, the article highlights the current role that universities play in supporting the Ministry of Foreign Affairs in the evaluation of development programs and projects in partner countries, as part of Universities' public engagement. This occurs through the creation of dedicated consulting bodies that gather institutional actors, civil society organizations, and university professors with thematic expertise in the field of development, including evaluation. The article concludes advancing future research questions to explore the effects that the collaboration between universities and the Italian cooperation system is producing, and the way universities can boost knowledge transfer for innovation and sustainable development. [ABSTRACT FROM AUTHOR]

The present evaluation analysis was conducted using a goal-free logic on 205 projects, implemented in different areas of Southern Italy benefiting from the financial support of the "CON IL SUD (FCS) Foundation", with the aim of highlighting some of the positive changes (expected or unexpected, direct or indirect) triggered by mid-long term projects, analysed using the Positive Thinking Approach as a research framework. In situations characterized by high levels of complexity, it is difficult to forecast how a change will manifest itself; therefore, the evaluative research should identify positive changes even when they occur unexpectedly. The author focuses on the different types of relational contexts in which the projects were embedded, with particular attention to how certain types of links between project partners and other stakeholders contributed, along with other factors, to generating local development in the territories. [ABSTRACT FROM AUTHOR]

The article presents and discusses the results of the evaluation of the effects of an intervention to strengthen socio-emotional skills carried out in Colombia in 2019 as part of the international cooperation project "La Paz Unica Esperanza Para el Desarrollo Economico y Social" (PUEDES). The evaluation is carried out with the difference-in-difference method, on a group of 754 adolescents aged between 10 and 18, attending secondary school in the Department of Putumayo. The competences being assessed are self-esteem, regulation of positive and negative emotions and empathy. The results show that the effects were positive in all skills and, observing the differentiated effect by gender, it was found that it was more intense among girls than boys. Furthermore, participatory processes have produced positive effects especially in girls, in particular as regards the management of negative emotions, confirming the relevance of the gender variable in this type of process. [ABSTRACT FROM AUTHOR]

In Italy, the Presidential Decree 80/2013 established the Schools' National Evaluation System. In 2018, INVALSI (project PON Valu.E2) launched an evaluation study to develop a Self-Evaluation Form for Provincial Centres for Adult Education (PCAE). The use of the participatory approach in the study involved the PCAEs, bringing out their needs and expectations, while ensuring full adherence to the process by all actors. This contribution presents the participatory exploration phase of the semantic field of the study aimed at defining the evaluation indicators and the self-evaluation form. The authors present the approach and discuss its implications. Finally, they reflect on the role that institutional subjects can play in promoting a space of connection between the bearers of different points of view, creating synergies between central and local authorities in the field of evaluation. [ABSTRACT FROM AUTHOR]

Evaluative research and research synthesis are connected from the drive to a constant reflection on the organicity of the available knowledge, its reliability, and the chance of considering it transferable beyond the original contest. The pluralism that characterizes evaluative research presents specific challenges related to the variety of approaches, designs, and questions. The reporting mode of processes and results - with the coexistence of heterogeneous literatures, lexicons, and approaches - require a specific reflection. Developments on the research synthesis front make it possible to identify different pathways to reach various objectives. Those developments are often related to collaborative initiatives that provide platforms for coordination, debate, and methodological development. The construction of collaborative networks involving the academic and professional community is the first step to respond to the main research synthesis challenges in the evaluative research field. [ABSTRACT FROM AUTHOR]