Search

This paper addresses non-academic training in evaluation with a twofold perspective. The first one will focus on the reconstruction of a map of the existing training offer in Italy. This map, updated at September 2022, will propose an overview of the existing courses in terms of both main features (e.g.: duration, structure, providers) and macro-contents. The second perspective will match the results of this analysis with the profile, overall and by competences, of the Expert in Evaluation of Policies, Programmes, Projects and Institutions as defined by the Italian Evaluation Society (A.I.V.). This match intends to explore the correspondence between the courses and the A.I.V. proposal. In such a way, an attempt will be made, on the one hand, to contextualize the courses with respect to A.I.V. evaluator's profile and competences and, on the other hand, to do the same with A.I.V. profile in relation to evaluation courses. Finally, some recommendations will be proposed on the basis of the evidence produced by the analysis. [ABSTRACT FROM AUTHOR]

The main objective of the paper is to address and explore a much-debated issue within the evaluation community: the idea of "democratic evaluation". What exactly is meant by this term? When can an evaluation be said to be democratic? Through the adoption of what behaviours should this ideal translate into concrete working practice? The essay aims to contribute to broadening the community's debate on these questions, which are central at a historical-political moment such as the post-emergence from Covid-19 and in the midst of an international crisis, that between Russia and Ukraine, which calls democratic evaluation into question in our country. On a general level, in fact, the situation of uncertainty due to the unpredictable effects of the pandemic, the social and economic crisis in the country and the consequent growing distrust of the citizens towards the administration seem to have increased the demand for accountability, leading to talk, as in the rest of Europe as well, of a "democratic deficit". On the other hand, recent administrative re-centralization, even assisted by current evaluative practices, only aggravates the inefficiency of public administration. Accountability, freedom and democracy are never given, they need to be defended, always regained: evaluation assumes a strategic role to this end; it can help bring citizens closer to public administration. The crisis between Russia and Ukraine is changing geopolitical balances; the effects of the war economy, inflation, the energy crisis, and the possible escalation of the conflict add to the uncertain situation due to the pandemic. New inequalities and an increase in poverty threaten to generate deep rifts in the country's socioeconomic fabric. Systemic shocks add to chronic problems: a demographic collapse, stagnation, job insecurity, and a lag in digitization. At the same time, the political system fails to represent and stabilize consensus over time. Governments also suffer from this instability, with the inability to design and implement structural reforms for the country, including that of the bureaucracy. Social complexity, chronic problems and systemic shocks, require the ability to apply effective public policies. State intervention and the ability of the bureaucracy to be efficient and responsive to social change are central issues for the health of democracy. Public policies move within a framework of very high complexity, in which the role of evaluation emerges, which should tend to the development of democratic accountability, illuminating decision-making choices through its methodological rigor, analytical capacity and scientific knowledge. The essay thus aims to address the potential of a democratic evaluation that can broaden the number of stakeholders, the field of values that are taken into account, the repertoire of research methods, the field of interest of policies, and the very definition of the objects of evaluations. [ABSTRACT FROM AUTHOR]

The paper wants to explore, using the case of the PRIN, what criteria are most influential in the evaluation of proposals submitted for funding, in particular what is the relevance of the criteria related to the scientific quality of the proposal in itself (originality, methodological rigor etc.) than the quality of the proposing team, or alternative criteria of evaluation such as the impact expected of the research or the geographical location of the proposers. It is assumed that there is a difference between the quality criteria that scientists belonging to a scientific community say they use when they work as evaluators in the process of peer review, and the perception that scientists themselves have of the importance of scientific quality when they are objects of a process of evaluation. This difference is based on the existence of mechanisms distorting the peer review that: a) derive from the internal organizational characteristics of the scientific field; b) tend to become deeper when rules circumscribe the open debate of the reviewers and selection committees, and introduce the use of technical constraints (eg. methods and rating scales and decisions based on average scores) favouring the emergence of factors other than the scientific quality. [ABSTRACT FROM AUTHOR]

The first relevant attempt to describe mixed methods (MM) in the field of evaluative research is attributable to Greene and Caracelli (1997). The authors distinguish various forms of MM starting from the purposes for which they are used. Since 1997 important developments occurred in academic and professional debates related to MM in evaluative research. Starting from the reconstruction of the debate on the usefulness and use of MM in evaluative research, this paper reflects on the ontological, epistemological, methodological, and axiological issues related to the evaluative research processes using MM. This consideration needs to involve an extended vision of the issues underlying the choices of specific methods in the research process. This is even more necessary in evaluative research that constitutionally engages with complex dilemmas, especially on the axiological level. [ABSTRACT FROM AUTHOR]

The paper, in light of the growing interest that the Third Mission of the University knows at international and national level, delves into the ways in which it has been detected and evaluated by ANVUR in recent years. It shows that this assessment is based on two logics, one of a linearincremental type (more is better) and one of adequacy of the activities carried out face to the needs of the territory and the internal resources of the University. The second appears to be the most useful for the learning function that the evaluation should carry out in order to continuously improve the universities in their positioning into their settlement area. Finally, it suggests some lines of work to use TM assessment for learning purposes. [ABSTRACT FROM AUTHOR]

The guiding hypothesis of the paper assumes that big data (BD) started affecting evaluation practices (formal and informal) in / of planning. The delay in developing 'public internet' for the benefit of the platforms prevents knowledge from being recognized as a common good. But, we must not underestimate the fact that the individual data in a social practice such as planning, have little value if not aggregated and aimed at offering innovative, and not only marketable, services. BDs open up new possibilities, but they also raise new issues of a cognitive, regulatory and decision-making type, as well as ethical and deontological. Four case studies provide evidence in this regard, particularly in the domains of mapping, mobility management and participation in public policies. These evidences help to identify relevant criteria for the evaluation of BD contributions in planning processes. [ABSTRACT FROM AUTHOR]

The paper describes a research work about evaluation of the acted and desired quality of professional activity within the school for foreigners of the multiethnic cultural center "La Tenda" in Milan. Competences useful for the professional development are the fulcrum of this evaluative research that has tried, not only to describe what implements the studied quality, but also to highlight the premises and consequences. In addition to the skills necessary for the professional mission, we therefore refer to the pre-requisites for carrying out the task: personal (eg: education), material (eg structural endowment) and intangible (eg: the ethical principles of the school). We refer to the facilitating elements and the consequent result that can be achieved by the work that the school teachers carry out (learning Italian and integrating with the new culture). [ABSTRACT FROM AUTHOR]

In line with a constructivist approach and a pratical participatory evaluation, the paper used brainstorming technique to define effectiveness' criteria of children adoption. 26 judges of the Juvenile Court of Bari took part in the evaluational brainstorming, which have been organized in a creative phase, a classification step and a final relevance analysis and group discussion. Applying the Lawshe formula, it has been possible identify the most shared success standards (children well-being: self-esteem, adjustment and secure attachment; adoptees' and parents' skill to revise their life story; adoptees' scholastic and social integration; a good parentsadoptees relationship) and failure ones (parents reject adoptees or adoptees abandon adoptive family). The study highlights suggestions regarding the flexible use of brainstorming technique and underlines the potentials to use a shared evaluation procedure in order to improve both adoption research and practices. [ABSTRACT FROM AUTHOR]

In prevention and health promotion there is an increasing demand for evaluation: especially in times of economic crisis, programs, interventions, policies are asked to be based on substantial evidence of efficacy and effectiveness. Ministry and regions, to perform their governance role, need mechanisms to monitor and control, planning, directing and evaluating. This paper intends to propose some methodological reflections both for those evaluators who are going to apply to this specific area and to operators of prevention and health promotion that want to deal with problems of evaluation. This is not to achieve a final comprehensive presentation of method and techniques (perhaps impossible in a single paper) but to raise some questions and reflections. [ABSTRACT FROM AUTHOR]

This paper examines the role of economic appraisal of education and training systems under the light of the current economic and financial crisis and of the relevant "exit policy" strategies. On the basis of the literature on the Post World War II "economic miracles", the paper confirms that a careful evaluation of education and training systems can be a powerful tool to transit to the "post crisis" situation and to set a growth path. However, evaluation of human capital preparation through education and training cannot be disjoined from evaluation of human capital utilization- viz. of labor and retirement policies. The current practice is to follow different methodologies, techniques and procedures for evaluating human capital preparation, on the one hand, and human capital utilization, on the other. The paper outlines two possible strategies to define a unified evaluation methodology with pertinent techniques and procedures. [ABSTRACT FROM AUTHOR]

The paper presents a discussion over various evaluation approaches and criteria in the field of Development Co-operation by the Italian Ministry of Foreign Affairs and by International Organizations such as FAO, IFAD and the European Commission (Europeaid). At the beginning the analysis considers the evolution of planning models based on resources supply (human, financial and material), and successively the paper focuses on models much more concerning the sustainable achievement of objectives and results, by means of participative approaches. Specific problems on the evaluation of development cooperation initiatives are presented; conclusions assess the necessities of a systematic utilization of the methods for the economic evaluation of investments which, at present, seem to be underemployed, especially by the Italian Development Co-operation. [ABSTRACT FROM AUTHOR]

The aim of this paper is to propose a reflection on specific learnings, arising from a cyclic evaluation of the same intervention. The proposed topic is the study of Placement of Vocational Training in Piedmont Region (IT), held from 2011 to 2014. Through a diachronic reading of the four annual reports of the Placement, are presented three dimensions of analysis: financial dynamics, employment outcomes and net impacts. This contribute aims to discuss learnings in terms of method and policy, starting by the main results emerging from the analysis of the three dimensions listed above. [ABSTRACT FROM AUTHOR]

In the last decades, a lot of resources for active labour market policies have been devoted to training programs. Even if the opinion is diffused that training provides both an increase in skills and (thus) employability, following the international literature the impact of training on working conditions is very low. Evidence from Italy is quite scarce. In this paper an evaluation from Piedmont (in the north-west of Italy) is presented. Using administrative data from the job training and the labour market information systems, the employment rates of two groups of unemployed enrolled at Public Employment Service offices are compared. The impact estimates, obtained via propensity score matching, are (except for some quite peculiar courses) close to zero. Results are slightly better in the case of long term courses. While presenting the research, main advantages and disadvantages of different evaluation designs are discussed. [ABSTRACT FROM AUTHOR]

The paper aims to show that the irresolution of the National Agency for the Evaluation of the University System and Research (ANVUR) referred to its institutional position derives from a ungoverned hybridization of the reform of the University with the type of evaluation that has been considered as part of the reform administrations. This irresolution destroys the technical role of the Agency, which prevents it improve the role that the law has assigned, of support of the activities of evaluation and self evaluation of universities; it hinders the diffusion of reflective practices, and, with this, the development of an effective institutional evaluation. After explaining the regulatory steps that have made this possible, the work focuses on the three evaluation exercises hitherto made, to show how the limitations and distortions with which ANVUR is born created obstacles to the diffusion of the culture of evaluation in the universities. [ABSTRACT FROM AUTHOR]

The evaluation of the quality of health services has become a topic of crucial importance in public policy and in the socio-health organizations called to play a proactive and participatory role in the process of innovation and improvement of their caring work. In this paper we present a research project evaluation, through the discussion of a case study carried out in a residence for elderly people, the objectives of which concerned two different levels of operation: the identification of indicators for the continuous development of the quality and the creation of effective tools for self-evaluation. [ABSTRACT FROM AUTHOR]

The paper presents an evaluation work carried out by Confcooperative Bolzano about the policy for Equal Opportunities implemented until April 2013. The approach used for evaluative research is constructivist and deals with the subject of study through the use of evaluation techniques combined into a theoretical and temporal succession scientifically rigorous and deemed to be consistent with the targets set. The results have led to a clarification of the achieved goals and the implementation of concrete development actions. [ABSTRACT FROM AUTHOR]

In Italy, the recently launched spending review process seems the only alternative to budgetary cutbacks centrally applied across the board. Fighting waste and corruption while making administrative agencies more effcient and accountable to citizens is a key priority for local and regional governments as well. Yet, current spending reviews are conducted on aggregated broad-based appropriations of public spending and are, then, unable to generate that "usable" knowledge, which regional and local policy makers and managers can rely on for policy design and implementation. Indeed, a centralistic and technocratic modus operandi, heavily relying on civil service downsizing, risks undermining spending review effectiveness as public oversight tool, and reinforcing coercive isomorphism at the expense of political legitimacy and cultural acceptance of administrative performance assessments. This paper addresses such questions as on how a democratic "would-be federal" state operates and to what extent consolidation of public finances can be reckoned as an overriding goal against efficiency and accountability considerations at the local level Three issues emerge are key to reform: (i) what human resources public administration should invest on to adequately govern localities as well as regions; (ii) what measures are effective to fight waste and corruption; and (iii) what reliability and plausibility current performance measurement have with respect to actual public services offered to citizens. [ABSTRACT FROM AUTHOR]

This paper presents a view of research evaluation from the perspective of a large, multidisciplinary public university in the United States. Faced with increasing demands for accountability to the public, such universities currently lack adequate approaches to research evaluation. In fact, many of the approaches to research evaluation tend to rely on disciplinary standards for what counts as excellent research. Although these standards protect the autonomy of disciplines, departments, and faculty members, they do not speak to a notion of accountability that goes beyond accountability to academic peers. Public universities in the US should follow the example of the US National Science Foundation and find a way to incorporate concerns for broader societal impacts into their approaches to research evaluation. [ABSTRACT FROM AUTHOR]

Physical inactivity has been linked with a number of chronic diseases affecting the population. Evidences arerising the concern and a number of nations have implemented policies promoting health increasing physical activity, using an interdisciplinary approach. New methodologies for assessing their efficacy have to consider the growing complexity and the multisectorial context of the setting. The purpose of this paper is to describe an evaluation project of physical activity promotion programs, which have been implemented in three different Italian regions: Veneto, Emilia-Romagna and Autonomous Province of Trento. The evaluation process has been specially designed for each region, taking into consideration its aims and the local context of each promotion plan. The work involved the main stakeholders and was planned bearing in mind the importance of long-term sustaina-bility. It is here proposed as a contribution for developing efficient and sustainable methods and operational tools for the evaluation of health promotion programs. [ABSTRACT FROM AUTHOR]

The paper describes the main characteristics of the Information System (SIS) of the cancer (breast, colorectal, cervical) screening programs (SP) in Italy. The SIS is organized in order to evaluate the quality and the effectiveness of the SP and is referred to different stakeholders and actors (national & local government, professionals, society) that can actually express different knowledge priorities. Different variables have been identified and annually collected by national surveys. The sources of data are the managing systems of the screening programs. The SIS produces several indicators, according to those identified by the European Guide Lines on quality assurance, which respond to the following evaluation and monitoring needs: * the SPis a path (disease management), not just performing a test; therefore we need to assess the quality of the whole path, and not only of the screening test performance; * the SPis directed to an asymptomatic population, hence it must take in account both the sensitivity and specificity * to evaluate the quality of the organization. The results of the evaluation are reported and/or summarized by different reporting system/tools in order to be used at different levels: service delivery, regional and national planning, governance assessment, generation of scientific evidence, accountability's purposes. [ABSTRACT FROM AUTHOR]

This contribute describes the evaluation of the project "Genitori Più -- Campagna di promozione della salute nei primi anni di vita" (Parents Plus -- a health promotion campaign), developed during 2008-2009 in several Italian Regions. Genitori Più main objectives were to increase the knowledge and to change some behaviors in the health professionals working on birth path and in the parents, focusing on 7 determinants that have been proved important for the health of the babies. The evaluation followed the whole project with the constant implication of the project team using structured questionnaires. The evaluation surveys involved 5.961 health professionals and 5.819 parents, in particular mothers. In this paper we show how evaluation in health promotion can give to decision-makers several elements for re-thinking and adjusting the programs, using different evaluation strategy like longitudinal match over time or across different territories or across different strata of population. The evaluation showed how knowledge and behaviors are different in each territories involved in the project and between social classes, and showed much higher levels in the areas where the preventive intervention were more rooted. [ABSTRACT FROM AUTHOR]

Correct assessment of critical variables influencing public investment final outcome is often a crucial procedure, especially in cases where the dynamics of such variables cannot be controlled by public agents, because they mainly depend on choices and behaviors of other operators. This is a typical situation occurring in development project assessment, characterized by a high uncertainty degree and by a scarce availability of information in relation to the behaviour of aleatory variables influencing investment final outcome. In such cases, risk analysis by directly using Montecarlo method could be misleading: on the contrary, we think that such an evaluation should be preceded by uncertainty analysis, allowing to generate new useful information for subsequent risk assessment. In this context, the paper presents one possible evaluative pattern, especially suited for development projects assessment, allowing first to understand the leeways of uncertainty between which investment decision must be taken and, then, to estimate investment risk and to generate useful information to help Decision Maker defining the most suitable economic tools to mitigate it. The proposed evaluative pattern is composed by four steps, based on different techniques and sequentially articulated, so that each step generates new useful information for developing the following one. Such a scheme allows both to deepen the information produced by the evaluative process, and to guarantee more decisional transparency. [ABSTRACT FROM AUTHOR]

The following paper provides an overview of the multi-criteria approach within a programme of Career centres development (i.e. University placement offices in Italy), focused on three new elements of interest: i) the analysis process has been designed and managed together with the Evalution staff and the Project team, in a combined/joined way that has allowed the Project team to learn more about the evaluation process other than the only method itself; ii) because of the Project team involvement the multi-criteria approach has been used in a bottom-up way, that has solved problems related to the understanding of the project and its different parts; iii) analysis has been conducted in two different dimensions, that have been developed in two distinctive evaluation matrixes. The use of two analysis dimensions has given the Evaluation team the opportunity to observe the different and specific elements of the Career centres structures and the activities of technical support delivered to Career Centres operators from two point of views. It also helped to design a bi-dimensional map of the services the Career centres deliver on both dimensions we considered. [ABSTRACT FROM AUTHOR]