Search

The focus of this study is on the development of a novel approach to evaluate the impact of the R4D Alliance at project and program level, including practical guidance to support more effective project design and impact for future R4D investment. A significant limitation amongst many current impact evaluation methodologies is that a narrow set of approaches is often used to evaluate complex development issues. This means that impacts may be only partially described and this is compounded by a lack of clarity around the validation and attribution of impacts to particular interventions - in some cases uncertainties may be larger than claimed benefits! Our mixed methods approach to impact evaluation overcomes many of these challenges, allowing us to evaluate and quantify the impact of the Alliance at program level and at the individual project level in a timely manner, and to the best of our knowledge is the only product of this type currently in use. Our approach demonstrates that with a small additional investment impacts can be successfully tested for at project completion and greater confidence levels for impact pathways can be achieved. These are essential if confidence in the value of R4D investment is to be maintained, and will contribute to the ongoing quality and effectiveness of Australian aid. We anticipate that this approach will raise the bar for future impact evaluations.

A joint Indonesian-Australian researcher meeting for the Infrastructure Cluster was held in Surabaya, Indonesia in January 2015. During this meeting a horizon scanning activity was undertaken with 20 members of the Cluster and two AIC representatives. The horizon scanning exercise aggregated participant concerns into common issues, drivers and megashocks.

The Energy Cluster joint Indonesian-Australian researcher workshop was held in Bandung, Indonesia in late November 2014. A horizon scanning activity was undertaken on 24th November with 35 members of the Cluster, including representatives from the AIC and CSIRO. Five of the representatives were from Australia, with two from The Australian National University, two from Monash University and one from the University of Melbourne. The remaining participants were from Indonesian institutions including: UNHAS, IPB, ITSN, ITB, UI and UGM. The horizon scan workshop exercise aggregated participants’ views into common issues, drivers and megashocks

In 2009-2014 the Department of Foreign Affairs and Trade (DFAT)-CSIRO Research for Development (R4D) Alliance (‘the Alliance’) brought together the research skills of CSIRO and its international research partners with the development knowledge and networks of the Australian aid program to enhance poverty reduction in the Southeast Asian region. The partnership provided an opportunity to develop innovative approaches to international aid delivery by better understanding the relationship between poverty and the environment. Six major collaborative projects were implemented, covering Vietnam (two projects), Bangladesh (one), Indonesia (two) and the Greater Mekong Region (one). The research investigated complex development challenges in the domains of climate, water resources, sustainable cities, and food security. This Australia Indonesia Centre (AIC) Small Grants project enabled us to refine an evaluation methodology developed as part of the Alliance. In this case, the two Indonesian projects and partners were the focus: Makassar Sustainable Urban Development (with Hasanuddin University), and Climate Adaptation Strategies for Rural Livelihoods in Nusa Tenggara Barat Province (with the University of Mataram). The Small Grants project involved repeating evaluation workshops and stakeholder interviews in April-June 2015 that had first been undertaken at the end of the Alliance projects in April-June 2014, thus enabling the tracking of progress along each project’s Impact Pathway 1 year after project completion. A significant limitation amongst many current impact evaluation methodologies is that a narrow set of approaches is often used to evaluate complex development issues. This means that impacts may be only partially described, and this is compounded by a lack of clarity around the validation and attribution of impacts to particular interventions. The Alliance approach applied mixed methods to overcome many of these challenges, allowing us to evaluate and quantify the impact of the individual projects, and to provide an opportunity for reflection by the project teams and hence further build capacity through learning. Our approach was based on three facets: 1. A Theory of Change and Impact Pathway exercise and diagram which created a ‘roadmap’ for each project’s assumed progress and related outputs, outcomes, impacts and goals.2. A self-reflection workshop amongst the Indonesian research partners which mapped key achievements against the project’s Impact Pathway, and discussed reasons for lack of progress and necessary remedial actions.3. An impact evaluation survey of the research team and boundary partners, which asked interviewees to score 18 indicators linked to phases of the Theory of Change and Impact Pathway, providing additional perspectives and triangulation of results from the self-reflection workshops.The results of the exercises demonstrated that with a small additional investment, project impacts can be effectively estimated and attributed. In addition, the process of self-reflection was shown to rekindle project teams’ efforts to maintain momentum, and to tackle barriers to impact that they had identified. We suggest that the Alliance methodology demonstrated by this Small Grants project could be usefully applied to other projects in the AIC portfolio.

For the Health Cluster a planning meeting with Australian Health Cluster Leads was held by teleconference on the 24th November 2014. In early December 2014 a series of stakeholder engagement meetings were undertaken in Jakarta, Indonesia with both Australian and Indonesian Cluster Leads. Following this consultation a joint Indonesian-Australian Cluster planning meeting was held on 11th December to undertake a horizon scanning exercise for the Cluster. The meeting was attended by members from the University of Sydney, University of Melbourne, UI, and UNAIR, CSIRO and the AIC. The results of these consultations were aggregated into shared issues, drivers and megashocks

The Australia-Indonesia Centre is collaboratively researching solutions to shared national challenges in four Clusters: * Energy* Infrastructure* Agriculture and Food* HealthIn 2015 a Cluster Investment Plan (CLIP) was developed for each Cluster. This was undertaken through planning workshops held in Australia and Indonesia. In November 2014, CSIRO was contracted by the AIC to assist with the Cluster planning process by applying its ‘foresighting’ method, which analysed key trends in each theme towards 2030 and beyond. The over-arching goal was ‘to support the development of CLIPs which focus on priority research areas of shared interest between Australia and Indonesia, and to promote integration, synergy and learning amongst the Cluster teams’. The objectives of the Foresighting Project were: 1. For each Cluster, analyse megatrends and megashocks with shared relevance for Australia and Indonesia to support the development of each CLIP;2. Develop exploratory scenarios with AIC partners of Australian and Indonesian regional development to identify and integrate research priorities within and between Clusters;3. Test and apply the DFAT-CSIRO Research for Development Alliance method to assess the learning and integration achieved amongst AIC researchers as a result of the foresighting.