March 2019 - UNITAR has reached over 50,000 people every year for the past three years. More and more people continue to benefit from the Institute’s programming through face-to-face trainings, e-learning courses, master’s degrees, conferences and more. 2018 was another record year, with nearly 84,901 beneficiaries reached, of which 72 per cent were learning-related beneficiaries.

While these successes are of course welcome, they present a challenge in assessing how well UNITAR is performing, especially regarding learning events. UNITAR always conducts evaluations of any learning event of two days or more, specifically measuring participants’ reactions and whether learning has taken place. It is often difficult to go any further than this and truly assess whether the learning has taken hold and had an impact. There is not necessarily always a direct line from training to results, and impact cannot be assumed.

UNITAR is interested in evaluating the impact of its programming, and one relatively quick way of doing this is through Impact Stories. These stories use the most significant change technique which is based on a qualitative, participatory approach and looks for stories of significant changes. By analysing the impact that one event has had in the medium-term for a few beneficiaries, these impact stories highlight the experiences of participants months after they have completed the course or training. The personal stories are supported by a larger sample of survey data.

Impact Stories are a way of going beyond the numbers and beyond the self-assessments taken at the end of courses, to hear the stories and real experiences of those who have taken part. They are not success stories as they try to get a full picture — the positives and the negatives of the learning events. Since the inception of the impact stories programme in 2018, the Planning, Performance Monitoring and Evaluation (PPME) unit of the Institute has published six stories, with more in production. Here are a few highlights of those already released.

Last December, the Institute published an Impact Story on a UNOSAT face-to-face training on Geospatial Information Technology for Evidence-Based Decision Making that took place in Bhutan. Unlike other impact stories, research for this story began only one month after the training was completed, so we did not necessarily expect impact to be visible. Nevertheless, in our interviews, we could already ascertain a certain degree of impact.

Sonam Tobgay, a Senior Survey Engineer with the National Land Commission of Bhutan, was one of the participants interviewed. Since the training, he has used techniques from the training as part of a spatial analysis his department conducts to measure the gradient of land.  He has also implemented the use of an app, which was first presented to him at the training, to coordinate with and monitor those working in the field with him in almost real time. The project has now increased in scale, and there have been discussions on implementing the use of these technologies at the national level. The overall greatest impact was in increased data reliability, in turn allowing for more robust analysis.

Impact was less visible with Pema Zangmo, another participant interviewed. This shows that impact is not the same for all participants, and that it is certainly not something that can be taken for granted as applying to all participants.  

While the Impact Story on the training in Bhutan was written only a matter of months after the end of the event, the Institute’s Impact Story on the online training organised by UNITAR’s Public Finance and Trade Programme in conjunction with FAO on Trade, Food Security and Nutrition for Southern and Eastern Africa was written over a year after the end of the event. The clearest impact that could be identified was the change in attitudes and confidence of the participants, new roles and responsibilities at work, and the development of a community of knowledge that has continued since the training.

The participants have been incorporating evidence-based decision making into trade and policy negotiations after the course in a way they were not doing before, and they credit the course for this. Kelvin Nkai from Kenya explains in the story how he approaches any important meeting or negotiation differently now. In both his professional and personal life, he takes more time and he ensures he has a solid understanding of all relevant information. An increase in confidence and competence was also found with Jean-Baptiste Ukwizagira, another participant interviewed. This change has resulted in greater responsibilities at work, and assignments that previously would have been unattainable.

The most recent Impact Story from the Institute, on National Adaptation Plans (NAP) for Climate Change from the Green Development and Climate Change Programme, is soon to be published. This story focuses on one training on this organised by the Institute: the National Adaptation Plans Building Climate Resilience in Agriculture Massive Open Online Course (NAP MOOC).

For this story, PPME interviewed Steven Banda, who works in agriculture and production in Malawi, to see how the NAP MOOC has had a practical impact at the grassroots level. The clearest impact is that Steven has been inspired to take up traditional alternative practices and techniques to replace artificial chemical fertilisers and pesticides. The course encouraged Steven to use techniques already known to him, namely the use of Msangu trees to return nutrients to the soil and the use of Neem leaves as a natural fertiliser. The story shows that people from all over the world and from many walks of life have benefited from the course in their own individual, locally relevant ways. The positive impact of this, for the individuals, for their organisations and ultimately for the climate and us all as a result, has already began.

There are several impact stories currently in the works, including one on the Pintando el Futuro project in Colombia organised by UNITAR’s Peacekeeping Training Programme, another on the Road Safety programme organised through the International Training Centres for Authorities and Leaders (CIFAL) Global Network, and one on a project to strengthen capacities for developing a national pollutant release and transfer register in Mongolia. Other stories are also planned. Each of these stories present their own unique challenges, from summer holidays in Colombia, where the Pintando el Futuro project and the tool “Viaje de Héroes y Heroínas” was developed for former child soldiers and other children and youth impacted by the Civil War in Colombia, to finding a Mongolian translator for the story on the pollutant register.

Each challenge results in new learning and approaches to assessing impact. Finding and highlighting impact is vital to ensure that UNITAR continues to be on the cutting edge of training and research.  Impact shows how programmes and projects are translated into results and shows that all the effort is worthwhile. Finally, and perhaps most importantly, Impact Stories allow the voices of those who participate in UNITAR’s courses to be heard. At the end of the day, it is their voices and their stories that matter most.