Working on outcomes: Themes and learning from the Impact Measurement Masterclass

Between April and June 2020, Quality Matters was invited by IDEA to deliver an Impact Measurement Masterclass, for formal and non-formal education practitioners to explore outcome measurement with Development Education (DE) organisations. The masterclass was delivered as a series of virtual workshops and conversations attended by 19 individuals representing 15 IDEA member organisations and individuals from across the sector.  From the outset, participants all agreed that demonstrating the outcomes of their work was important, but that organisations need to take several factors into account for an outcome measurement process to be successful and meaningful for both participants and staff alike.

Outcome measurement can be defined as measuring the change that has occurred, because of a specific activity or project, for a range of stakeholders. It involves understanding both the positive and negative, and intended or unintended outcomes, created by this work. Analysing outcome data can support organisations with their future strategic planning, ongoing quality improvement and decision-making about a project or activity.

To look at how an outcome-focused approach might be used by DE organisations, we will explore a few issues addressed in the training.

Identifying the changes that matter most. For DE organisations, a strength is that Result-based Frameworks (RFs) and Result Chains are widely used to define the outcomes and outputs of projects and activities. For charities and non-profit organisations, a theory of change (or other Project Management tools such as RFs) is an integral step in identifying and defining outcomes to be generated by their activities or programmes. At the start of an outcome measurement process, such a framework can help an organisation to answer key research questions, such as:

  • What are the problems this activity is trying to address?
  • What are the outcomes that matter most to stakeholders?
  • How does your organisation's activity create these outcomes?
  • How do you know if people have achieved these outcomes?

By answering these questions, an organisation can establish the scope of its outcome measurement process. It's certainly a good starting place for engaging in a broader discussion with a staff team about what an organisation is trying to accomplish and setting a proportionate scope, including the number of stakeholders who will be involved. Establishing the scope also came up in terms of trying to decide on measuring outcomes for individual learners,or taking a broader approach to measuring changes for other stakeholder groups. It was recognised that outcomes related to policy and advocacy work often require different indicators to methods for measuring behavioural or attitudinal outcomes for participants.

Finding the right tools for the job. Training participants were interested in more examples of tools used for measuring the extent of change, or when an outcome has been achieved (or not). There are outcome measurement tools used within the Irish Development Education sector that participants could point to as examples. For instance, World Wise Global Schools developed a self-assessment tool to measure changes in both students’ and teachers’ skills, knowledge, attitudes and actions.

We also looked at examples of good practice, from other sectors in or outside of Ireland, to better understand how organisations can measure social and behavioural outcomes. For the Impact Measurement Masterclass, Quality Matters prepared a report with examples of various tools used in Development Education work. While DE organisations have the necessary skills and practical experience with undertaking research and evaluation, some may not be familiar with validated tools which are available and may be used for measuring outcomes.

Measuring intangibles can be difficult. Using outcome measurement as a way of measuring change relating to 'intangible' or 'soft' outcomes (such as changes in attitudes or beliefs) resonated with many participants. Most organisations use a combination of outputs, quantitative data and feedback from their beneficiaries to show the impact of their work, but some feel a more focused approach to demonstrating educational, social, attitudinal and behavioural outcomes would be useful. The first step is answering simple questions about which specific activity you are measuring the impact of, and which groups of individuals (or stakeholders) experience this change.

Some things are easier to measure than others. The truth is that, for many charities and non-profit organisations, there is no single method, survey or tool to measure outcomes for an intervention. As we know, different organisations require different approaches to effectively measure their work. But, if organisations don't have the time or resources to invest in developing appropriate surveys, questionnaires or measurement tools, they won't be able to work out how to measure their 'intangible' or 'soft' outcomes. Many participants recognised there are benefits to outcome measurement and the growing role for IDEA may be to help organisations with building this capacity and solving these practical challenges in years to come.

Opportunities to learn and improve. It may seem obvious, but one of the interesting discussion points from the Masterclass workshops was how organisations would use their outcome data as an opportunity to learn and improve, and to demonstrate to funders how programmes are effective. Many participants discussed how their organisations were already collecting data used for a variety of purposes, and how this data may already demonstrate both the effectiveness and outcomes generated by their work. In this context, the real challenge of starting an outcome-focussed approach is determining what resources are available, what data is already being collected and what will be measured by an organisation.

As for what might encourage more DE organisations to measure outcomes, or help to decide if outcome measurement is worthwhile, an outcome-focussed approach will support any organisations to make evidence-informed decisions about their interventions and to test if this work is making a difference for their stakeholder groups. Don't try shooting for the stars; think about outcome measurement being proportionate to your work (and staff resources) and finding simple means of undertaking outcome measurement with your stakeholders, so they find this experience rewarding, meaningful and positive.

Philip Isard is a Senior Project Specialist with Quality Matters

May 2, 2025
Dates: Thursday 10 July,10.00am-6.00pm and Friday 11 July, 10.00am-3.00pm Location: An Tairseach, Wicklow IDEA is thrilled to open applications for our second iteration of the IDEA's Lab. In this first of its kind residential Lab, we will create space where members can bring their ideas for projects and try out new planning and design methods. Participants will have the opportunity to work on an idea from scratch to (almost) finish with the support of expert trainers and learn through group exchange in a welcoming and inspiring setting. Participants will come out of the training with: An early project design proofed, tested and built through group feedback, iteration, trialling and support. New approaches in designing innovative projects they can apply to their workplace Headspace and time to work on the design of an innovative project A chance to actively learn through group exchange and strengthen the sense of an innovative community of practice in Ireland How? Please email anya@ideaonline.ie to request an application form. Requesting an application form does not commit you to applying. The deadline for returning application form (including organisational sign off letter) is Friday 23 May, midnight . You will find further details in the application form. Participants must register as a team of 2min.- 4 max. Participants without a team member will be asked to pair up with other individual participants or find a partner outside their organisation. Cost : This training is funded by Irish Aid at the Department of Foreign Affairs and offered at a reduced rate. This training is only available to IDEA members. Non-IDEA members can participant as part of an IDEA member team. Fees for members of IDEA is €50  Irish Aid is the Government’s overseas development programme which supports partners working in some of the world’s poorest countries. Irish Aid also supports global citizenship education in Ireland to encourage learning and public engagement with global issues the ideas, opinions and comments expressed in this training are entirely the responsibility of its author(s) and do not necessarily represent or reflect DFA policy. Timeline: Applications open: 07 May Applications close: 23 May, midnight Preparatory session with trainer (30min, online): Mid June Lab arrival: Where possible we advise arrival on Wednesday 09 July in the evening. Accommodation and food will be provided from 09 July, 5.00pm - 11 July, 3.00pm Lab: 10 July,10.00am-6.00pm. 11 July, 10.00am-3.00pm Please don’t hesitate to get in touch at anya@ideaonline.ie if any of the above pose a barrier to your participation, or if you have any other questions.
April 29, 2025
Wednesday 18 June, 10.00am - 4.00pm Wynn’s Hotel, Abbey Street Lower, Dublin 1
April 7, 2025
Last week, IDEA hosted the second webinar from our 2025 series, ‘Exploring Contemporary Crises and Issues through Global Citizenship Education, ' focusing on Palestine. If you missed out, you can watch the full video below!