Working on outcomes: Themes and learning from the Impact Measurement Masterclass
Between April and June 2020, Quality Matters was invited by IDEA to deliver an Impact Measurement Masterclass, for formal and non-formal education practitioners to explore outcome measurement with Development Education (DE) organisations. The masterclass was delivered as a series of virtual workshops and conversations attended by 19 individuals representing 15 IDEA member organisations and individuals from across the sector. From the outset, participants all agreed that demonstrating the outcomes of their work was important, but that organisations need to take several factors into account for an outcome measurement process to be successful and meaningful for both participants and staff alike.
Outcome measurement can be defined as measuring the change that has occurred, because of a specific activity or project, for a range of stakeholders. It involves understanding both the positive and negative, and intended or unintended outcomes, created by this work. Analysing outcome data can support organisations with their future strategic planning, ongoing quality improvement and decision-making about a project or activity.
To look at how an outcome-focused approach might be used by DE organisations, we will explore a few issues addressed in the training.
Identifying the changes that matter most. For DE organisations, a strength is that Result-based Frameworks (RFs) and Result Chains are widely used to define the outcomes and outputs of projects and activities. For charities and non-profit organisations, a theory of change (or other Project Management tools such as RFs) is an integral step in identifying and defining outcomes to be generated by their activities or programmes. At the start of an outcome measurement process, such a framework can help an organisation to answer key research questions, such as:
- What are the problems this activity is trying to address?
- What are the outcomes that matter most to stakeholders?
- How does your organisation's activity create these outcomes?
- How do you know if people have achieved these outcomes?
By answering these questions, an organisation can establish the scope of its outcome measurement process. It's certainly a good starting place for engaging in a broader discussion with a staff team about what an organisation is trying to accomplish and setting a proportionate scope, including the number of stakeholders who will be involved. Establishing the scope also came up in terms of trying to decide on measuring outcomes for individual learners,or taking a broader approach to measuring changes for other stakeholder groups. It was recognised that outcomes related to policy and advocacy work often require different indicators to methods for measuring behavioural or attitudinal outcomes for participants.
Finding the right tools for the job. Training participants were interested in more examples of tools used for measuring the extent of change, or when an outcome has been achieved (or not). There are outcome measurement tools used within the Irish Development Education sector that participants could point to as examples. For instance, World Wise Global Schools developed a self-assessment tool to measure changes in both students’ and teachers’ skills, knowledge, attitudes and actions.
We also looked at examples of good practice, from other sectors in or outside of Ireland, to better understand how organisations can measure social and behavioural outcomes. For the Impact Measurement Masterclass, Quality Matters prepared a report with examples of various tools used in Development Education work. While DE organisations have the necessary skills and practical experience with undertaking research and evaluation, some may not be familiar with validated tools which are available and may be used for measuring outcomes.
Measuring intangibles can be difficult. Using outcome measurement as a way of measuring change relating to 'intangible' or 'soft' outcomes (such as changes in attitudes or beliefs) resonated with many participants. Most organisations use a combination of outputs, quantitative data and feedback from their beneficiaries to show the impact of their work, but some feel a more focused approach to demonstrating educational, social, attitudinal and behavioural outcomes would be useful. The first step is answering simple questions about which specific activity you are measuring the impact of, and which groups of individuals (or stakeholders) experience this change.
Some things are easier to measure than others. The truth is that, for many charities and non-profit organisations, there is no single method, survey or tool to measure outcomes for an intervention. As we know, different organisations require different approaches to effectively measure their work. But, if organisations don't have the time or resources to invest in developing appropriate surveys, questionnaires or measurement tools, they won't be able to work out how to measure their 'intangible' or 'soft' outcomes. Many participants recognised there are benefits to outcome measurement and the growing role for IDEA may be to help organisations with building this capacity and solving these practical challenges in years to come.
Opportunities to learn and improve. It may seem obvious, but one of the interesting discussion points from the Masterclass workshops was how organisations would use their outcome data as an opportunity to learn and improve, and to demonstrate to funders how programmes are effective. Many participants discussed how their organisations were already collecting data used for a variety of purposes, and how this data may already demonstrate both the effectiveness and outcomes generated by their work. In this context, the real challenge of starting an outcome-focussed approach is determining what resources are available, what data is already being collected and what will be measured by an organisation.
As for what might encourage more DE organisations to measure outcomes, or help to decide if outcome measurement is worthwhile, an outcome-focussed approach will support any organisations to make evidence-informed decisions about their interventions and to test if this work is making a difference for their stakeholder groups. Don't try shooting for the stars; think about outcome measurement being proportionate to your work (and staff resources) and finding simple means of undertaking outcome measurement with your stakeholders, so they find this experience rewarding, meaningful and positive.
Philip Isard is a Senior Project Specialist with Quality Matters
