Evaluation: Practice to Theory

Feb 4th and 5th, Dublin and Belfast

I really enjoyed the 2 days of training we did with Louise Robinson and Susan Gallwey on Evaluation. It was a half-day training on evaluation, combining Louise's practical demonstration of the toolkit "How do we know it's Working?" with Susan's thoughful reflections on the challenges of evaluating Dev Ed.

We got some great discussions going and participants had plenty of feedback at the end of the day, so I wanted to use this blog post to highlight some of those key questions and invite readers to comment and suggest possible follow-up activities that IDEA could coordinate. The entire feedback from both days is available to download in pdf format at the end of the report in our News section. However, if you don't want to read through all that- here is an extract! I've divided it into key challenges identified and the questions that arose from discussion of them.

CHALLENGE 1: EVALUATORS HAVE THEIR OWN ASSUMPTIONS

How do you build objectivity into the rigour of the evaluation?

The participant’s assumptions, the evaluators assumptions should be recognised.

Who decides what is good DE?

Deciding WHAT we want to measure in a baseline assessment forces us as practitioners to really think about our goals for DE and our assumptions about "successful" DE.

CHALLENGE 2: HOW TO ALLOW MISTAKES AND ENCOURAGE RISK-TAKING

Capturing individual learning and feeding a COLLECTIVE sense of purpose.

Message that learning is not a ‘judgement’

How to say its NOT working, take the risk, mistakes are good

Time is needed for evaluation, it is diffiult for us to go back (to learners) after 2 or 4 years without long-term funding.

CHALLENGE 3: HOW DO WE KNOW IT WAS OUR INTERVENTION THAT CAUSED CHANGE?

If I used this toolkit how would I know what my project had contributed compared to overall school curriculum?

Learning does not stop outside the classroom. Maybe we should acknowledge that we can't take credit for what anyone learns. It's just a step along the way.

We should ask students/ participants what THEY believe is the effect

Should we have a control group as a means to compare?

Control groups are impossible- there are too many variables (for one thing).

CHALLENGE 4: MEASURING CHANGES IN BEHAVIOUR VS ACTUAL LEARNING

I think measuring both (changes in behaviour/ actions AND learning) are needed. The difficulty is in measuring change in behaviour and actions needed.

The "how" of learning is important as it is transferable, this toolkit is good for measuring progress and changing attitudes.

(We could be) trying to measure "real" learning though comments instead of just counting beans/ sheep?

CHALLENGE 5: EVALUATING FROM A SOUTHERN PERSPECTIVE

This is a key challenge if we don't want to put forward our own notion of development- we need a non Northern/ Western input!

Whose perspective really counts? Are teachers equipped to integrate the Southern perspective?

Not easy to do

How would you do this without multicultural/ Southern practitioners present?

CHALLENGE 6: NEED FOR ACTION/ REFLECTION

(In the toolkit) the desired outcomes of activities are quite determined. Where is the room for negotiation?

What is the prupose of Dev Ed- education/ awareness or political/ action-oriented.

If Dev Ed includes focus on action/ change the toolkit had liitle or no focus on action... rather on attitudes of children.

There is an emphasis on action (in the toolkit) both in the techniques and the activities and suggestions imply action and agency.

CHALLENGE 7: DON'T KNOWS ARE AN INDICATOR OF SUCCESS?

"Don't know" can be a powerful place to be... it can highlight the need to dig deeper.

There is a tension between an open-ended "don't know" and taking action

(A challenge is to) see dealing with uncertainty as a skill.

(If our aim is) giving tools to challenge for themselves, how do we capture this?