Tools and perspectives for evaluation

Transcript

1. Tools/Perspectives for Evaluation James Atherton October 2014
2. Gibbs’ cycle Description What happened? Conclusion What else could you have done? Action Plan If it arose again, what would you do? Feelings What were you thinking and feeling? Evaluation What was good and bad about the experience? Analysis What sense can you make of the situation? Just a reminder of the components of an evaluation and development cycle, after Graham Gibbs
3. Brookfield’s four “lenses” There’s nothing exclusive about choosing these particular lenses or perspectives. It is their multiplicity which is important. Brookfield S D (1995) Becoming a Critically Reflective Teacher San Francisco; Jossey- Bass
4. Four lenses •Autobiographical • Student • Peer • Theoretical
5. Four lenses •Autobiographical • Student • Peer • Theoretical This lens is the default for most of us—seeing from the perspective of our own experience as the teacher.
6. Four lenses • Autobiographical •Student • Peer • Theoretical There has recently been an emphasis on the “student experience” particularly in HE
7. This is the formal curriculum
8. This is the formal curriculum This is broadly where it works; you set out to teach something , and the students learn it
9. This is the formal curriculum This is broadly where it works; you set out to teach something , and the students learn it This is what you teach but the students do not learn; it’s a waste of time
10. This is the formal curriculum This is broadly where it works; you set out to teach something , and the students learn it This is what you teach but the students do not learn; it’s a waste of time And this is the hidden curriculum; stuff you did not set out to teach, but the students learned anyway.
11. Four lenses • Autobiographical • Student •Peer • Theoretical We teach as part of a community of practice, living with all the tensions that entails. Assumptions about teaching and learning… Concern for inclusivity, and for standards
12. Based on Wenger E (1998) Communities of Practice Cambridge; CUP p. 63 Participation Reification meaning world experience negotiation living in the world membership acting interacting mutuality forms points of focus documents monuments instruments projection This is Etienne Wenger’s take on a community of practice. Much of it is not formally designed as an organisation; it develops organically...
13. Based on Wenger E (1998) Communities of Practice Cambridge; CUP p. 63 Participation Reification meaning world experience negotiation living in the world membership acting interacting mutuality forms points of focus documents monuments instruments projection ...out of the interaction of participants, and the way in which procedures take on a life of their own and may even take physical form (reification)
14. Hunting Assumptions Assumption 1 • It’s common sense to cut lecturing down to a minimum, since lecturing induces passivity in students and kills critical thinking Assumption 2 • It’s common sense that students like group discussion because they feel involved and respected in such a setting. Discussion methods build on principles of participatory, active learning. Assumption 3 • It’s common sense that respectful, empathic teachers will downplay their position of presumed superiority and acknowledge their students as co-teachers. Etc…. (Brookfield, 1995) In the same vein, Brookfield encourages us to explore the taken-for-granted assumptions implicit in the curriculum and pedagogic approach: where’s the evidence for all this stuff? What’s it say about us that we believe it?
15. Espoused theories and theories-in-use Argyris and Schön differentiate between • espoused theories: what people say they are doing, and • theories-in-use: what they are “in fact” doing, as it might appear to an informed outsider So what are the theories-in-use in the curriculum? Note that theories-in-use are often regarded as “inferior” to espoused theories, both technically and morally. But sometimes they are actually better—we just don’t know how to explain and communicate them.
16. Working myths • When theories-in-use are articulated • usually as stories • they become taken-for-granted “working myths” Ideas are shared in communities of practice not as formal theories, but as stories which embody values
17. Four lenses • Autobiographical • Student • Peer •Theoretical We’ve touched on this lens, implicitly, but as the preceding point about stories makes clear, it’s perhaps the least important one...
18. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact (results) Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler To end with a reminder of what you already know, just as we started with one.
19. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Did the students/participants enjoy it?
20. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Did the students/participants enjoy it? From evaluation instruments
21. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Did they actually learn anything?
22. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Did they actually learn anything? From assessment results
23. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Do they do anything differently?
24. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Do they do anything differently? From observation of subsequent practice
25. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Does that make any difference to the performance of the organisation?
26. Kirkpatrick 1. Reaction 2. Learning 3. Behaviour 4. Impact Kirkpatrick D and Kirkpatrick J (2006) Evaluating Training Programs: The Four Levels (3rd edn.) NY; Berrett-Koehler Does that make any difference to the performance of the organisation? From ?