Evaluation as reflective practice

A presentation by Camilla Child and Dione Hills as part of our new series of lunchtime dialogues on The Dynamics of Evaluation, which contributes to the celebration of the International Year of Evaluation.

Transcript

1. Evaluation as Reflective practice A Dynamics of Evaluation Dialogue Dione Hills Camilla Child
2. What this event is about The series: •  Bringing together evaluation and consultancy practice •  Exploring this interface with others working in the field •  Considering the potential for a future professional development offer
3. Today’s event How can we make best use of our personal experience, alongside evaluation skills, to support learning and change? Reflective practice and evaluation Dione Hills The perspective from an organisational consultant Camilla Child An experience of reflective practice Feedback
4. Reflective Practice: its role in Evaluation
5. Reflective Practice is •  Central to the evaluation process •  Implicit in guidelines on professional practice •  At the heart of recent professional development activities
6. Reflective practice involves bringing feelings and implicit experience to the surface “Bridging the gap between the ‘high ground’ of academic rigour and the ‘lowland of messy practice’ requires ‘professional artistry’, an artistry which should be informed by ‘reflective practice’”. Donald Schön (1983): The Reflective Practitioner: How professionals think in action.
7. Reflective practice as central to evaluation activities Rationale Objectives Appraisal Monitoring Evaluation Feedback *ROAMEF cycle from Treasury Green Book Opportunity to reflect and learn conceptuali se test experience reflect
8. Reflective Practice is generally part of the competency frameworks developed by Evaluation societies *From UKES Evaluation capabilities framework 2013 http://www.evaluation.org.uk/assets/UKES%20Evaluation%20Capabilities%20Framework%20January%202013.pdf PROFESSIONAL PRACTICE 2.25 Displays impartiality in conducting and reporting evaluation 2.26 Manages conflicts of interests and values fairly 3. QUALITIES AND DISPOSITIONS 3.2 Exercises sound, rigorous and fair judgment 3.4 Displays independence of mind and integrity especially when evaluation challenged 3.6 Displays self-knowledge and pursues professional development
9. Reflective practice is part of a process for enhancing professionalism in evaluation being developed by European and UK evaluation societies (EES and UKES) Practice standards Practitioner skills Ethical guidelines Personal conduct Capability frameworks Knowledge base Profession al application Developing the individual 9
10. Voluntary Evaluator Peer Review (VEPR) is about reflection and development rather than designation Reflective practice Peer review è Accountabilit y & professional development 10
11. Reflective practice bridges the gap between theory, skills, values and methods 11 methods Interpersonal skills and values theory Bridges the gaps in evaluation “Bridging the gap between the ‘high ground’ of academic rigour and the ‘lowland of messy practice’ “If you can keep your head when all about you are losing theirs….you don’t understand the situation”. Evan’s law
12. How do evaluators learn reflective practice? •  On ‘research or evaluation methods’ courses? •  Reading (journal articles, methods text books)? •  Professional development opportunities? •  ‘On the job’? •  Prior / outside work experience?
13. Reflective practice uses subjective and objective data (four quadrants integral map by Ken Wilber) Research training/text books often focus attention on the right hand side: objective knowledge and data (and away from the left side)
14. But it can be difficult to be reflective when we….. Are under time pressure – to produce, analyse and report on ‘data’ Have tensions in the evaluation team
15. Reflective Practice – the perspective from organisational consultantcy
16. What is reflective practice? •  Reflective practice is the capacity to reflect on action so as to engage in a process of continuous learning. (Schon 1983) •  It allows you to be a part of an unfolding drama as well as providing opportunities to be sitting in the audience watching it •  Is about creating awareness of the self in relation to the system with which we are working so that we can work better with that system or set of relationships. Self as instrument/instrumentality •  Offers the opportunity to think and take stock
17. Different models of reflective practice •  Kolb (1975) Experiential learning •  Argyris and Schon (1978) Single and Double loop learning •  Schon (1983) – Reflection-in-Action, Reflection-on-Action •  Brookfield (1998) Seeing through different lenses •  Johns (1995) Looking out, looking in
18. Carper (1978) – fundamental ways of knowing Many sources of data for ‘knowing’ •  Empirical - factual knowledge from science, or other external sources, that can be empirically verified. •  Personal - knowledge and attitudes derived from personal self understanding and empathy, including imagining one's self in the patient's position. •  Ethical - attitudes and knowledge derived from an ethical framework, including an awareness of moral questions and choices. •  Aesthetic - awareness of the immediate situation, in this sense is used to mean "relating to the here and now", from the Greek meaning "I perceive, feel, sense";
19. Audience Fitz Actor Henry Benjamin Anna Stage Manager Actor Actor Off stage/ Outside Humphrey Jane Rehearsal Rooms (Front) stage Neil Actor Back stage Audience Audience The wings Back stage Goffman (1959) – Stage behaviour
20. Roles given to and/or taken up by evaluator –  Expert: client plays inactive role, key decisions and technical control lies with consultant (evaluator), two way communication limited, client’s role to ‘judge and evaluate’ after the fact –  Pair of Hands: all key decisions taken by client who specifies actions required, consultant (evaluator) role is to apply specialist knowledge, two way communication limited –  Collaborator : interdependency, data collection and analysis jointly undertaken, implementation by joint discussion and agreement Peter Block: Flawless consulting
21. What can happen to an evaluator? •  Crossing the Boundary into the system  Crossing  boundaries  induces  mul1ple  responses  and  feelings.  These    feelings    may  be  heightened,  some1mes  difficult,    but  always  useful    to  no1ce  in  oneself  and  in  others   •  Being too far in •  Too far out •  On the boundary
22. What can happen to evaluation teams This can be understood to be a fractal of the system as the same patterns repeat through the system Parallel process – the dynamics within an organisation are reproduced between the organisation and a consultant [or evaluator], or within the consultant [evaluation] team
23. Ways of supporting reflective practice •  Reflective notes (as part of the reporting on interviews or field observations) •  One to one - with someone not directly involved in the evaluation/issue – supervision/coach/colleague •  In teams – taking time to reflect on feelings/general observations as well as on evaluation ‘content’ •  Recognising that members of the team might have a ‘valence’ to take on feelings, or identify closely with one part, of the system –  Using this as data (rather than rejecting or scapegoating them) •  Feedback reflections (appropriately) to clients to test out whether this resonates or is helpful. (May depend on role taken up)
24. Exercise in reflective practice
25. Practical exercise in reflection •  Think of an evaluation in which you have been involved (as evaluator, commissioner, or being evaluated) that you experienced as challenging –  At what point did these challenges arise? –  What feelings came up for you? •  Can you see any link between the challenges and feelings you experienced and: –  The nature of the programme/project/work situation itself (e.g. lack of clear boundaries, any conflicts or disagreements between different groups involved, denial of difficulties coming up) –  The characteristics of the ‘issue’ or client group that the programme/ project/policy was concerned with –  What your brought personally to the situation (e.g. personal or professional experience, professional identity, personal values)
26. In pairs A •  Briefly tell your partner about the situation and what came up for you when reflecting on this and the questions we posed B Do: •  Listen to both the content and ‘feeling tone’ of what A is telling you •  Support A through empathising ( e.g. ‘that must have been difficult’) •  Ask clarifying questions (e.g. ‘Can you say a bit more about….’or ‘how did you feel about that?) Don’t: •  Give advice or try to ‘sort out’ the problem
27. In groups of four •  Discuss •  Generally – how did you find this exercise? Did you find it easy to talk about this situation? •  Did any new insights come up? •  Was there anything particular in the way your partner responded that helped you talk about the situation, or gain new insights or understanding? •  How do you now feel about the ‘challenging situation’ you described
28. Thank you and see you next time!