Five Faceted ToC by Richard Allen

Transcript

1. Complexity, Evaluation and Theory of Change 17th October 2017 Dr Kerstin Junge & Richard Allen from the Tavistock Institute
2. Complexity, Evaluation & Theory of Change The Tavistock Festival 17th Oct 2017, Swiss Church , 4pm to 5.30pm Richard Allen, Dione Hills and Kerstin Junge
3. Scope of today’s session • How we got here • Challenges of evaluating in complexity and where Theory of Change fits in • Questions and Discussion
4. How we got here
5. Systems thinking to complexity strands in Tavistock evaluation approach Systems theory New evaluation approaches emerging to address complex world Lewin: Field theory Complexity theory Complexity: from Latin complexus ‘intertwined’ or ‘plaited’ Action research Action orientated, participative, theory and system based, evaluation projects System based, Socio- technical, Socio-ecological, System psychoanalytic thinking Interdisciplinary practice: sociologists, anthropologists, psychotherapists, engineers, psychologists economists, political scientists (amongst others….)
6. Systems theory and Action research “If you truly want to understand something, try to change it” Kurt Lewin
7. Systems thinking and action research at the Tavistock Institute • Applied systems thinking in action research projects (1947 through to early 1980’s) • Open systems and turbulent environments (Trist, Emery, Miller) • Socio-ecological and Socio technical action research • System psychodynamics and group relations
8. Complex adaptive systems Santa Fe Institute founded 1984 Stacey matrix
9. New evaluation approaches for a complex world 1950’s: Kirkpatrick’s four levels of learning evaluation 1960’s: Stufflebeam’s CIPP (context, input, process and product) evaluation 1970’s: Parlett and Hamilton’s: Illuminative evaluation Quinn Patton: Utilisation focused evaluation (1978) 1980’s: Guba and Lincoln’s Fourth Generation evaluation (1986) 1990’s: Theory based evaluation and theory of change evaluation (Aspen Round table, Weiss 1997) Realistic evaluation (Pawson and Tilley 1997) 2000’s: Renewed interest in ‘evidence based policy’. Centre for evaluating complexity across the Nexus (CECAN) set up
10. Evaluation at the Tavistock Institute 1980’s + We start to undertake evaluations building on our action research and systems orientation 1990’s + EDRU (Evaluation, Development and Research Unit) set up to explore innovative evaluation methods. Participative, Theory of Change and Realistic Evaluation approaches incorporated into evaluation projects 2000 + Many national and international evaluation projects undertaken , plus evaluation guides and trainings produced 2015 + ‘Dynamics of evaluation’ workshops developed. Join CECAN as partner and become involved in case study activities 1995: ‘Evaluation’ journal launched 1996: UK Evaluation society set up
11. What brings you here? Why are you interested in complexity and evaluation?
12. Challenges of evaluating in complexity and where Theory of Change fits in
13. The challenges • How do we understand and capture change? • The (perceived) TOC paradox • The emergent and dealing with the ‘missing middle’ • Managing ourselves and others
14. Challenge 1: understanding and capturing change
15. Challenge 1: understanding and capturing change Evaluation approach Evaluation question Result Before and after design Has the twins’ approach to Mark Zuckerberg achieved the intended result? No Experimental study What works when creating a dating website? X (Inappropriate design) Theory of change What outcome(s), if any, can be attributed to the twin’s approach to Mark Zuckerberg? How an why was this changed produced? No direct effect but indirect effect in the form of an innovative social networking platform transforming how people socialise online.
16. I live in a self organizing system which means I’m on the edge of chaos Cause does not = effect I can see a lot of nested systems in here Everywhere I look there are repeated irregular forms It’s all a bit fractal! I’m interested in the emergent Challenge 1: understanding and capturing change
17. So the question is… When we are faced with (overwhelming) complexity, (how) can theory of change help us (re-)gain a ‘good enough’ sense of ‘order’ to design and implement (an intervention and evaluation)?
18. Challenge 2 – The (perceived) Theory of Change Paradox
19. What is theory of change ? A theory of change explains how an intervention (a project, a programme, a policy, a strategy) is understood to contribute to a chain of results that produce the intended or actual impacts. Evaluation tests the theory by investigating how what happens in one link affects what happens in other links.
20. Programme theory + implementation theory = Theory of Change (ToC) • “I call the combination of program theory and implementation theory the program’s ‘theories of change’” (Weiss, 1995, p.58). • Change is generated not by the programme activities but by “the response the activities generate”(Weiss 1997, p.46) Log frame? Logic model? IMPLEMENTATION PATHWAY?
21. Working with assumptions “Every programme is packed with beliefs, assumptions and hypotheses about how change happens, the way humans work, or organisations, or political systems, or eco-systems. Theory of change is about articulating these many underlying assumptions about how change will happen in a programme.” (Patricia Rogers)
22. Theory of change and complexity • When working with complexity we need to take a systemic perspective on programmes, policies and projects • Images / theory of change maps help us create ‘good enough’ maps • But we need to find ways to not only use linear models to represent complex initiatives
23. Complex open system Dealing with complex system emergent phenomena e.g. poor childhood attachment (Westhorp 2012). Processes at one system level generate outcomes at another FORMING RELATIONSHIPS SOCIAL JUDGEMENTS ATTACHMENT STYLE Theory of change, complexity and boundaries Layered theories
24. An example Young Adults’ indebtedness System emergent phenomenon: Young Adults’ low level of Fincap Adapted from Ramlingham et al (2014) “From best practice to best fit – Understanding and navigating wicked problems in international development, p.26, ODI Working Paper.
25. Theory of change map for MAS Programme theory: The coordinated feeding of ‘what works’ knowledge to a virtual network of stakeholders via Financial Capability Steering Groups can develop financial capability practice and influence policy change Continuous development and learning ‘WHAT WORKS’ KNOWLEDGE INFLUENCE NETWORK DEVELOPMENT STRATEGY COORDINATION via Steering Groups Assumptions: MAS’s agency in stimulating a virtual network
26. MAS – Meso Theory of Change ‘WHAT WORKS’ KNOWLEDGE Provide knowledge of money management when starting work – a “teachable moment”. Induction will explain salaries, auto enrolment, differences between loans and overdrafts and advice on saving. INFLUENCE Influencing employer networks to makes this part of their management and inductions practices by including it in standards. NETWORK Employer networks pick this issue up and feed it into their members’ HR management e.g. CBI, IoD, FSB, BCC and Unions STRATEGY COORDINATION The strategy communicates knowledge on the relationship between job sustainability and debt management and uses contacts to promote an interest in this Continuous development and learning from effective support for YAs at key points of transition ToC – If effective support is available at key points of transition (e.g. starting work), and Young Adults engage with this support, then it will strengthen their financial capability and financial resilience
27. Reaching the butterfly • Context – What is contributing to indebtedness? • What is our agency? • How do we think change happens? • Assumptions? • Which parts of the system can we plausibly address • What theories can help us?
28. • What are your thoughts and responses to what you’ve heard so far?
29. Challenge 3 - Dealing with the emergent
30. Aspects of emergence • Complexity means working with non-linearity ─ See facebook story: results may be unpredictable, indirect and only emerge over time • An intervention is prone to adapt to context ─ Different actors interact locally, potentially producing different results • So, how do we plan for and / or analyse results using theory of change?
31. Dealing with emergence - the “missing middle”
32. “Missing Middle” defined “‘Missing Middle’ – how the immediate results of a programme influence changes at other levels to influence outcomes and impact in the longer-term.” (Isobel Vogel, 2012. Review of the use of ‘Theory of Change’ in international development. DFID., p. 19)
33. Dealing with the ‘missing middle’: transition points TP0 TP1 TP2 TP3 Poor awareness of indebtedness TPs – Changed state – from information givers Love to see – like to see – fear to see Improved knowledge and practice Network emerging Testing Development & Learning – Network building More agencies involved in multiple schemes – Shift to policy and legislation
34. Where the logic model fits in the missing middle Extracting from ‘what works’ knowledge Developing the dissemination strategy with the steering group Disseminating to targeted parts of the virtual network. Could include commissioning by MAS Implementation of contextually adapted practice Evaluation Influencing – changing practice at policy level Practice becomes established at institutional level Strengthened Fincap for YAs starting work ToC – If effective support is available at key points of transition (e.g. starting work), and Young Adults engage with this support, then it will strengthen their financial capability and financial resilience
35. Dealing with emergence – bounding and simplification “Drawing boundaries around systems and identifying their component elements provides the boundary within which an evaluator will work and identifies elements about which data might be collected.“ (Westhorp, 2012)
36. An example TCBL is a project funded by the European Union's Horizon 2020 programme. It aims to transform the Textiles and Clothing industry, with the objective of bringing 5% of production capacity back to Europe by 2025. http://tcbl.eu/
37. Dealing with the emergence: defining and measuring outcomes • Programme theory and implementation theory useful to guide design of indicator system • Understanding the theoretical foundations of an intervention as represented in the programme theory helps find most suitable overarching framework for an indicator system • Implementation theory can help shape the set of monitoring indicators
38. Dealing with emergence: defining and measuring outcomes (TCBL) • Programme theory primarily drawing on ecosystems thinking: ─ Single indicators or measures are unlikely to be sufficient as it becomes harder to define what ‘success’ is, let alone how to measure it. ─ monitor a range of social, economic and cultural issues as well as ecological issues to allow for adaptive management and learning • Implementation theory reveals ─ Quantitative output, outcome and impact targets (over time) • From that, created multi-dimensional indicator system, including: ─ Key Progress Monitoring Indicators : a list of core outputs specified in the project proposal (and contract) ─ ToC Distance Travelled indicators: provide an indication, at a point in time, of the ‘distance travelled’ by the project in relation to the ‘change journey’ set out in the TCBL baseline Theory of Change.
39. Dealing with emergence III: context sensitivity • As individuals and other units interact locally, results likely to differ by context • Multiple and / or longitudinal case studies to test outcomes and pathways in different settings ─ each case study may have its own theory of change ─ Linked, however, to overarching programme / project / policy ToC • Scenario based ‘effective principles
40. Dealing with emergence III – context sensitivity and working at the programme boundary Boundary partners -- individuals, groups, and organizations directly engaged and who can influence/generate outcomes Identifies changes in the behaviour of people, groups and organizations worked with by development programmes and what those changes result in.
41. Dealing with emergence: iterating the theory of change • Necessary to update programme and implementation theories at regular intervals ─ Incorporating evaluation data • Participative to: ─ Identify possible ‘unanticipated’ or unpredictable outcomes from the beginning and co-construct the story involving as many parts of the system as possible • Iterations become part of the evidence base • Outcomes then judged against decision taken during implementation not the initial theory of change ─ Joint interpretation and sense-making with programme actors to bring multiple perspectives and hence explanations for change
42. Back to TCBL
43. Dealing with the emergence: some (new) analytical techniques • Contribution analysis systematically tests plausibility of hypotheses and counter-hypotheses ─ BUT: how far practical to do this on all aspects of a complex programme? • Process tracing focuses on ‘clues’ within a case ─ BUT: does it really work in super complex real time cases? • Agent based modelling can help test hypotheses about causation by simulating complex processes ─ BUT: if we’re simulating we’re not evidencing?
44. Challenge 4: How do we manage ourselves and others in the space of uncertainty? • Working with multiple stakeholders ─ Essential to capture difference ─ Challenge of power dynamics • Fractals ─ Dynamics of the wider system may be found in the team ─ Recognising, acknowledging, working with very sophisticated work. Needs ‘external’ consultant? • Staying at the boundary ─ How far is too far in the system? ─ When are we too far out?
45. Concluding reflections • Designing and evaluating in the space of complexity means understanding both – programme theory and implementation theory and see those working in dialogue with each other. • Evaluating from a complexity perspective only has a chance of working if there is a shared acceptance / understanding of what this really means • To an extent it is the role of the evaluator to ‘hold’ the complexity for the client system, partners, others ─ We become change agents ourselves
46. Over to you… • Questions, thoughts, reflections?