Lunchtime Talk: Handling Complexity

Powerpoint from lunchtime event is the third of four dialogues in the Dynamics of Evaluation series, and part of our celebration of 2015 International Year of Evaluation. Each event is taking a different perspective on challenges that can occur in evaluation practice and how these might be addressed. In this third dialogue, we plan to explore with you how far a better understanding of complexity theory might help us make sense of, and handle some of these challenges.

Transcript

1. Handling complexity in evaluation A Dynamics of Evaluation Dialogue Eliat Aram Dione Hills Kerstin Junge
2. The dynamics of evaluation series • Bringing together evaluation and consultancy practice • Exploring this interface with others working in the field • Considering the potential for a future professional development offer
3. What are complex adaptive systems Today’s event Can a better understanding of complexity help us navigate the uncertainties and paradoxes encountered during an evaluation?
4. Recent developments in addressing complexity in evaluation An evaluator’s perspective Dione
5. The past • Evaluators have always had to tussle with the evaluation of complex programmes and policies • Theory based evaluation strategies developed from early 1990 onwards to address this challenge (as alternative to experimental methods) • Had to compete with view that experimental designs were ‘gold standard’ for demonstrating causality • Theory based strategies criticised as being: – Better at tracking progress and process than providing evidence of impacts. – Stronger on qualitative than quantitative data
6. The present • Growing demand for evaluation and ‘evidence based policy’ with emphasis on impact and economic evaluation • Evaluation of growing importance in sectors where it was previously weak (e.g. transport and environment sectors) and where quantitate and economic data very important • ‘Experimental designs’ still seen in many circles as the ‘gold standard’ • But experimental methods not suitable where programmes or policies are ‘complex’ • New approaches now being sought to address this challelnge (e.g. in UK new centre is being funded to find new ways of evaluating complex policies and programmes in environmental sector)
7. What is a complex adaptive system A consultant’s perspective Eliat
8. Key characteristics of a complex adaptive system • organisms made up of many interacting agents • Non-linear and self-organising with emergent futures • Self organisation means that agents interact locally and that it is in this local interaction that global patterns emerge without any global blueprint, design or programme. The system evolves in an intrinsically unpredictable manner into an undetermined future. They are interdependent, with local action at one scale having unpredictable consequences at all scales through complex relationships over time. They co-create their future. • Interactions, taking the form of both positive and negative feedback, can, broadly speaking, display three dynamics: stable to the point of rigidity; unstable to the point of disintegration; and, paradoxically, patterns that are both stable and unstable simultaneously.
9. • The paradoxical dynamic has become known as the “edge of chaos”, a dynamic where order and disorder co-exist. (Waldrop, 1992: 12).
10. • When the agents in a complex adaptive system differ from each other, the system displays the capacity to transform itself. It is only at a critical level of diversity that a system can produce novelty (Allen, 1998 a & b). • Such systems are adaptive in that they do not simply respond to events, but evolve or learn. Each agent is guided by its own schema, or rules of behaviour, and also by schema shared with other agents.
11. Organisations as complex adaptive systems: Agents or entities interacting in nature produce coherence that emerges in these interactions even though there is no blueprint or programme determining that coherence. This interaction is inherently paradoxical and novelty emerges only when there is enough diversity and enough constraint, so that order and disorder, creativity and destruction are inextricably linked in the creative process. The process is self-referential in the sense that interaction causes patterns that are recognisable. The process maintains continuity and is potentially transformational, that is, novel. Some (Goodwin, 1994) hold that in nature, such interaction occurs not primarily in order to survive but as the creative expression of identity.
12. Systems and Organisations as fractals • A fractal is a never-ending pattern. Fractals are infinitely complex patterns that are self-similar across different scales. They are created by repeating a simple process over and over in an ongoing feedback loop. Fractals are images of dynamic systems – the pictures of the edge of chaos. • Understanding systems and organisations as fractals challenges our understanding of: – Boundaries – Role of Evaluator / consultant as a boundary manager – System the notion of a parallel process
13. Different evaluation strategies and complexity An evaluator’s perspective Kerstin
14. How evaluators deal with complexity (I) – looking at the nature of interventions Source: Rogers, P (2008) Using Program Theory to evaluate complicated and complex aspects of interventions, Evaluation, Vol 14 (1), p 31
15. Different evaluation strategies and their ability to ‘cope’ with complexity • Before and after outcome designs can give information whether the intended results of the intervention have been achieved. – Used in ‘simple’ interventions, e.g. introducing speed bumps to slow traffic • Experimental designs – Work where there’s a high degree of knowledge on intended outcomes, standardisation of intervention, and little or no context interference is assumed or can be expected • Theory based evaluation strategies (theory of change, realist evaluation, contribution analysis) – Work with context dependence, multiple activities (and strands), change, and (uncertain) outcomes – Thinking about contribution rather than attribution
16. How evaluators deal with complexity (II) – towards a complexity theory position for evaluation? • Emergence – an intervention is prone to adapt to context, learning by those implementing it, …outcomes are not pre-defined (or pre-definable) – Evaluators to support learning through real time evaluation and produce rich descriptions from which others can learn? (Ling, 2012) • Systems – recognition that social interventions are systems themselves that are nested in social systems and sub-systems and that outcomes (as well as the intervention itself) are influenced by these systems – “Drawing boundaries around systems and identifying their component elements provides the boundary within which an evaluator will work and identifies elements about which data might be collected.“ (Westhorp, 2012) • Context dependence – intervention implementation and outcomes are dependent on the context in which they are situated – Contexts are systems, and understanding interactions of these systems is the task of the complexity-minded evaluator (Ling, 2012) • Uncertainty – about causation, likely / possible / expected outcomes – Use of ‘contribution stories’ in a theory based evaluation to extract practitioners’ theores about causation (Ling, 2012) – Use ‘substantive theories’ (that theories specify which elements of the system are important for change processes) to guide data collection. (Westhorp, 2012)
17. A practice example: Impact evaluation of an EU-funded R+D project on the use of social media in emergencies • Emergent – a broad road map towards outputs specified in proposal, but detail on activities, content etc shaping as the project develops • Uncertainty – broad ideas for (desired) outcomes, but not detailed or necessarily backed up by much evidence; outcomes to an extent emergent themselves • Systems and context – implemented in different EU member states, different emergency management (work) systems and scenarios, outputs to be affecting individual citizens as well as emergency management organisations (also involved in the project)
18. Evaluation strategy: a ‘real time’ theory of change impact evaluation Programme feature Impact evaluation strategy Emergence • Extracting embedded theory about causation from consortium partners, at different time points in the project • Feeding research outputs back into design of impact work Uncertainty • Review of scientific literature on impact of social media in emergencies • Multiple and successive in-depth case studies in different contexts and involving both citizens and emergency services to successively understand range of key outcomes and built an evidence base on those • Bringing in theories of behaviour change to explain change Systems and context • All impact evaluation activities considering individual, organisational, as well as geographic / administrative levels
19. Managing and evaluating (in) uncertainty what sense are you making of these ideas? A reflective activity
20. Personal reflection (10’) • Can you think of an evaluation that you were involved in where using a complexity lens could be/ have been helpful or suitable? • If yes, how have you coped with the challenges that a complexity lens throws up? • If not, what would you have changed or done differently had you been able to think in complexity theory terms?
21. In pairs (15’) Share your cases with each other, note differences and similarities and how you each coped in that situation
22. In whole table group (15’) Discuss • How do you now feel about the ‘challenging situation’ you described • Did any new insights come up as you were grappling with the concepts of a complexity approach? • Generally – how did you find this exercise?
23. In plenary 15’ Thank you! See you next time.