Lunchtime Talk: Handling Complexity

Powerpoint from lunchtime event is the third of four dialogues in the Dynamics of Evaluation series, and part of our celebration of 2015 International Year of Evaluation. Each event is taking a different perspective on challenges that can occur in evaluation practice and how these might be addressed. In this third dialogue, we plan to explore with you how far a better understanding of complexity theory might help us make sense of, and handle some of these challenges.

Transcript

1. Handling complexity
   in evaluation
   A Dynamics of Evaluation Dialogue
   Eliat Aram
   Dione Hills
   Kerstin Junge
   

   View Slide

2. The dynamics of
   evaluation series
   • Bringing together evaluation
   and consultancy practice
   • Exploring this interface with
   others working in the field
   • Considering the potential for
   a future professional
   development offer
   

   View Slide

3. What are complex
   adaptive systems
   Today’s event
   Can a better understanding of complexity help us
   navigate the uncertainties and paradoxes encountered
   during an evaluation?
   

   View Slide

4. Recent developments in addressing
   complexity in evaluation
   An evaluator’s perspective
   Dione
   

   View Slide

5. The past
   • Evaluators have always had to tussle with the evaluation of complex
   programmes and policies
   • Theory based evaluation strategies developed from early 1990 onwards to
   address this challenge (as alternative to experimental methods)
   • Had to compete with view that experimental designs were ‘gold standard’ for
   demonstrating causality
   • Theory based strategies criticised as being:
   – Better at tracking progress and process than providing evidence of
   impacts.
   – Stronger on qualitative than quantitative data
   

   View Slide

6. The present
   • Growing demand for evaluation and ‘evidence based policy’ with emphasis
   on impact and economic evaluation
   • Evaluation of growing importance in sectors where it was previously weak
   (e.g. transport and environment sectors) and where quantitate and
   economic data very important
   • ‘Experimental designs’ still seen in many circles as the ‘gold standard’
   • But experimental methods not suitable where programmes or policies are
   ‘complex’
   • New approaches now being sought to address this challelnge (e.g. in UK
   new centre is being funded to find new ways of evaluating complex policies
   and programmes in environmental sector)
   

   View Slide

7. What is a complex adaptive
   system
   A consultant’s perspective
   Eliat
   

   View Slide

8. Key characteristics of a complex adaptive
   system
   • organisms made up of many interacting agents
   • Non-linear and self-organising with emergent futures
   • Self organisation means that agents interact locally and that it is in
   this local interaction that global patterns emerge without any global
   blueprint, design or programme. The system evolves in an
   intrinsically unpredictable manner into an undetermined future. They
   are interdependent, with local action at one scale having
   unpredictable consequences at all scales through complex
   relationships over time. They co-create their future.
   • Interactions, taking the form of both positive and negative feedback,
   can, broadly speaking, display three dynamics: stable to the point of
   rigidity; unstable to the point of disintegration; and, paradoxically,
   patterns that are both stable and unstable simultaneously.
   

   View Slide

9. • The paradoxical dynamic has become known as the “edge of
   chaos”, a dynamic where order and disorder co-exist.
   (Waldrop, 1992: 12).
   

   View Slide

10. • When the agents in a complex adaptive system differ
    from each other, the system displays the capacity to
    transform itself. It is only at a critical level of diversity that
    a system can produce novelty (Allen, 1998 a & b).
    • Such systems are adaptive in that they do not simply
    respond to events, but evolve or learn. Each agent is
    guided by its own schema, or rules of behaviour, and
    also by schema shared with other agents.
    

    View Slide

11. Organisations as complex adaptive systems:
    Agents or entities interacting in nature produce coherence that
    emerges in these interactions even though there is no blueprint or
    programme determining that coherence.
    This interaction is inherently paradoxical and novelty emerges only
    when there is enough diversity and enough constraint, so that order
    and disorder, creativity and destruction are inextricably linked in the
    creative process.
    The process is self-referential in the sense that interaction causes
    patterns that are recognisable. The process maintains continuity and is
    potentially transformational, that is, novel.
    Some (Goodwin, 1994) hold that in nature, such interaction occurs not
    primarily in order to survive but as the creative expression of identity.
    

    View Slide

12. Systems and Organisations
    as fractals
    • A fractal is a never-ending pattern. Fractals are infinitely complex
    patterns that are self-similar across different scales. They are
    created by repeating a simple process over and over in an ongoing
    feedback loop. Fractals are images of dynamic systems – the
    pictures of the edge of chaos.
    • Understanding systems and organisations as fractals challenges our
    understanding of:
    – Boundaries
    – Role of Evaluator / consultant as a boundary manager
    – System the notion of a parallel process
    

    View Slide

13. Different evaluation strategies
    and complexity
    An evaluator’s perspective
    Kerstin
    

    View Slide

14. How evaluators deal with complexity (I) –
    looking at the nature of interventions
    Source: Rogers, P (2008) Using Program Theory to evaluate complicated and complex aspects of interventions,
    Evaluation, Vol 14 (1), p 31
    

    View Slide

15. Different evaluation strategies and their ability
    to ‘cope’ with complexity
    • Before and after outcome designs can give information whether the
    intended results of the intervention have been achieved.
    – Used in ‘simple’ interventions, e.g. introducing speed bumps to
    slow traffic
    • Experimental designs
    – Work where there’s a high degree of knowledge on intended
    outcomes, standardisation of intervention, and little or no context
    interference is assumed or can be expected
    • Theory based evaluation strategies (theory of change, realist
    evaluation, contribution analysis)
    – Work with context dependence, multiple activities (and strands),
    change, and (uncertain) outcomes
    – Thinking about contribution rather than attribution
    

    View Slide

16. How evaluators deal with complexity (II) – towards a
    complexity theory position for evaluation?
    • Emergence – an intervention is prone to adapt to context, learning by those
    implementing it, …outcomes are not pre-defined (or pre-definable)
    – Evaluators to support learning through real time evaluation and produce rich
    descriptions from which others can learn? (Ling, 2012)
    • Systems – recognition that social interventions are systems themselves that are
    nested in social systems and sub-systems and that outcomes (as well as the
    intervention itself) are influenced by these systems
    – “Drawing boundaries around systems and identifying their component
    elements provides the boundary within which an evaluator will work and
    identifies elements about which data might be collected.“ (Westhorp, 2012)
    • Context dependence – intervention implementation and outcomes are
    dependent on the context in which they are situated
    – Contexts are systems, and understanding interactions of these systems is the
    task of the complexity-minded evaluator (Ling, 2012)
    • Uncertainty – about causation, likely / possible / expected outcomes
    – Use of ‘contribution stories’ in a theory based evaluation to extract
    practitioners’ theores about causation (Ling, 2012)
    – Use ‘substantive theories’ (that theories specify which elements of the system
    are important for change processes) to guide data collection. (Westhorp,
    2012)
    

    View Slide

17. A practice example:
    Impact evaluation of an EU-funded R+D project on
    the use of social media in emergencies
    • Emergent – a broad road map towards outputs specified in
    proposal, but detail on activities, content etc shaping as the project
    develops
    • Uncertainty – broad ideas for (desired) outcomes, but not detailed
    or necessarily backed up by much evidence; outcomes to an extent
    emergent themselves
    • Systems and context – implemented in different EU member
    states, different emergency management (work) systems and
    scenarios, outputs to be affecting individual citizens as well as
    emergency management organisations (also involved in the project)
    

    View Slide

18. Evaluation strategy: a ‘real time’ theory of
    change impact evaluation
    Programme
    feature
    Impact evaluation strategy
    Emergence • Extracting embedded theory about causation from
    consortium partners, at different time points in the
    project
    • Feeding research outputs back into design of impact
    work
    Uncertainty • Review of scientific literature on impact of social media
    in emergencies
    • Multiple and successive in-depth case studies in
    different contexts and involving both citizens and
    emergency services to successively understand range of
    key outcomes and built an evidence base on those
    • Bringing in theories of behaviour change to explain
    change
    Systems and
    context
    • All impact evaluation activities considering individual,
    organisational, as well as geographic / administrative
    levels
    

    View Slide

19. Managing and evaluating (in)
    uncertainty
    what sense are you making of these
    ideas?
    A reflective activity
    

    View Slide

20. Personal reflection (10’)
    • Can you think of an evaluation that you were involved in
    where using a complexity lens could be/ have been
    helpful or suitable?
    • If yes, how have you coped with the challenges that a
    complexity lens throws up?
    • If not, what would you have changed or done differently
    had you been able to think in complexity theory terms?
    

    View Slide

21. In pairs (15’)
    Share your cases with each other,
    note differences and similarities and
    how you each coped in that situation
    

    View Slide

22. In whole table group (15’)
    Discuss
    • How do you now feel about the ‘challenging situation’ you described
    • Did any new insights come up as you were grappling with the
    concepts of a complexity approach?
    • Generally – how did you find this exercise?
    

    View Slide

23. In plenary 15’
    Thank you!
    See you next time.
    

    View Slide