Five Faceted ToC by Richard Allen

Transcript

1. Complexity, Evaluation and Theory of Change
   17th October 2017
   Dr Kerstin Junge & Richard Allen from the
   Tavistock Institute
   

   View Slide

2. Complexity, Evaluation & Theory of Change
   The Tavistock Festival 17th Oct 2017, Swiss Church , 4pm to
   5.30pm
   Richard Allen, Dione Hills and Kerstin Junge
   

   View Slide

3. Scope of today’s session
   • How we got here
   • Challenges of evaluating in complexity and
   where Theory of Change fits in
   • Questions and Discussion
   

   View Slide

4. How we got here
   

   View Slide

5. Systems thinking to complexity strands in
   Tavistock evaluation approach
   Systems
   theory
   New
   evaluation
   approaches
   emerging to
   address
   complex
   world
   Lewin: Field
   theory
   Complexity
   theory
   Complexity: from Latin complexus ‘intertwined’ or ‘plaited’
   Action research
   Action orientated,
   participative,
   theory and system
   based, evaluation
   projects
   System based,
   Socio- technical,
   Socio-ecological,
   System
   psychoanalytic
   thinking
   Interdisciplinary practice:
   sociologists, anthropologists, psychotherapists, engineers, psychologists
   economists, political scientists (amongst others….)
   

   View Slide

6. Systems theory and Action research
   “If you truly want to
   understand something,
   try to change it” Kurt
   Lewin
   

   View Slide

7. Systems thinking and action research
   at the Tavistock Institute
   • Applied systems thinking in
   action research projects (1947
   through to early 1980’s)
   • Open systems and turbulent
   environments (Trist, Emery,
   Miller)
   • Socio-ecological and Socio
   technical action research
   • System psychodynamics and
   group relations
   

   View Slide

8. Complex adaptive systems
   Santa Fe Institute founded 1984
   Stacey matrix
   

   View Slide

9. New evaluation approaches for a complex world
   1950’s: Kirkpatrick’s four levels of learning
   evaluation
   1960’s: Stufflebeam’s CIPP (context, input, process
   and product) evaluation
   1970’s: Parlett and Hamilton’s: Illuminative
   evaluation
   Quinn Patton: Utilisation focused evaluation
   (1978)
   1980’s: Guba and Lincoln’s Fourth Generation
   evaluation (1986)
   1990’s: Theory based evaluation and theory of
   change evaluation (Aspen Round table, Weiss
   1997)
   Realistic evaluation (Pawson and Tilley 1997)
   2000’s: Renewed interest in ‘evidence based
   policy’.
   Centre for evaluating complexity across the
   Nexus (CECAN) set up
   

   View Slide

10. Evaluation at the Tavistock Institute
    1980’s + We start to undertake evaluations
    building on our action research and
    systems orientation
    1990’s + EDRU (Evaluation, Development
    and Research Unit) set up to explore
    innovative evaluation methods.
    Participative, Theory of Change and
    Realistic Evaluation approaches
    incorporated into evaluation projects
    2000 + Many national and international
    evaluation projects undertaken , plus
    evaluation guides and trainings produced
    2015 + ‘Dynamics of evaluation’
    workshops developed.
    Join CECAN as partner and become
    involved in case study activities
    1995: ‘Evaluation’
    journal launched
    1996: UK
    Evaluation society
    set up
    

    View Slide

11. What brings you here?
    Why are you interested in complexity and
    evaluation?
    

    View Slide

12. Challenges of evaluating in complexity
    and where Theory of Change fits in
    

    View Slide

13. The challenges
    • How do we understand and capture change?
    • The (perceived) TOC paradox
    • The emergent and dealing with the ‘missing
    middle’
    • Managing ourselves and others
    

    View Slide

14. Challenge 1: understanding and capturing
    change
    

    View Slide

15. Challenge 1: understanding and capturing
    change
    Evaluation approach Evaluation question Result
    Before and after design Has the twins’ approach to
    Mark Zuckerberg achieved the
    intended result?
    No
    Experimental study What works when creating a
    dating website?
    X
    (Inappropriate design)
    Theory of change What outcome(s), if any, can be
    attributed to the twin’s
    approach to Mark Zuckerberg?
    How an why was this changed
    produced?
    No direct effect but
    indirect effect in the form
    of an innovative social
    networking platform
    transforming how people
    socialise online.
    

    View Slide

16. I live in a self
    organizing
    system which
    means I’m on the
    edge of chaos
    Cause
    does not =
    effect
    I can see
    a lot of
    nested
    systems
    in here
    Everywhere
    I look there
    are
    repeated
    irregular
    forms
    It’s all a
    bit fractal!
    I’m
    interested
    in the
    emergent
    Challenge 1: understanding and capturing
    change
    

    View Slide

17. So the question is…
    When we are faced with (overwhelming)
    complexity,
    (how) can theory of change help us
    (re-)gain a ‘good enough’ sense of
    ‘order’ to design and implement (an
    intervention and evaluation)?
    

    View Slide

18. Challenge 2 – The (perceived) Theory of Change
    Paradox
    

    View Slide

19. What is theory of change ?
    A theory of change explains how an intervention (a
    project, a programme, a policy, a strategy) is
    understood to contribute to a chain of results that
    produce the intended or actual impacts.
    Evaluation tests the theory by investigating how
    what happens in one link affects what happens in
    other links.
    

    View Slide

20. Programme theory + implementation theory
    = Theory of Change (ToC)
    • “I call the combination of program theory and implementation
    theory the program’s ‘theories of change’” (Weiss, 1995, p.58).
    • Change is generated not by the programme activities but by “the
    response the activities generate”(Weiss 1997, p.46)
    Log frame?
    Logic model?
    IMPLEMENTATION PATHWAY?
    

    View Slide

21. Working with assumptions
    “Every programme is packed with
    beliefs, assumptions and
    hypotheses about how change
    happens, the way humans work,
    or organisations, or political
    systems, or eco-systems. Theory
    of change is about articulating
    these many underlying
    assumptions about how change
    will happen in a programme.”
    (Patricia Rogers)
    

    View Slide

22. Theory of change and complexity
    • When working with complexity we need to
    take a systemic perspective on programmes,
    policies and projects
    • Images / theory of change maps help us
    create ‘good enough’ maps
    • But we need to find ways to not only use
    linear models to represent complex initiatives
    

    View Slide

23. Complex open system
    Dealing with complex system emergent
    phenomena e.g. poor childhood
    attachment (Westhorp 2012).
    Processes at one system level
    generate outcomes at another
    FORMING RELATIONSHIPS
    SOCIAL JUDGEMENTS
    ATTACHMENT STYLE
    Theory of change, complexity and boundaries
    Layered theories
    

    View Slide

24. An example Young Adults’ indebtedness
    System
    emergent
    phenomenon:
    Young Adults’
    low level of
    Fincap
    Adapted from Ramlingham et
    al (2014) “From best practice
    to best fit – Understanding
    and navigating wicked
    problems in international
    development, p.26, ODI
    Working Paper.
    

    View Slide

25. Theory of change map for MAS
    Programme theory: The coordinated feeding of ‘what works’ knowledge to a virtual
    network of stakeholders via Financial Capability Steering Groups can develop
    financial capability practice and influence policy change
    Continuous
    development
    and learning
    ‘WHAT WORKS’
    KNOWLEDGE
    INFLUENCE
    NETWORK
    DEVELOPMENT
    STRATEGY
    COORDINATION
    via Steering
    Groups
    Assumptions: MAS’s agency in stimulating a virtual
    network
    

    View Slide

26. MAS – Meso Theory of Change
    ‘WHAT WORKS’ KNOWLEDGE
    Provide knowledge of money management
    when starting work – a “teachable
    moment”. Induction will explain salaries,
    auto enrolment, differences between loans
    and overdrafts and advice on saving.
    INFLUENCE
    Influencing employer networks
    to makes this part of their
    management and inductions
    practices by including it in
    standards.
    NETWORK
    Employer networks pick this issue
    up and feed it into their members’
    HR management e.g. CBI, IoD,
    FSB, BCC and Unions
    STRATEGY COORDINATION
    The strategy communicates
    knowledge on the relationship
    between job sustainability and debt
    management and uses contacts to
    promote an interest in this
    Continuous
    development and
    learning from
    effective support for
    YAs at key points of
    transition
    ToC – If effective support is available at key points of transition
    (e.g. starting work), and Young Adults engage with this support,
    then it will strengthen their financial capability and financial
    resilience
    

    View Slide

27. Reaching the butterfly
    • Context – What is contributing to
    indebtedness?
    • What is our agency?
    • How do we think change happens?
    • Assumptions?
    • Which parts of the system can we plausibly
    address
    • What theories can help us?
    

    View Slide

28. • What are your thoughts and responses to
    what you’ve heard so far?
    

    View Slide

29. Challenge 3 - Dealing with the emergent
    

    View Slide

30. Aspects of emergence
    • Complexity means working with non-linearity
    ─ See facebook story: results may be unpredictable,
    indirect and only emerge over time
    • An intervention is prone to adapt to context
    ─ Different actors interact locally, potentially
    producing different results
    • So, how do we plan for and / or analyse
    results using theory of change?
    

    View Slide

31. Dealing with emergence - the “missing middle”
    

    View Slide

32. “Missing Middle” defined
    “‘Missing Middle’ – how the immediate
    results of a programme influence changes
    at other levels to influence outcomes and
    impact in the longer-term.”
    (Isobel Vogel, 2012. Review of the use of ‘Theory of Change’ in
    international development. DFID., p. 19)
    

    View Slide

33. Dealing with the ‘missing middle’: transition
    points
    TP0 TP1 TP2 TP3
    Poor
    awareness of
    indebtedness
    TPs – Changed state – from information givers
    Love to see – like to see – fear to see
    Improved
    knowledge and
    practice
    Network
    emerging
    Testing
    Development
    & Learning –
    Network
    building
    More agencies
    involved in multiple
    schemes – Shift to
    policy and
    legislation
    

    View Slide

34. Where the logic model fits in the
    missing middle
    Extracting from
    ‘what works’
    knowledge
    Developing the
    dissemination
    strategy with
    the steering
    group
    Disseminating
    to targeted
    parts of the
    virtual network.
    Could include
    commissioning
    by MAS
    Implementation
    of contextually
    adapted
    practice
    Evaluation
    Influencing –
    changing
    practice at
    policy level
    Practice
    becomes
    established at
    institutional
    level
    Strengthened
    Fincap for YAs
    starting work
    ToC – If effective support is
    available at key points of
    transition (e.g. starting work), and
    Young Adults engage with this
    support, then it will strengthen
    their financial capability and
    financial resilience
    

    View Slide

35. Dealing with emergence – bounding and
    simplification
    “Drawing boundaries around
    systems and identifying their
    component elements
    provides the boundary
    within which an evaluator
    will work and identifies
    elements about which data
    might be collected.“
    (Westhorp, 2012)
    

    View Slide

36. An example
    TCBL is a project funded by the European
    Union's Horizon 2020 programme. It aims to
    transform the Textiles and Clothing industry,
    with the objective of bringing 5% of production
    capacity back to Europe by 2025.
    http://tcbl.eu/
    

    View Slide

37. Dealing with the emergence: defining and
    measuring outcomes
    • Programme theory and implementation theory
    useful to guide design of indicator system
    • Understanding the theoretical foundations of an
    intervention as represented in the programme
    theory helps find most suitable overarching
    framework for an indicator system
    • Implementation theory can help shape the set
    of monitoring indicators
    

    View Slide

38. Dealing with emergence: defining and
    measuring outcomes (TCBL)
    • Programme theory primarily drawing on ecosystems thinking:
    ─ Single indicators or measures are unlikely to be sufficient as it
    becomes harder to define what ‘success’ is, let alone how to measure
    it.
    ─ monitor a range of social, economic and cultural issues as well as
    ecological issues to allow for adaptive management and learning
    • Implementation theory reveals
    ─ Quantitative output, outcome and impact targets (over time)
    • From that, created multi-dimensional indicator system, including:
    ─ Key Progress Monitoring Indicators : a list of core outputs specified in
    the project proposal (and contract)
    ─ ToC Distance Travelled indicators: provide an indication, at a point in
    time, of the ‘distance travelled’ by the project in relation to the
    ‘change journey’ set out in the TCBL baseline Theory of Change.
    

    View Slide

39. Dealing with emergence III: context sensitivity
    • As individuals and other units interact locally,
    results likely to differ by context
    • Multiple and / or longitudinal case studies to
    test outcomes and pathways in different settings
    ─ each case study may have its own theory of change
    ─ Linked, however, to overarching programme / project
    / policy ToC
    • Scenario based ‘effective principles
    

    View Slide

40. Dealing with emergence III – context sensitivity and working at
    the programme boundary
    Boundary partners -- individuals, groups,
    and organizations directly engaged and who
    can influence/generate outcomes
    Identifies changes in the behaviour of
    people, groups and organizations
    worked with by development
    programmes and what those changes
    result in.
    

    View Slide

41. Dealing with emergence: iterating the theory of
    change
    • Necessary to update programme and implementation
    theories at regular intervals
    ─ Incorporating evaluation data
    • Participative to:
    ─ Identify possible ‘unanticipated’ or unpredictable outcomes
    from the beginning and co-construct the story involving as
    many parts of the system as possible
    • Iterations become part of the evidence base
    • Outcomes then judged against decision taken during
    implementation not the initial theory of change
    ─ Joint interpretation and sense-making with programme actors
    to bring multiple perspectives and hence explanations for
    change
    

    View Slide

42. Back to TCBL
    

    View Slide

43. Dealing with the emergence: some (new)
    analytical techniques
    • Contribution analysis systematically tests plausibility
    of hypotheses and counter-hypotheses
    ─ BUT: how far practical to do this on all aspects of a
    complex programme?
    • Process tracing focuses on ‘clues’ within a case
    ─ BUT: does it really work in super complex real time cases?
    • Agent based modelling can help test hypotheses
    about causation by simulating complex processes
    ─ BUT: if we’re simulating we’re not evidencing?
    

    View Slide

44. Challenge 4: How do we manage ourselves and
    others in the space of uncertainty?
    • Working with multiple stakeholders
    ─ Essential to capture difference
    ─ Challenge of power dynamics
    • Fractals
    ─ Dynamics of the wider system may be found in the team
    ─ Recognising, acknowledging, working with very
    sophisticated work. Needs ‘external’ consultant?
    • Staying at the boundary
    ─ How far is too far in the system?
    ─ When are we too far out?
    

    View Slide

45. Concluding reflections
    • Designing and evaluating in the space of complexity
    means understanding both – programme theory and
    implementation theory and see those working in
    dialogue with each other.
    • Evaluating from a complexity perspective only has a
    chance of working if there is a shared acceptance /
    understanding of what this really means
    • To an extent it is the role of the evaluator to ‘hold’ the
    complexity for the client system, partners, others
    ─ We become change agents ourselves
    

    View Slide

46. Over to you…
    • Questions, thoughts, reflections?
    

    View Slide