$30 off During Our Annual Pro Sale. View Details »

Five Faceted ToC by Richard Allen

Five Faceted ToC by Richard Allen

Tavistock Institute

April 10, 2018
Tweet

More Decks by Tavistock Institute

Other Decks in Education

Transcript

  1. Complexity, Evaluation and Theory of Change
    17th October 2017
    Dr Kerstin Junge & Richard Allen from the
    Tavistock Institute

    View Slide

  2. Complexity, Evaluation & Theory of Change
    The Tavistock Festival 17th Oct 2017, Swiss Church , 4pm to
    5.30pm
    Richard Allen, Dione Hills and Kerstin Junge

    View Slide

  3. Scope of today’s session
    • How we got here
    • Challenges of evaluating in complexity and
    where Theory of Change fits in
    • Questions and Discussion

    View Slide

  4. How we got here

    View Slide

  5. Systems thinking to complexity strands in
    Tavistock evaluation approach
    Systems
    theory
    New
    evaluation
    approaches
    emerging to
    address
    complex
    world
    Lewin: Field
    theory
    Complexity
    theory
    Complexity: from Latin complexus ‘intertwined’ or ‘plaited’
    Action research
    Action orientated,
    participative,
    theory and system
    based, evaluation
    projects
    System based,
    Socio- technical,
    Socio-ecological,
    System
    psychoanalytic
    thinking
    Interdisciplinary practice:
    sociologists, anthropologists, psychotherapists, engineers, psychologists
    economists, political scientists (amongst others….)

    View Slide

  6. Systems theory and Action research
    “If you truly want to
    understand something,
    try to change it” Kurt
    Lewin

    View Slide

  7. Systems thinking and action research
    at the Tavistock Institute
    • Applied systems thinking in
    action research projects (1947
    through to early 1980’s)
    • Open systems and turbulent
    environments (Trist, Emery,
    Miller)
    • Socio-ecological and Socio
    technical action research
    • System psychodynamics and
    group relations

    View Slide

  8. Complex adaptive systems
    Santa Fe Institute founded 1984
    Stacey matrix

    View Slide

  9. New evaluation approaches for a complex world
    1950’s: Kirkpatrick’s four levels of learning
    evaluation
    1960’s: Stufflebeam’s CIPP (context, input, process
    and product) evaluation
    1970’s: Parlett and Hamilton’s: Illuminative
    evaluation
    Quinn Patton: Utilisation focused evaluation
    (1978)
    1980’s: Guba and Lincoln’s Fourth Generation
    evaluation (1986)
    1990’s: Theory based evaluation and theory of
    change evaluation (Aspen Round table, Weiss
    1997)
    Realistic evaluation (Pawson and Tilley 1997)
    2000’s: Renewed interest in ‘evidence based
    policy’.
    Centre for evaluating complexity across the
    Nexus (CECAN) set up

    View Slide

  10. Evaluation at the Tavistock Institute
    1980’s + We start to undertake evaluations
    building on our action research and
    systems orientation
    1990’s + EDRU (Evaluation, Development
    and Research Unit) set up to explore
    innovative evaluation methods.
    Participative, Theory of Change and
    Realistic Evaluation approaches
    incorporated into evaluation projects
    2000 + Many national and international
    evaluation projects undertaken , plus
    evaluation guides and trainings produced
    2015 + ‘Dynamics of evaluation’
    workshops developed.
    Join CECAN as partner and become
    involved in case study activities
    1995: ‘Evaluation’
    journal launched
    1996: UK
    Evaluation society
    set up

    View Slide

  11. What brings you here?
    Why are you interested in complexity and
    evaluation?

    View Slide

  12. Challenges of evaluating in complexity
    and where Theory of Change fits in

    View Slide

  13. The challenges
    • How do we understand and capture change?
    • The (perceived) TOC paradox
    • The emergent and dealing with the ‘missing
    middle’
    • Managing ourselves and others

    View Slide

  14. Challenge 1: understanding and capturing
    change

    View Slide

  15. Challenge 1: understanding and capturing
    change
    Evaluation approach Evaluation question Result
    Before and after design Has the twins’ approach to
    Mark Zuckerberg achieved the
    intended result?
    No
    Experimental study What works when creating a
    dating website?
    X
    (Inappropriate design)
    Theory of change What outcome(s), if any, can be
    attributed to the twin’s
    approach to Mark Zuckerberg?
    How an why was this changed
    produced?
    No direct effect but
    indirect effect in the form
    of an innovative social
    networking platform
    transforming how people
    socialise online.

    View Slide

  16. I live in a self
    organizing
    system which
    means I’m on the
    edge of chaos
    Cause
    does not =
    effect
    I can see
    a lot of
    nested
    systems
    in here
    Everywhere
    I look there
    are
    repeated
    irregular
    forms
    It’s all a
    bit fractal!
    I’m
    interested
    in the
    emergent
    Challenge 1: understanding and capturing
    change

    View Slide

  17. So the question is…
    When we are faced with (overwhelming)
    complexity,
    (how) can theory of change help us
    (re-)gain a ‘good enough’ sense of
    ‘order’ to design and implement (an
    intervention and evaluation)?

    View Slide

  18. Challenge 2 – The (perceived) Theory of Change
    Paradox

    View Slide

  19. What is theory of change ?
    A theory of change explains how an intervention (a
    project, a programme, a policy, a strategy) is
    understood to contribute to a chain of results that
    produce the intended or actual impacts.
    Evaluation tests the theory by investigating how
    what happens in one link affects what happens in
    other links.

    View Slide

  20. Programme theory + implementation theory
    = Theory of Change (ToC)
    • “I call the combination of program theory and implementation
    theory the program’s ‘theories of change’” (Weiss, 1995, p.58).
    • Change is generated not by the programme activities but by “the
    response the activities generate”(Weiss 1997, p.46)
    Log frame?
    Logic model?
    IMPLEMENTATION PATHWAY?

    View Slide

  21. Working with assumptions
    “Every programme is packed with
    beliefs, assumptions and
    hypotheses about how change
    happens, the way humans work,
    or organisations, or political
    systems, or eco-systems. Theory
    of change is about articulating
    these many underlying
    assumptions about how change
    will happen in a programme.”
    (Patricia Rogers)

    View Slide

  22. Theory of change and complexity
    • When working with complexity we need to
    take a systemic perspective on programmes,
    policies and projects
    • Images / theory of change maps help us
    create ‘good enough’ maps
    • But we need to find ways to not only use
    linear models to represent complex initiatives

    View Slide

  23. Complex open system
    Dealing with complex system emergent
    phenomena e.g. poor childhood
    attachment (Westhorp 2012).
    Processes at one system level
    generate outcomes at another
    FORMING RELATIONSHIPS
    SOCIAL JUDGEMENTS
    ATTACHMENT STYLE
    Theory of change, complexity and boundaries
    Layered theories

    View Slide

  24. An example Young Adults’ indebtedness
    System
    emergent
    phenomenon:
    Young Adults’
    low level of
    Fincap
    Adapted from Ramlingham et
    al (2014) “From best practice
    to best fit – Understanding
    and navigating wicked
    problems in international
    development, p.26, ODI
    Working Paper.

    View Slide

  25. Theory of change map for MAS
    Programme theory: The coordinated feeding of ‘what works’ knowledge to a virtual
    network of stakeholders via Financial Capability Steering Groups can develop
    financial capability practice and influence policy change
    Continuous
    development
    and learning
    ‘WHAT WORKS’
    KNOWLEDGE
    INFLUENCE
    NETWORK
    DEVELOPMENT
    STRATEGY
    COORDINATION
    via Steering
    Groups
    Assumptions: MAS’s agency in stimulating a virtual
    network

    View Slide

  26. MAS – Meso Theory of Change
    ‘WHAT WORKS’ KNOWLEDGE
    Provide knowledge of money management
    when starting work – a “teachable
    moment”. Induction will explain salaries,
    auto enrolment, differences between loans
    and overdrafts and advice on saving.
    INFLUENCE
    Influencing employer networks
    to makes this part of their
    management and inductions
    practices by including it in
    standards.
    NETWORK
    Employer networks pick this issue
    up and feed it into their members’
    HR management e.g. CBI, IoD,
    FSB, BCC and Unions
    STRATEGY COORDINATION
    The strategy communicates
    knowledge on the relationship
    between job sustainability and debt
    management and uses contacts to
    promote an interest in this
    Continuous
    development and
    learning from
    effective support for
    YAs at key points of
    transition
    ToC – If effective support is available at key points of transition
    (e.g. starting work), and Young Adults engage with this support,
    then it will strengthen their financial capability and financial
    resilience

    View Slide

  27. Reaching the butterfly
    • Context – What is contributing to
    indebtedness?
    • What is our agency?
    • How do we think change happens?
    • Assumptions?
    • Which parts of the system can we plausibly
    address
    • What theories can help us?

    View Slide

  28. • What are your thoughts and responses to
    what you’ve heard so far?

    View Slide

  29. Challenge 3 - Dealing with the emergent

    View Slide

  30. Aspects of emergence
    • Complexity means working with non-linearity
    ─ See facebook story: results may be unpredictable,
    indirect and only emerge over time
    • An intervention is prone to adapt to context
    ─ Different actors interact locally, potentially
    producing different results
    • So, how do we plan for and / or analyse
    results using theory of change?

    View Slide

  31. Dealing with emergence - the “missing middle”

    View Slide

  32. “Missing Middle” defined
    “‘Missing Middle’ – how the immediate
    results of a programme influence changes
    at other levels to influence outcomes and
    impact in the longer-term.”
    (Isobel Vogel, 2012. Review of the use of ‘Theory of Change’ in
    international development. DFID., p. 19)

    View Slide

  33. Dealing with the ‘missing middle’: transition
    points
    TP0 TP1 TP2 TP3
    Poor
    awareness of
    indebtedness
    TPs – Changed state – from information givers
    Love to see – like to see – fear to see
    Improved
    knowledge and
    practice
    Network
    emerging
    Testing
    Development
    & Learning –
    Network
    building
    More agencies
    involved in multiple
    schemes – Shift to
    policy and
    legislation

    View Slide

  34. Where the logic model fits in the
    missing middle
    Extracting from
    ‘what works’
    knowledge
    Developing the
    dissemination
    strategy with
    the steering
    group
    Disseminating
    to targeted
    parts of the
    virtual network.
    Could include
    commissioning
    by MAS
    Implementation
    of contextually
    adapted
    practice
    Evaluation
    Influencing –
    changing
    practice at
    policy level
    Practice
    becomes
    established at
    institutional
    level
    Strengthened
    Fincap for YAs
    starting work
    ToC – If effective support is
    available at key points of
    transition (e.g. starting work), and
    Young Adults engage with this
    support, then it will strengthen
    their financial capability and
    financial resilience

    View Slide

  35. Dealing with emergence – bounding and
    simplification
    “Drawing boundaries around
    systems and identifying their
    component elements
    provides the boundary
    within which an evaluator
    will work and identifies
    elements about which data
    might be collected.“
    (Westhorp, 2012)

    View Slide

  36. An example
    TCBL is a project funded by the European
    Union's Horizon 2020 programme. It aims to
    transform the Textiles and Clothing industry,
    with the objective of bringing 5% of production
    capacity back to Europe by 2025.
    http://tcbl.eu/

    View Slide

  37. Dealing with the emergence: defining and
    measuring outcomes
    • Programme theory and implementation theory
    useful to guide design of indicator system
    • Understanding the theoretical foundations of an
    intervention as represented in the programme
    theory helps find most suitable overarching
    framework for an indicator system
    • Implementation theory can help shape the set
    of monitoring indicators

    View Slide

  38. Dealing with emergence: defining and
    measuring outcomes (TCBL)
    • Programme theory primarily drawing on ecosystems thinking:
    ─ Single indicators or measures are unlikely to be sufficient as it
    becomes harder to define what ‘success’ is, let alone how to measure
    it.
    ─ monitor a range of social, economic and cultural issues as well as
    ecological issues to allow for adaptive management and learning
    • Implementation theory reveals
    ─ Quantitative output, outcome and impact targets (over time)
    • From that, created multi-dimensional indicator system, including:
    ─ Key Progress Monitoring Indicators : a list of core outputs specified in
    the project proposal (and contract)
    ─ ToC Distance Travelled indicators: provide an indication, at a point in
    time, of the ‘distance travelled’ by the project in relation to the
    ‘change journey’ set out in the TCBL baseline Theory of Change.

    View Slide

  39. Dealing with emergence III: context sensitivity
    • As individuals and other units interact locally,
    results likely to differ by context
    • Multiple and / or longitudinal case studies to
    test outcomes and pathways in different settings
    ─ each case study may have its own theory of change
    ─ Linked, however, to overarching programme / project
    / policy ToC
    • Scenario based ‘effective principles

    View Slide

  40. Dealing with emergence III – context sensitivity and working at
    the programme boundary
    Boundary partners -- individuals, groups,
    and organizations directly engaged and who
    can influence/generate outcomes
    Identifies changes in the behaviour of
    people, groups and organizations
    worked with by development
    programmes and what those changes
    result in.

    View Slide

  41. Dealing with emergence: iterating the theory of
    change
    • Necessary to update programme and implementation
    theories at regular intervals
    ─ Incorporating evaluation data
    • Participative to:
    ─ Identify possible ‘unanticipated’ or unpredictable outcomes
    from the beginning and co-construct the story involving as
    many parts of the system as possible
    • Iterations become part of the evidence base
    • Outcomes then judged against decision taken during
    implementation not the initial theory of change
    ─ Joint interpretation and sense-making with programme actors
    to bring multiple perspectives and hence explanations for
    change

    View Slide

  42. Back to TCBL

    View Slide

  43. Dealing with the emergence: some (new)
    analytical techniques
    • Contribution analysis systematically tests plausibility
    of hypotheses and counter-hypotheses
    ─ BUT: how far practical to do this on all aspects of a
    complex programme?
    • Process tracing focuses on ‘clues’ within a case
    ─ BUT: does it really work in super complex real time cases?
    • Agent based modelling can help test hypotheses
    about causation by simulating complex processes
    ─ BUT: if we’re simulating we’re not evidencing?

    View Slide

  44. Challenge 4: How do we manage ourselves and
    others in the space of uncertainty?
    • Working with multiple stakeholders
    ─ Essential to capture difference
    ─ Challenge of power dynamics
    • Fractals
    ─ Dynamics of the wider system may be found in the team
    ─ Recognising, acknowledging, working with very
    sophisticated work. Needs ‘external’ consultant?
    • Staying at the boundary
    ─ How far is too far in the system?
    ─ When are we too far out?

    View Slide

  45. Concluding reflections
    • Designing and evaluating in the space of complexity
    means understanding both – programme theory and
    implementation theory and see those working in
    dialogue with each other.
    • Evaluating from a complexity perspective only has a
    chance of working if there is a shared acceptance /
    understanding of what this really means
    • To an extent it is the role of the evaluator to ‘hold’ the
    complexity for the client system, partners, others
    ─ We become change agents ourselves

    View Slide

  46. Over to you…
    • Questions, thoughts, reflections?

    View Slide