theory New evaluation approaches emerging to address complex world Lewin: Field theory Complexity theory Complexity: from Latin complexus ‘intertwined’ or ‘plaited’ Action research Action orientated, participative, theory and system based, evaluation projects System based, Socio- technical, Socio-ecological, System psychoanalytic thinking Interdisciplinary practice: sociologists, anthropologists, psychotherapists, engineers, psychologists economists, political scientists (amongst others….)
Applied systems thinking in action research projects (1947 through to early 1980’s) • Open systems and turbulent environments (Trist, Emery, Miller) • Socio-ecological and Socio technical action research • System psychodynamics and group relations
levels of learning evaluation 1960’s: Stufflebeam’s CIPP (context, input, process and product) evaluation 1970’s: Parlett and Hamilton’s: Illuminative evaluation Quinn Patton: Utilisation focused evaluation (1978) 1980’s: Guba and Lincoln’s Fourth Generation evaluation (1986) 1990’s: Theory based evaluation and theory of change evaluation (Aspen Round table, Weiss 1997) Realistic evaluation (Pawson and Tilley 1997) 2000’s: Renewed interest in ‘evidence based policy’. Centre for evaluating complexity across the Nexus (CECAN) set up
undertake evaluations building on our action research and systems orientation 1990’s + EDRU (Evaluation, Development and Research Unit) set up to explore innovative evaluation methods. Participative, Theory of Change and Realistic Evaluation approaches incorporated into evaluation projects 2000 + Many national and international evaluation projects undertaken , plus evaluation guides and trainings produced 2015 + ‘Dynamics of evaluation’ workshops developed. Join CECAN as partner and become involved in case study activities 1995: ‘Evaluation’ journal launched 1996: UK Evaluation society set up
Result Before and after design Has the twins’ approach to Mark Zuckerberg achieved the intended result? No Experimental study What works when creating a dating website? X (Inappropriate design) Theory of change What outcome(s), if any, can be attributed to the twin’s approach to Mark Zuckerberg? How an why was this changed produced? No direct effect but indirect effect in the form of an innovative social networking platform transforming how people socialise online.
on the edge of chaos Cause does not = effect I can see a lot of nested systems in here Everywhere I look there are repeated irregular forms It’s all a bit fractal! I’m interested in the emergent Challenge 1: understanding and capturing change
explains how an intervention (a project, a programme, a policy, a strategy) is understood to contribute to a chain of results that produce the intended or actual impacts. Evaluation tests the theory by investigating how what happens in one link affects what happens in other links.
• “I call the combination of program theory and implementation theory the program’s ‘theories of change’” (Weiss, 1995, p.58). • Change is generated not by the programme activities but by “the response the activities generate”(Weiss 1997, p.46) Log frame? Logic model? IMPLEMENTATION PATHWAY?
and hypotheses about how change happens, the way humans work, or organisations, or political systems, or eco-systems. Theory of change is about articulating these many underlying assumptions about how change will happen in a programme.” (Patricia Rogers)
we need to take a systemic perspective on programmes, policies and projects • Images / theory of change maps help us create ‘good enough’ maps • But we need to find ways to not only use linear models to represent complex initiatives
poor childhood attachment (Westhorp 2012). Processes at one system level generate outcomes at another FORMING RELATIONSHIPS SOCIAL JUDGEMENTS ATTACHMENT STYLE Theory of change, complexity and boundaries Layered theories
low level of Fincap Adapted from Ramlingham et al (2014) “From best practice to best fit – Understanding and navigating wicked problems in international development, p.26, ODI Working Paper.
feeding of ‘what works’ knowledge to a virtual network of stakeholders via Financial Capability Steering Groups can develop financial capability practice and influence policy change Continuous development and learning ‘WHAT WORKS’ KNOWLEDGE INFLUENCE NETWORK DEVELOPMENT STRATEGY COORDINATION via Steering Groups Assumptions: MAS’s agency in stimulating a virtual network
knowledge of money management when starting work – a “teachable moment”. Induction will explain salaries, auto enrolment, differences between loans and overdrafts and advice on saving. INFLUENCE Influencing employer networks to makes this part of their management and inductions practices by including it in standards. NETWORK Employer networks pick this issue up and feed it into their members’ HR management e.g. CBI, IoD, FSB, BCC and Unions STRATEGY COORDINATION The strategy communicates knowledge on the relationship between job sustainability and debt management and uses contacts to promote an interest in this Continuous development and learning from effective support for YAs at key points of transition ToC – If effective support is available at key points of transition (e.g. starting work), and Young Adults engage with this support, then it will strengthen their financial capability and financial resilience
indebtedness? • What is our agency? • How do we think change happens? • Assumptions? • Which parts of the system can we plausibly address • What theories can help us?
See facebook story: results may be unpredictable, indirect and only emerge over time • An intervention is prone to adapt to context ─ Different actors interact locally, potentially producing different results • So, how do we plan for and / or analyse results using theory of change?
of a programme influence changes at other levels to influence outcomes and impact in the longer-term.” (Isobel Vogel, 2012. Review of the use of ‘Theory of Change’ in international development. DFID., p. 19)
TP3 Poor awareness of indebtedness TPs – Changed state – from information givers Love to see – like to see – fear to see Improved knowledge and practice Network emerging Testing Development & Learning – Network building More agencies involved in multiple schemes – Shift to policy and legislation
from ‘what works’ knowledge Developing the dissemination strategy with the steering group Disseminating to targeted parts of the virtual network. Could include commissioning by MAS Implementation of contextually adapted practice Evaluation Influencing – changing practice at policy level Practice becomes established at institutional level Strengthened Fincap for YAs starting work ToC – If effective support is available at key points of transition (e.g. starting work), and Young Adults engage with this support, then it will strengthen their financial capability and financial resilience
systems and identifying their component elements provides the boundary within which an evaluator will work and identifies elements about which data might be collected.“ (Westhorp, 2012)
Union's Horizon 2020 programme. It aims to transform the Textiles and Clothing industry, with the objective of bringing 5% of production capacity back to Europe by 2025. http://tcbl.eu/
theory and implementation theory useful to guide design of indicator system • Understanding the theoretical foundations of an intervention as represented in the programme theory helps find most suitable overarching framework for an indicator system • Implementation theory can help shape the set of monitoring indicators
theory primarily drawing on ecosystems thinking: ─ Single indicators or measures are unlikely to be sufficient as it becomes harder to define what ‘success’ is, let alone how to measure it. ─ monitor a range of social, economic and cultural issues as well as ecological issues to allow for adaptive management and learning • Implementation theory reveals ─ Quantitative output, outcome and impact targets (over time) • From that, created multi-dimensional indicator system, including: ─ Key Progress Monitoring Indicators : a list of core outputs specified in the project proposal (and contract) ─ ToC Distance Travelled indicators: provide an indication, at a point in time, of the ‘distance travelled’ by the project in relation to the ‘change journey’ set out in the TCBL baseline Theory of Change.
other units interact locally, results likely to differ by context • Multiple and / or longitudinal case studies to test outcomes and pathways in different settings ─ each case study may have its own theory of change ─ Linked, however, to overarching programme / project / policy ToC • Scenario based ‘effective principles
the programme boundary Boundary partners -- individuals, groups, and organizations directly engaged and who can influence/generate outcomes Identifies changes in the behaviour of people, groups and organizations worked with by development programmes and what those changes result in.
to update programme and implementation theories at regular intervals ─ Incorporating evaluation data • Participative to: ─ Identify possible ‘unanticipated’ or unpredictable outcomes from the beginning and co-construct the story involving as many parts of the system as possible • Iterations become part of the evidence base • Outcomes then judged against decision taken during implementation not the initial theory of change ─ Joint interpretation and sense-making with programme actors to bring multiple perspectives and hence explanations for change
analysis systematically tests plausibility of hypotheses and counter-hypotheses ─ BUT: how far practical to do this on all aspects of a complex programme? • Process tracing focuses on ‘clues’ within a case ─ BUT: does it really work in super complex real time cases? • Agent based modelling can help test hypotheses about causation by simulating complex processes ─ BUT: if we’re simulating we’re not evidencing?
the space of uncertainty? • Working with multiple stakeholders ─ Essential to capture difference ─ Challenge of power dynamics • Fractals ─ Dynamics of the wider system may be found in the team ─ Recognising, acknowledging, working with very sophisticated work. Needs ‘external’ consultant? • Staying at the boundary ─ How far is too far in the system? ─ When are we too far out?
complexity means understanding both – programme theory and implementation theory and see those working in dialogue with each other. • Evaluating from a complexity perspective only has a chance of working if there is a shared acceptance / understanding of what this really means • To an extent it is the role of the evaluator to ‘hold’ the complexity for the client system, partners, others ─ We become change agents ourselves
t_i_h_r
vyadav
sapi_kawahara
masakiokuda
yatabe
iqbocchi
masakat0
matleenalaakso
datascientistsociety
mfcabrera
otsuki
territorium
colly
rkaga
codingconduct
jacobian
seathinner
stephenakadiri
soysaucechin
jonrohan
jdejongh
katarinadahlin
.png){kind=avatar}
helenjbeal
mfonobong
