The Making of a New-Age Evaluator: How evaluations can be conceptualised & practiced

TIHR Lunchtime Talk – 3rd September 2014

Ravinder Kumar
Sr. Research Fellow, Natural Resources Institute, University of Greenwich

Transcript

1. The Making of a New-Age Evaluator: How evaluations can be conceptualised & practiced Lunchtime Talk – 3rd September 2014 Ravinder Kumar Sr. Research Fellow, Natural Resources Institute, University of Greenwich
2. NRI's  mission  is  to  provide  dis1nc1ve,   high  quality  and  relevant  research,   consultancy,  teaching  and  advice  in   support  of  sustainable  development,   economic  growth  and  poverty   reduc1on.”  
3. Context of Evaluations -1 —  Evaluations are for social change as they seek to improve decision making, resource allocations and accountability at all levels —  It seems that evaluations have partially served the society with these functions This situation can be very different for next 50 years of development co-operation…
4. Context of Evaluations -2 Overall quality of USAID evaluation reports (2013) — Average Overall Score: 5.93 out of 10; Historical Comparison: 1983 USAID Meta-Evaluation: Average Score: 53.8 out of 100 27 UNIFEM evaluations conducted during 2004 and 2008 — 8% excellent; 50% good and rest were average or weak evaluations Aid for Trade – Meta Evaluation 2011 — …Evaluations of aid-for-trade operations do not say much about trade…assessed project implementation /operations without medium and long term impacts… Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) — The full potential benefit of humanitarian evaluations is not being realised…
5. Evaluation Ecosystem Commissioners, doers, regulators, recipients and users of evaluations COMMISSIONING EVALUATION POLICY EVALUATION RECEPIENT Overall Intent, TOR, F r a m i n g e v a l u a t i o n Questions, Budgets, RFP process, Hiring process, engagement process Supporting policy and funding s t r e a m s ; p l u r a l i t y o f methodologies, meta evaluation Leadership support, internal demand for e v a l u a t i o n s , O r g . structure, Mgt. response, Champions DESIGN & DELIVERY EVALUATION USERS A p p r o p r i a t e n e s s & timeliness of the process of design and deliver y; Evaluator’s capacities and roles How evaluation use is broad-based, with users at different levels (thematically & geographically) EVALUATION+ EVALUATION SOCIETIES EVALUATION OF EVALUATION What happens once evaluation repor t is delivered – dissemination, follow on engagements R e g u l a t i o n F u n c t i o n ; Professionalization, Capacity building, evaluation advocacy, stimulating demand for evidence Improving evaluation profession through meta evaluation and tracking of use What and Who influences Evaluation Quality?
6. The Questions that arises… Given that evaluators are just one of the axis of change, that soon fade away… — Can evaluators really influence the evaluations in terms of quality and use — Can evaluators really play multiple and independent roles during and after evaluations… — How can evaluators deal with ‘tricky situations’ — How differently evaluations need to conceptualised and practiced, wherein evaluators can actually play multiple and independent roles
7. Evaluators’ Role is currently determined by… —  Methods and tools —  Evaluation model —  Relationship of evaluator with the commissioner or recipient —  Situational response during the evaluation process All this leads to methodological-role myopia and inflexibility
8. How roles gets compromised? Evaluators becoming technical servant to the methodologies rather than demonstrating servant-leadership in the evaluation process…
9. Situations that we encounter-1 Evaluation being done as a routine, customary process, not expected to go further than a report —  Expected role: Supplier, Researcher, Synthesiser, —  Can one play other roles like collaborator, facilitator, designer, influencer etc. —  Lesson: Create an engagement process at different stages of evaluation, for stakeholders to appreciate the value of evaluation —  My relevant experience: social mobilisation, urban governance and SME projects
10. Situations that we encounter-2 Evaluation designs are supplied a-priori with little flexibility to evaluator or users to influence the design or the process —  Expected role: Supplier, Researcher, Synthesiser, —  Can one play other roles like engager, educator, designer, mediator —  Lesson: Create a pre-evaluation step in the evaluation for consultation and co-development of the evaluation design —  My relevant experience: Cassava, MSME evaluations
11. Situations that we encounter-3 Recipients or funders of the evaluations are expecting collusion —  Expected role: Political manager, Advocate of the client —  Can one play other roles like neutral player, sincere facilitator and mediator etc. —  Lesson: Given this challenging context, either opt out or identify champions within the client organisations who are interested in the real story and work with them —  My relevant experience: Watersheds, Rural Development Department of Government, Forest Department
12. Situations that we encounter-4 In an internal evaluation, top management intends to prove impacts at all costs — Expected role: Expert who can vouch for the findings, Selective researchers who can ‘play’ with data — Can one play other roles like activist, investigator, destroyer /transformer, sincere facilitator and mediator etc. — Lesson: Leave the stage for others who are willing and / or are capable to manage it — My relevant experience: Private sector organisation, HIV/ AIDS Government and Civil society organisation (in two cases, I excused myself, and in one case, became an activist to stop the project as serious problems were witnessed)
13. Situations that we encounter-5 Evaluations are under-funded with large expectations —  Expected role: Researcher, Facilitator, Designer, Advocate —  Can one play these expected roles sufficiently well, given the resource limitations —  Lesson: Make a good choice - Take limited numbers of such strategic works and contribute to the expected roles fully, with pro-bono time —  My relevant experience: Climate adaptations, Natural Resource management, Water sector evaluations (these are among the best evaluations that I have carried out)
14. Situations that we encounter-6 Where ‘difficult context’ becomes standard excuse for under- performance — Expected role: Benign Researcher, Supplier — Can one play the devil’s advocate in such situations — Lesson: Persuade the client /recipient /user of evaluation to organise post-evaluation series of events for broad- based consensus building on evaluation findings — My relevant experience: Agriculture – Public sector evaluations
15. Situations that we encounter-7 When, despite everything going right, evaluations still fall short on programme or policy impact —  Expected role: Facilitator, Collaborator —  Can one play the roles of designer and disseminator —  Lesson: Persuade the client /recipient /user of evaluation to publish the evaluation report or allow for you to publish, share widely in relevant fora —  My relevant experience: Cotton, Social Protection, Teacher Absenteeism
16. Performing a larger role than expected - what makes it difficult? —  Going beyond the ‘remit’ —  Funding constraints —  Institutional constraints —  Capacities of evaluator —  Contractual obligations (esp. non-disclosure related) —  Methodological driven role myopia —  Intellectual arrogance of evaluator —  Stage-management of the evaluation process and evaluators’ naivetés or inability to prevent this —  Payments of evaluators withheld!
19. Can new-age evaluator become the norm? —  Evaluators have power, and with it comes servant- leadership responsibilities —  Three primary roles and multiple secondary roles in various stages of evaluations can be performed when conditions are available —  As shown in the paper, independence and conditions can be developed as well through additional process steps in an evaluation —  However it does not always remain within the evaluator’s sphere of control or decision-making —  A new paradigm of evaluation ecosystem may make it easier for evaluator to play their desired roles effectively…
23. THANK YOU Ready to Jump!