Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

Transcript

1. Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners Bruno Cartaxo [bfsc@cin.ufpe.br] Gustavo Pinto [gustavo.pinto@ifpa.edu.br] Elton Vieira [elton.vieira@olinda.ifpe.edu.br] Sérgio Soares [scbs@cin.ufpe.br]
2. MOTIVATION 2
3. Evidence Based Software Engineering (EBSE) was introduced as a way to integrate the best research evidence with practice1 Some researches argue that systematic reviews lack connection with practice2,3,4 1. Kitchenham et al. Evidence-Based Software Engineering, ICSE 2004 2. Hassler et al. Outcomes of a community workshop to identify and rank barriers to the systematic literature review process, EASE 2014 3. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011 4. Santos and Da Silva. Motivation to perform systematic reviews and their impact on software engineering practice, ESEM 2013 3
4. BACKGROUND 4
5. EBM researchers argue that the systematic review process is time-consuming1 neglects the final report2 They have been developing alternative methods... Rapid Reviews7 Briefings and Summaries3,4 5 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Khangura et al. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 2012 3. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2015 4. Chambers et al. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012
6. 6 56% of rapid reviews were conducted on the last 3 years
7. Some Software Engineering researchers are also starting to discuss ways to better transfer knowledge/technology from research to practice 7
8. GOAL 8
9. Develop and evaluate a medium to transfer knowledge from systematic reviews to software engineering practitioners 9
10. METHOD 10
11. Systematic Reviews Selection Systematic Review Data Extraction Evidence Briefings Generation Evidence Briefings Evaluation 11 1 2 3 4
12. Systematic Reviews Selection Systematic Review Data Extraction Evidence Briefings Generation Evidence Briefings Evaluation 12 1 2 3 4
13. 120 Systematic reviews1 32 Systematic reviews with guidelines1 24 Systematic reviews with search string 12 Selected systematic reviews Practitioners guildelines filter Search string filter Random Filter 13 Systematic Reviews Selection 3 Global software development 2 Agile software development 1 Software testing 1 Software requirements 1 Model based software development 1 Software development productivity 1 Cost and effort estimation 1 Code duplication 1 Software engineering knowledge management 1 1. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011
14. Systematic Reviews Selection Systematic Review Data Extraction Evidence Briefings Generation Evidence Briefings Evaluation 14 1 2 3 4
15. • Paper title • Research goals • Research findings • Paper citation 15 Systematic Review Data Extraction 2 ORIGINAL The effectiveness of pair programming: A meta-analysis BRIEFING The effectiveness of pair programming TEMPLATE This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review. BRIEFING This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review. FINDING 1 One of the most interesting observations is that the pairing up of individuals seems to elevate the junior pairs up to near senior pair performance. FINDING 2 If you do not know the seniority or skill levels of your programmers, but do have a feeling for task complexity, then employ pair programming either when task complexity is low and time is of the essence, or when task complexity is high and correctness is important.
16. Systematic Reviews Selection Systematic Review Data Extraction Evidence Briefings Generation Evidence Briefings Evaluation 16 1 2 3 4
17. Defined by a design specialist researcher • Best Practices1 • Gestalt Principles2 • Information Design Principles3 17 3 Evidence Briefings Generation 1. Ambrose et al. Basics Design 08: Design Thinking. 2009 2. Lupton et al. Graphic Design: The New Basics. Princeton Architectural Press, 2015 3. Tondreau. Layout Essentials: 100 Design Principles for Using Grids. Rockport Publishers, 2011
18. 18 3 Evidence Briefings Generation 1. The title of the briefing 2. The goal of the briefing 3. The findings extracted from the original review 4. An informative box with general information 5. The reference to the original review 6. The logos of our research group and university Microsoft Word and Adobe InDesign Templates under CC-BY http://cin.ufpe.br/eseg/briefings
19. Systematic Reviews Selection Systematic Review Data Extraction Evidence Briefings Generation Evidence Briefings Evaluation 19 1 2 3 4
20. How practitioners and researchers perceive the content and format of the Evidence Briefings? 20 RESEARCH QUESTION 4 Evidence Briefings Evaluation
21. • Platform with high quality professional Q&A (Questions & Answers) communities • Very diverse users distributed all around the world, with wide variety of ages, genders, and professional experience1 • It has been target of many recent empirical studies2,3 21 4 Evidence Briefings Evaluation 1. Developer survey results, 2016. Available at: http://stackoverflow.com/research/developer-survey-2016 2. Mamykina et al. Design lessons from the fastest Q&A site in the west. In SIGCHI, 2011 3. Pinto et al. Mining questions about software energy consumption. MSR, 2014
22. 22 4 Evidence Briefings Evaluation Survey with StackExchange Users INSTRUMENT Questionnaire with open and closed questions POPULATION 473 StackExchange users 146 (30.8%) users provide contact in their public profile SAMPLE 32 (21.9%) out of 146 contacted StackExchange users To improve response rate, we employed majority of the principles listed by Smith et al 1 • Reciprocity • Brevity • Social Benefit2 1. Smith et al. Improving developer participation rates in surveys. In CHASE, 2013 2. Mello et al . Investigating samples representativeness for an online experiment in java code search. In ESEM, 2015
23. 23 4 Evidence Briefings Evaluation 1. Smith et al. Improving developer participation rates in surveys. In CHASE, 2013 2. Mello et al . Investigating samples representativeness for an online experiment in java code search. In ESEM, 2015 Survey with Systematic Reviews’ Authors INSTRUMENT Questionnaire open and closed questions POPULATION 22 authors of the 12 selected systematic reviews SAMPLE 7 (31%) out of 22 authors that answered the questionnaire
24. RESULTS 24
25. Survey with StackExchange Users 25
26. Q1. What is your current position? Q2. How many years of experience do you have in your current position? 26 DEMOGRAPHICS Survey with StackExchange Users Q4. What is your level of educational attainment?
27. Q8. If you answered Yes for the last question, for what reason you read a systematic review paper? Q5. How often do you use StackExchange websites? Q6. How often do you read software engineering research papers? Q7. Have you ever read a systematic review paper? 27 Survey with StackExchange Users MEDIUMS TO ACQUIRE KNOWLEDGE
28. Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange? Q14. How do you compare the answers from the StackExchange community to the findings presented in the briefing? Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing? 28 Survey with StackExchange Users PERCEPTIONS ON BRIEFINGS’ CONTENTS
29. Q15. How easy do you think it was to find the information in the briefing? Q16. Is the briefing interface clear and understandable? Q17. Does the briefing look reliable? 29 Survey with StackExchange Users PERCEPTIONS ON BRIEFINGS’ FORMAT
30. Survey with Systematic Reviews’ Authors 30
31. Q1. How important for you is to share research results to practitioners? Q2. How often do you share research results to practitioners? 31 Survey with Systematic Reviews’ Authors SHARING RESEARCH WITH PRACTITIONERS
32. Q4. How does the briefing that we sent to you cover the main findings of your paper? 32 Survey with Systematic Reviews’ Authors PERCEPTIONS ON BRIEFINGS’ CONTENTS *Q5. If you have answered “poor” or ”very poor” in the previous question, please describe why is it the case. NONE
33. Q6. How easy do you think it was to find the information in the briefing? Q7. Is the briefing interface clear and understandable? Q8. Does the briefing look reliable? 33 Survey with Systematic Reviews’ Authors PERCEPTIONS ON BRIEFINGS’ FORMAT
34. DISCUSSIONS 34
35. MAIN FINDINGS 35 StackExchange users are industry focused (87%), highly educated (83% BSc) and considerably experienced (53% +5ys) The briefings we generated well covered the main findings of the original systematic reviews (72% Good or VGood) Practitioners rarely use research papers (69% never or few), but they value software engineering research (62%) Researchers want to transfer knowledge (100%), but not all of them do so (42% weekly) Researchers and practitioners positively evaluated the evidence briefings “Many thanks [...] This is a good direction! I support this” “What a nice way to put it! Thank you for having taken a look at our paper.” “I think your format captures the essence really well, in a way that practitioners may find useful and understandable!”
36. LIMITATIONS AND THREATS 36 • We are not the authors of the systematic reviews we generated the briefings • 32 practitioners and 7 researchers answered the surveys • Absence of comparison between research paper and briefings
37. CONCLUDING REMARKS 37
38. Proposed Evidence Briefings as medium to transfer knowledge from systematic reviews to practice Evaluated Evidence Briefings with researchers and practitioners Evidence Briefings were positively evaluated by researchers and practitioners Evidence Briefings can play a role on transferring knowledge from systematic reviews to practice Microsoft Word and Adobe InDesign Templates at http://cin.ufpe.br/eseg/briefings FUTURE WORK: • Update this study to SRs until 2016 • Conduct Rapid Reviews together with practitioners 38
39. THANK YOU Bruno Cartaxo bfsc@cin.ufpe.br 39