Ph.D. Thesis: A Model To Transfer Knowledge From Research To Software Engineering Practice Based On Rapid Reviews And Evidence Briefings

Transcript

1. A MODEL TO TRANSFER KNOWLEDGE FROM RESEARCH TO SOFTWARE ENGINEERING PRACTICE BASED ON RAPID REVIEWS AND EVIDENCE BRIEFINGS Advisor: Sérgio Soares Co-Advisor: Gustavo Pinto Bruno Cartaxo Ph.D. Thesis 1
2. MUCH HAS BEEN DISCUSSED ABOUT THE IMPORTANCE OF MAKING SOFTWARE ENGINEERING RESEARCH CLOSER TO PRACTICE 2
4. WE ARE TRANSITIONING TO SOME BELIEVE A NEW ERA IN SOFTWARE ENGINEERING RESEARCH 4
6. IT IS NOT A COINCIDENCE THAT THE GOAL OF EVIDENCE BASED SOFTWARE ENGINEERING STRONGLY TIED TO PRACTICE 6
8. HOWEVER.... MANY ARE CLAIMING THAT SOFTWARE ENGINEERING RESEARCH LACKS CONNECTION WITH PRACTICE IN PARTICULAR SECONDARY STUDIES 8
9. HOW SECONDARY STUDIES IN SOFTWARE ENGINEERING COVER THE ISSUES PRACTITIONERS FACE IN PRACTICE? GRQ1 9
10. TO ANSWER THAT QUESTION, WE HAVE TO • IDENTIFY SECONDARY STUDIES • IDENTIFY PRACTITIONERS ISSUES • COMPARE THEM 10
11. IDENTIFYING SECONDARY STUDIES (SSs) 2009 2010 2011 11
12. Over 100 high-quality, professional Question & Answer (Q&A) websites IDENTIFYING PRACTITIONERS ISSUES 12
14. ES1 - REPOSITORY MINING SSs SELECTION SEARCH STRINGS EXTRACTION STACK EXCHANGE WEBSITES SELECTION SEARCH FOR PRACTITIONERS ISSUES FALSE POSITIVES EXCLUSION COVERAGE ANALYSIS 14
15. Secondary Studies Selection 120 SSs from tertiary studies1,2,3 32 SSs with guidelines1 24 Selected SSs Practitioners guidelines filter Search string filter 1. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011 2. Kitchenham et al. Systematic literature reviews in software engineering - a systematic literature review. IST, 2009 3. Kitchenham, et al. Systematic literature reviews in software engineering – a tertiary study. IST, 2010 15
16. Stack Exchange Websites Selection • Programmers (PROG) • Project Management (PM) • Quality Assurance & Testing (SQA) • Reverse Engineering (RE) • Software Recommendations (SREC) Search for Practitioners Issues 46,743 Issues 26,687 Issues 1,860 Issues 424 Selected Issues Questions’ score filter Search strings filter False-positive filter 16
17. Coverage Analysis 17
18. OVERALL COVERAGE 14.1% 18
19. SOFTWARE ENGINEERING AREAS COVERAGE 19
20. RECURRENT ISSUES 1. Applicability of agile in specific project context (19 issues) 2. Introducing and adapting a software development process (14 issues) 3. Strategies to cost and effort estimation in specific contexts (13 issues) 4. Mixing agile with traditional methods/practices (10 issues) 5. Tools to support software testing with specific features (10 issues) 20
21. IN SUMMARY The results suggest the existence of a gap between SSs and SE practice • 1.75% of most relevant practitioners issues are related to the selected SSs • 14.1% of the 424 practitioners issues related to the selected SSs are covered • 45 recurrent issues identified Guidelines for conducting SSs considering practitioners issues 21
22. Part of the results of this coverage analysis were PUBLISHED in the 14th International Conference on Mining Software Repositories (MSR 2017) The extended version presented in here was submitted and is UNDER REVIEW in the Journal of Brazilian Computer Society (JBCS) 22
23. IN THIS CONTEXT THE GOAL OF THIS THESIS IS... 23
24. To propose, evaluate, and discuss a model to transfer knowledge from secondary studies to software engineering practice 24
26. The model we are proposing was published in Information and Software Technology journal 26
27. MEDIUMS TO TRANSFER KNOWLEDGE Some researchers in medicine argue that the final reports of SSs are often neglected1 This has led to many proposals of alternative mediums to transfer knowledge that better fit practitioners' needs, than the traditional research paper format1,2,3 1. Khangura et al. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 2012 2. Chambers et al. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012 3. Young et al. A guide for developing plain-language and contextual summaries of systematic reviews in agri-food public health. Foodborne pathogens and disease, 2014 27
28. Evidence Summary1 Briefing2 Contextual Summary3 1. Khangura et al. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 2012 2. Chambers et al. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012 3. Young et al. A guide for developing plain-language and contextual summaries of systematic reviews in agri-food public health. Foodborne pathogens and disease, 2014 28
29. Defined by a design specialist researcher • Best Practices1 • Gestalt Principles2 • Information Design Principles3 1. Ambrose et al. Basics Design 08: Design Thinking. 2009 2. Lupton et al. Graphic Design: The New Basics. Princeton Architectural Press, 2015 3. Tondreau. Layout Essentials: 100 Design Principles for Using Grids. Rockport Publishers, 2011 Elton Vieira EVIDENCE BRIEFING 29
30. 1. The title of the briefing 2. The goal of the briefing 3. The findings extracted from the original review 4. An informative box with general information 5. The reference to the original review 6. The logos of our research group and university Microsoft Word and Adobe InDesign Templates under CC-BY http://cin.ufpe.br/eseg/briefings EVIDENCE BRIEFING 30
31. WHAT ARE THE PERCEPTIONS OF RESEARCHERS AND PRACTITIONERS ON EVIDENCE BRIEFINGS AS TRANSFER MEDIUMS? GRQ2 31
32. • What are the PRACTITIONERS perceptions on Evidence Briefings format and content? • What are the RESEARCHERS perceptions on Evidence Briefings format and content? • What are the RESEARCHERS perceptions on the process of CREATING Evidence Briefings? 32
33. 120 Secondary Studies1,2,3 32 Seconadary Studies with guidelines1 24 Secondary Studies with search string 12 Selected Secondary Studies Practitioners guildelines filter Search string filter Random Filter 1. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011 2. Kitchenham et al. Systematic literature reviews in software engineering - a systematic literature review. IST, 2009 3. Kitchenham, et al. Systematic literature reviews in software engineering – a tertiary study. IST, 2010 33
34. What are the PRACTITIONERS perceptions on Evidence Briefings format and content? INSTRUMENT Questionnaire with open and closed questions POPULATION 473 Stack Exchange users 146 (30.8%) users provide contact in their public profile SAMPLE 32 (21.9%) out of 146 contacted Stack Exchange users LIMITATIONS • Some Stack Exchange users could be contacted • Traditional paper format was not compared to Evidence Briefings ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 34
35. S1Q5. How often do you use Stack Exchange websites? S1Q6. How often do you read software engineering research papers? S1Q7. Have you ever read a secondary study paper? MEDIUMS TO ACQUIRE KNOWLEDGE ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 35
36. S1Q15. How easy do you think it was to find the information in the briefing? S1Q16. Is the briefing interface clear and understandable? S1Q17. Does the briefing look reliable? PERCEPTIONS ON BRIEFINGS’ FORMAT ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 36
37. What are the RESEARCHERS perceptions on Evidence Briefings format and content? INSTRUMENT Questionnaire with open and closed questions POPULATION 22 authors of the 12 selected secondary studies SAMPLE 7 (31%) out of 22 authors that answered the questionnaire LIMITATIONS • Some authors could not be contacted • The bad design case was not compared to Evidence Briefings ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 37
38. S2Q1. How important for you is to share research results to practitioners? S2Q2. How often do you share research results to practitioners? SHARING RESEARCH WITH PRACTITIONERS ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 38
39. S2Q4. How does the briefing that we sent to you cover the main findings of your paper? PERCEPTIONS ON BRIEFINGS’ CONTENTS *S2Q5. If you have answered “poor” or ”very poor” in the previous question, please describe why is it the case. NONE ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 39
40. S2Q6. How easy do you think it was to find the information in the briefing? S2Q7. Is the briefing interface clear and understandable? S2Q8. Does the briefing look reliable? PERCEPTIONS ON BRIEFINGS’ FORMAT ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 40
41. SBCARS 2017 SBES 2017 SAST 2017 41
42. What are the RESEARCHERS perceptions on the process of CREATING Evidence Briefings? INSTRUMENT Questionnaire with open and closed questions POPULATION Authors of the 115 papers published in EASE 2017, SBES 2017, SBCARS 2017, and SAST 2017 SAMPLE 44 (38%) out of 115 authors that answered the questionnaire LIMITATIONS • Authorship of some papers published in those conferences • Later invitation to EASE authors ES4 - SURVEY ON THE PERCEPTIONS CREATING 42
43. S3Q5: How much time was needed to create the briefing? PERCEPTIONS ON EVIDENCE BRIEFING CREATION S3Q6: How difficult was to create the briefing? ES4 - SURVEY ON THE PERCEPTIONS CREATING 43
44. S3Q8: How do you evaluate your final briefing? S3Q10: Was one single page enough to summarize your results? PERCEPTIONS ON EVIDENCE BRIEFING CREATION ES4 - SURVEY ON THE PERCEPTIONS CREATING 44
45. IN SUMMARY • Researchers want to transfer knowledge to practice but not all of them do so • Some researchers consider scientific writing is an effective way to transfer knowledge to practice, but few practitioners read research papers • Both researchers and practitioners positively evaluated the Evidence Briefings • Effort to create Evidence Briefings is non-trivial but researchers still are willing to create them 45
46. IMPLICATIONS • Researchers • Practitioners • Conference Chairs 46
47. The results of ES2 and ES3 were PUBLISHED in the 10th International Conference on Empirical Software Engineering and Measurement (ESEM 2016). The extended version, with the results of ES4, was submitted and is UNDER REVIEW in the Empirical Software Engineering (EMSE) journal. 47
48. ALTERNATIVE WAYS TO PRODUCE KNOWLEDGE Some researchers in medicine argue that Secondary Study do not consider some of the important practitioners constraints, like time and problem-orientation 1 This has led to many proposals of alternative ways to produce knowledge that better fit practitioners needs1,2,3 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2016 48
49. RRs are lightweight Secondary Studies (SSs) focused on delivering evidence to practitioners in a timely manner 1,2 Some steps of Full Secondary Studies (FSSs) are deliberately omitted or simplified in RRs to achieve their proposed goal RAPID REVIEWS 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2016 49
50. ON RAPID REVIEWS IN MEDICINE THERE IS A GROWING INTEREST... 50
51. 56% of rapid reviews were conducted on the last 3 years 51
53. CHARACTERISTICS • They reduce costs of heavyweight methods • They deliver evidence in a timely manner • They are performed in close collaboration with practitioners • They aim to provide evidence to issues practitioners are facing in practice • They report results through appealing mediums 53
54. WHAT ARE THE PERCEPTIONS OF PRACTITIONERS ON THE KNOWLEDGE PRODUCED WITH RAPID REVIEWS AS TRANSFER OBJECTS? GRQ3 54
55. ES5 - ACTION RESEARCH DIAGNOSIS Semi-Structured Interview Low Customer Collaboration PLANNING RR Protocol Definition RR planned to one week INTERVENTION 6 days of RR Execution Workshop to Present Results EVALUATION Introduction Interview Adoption Interview REFLECTION Input to the 2nd RR COMPANY Applied-research institute in Recife, Brazil PROJECT Development of a system to monitor reusable packages during the entire production chain of the automotive industry PARTICIPANTS • Projects coordinator - MSc + 15 years of experience • Project manager - MSc + 2.5 years of experience LIMITATIONS • Perceptions vs. Effects • Collective Interviews 55
56. ES6 - ACTION RESEARCH DIAGNOSIS Semi-Structured Interview Low Team Motivation PLANNING RR Protocol Definition RR planned to one week INTERVENTION 8 days of RR Execution Workshop to Present Results EVALUATION Introduction Interview REFLECTION Synthesis of what was learned in the two RRs COMPANY Software development company in Recife, Brazil PROJECT Development of a platform to create and publish e-books for basic schools in the entire Brazilian territory PARTICIPANTS • IT director - MSc + 12 years of experience • Project manager - BSc + 8 years of experience LIMITATIONS • Perceptions vs. Effects • Collective Interviews • No Adoption Interview 56
57. Perceptions on Introducing Rapid Reviews 57
58. Perceptions on Introducing Rapid Reviews 58
59. Perceptions on Introducing Rapid Reviews 59
60. Perceptions on Introducing Rapid Reviews 60
61. Perceptions on Introducing Rapid Reviews 61
62. Perceptions on Adopting Rapid Reviews Evidence WHAT WORKED • Some strategies were successfully applied • Evidence Briefing as a reference material • Evidence Briefing to discuss with team members • Motivated them to seek additional knowledge WHAT DID NOT WORK • Some strategies turned out to be not applicable ◦ e-collaboration tools ◦ Just Demos 62
63. Asking the Right Questions Exploratory Questions • ES5-RQ4: What are the strategies to improve customer collaboration in software development practice? • ES6-RQ2: What are the strategies to enhance software development teams motivation? Motivational Questions • ES5-RQ1: What are the benefits of customer collaboration in software development practice? • ES5-RQ2: What are the problems caused by low customer collaboration in the software development practice? 63
64. The results of ES5 were PUBLISHED at the 22th International Conference on Evaluation and Measurement in Software Engineering (EASE 2018) 64
65. “it may be time to reverse the perspective and start from the consumer’s perspective: for practitioners of software engineering, what problems, not yet satisfactorily answered by software engineering theory, could benefit, in the search for answers, from empirical studies?” Bertrand Meyer, 2018 IN CONCLUSION... 65
66. Meyer’s voice certainly is not alone Many other researchers are starting to recognize practice orientation as the next long way ahead1,2,3,4 Unfortunately, Considering the results of our coverage analysis Today we are just scratching the surface of the problem 1.75% of coverage considering 26,000+ most relevant practitioners issues 1. Beecham, et al. Making software engineering research relevant. Computer, 2014 2. Duarte et al. Patterns of cooperative technology development and transfer for software-engineering-in-the-large. SERIP, 2015 3. Laird et al. Transferring software engineering research into industry: The stevens way. SERIP, 2015 4. Santos et al. Motivation to perform systematic reviews and their impact on software engineering practice. ESEM, 2013 66
67. Based on the, mostly positive, results of six empirical studies We believe the model we proposed, enables researchers to get closer to practice And considering the inherent limitations of RRs and EBs 67
68. FUTURE DIRECTIONS • More RRs in SE • Compare RRs with FSSs • Perception of researchers on RRs • Impact of RRs in practice • Creation of EBs templates for different audiences • Graphical synthesis • Coverage analysis for specific SE areas 68
69. 1 journal paper was published at IST 3 conference papers were published at MSR, ESEM, and EASE 3 doctoral symposium paper was published at ESEM, EASE, and ESELAW 2 journal papers are under review at EMSE and JBCS PUBLICATIONS RELATED TO THIS THESIS PUBLICATIONS OUT OF THE SCOPE OF THIS THESIS 2 conference papers were published at ESEM and ESELAW 2 conference papers were published at EASE 69
71. THANK YOU