Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ph.D. Thesis: A Model To Transfer Knowledge From Research To Software Engineering Practice Based On Rapid Reviews And Evidence Briefings

Ph.D. Thesis: A Model To Transfer Knowledge From Research To Software Engineering Practice Based On Rapid Reviews And Evidence Briefings

Bruno Cartaxo

May 11, 2018
Tweet

More Decks by Bruno Cartaxo

Other Decks in Research

Transcript

  1. A MODEL TO TRANSFER KNOWLEDGE FROM RESEARCH TO SOFTWARE ENGINEERING

    PRACTICE BASED ON RAPID REVIEWS AND EVIDENCE BRIEFINGS Advisor: Sérgio Soares Co-Advisor: Gustavo Pinto Bruno Cartaxo Ph.D. Thesis 1
  2. MUCH HAS BEEN DISCUSSED ABOUT THE IMPORTANCE OF MAKING SOFTWARE

    ENGINEERING RESEARCH CLOSER TO PRACTICE 2
  3. 3

  4. WE ARE TRANSITIONING TO SOME BELIEVE A NEW ERA IN

    SOFTWARE ENGINEERING RESEARCH 4
  5. 5

  6. IT IS NOT A COINCIDENCE THAT THE GOAL OF EVIDENCE

    BASED SOFTWARE ENGINEERING STRONGLY TIED TO PRACTICE 6
  7. 7

  8. TO ANSWER THAT QUESTION, WE HAVE TO • IDENTIFY SECONDARY

    STUDIES • IDENTIFY PRACTITIONERS ISSUES • COMPARE THEM 10
  9. 13

  10. ES1 - REPOSITORY MINING SSs SELECTION SEARCH STRINGS EXTRACTION STACK

    EXCHANGE WEBSITES SELECTION SEARCH FOR PRACTITIONERS ISSUES FALSE POSITIVES EXCLUSION COVERAGE ANALYSIS 14
  11. Secondary Studies Selection 120 SSs from tertiary studies1,2,3 32 SSs

    with guidelines1 24 Selected SSs Practitioners guidelines filter Search string filter 1. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011 2. Kitchenham et al. Systematic literature reviews in software engineering - a systematic literature review. IST, 2009 3. Kitchenham, et al. Systematic literature reviews in software engineering – a tertiary study. IST, 2010 15
  12. Stack Exchange Websites Selection • Programmers (PROG) • Project Management

    (PM) • Quality Assurance & Testing (SQA) • Reverse Engineering (RE) • Software Recommendations (SREC) Search for Practitioners Issues 46,743 Issues 26,687 Issues 1,860 Issues 424 Selected Issues Questions’ score filter Search strings filter False-positive filter 16
  13. RECURRENT ISSUES 1. Applicability of agile in specific project context

    (19 issues) 2. Introducing and adapting a software development process (14 issues) 3. Strategies to cost and effort estimation in specific contexts (13 issues) 4. Mixing agile with traditional methods/practices (10 issues) 5. Tools to support software testing with specific features (10 issues) 20
  14. IN SUMMARY The results suggest the existence of a gap

    between SSs and SE practice • 1.75% of most relevant practitioners issues are related to the selected SSs • 14.1% of the 424 practitioners issues related to the selected SSs are covered • 45 recurrent issues identified Guidelines for conducting SSs considering practitioners issues 21
  15. Part of the results of this coverage analysis were PUBLISHED

    in the 14th International Conference on Mining Software Repositories (MSR 2017) The extended version presented in here was submitted and is UNDER REVIEW in the Journal of Brazilian Computer Society (JBCS) 22
  16. To propose, evaluate, and discuss a model to transfer knowledge

    from secondary studies to software engineering practice 24
  17. 25

  18. MEDIUMS TO TRANSFER KNOWLEDGE Some researchers in medicine argue that

    the final reports of SSs are often neglected1 This has led to many proposals of alternative mediums to transfer knowledge that better fit practitioners' needs, than the traditional research paper format1,2,3 1. Khangura et al. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 2012 2. Chambers et al. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012 3. Young et al. A guide for developing plain-language and contextual summaries of systematic reviews in agri-food public health. Foodborne pathogens and disease, 2014 27
  19. Evidence Summary1 Briefing2 Contextual Summary3 1. Khangura et al. Evidence

    summaries: the evolution of a rapid review approach. Systematic Reviews, 2012 2. Chambers et al. A framework for production of systematic review based briefings to support evidence-informed decision-making. Systematic reviews, 2012 3. Young et al. A guide for developing plain-language and contextual summaries of systematic reviews in agri-food public health. Foodborne pathogens and disease, 2014 28
  20. Defined by a design specialist researcher • Best Practices1 •

    Gestalt Principles2 • Information Design Principles3 1. Ambrose et al. Basics Design 08: Design Thinking. 2009 2. Lupton et al. Graphic Design: The New Basics. Princeton Architectural Press, 2015 3. Tondreau. Layout Essentials: 100 Design Principles for Using Grids. Rockport Publishers, 2011 Elton Vieira EVIDENCE BRIEFING 29
  21. 1. The title of the briefing 2. The goal of

    the briefing 3. The findings extracted from the original review 4. An informative box with general information 5. The reference to the original review 6. The logos of our research group and university Microsoft Word and Adobe InDesign Templates under CC-BY http://cin.ufpe.br/eseg/briefings EVIDENCE BRIEFING 30
  22. • What are the PRACTITIONERS perceptions on Evidence Briefings format

    and content? • What are the RESEARCHERS perceptions on Evidence Briefings format and content? • What are the RESEARCHERS perceptions on the process of CREATING Evidence Briefings? 32
  23. 120 Secondary Studies1,2,3 32 Seconadary Studies with guidelines1 24 Secondary

    Studies with search string 12 Selected Secondary Studies Practitioners guildelines filter Search string filter Random Filter 1. Da Silva et al. Six years of systematic literature reviews in software engineering: An updated tertiary study, IST, 2011 2. Kitchenham et al. Systematic literature reviews in software engineering - a systematic literature review. IST, 2009 3. Kitchenham, et al. Systematic literature reviews in software engineering – a tertiary study. IST, 2010 33
  24. What are the PRACTITIONERS perceptions on Evidence Briefings format and

    content? INSTRUMENT Questionnaire with open and closed questions POPULATION 473 Stack Exchange users 146 (30.8%) users provide contact in their public profile SAMPLE 32 (21.9%) out of 146 contacted Stack Exchange users LIMITATIONS • Some Stack Exchange users could be contacted • Traditional paper format was not compared to Evidence Briefings ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 34
  25. S1Q5. How often do you use Stack Exchange websites? S1Q6.

    How often do you read software engineering research papers? S1Q7. Have you ever read a secondary study paper? MEDIUMS TO ACQUIRE KNOWLEDGE ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 35
  26. S1Q15. How easy do you think it was to find

    the information in the briefing? S1Q16. Is the briefing interface clear and understandable? S1Q17. Does the briefing look reliable? PERCEPTIONS ON BRIEFINGS’ FORMAT ES2 - SURVEY ON THE PERCEPTIONS OF PRACTITIONERS 36
  27. What are the RESEARCHERS perceptions on Evidence Briefings format and

    content? INSTRUMENT Questionnaire with open and closed questions POPULATION 22 authors of the 12 selected secondary studies SAMPLE 7 (31%) out of 22 authors that answered the questionnaire LIMITATIONS • Some authors could not be contacted • The bad design case was not compared to Evidence Briefings ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 37
  28. S2Q1. How important for you is to share research results

    to practitioners? S2Q2. How often do you share research results to practitioners? SHARING RESEARCH WITH PRACTITIONERS ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 38
  29. S2Q4. How does the briefing that we sent to you

    cover the main findings of your paper? PERCEPTIONS ON BRIEFINGS’ CONTENTS *S2Q5. If you have answered “poor” or ”very poor” in the previous question, please describe why is it the case. NONE ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 39
  30. S2Q6. How easy do you think it was to find

    the information in the briefing? S2Q7. Is the briefing interface clear and understandable? S2Q8. Does the briefing look reliable? PERCEPTIONS ON BRIEFINGS’ FORMAT ES3 - SURVEY ON THE PERCEPTIONS OF RESEARCHERS 40
  31. What are the RESEARCHERS perceptions on the process of CREATING

    Evidence Briefings? INSTRUMENT Questionnaire with open and closed questions POPULATION Authors of the 115 papers published in EASE 2017, SBES 2017, SBCARS 2017, and SAST 2017 SAMPLE 44 (38%) out of 115 authors that answered the questionnaire LIMITATIONS • Authorship of some papers published in those conferences • Later invitation to EASE authors ES4 - SURVEY ON THE PERCEPTIONS CREATING 42
  32. S3Q5: How much time was needed to create the briefing?

    PERCEPTIONS ON EVIDENCE BRIEFING CREATION S3Q6: How difficult was to create the briefing? ES4 - SURVEY ON THE PERCEPTIONS CREATING 43
  33. S3Q8: How do you evaluate your final briefing? S3Q10: Was

    one single page enough to summarize your results? PERCEPTIONS ON EVIDENCE BRIEFING CREATION ES4 - SURVEY ON THE PERCEPTIONS CREATING 44
  34. IN SUMMARY • Researchers want to transfer knowledge to practice

    but not all of them do so • Some researchers consider scientific writing is an effective way to transfer knowledge to practice, but few practitioners read research papers • Both researchers and practitioners positively evaluated the Evidence Briefings • Effort to create Evidence Briefings is non-trivial but researchers still are willing to create them 45
  35. The results of ES2 and ES3 were PUBLISHED in the

    10th International Conference on Empirical Software Engineering and Measurement (ESEM 2016). The extended version, with the results of ES4, was submitted and is UNDER REVIEW in the Empirical Software Engineering (EMSE) journal. 47
  36. ALTERNATIVE WAYS TO PRODUCE KNOWLEDGE Some researchers in medicine argue

    that Secondary Study do not consider some of the important practitioners constraints, like time and problem-orientation 1 This has led to many proposals of alternative ways to produce knowledge that better fit practitioners needs1,2,3 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2016 48
  37. RRs are lightweight Secondary Studies (SSs) focused on delivering evidence

    to practitioners in a timely manner 1,2 Some steps of Full Secondary Studies (FSSs) are deliberately omitted or simplified in RRs to achieve their proposed goal RAPID REVIEWS 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2016 49
  38. 52

  39. CHARACTERISTICS • They reduce costs of heavyweight methods • They

    deliver evidence in a timely manner • They are performed in close collaboration with practitioners • They aim to provide evidence to issues practitioners are facing in practice • They report results through appealing mediums 53
  40. WHAT ARE THE PERCEPTIONS OF PRACTITIONERS ON THE KNOWLEDGE PRODUCED

    WITH RAPID REVIEWS AS TRANSFER OBJECTS? GRQ3 54
  41. ES5 - ACTION RESEARCH DIAGNOSIS Semi-Structured Interview Low Customer Collaboration

    PLANNING RR Protocol Definition RR planned to one week INTERVENTION 6 days of RR Execution Workshop to Present Results EVALUATION Introduction Interview Adoption Interview REFLECTION Input to the 2nd RR COMPANY Applied-research institute in Recife, Brazil PROJECT Development of a system to monitor reusable packages during the entire production chain of the automotive industry PARTICIPANTS • Projects coordinator - MSc + 15 years of experience • Project manager - MSc + 2.5 years of experience LIMITATIONS • Perceptions vs. Effects • Collective Interviews 55
  42. ES6 - ACTION RESEARCH DIAGNOSIS Semi-Structured Interview Low Team Motivation

    PLANNING RR Protocol Definition RR planned to one week INTERVENTION 8 days of RR Execution Workshop to Present Results EVALUATION Introduction Interview REFLECTION Synthesis of what was learned in the two RRs COMPANY Software development company in Recife, Brazil PROJECT Development of a platform to create and publish e-books for basic schools in the entire Brazilian territory PARTICIPANTS • IT director - MSc + 12 years of experience • Project manager - BSc + 8 years of experience LIMITATIONS • Perceptions vs. Effects • Collective Interviews • No Adoption Interview 56
  43. Perceptions on Adopting Rapid Reviews Evidence WHAT WORKED • Some

    strategies were successfully applied • Evidence Briefing as a reference material • Evidence Briefing to discuss with team members • Motivated them to seek additional knowledge WHAT DID NOT WORK • Some strategies turned out to be not applicable ◦ e-collaboration tools ◦ Just Demos 62
  44. Asking the Right Questions Exploratory Questions • ES5-RQ4: What are

    the strategies to improve customer collaboration in software development practice? • ES6-RQ2: What are the strategies to enhance software development teams motivation? Motivational Questions • ES5-RQ1: What are the benefits of customer collaboration in software development practice? • ES5-RQ2: What are the problems caused by low customer collaboration in the software development practice? 63
  45. The results of ES5 were PUBLISHED at the 22th International

    Conference on Evaluation and Measurement in Software Engineering (EASE 2018) 64
  46. “it may be time to reverse the perspective and start

    from the consumer’s perspective: for practitioners of software engineering, what problems, not yet satisfactorily answered by software engineering theory, could benefit, in the search for answers, from empirical studies?” Bertrand Meyer, 2018 IN CONCLUSION... 65
  47. Meyer’s voice certainly is not alone Many other researchers are

    starting to recognize practice orientation as the next long way ahead1,2,3,4 Unfortunately, Considering the results of our coverage analysis Today we are just scratching the surface of the problem 1.75% of coverage considering 26,000+ most relevant practitioners issues 1. Beecham, et al. Making software engineering research relevant. Computer, 2014 2. Duarte et al. Patterns of cooperative technology development and transfer for software-engineering-in-the-large. SERIP, 2015 3. Laird et al. Transferring software engineering research into industry: The stevens way. SERIP, 2015 4. Santos et al. Motivation to perform systematic reviews and their impact on software engineering practice. ESEM, 2013 66
  48. Based on the, mostly positive, results of six empirical studies

    We believe the model we proposed, enables researchers to get closer to practice And considering the inherent limitations of RRs and EBs 67
  49. FUTURE DIRECTIONS • More RRs in SE • Compare RRs

    with FSSs • Perception of researchers on RRs • Impact of RRs in practice • Creation of EBs templates for different audiences • Graphical synthesis • Coverage analysis for specific SE areas 68
  50. 1 journal paper was published at IST 3 conference papers

    were published at MSR, ESEM, and EASE 3 doctoral symposium paper was published at ESEM, EASE, and ESELAW 2 journal papers are under review at EMSE and JBCS PUBLICATIONS RELATED TO THIS THESIS PUBLICATIONS OUT OF THE SCOPE OF THIS THESIS 2 conference papers were published at ESEM and ESELAW 2 conference papers were published at EASE 69
  51. 70