Hitchhikers need free vehicles! Shared repositories for statistical analysis in SBST

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Kapfhammer2016/

Transcript

1. Hitchhikers Need Free Vehicles! Gregory M. Kapfhammer Phil McMinn and Chris J. Wright shared by under a ickr photo BergsPix Creative Commons ( BY-ND ) license
2. Randomization
3. Randomization Inherent in SBST techniques
4. Randomization Inherent in SBST techniques Necessitates careful experiment design
5. Randomization Inherent in SBST techniques Necessitates careful experiment design Statistical analysis of results required!
6. When I say "statistics" ...
9. But, we need statistical analysis!
11. Statistical Analysis shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
12. Statistical Analysis A Hitchhiker's Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
13. Statistical Analysis A Hitchhiker's Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering Arcuri and Briand recommend statistical techniques shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
14. Statistical Analysis A Hitchhiker's Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering Arcuri and Briand recommend statistical techniques Code snippets provided in the R language shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
15. Statistical Analysis A Hitchhiker's Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering Arcuri and Briand recommend statistical techniques Code snippets provided in the R language A tremendous asset to the SBST community! shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
16. Subtleties of Statistical Analysis shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
17. Subtleties of Statistical Analysis Well-meaning researchers may make small mistakes shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
18. Subtleties of Statistical Analysis Well-meaning researchers may make small mistakes Marco Torchiano revealed paradoxical e ect sizes shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
19. Hitchhikers Need Vehicles
20. Hitchhikers Need Vehicles Shared repositories of statistical code
21. Hitchhikers Need Vehicles Shared repositories of statistical code Well-tested implementations of procedures
22. Hitchhikers Need Vehicles Shared repositories of statistical code Well-tested implementations of procedures Additional documentation and guidelines
23. Hitchhikers Need Vehicles Shared repositories of statistical code Well-tested implementations of procedures Additional documentation and guidelines Replication packages for completed analyses
24. Why is This Important?
25. Enhance the Maturity of the SBST Field
26. Supporting Tools and Platforms?
28. Suggestions shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
29. Suggestions Use GitHub to store data and analysis code shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
30. Suggestions Use GitHub to store data and analysis code Create R packages using devtools shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
31. Suggestions Use GitHub to store data and analysis code Create R packages using devtools Reveal your full analysis with RMarkdown shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
32. Suggestions Use GitHub to store data and analysis code Create R packages using devtools Reveal your full analysis with RMarkdown Use "best of breed" tools to support your work! shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
33. Carefully pick your analysis team ...
35. "Hadleyverse" shared by Publicly available photo Hadley Wickham
36. "Hadleyverse" dplyr for fast data manipulation shared by Publicly available photo Hadley Wickham
37. "Hadleyverse" dplyr for fast data manipulation tidyr for disciplined data restructuring shared by Publicly available photo Hadley Wickham
38. "Hadleyverse" dplyr for fast data manipulation tidyr for disciplined data restructuring ggplot2 for impressive data visualization shared by Publicly available photo Hadley Wickham
39. "Hadleyverse" dplyr for fast data manipulation tidyr for disciplined data restructuring ggplot2 for impressive data visualization Or, use the languages and packages you prefer shared by Publicly available photo Hadley Wickham
40. "Hadleyverse" dplyr for fast data manipulation tidyr for disciplined data restructuring ggplot2 for impressive data visualization Or, use the languages and packages you prefer But, seriously, Hadley Wickham's code is awesome! shared by Publicly available photo Hadley Wickham
41. Where do we go next?
43. Let's Talk shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
44. Let's Talk What statistical analysis do you regularly perform? shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
45. Let's Talk What statistical analysis do you regularly perform? What is needed to move the SBST community forward? shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
46. Let's Talk What statistical analysis do you regularly perform? What is needed to move the SBST community forward? What types of vehicles do hitchhikers really need? shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
47. Questions shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
48. Questions Sharing data sets larger than what GitHub supports? shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
49. Questions Sharing data sets larger than what GitHub supports? Use Git Large File Storage (LFS) shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
50. Questions Sharing data sets larger than what GitHub supports? Use Git Large File Storage (LFS) Why don't we release scripts for running experiments? shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license
51. Questions Sharing data sets larger than what GitHub supports? Use Git Large File Storage (LFS) Why don't we release scripts for running experiments? They are often customized. But, yes, we should! shared by under a Stocksnap.io photo Alejandro Escamilla Public Domain license