Software Analytics with Data Science on Software Data

Data Science has demonstrated its value in extracting insights from business data, raising the question: Why not apply these principles to our software systems’ data? In this talk, I’ll introduce you to the world of Software Analytics. We’ll explore how to extract valuable insights from software data by using tools and techniques from data science to get rid of big, systemic problems in our software systems.

You’ll learn how to leverage scientific thinking, manage the analysis process, and apply literate statistical programming to analyze software systems in an understandable way. Or to put it in the words of software developers: We automate the analysis of large software systems using open-source tools like Jupyter Notebook, Python, pandas, jQAssistant, and Neo4j. I’ll also demonstrate through live-coding how we can gain new insights from data sources like Git repositories, performance measurements, or source code.

Join me in this session to acquire your starter kit for uncovering deeply hidden issues and change the way of improving systems with data-driven software analysis!

Transcript

1. Software Analytics with Data Science on Software Data BOBKONF 2024, 15.03.2024 Markus Harrer Software Evolutionist @ INNOQ Social: markusharrer.de
2. The original horror show IMAGE: sethJreid/Pixabay LEGACY systemS
3. According to https://pkruchten.files.wordpress.com/2013/12/kruchten-colours-yow-sydney.pdf, Icons from https://game-icons.net/, CC BY 3.0 license
5. Adapted from Ray Koopa https://commons.wikimedia.org/wiki/File:Lcars_wallpaper.svg (CC BY-SA 4.0) THE ULTIMATE QUALITY DASHBOARD SHIPPING APP DEVELOPER'S SANITY PROJECT'S BUDGET
6. THE
10. A NEW HOPE
11. Frequency Questions Importance Answering YOUR SPECIFIC questions Use standard tools for general questions Option 2: Use Software Analytics to answer your very specific questions! Option 1: Just ignore the other questions
12. JUDGMENT DAY Typical ISSUES TO TERMINATE  Spotting parts in source code no one knows of anymore  Finding root causes of performance bottlenecks  Identifying alternative modularization options  Showing the progress of long-living restructurings  Measuring the community activity around open source software  <your very specific analysis in your very specific situation>
13. THE SOFTWARE DATA OF OF DEVELOPERS
14. Substantial expertise Data Science Data Science Venn diagram by Drew Conway (simplified)
15. = A WAY TO IMPLEMENT SOLID SOFTWARE ANALYTICS R E P R O D U C I B L E D A T A S C I E N C E open comprehensible systematic automated
17. 3 Python 1 ... and matplotlib, numpy, scikit-learn, NLTK, Pygments, py2neo, requests, BeautifulSoup, Pygal ...
18. code and data in love Computational Notebook
19. Computational Notebook Jupyter Notebook Context Idea Analysis Conclusion Data-driven Software Analysis
20. Literate Programming with Jupyter Notebook
21. Attribution: Tobias ToMar Maier, https://commons.wikimedia.org/wiki/File:VHS_tape_with_time_scale.jpg Demo I Miami Cops Police Academy
22. jQAssistant Neo4j Graph Data Science
23. :Class :Method :Field https://github.com/JavaOnAutobahn/spring-petclinic public class Pet { private LocalDate birthDate; public LocalDate getBirthDate(){ return this.birthDate; } public void setBirthDate(LocalDate birthDate){ this.birthDate = birthDate; }
24. :Class :Method :Field :Entity @Entity @Table(name = "pets") public class Pet { https://github.com/JavaOnAutobahn/spring-petclinic
25. :Class Business Subdomain :Method :Field 2 findings 5 changes :Entity 100% usage birthDate name
26. 16 types 17 findings 15 changes 70% usage 5 types 39 findings 51 changes 80% usage A perspective, where also managers can reason about!
27. Attribution: Tobias ToMar Maier, https://commons.wikimedia.org/wiki/File:VHS_tape_with_time_scale.jpg Demo II Terminator Flash Gordon
28. The return of reason The Two Tips The fellowship of the bling A u t o m a t i o n Meta Metric Number of solved problems NO Tool To Rule Them All O p e n e s s become the of the T by analyzing software in a data-driven way
29. ASK ' EM ALL @feststelltaste
30. Appendix
31. Jupyter notebook, python, pandas, matplotlib Repo https://github.com/feststelltaste/software-analytics/tree/master/demos/20240315_BOBKonf_2024 jQAssistant & Neo4j Repo Spring PetClinic https://github.com/JavaOnAutobahn/spring-petclinic Demos
32. More on Software Analytics softwareanalytics.de
33. More on Software Analytics Adam Tornhill: Software X-Ray Tim Menzies, Laurie Williams, Thomas Zimmermann: Perspectives on Data Science for Software Engineering Christian Bird, Tim Menzies, Thomas Zimmermann: The Art and Science of Analyzing Software Data
34. More on Data Science Jeff Leek: The Elements of Data Analytic Style Roger D. Peng: Report Writing for Data Science in R Wes McKinney: Python for Data Analysis
35. More on Graph Analytics Mark Needham & Amy Hodler: Graph Algorithms https://neo4j.com/product/graph-data-science/
36. Paper about jQAssistant/Neo4j https://easychair.org/publications/preprint/893N
37. Thank you very much! innoQ Deutschland GmbH Krischerstr. 100 40789 Monheim am Rhein Germany +49 2173 3366-0 Ohlauer Str. 43 10999 Berlin Germany Ludwigstr. 180E 63067 Offenbach Germany Kreuzstr. 16 80331 Munich Germany Gewerbestr. 11 CH-6330 Cham Switzerland +41 41 743 01 11 Albulastr. 55 8048 Zurich Switzerland innoQ Schweiz GmbH Markus Harrer markus.harrer@innoq.com @feststelltaste