GrimoireLab: A tool for Open Development Analytics

Talk at Linux Foundation Collab Summit (Tahoe, CA, USA, March 30th 2016). Current state of GrimoireLab, and its relationship with Open Development Analytics.

Transcript

1. GrimoireLab A toolset for Open Development Analytics Jesus M. Gonzalez-Barahona jgb@bitergia.com @jgbarah Bitergia / LibreSoft (URJC) Linux Foundation Collaboration Summit Lake Tahoe (CA, USA), March 30th 2016 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 1 / 36
2. Structure of the presentation 1 A bit of context 2 A personal story 3 Open Development Analytics 4 MetricsGrimoire 5 GrimoireLab 6 Preview Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 2 / 36
3. A bit of context Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 3 / 36
4. Me and my two hats Uni Rey Juan Carlos: LibreSoft research team Understanding free, open source software Data analytics approach Bitergia: From research to the real world Understanding software development Data analytics approach http://gsyc.es/~jgb Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 4 / 36
5. The company The software development analytics company dashboards reports consultancy ... http://bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 5 / 36
6. A personal story Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 6 / 36
7. Counting lines of code http://www.dwheeler.com/sloc/redhat62-v1/redhat62sloc.html Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 7 / 36
8. Counting lines of code (revisited) http://gsyc.es/~mortuno/articulos/counting_potatoes.pdf Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 8 / 36
9. Key factors We could answer an interesting question, with data analytics (data, methodology) The software was FLOSS (sloccount) We could reproduce the study easily We could work on top of it easily We could apply new ideas We could improve the software We could automate everything Applying free, open source software principles to analytics Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 9 / 36
10. Open Development Analytics Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 10 / 36
11. Software development “the open source way” Common set of tools (diverse, but not that large): git / svn / hg Bugzilla / Jira / GitHub Gerrit Mailman / Gmane / StackOverflow IRC / Slack ... Similar processes: bug fixing coordination using tickets pre-merge code review general discussion in mailing lists support / meetings in IRC / Slack / StackOverflow ... Collection and analysis of data is possible Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 11 / 36
12. From open development to open development analytics Information about code, community, development for open development projects can be retrieved, organized, analyzed Let’s publish analytics results & data Open Development Analytics: A new standard for transparency Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 12 / 36
13. Getting data, making it useful But data lives in software development systems It has to be retrieved... ...from many different systems It has to be analyzed It has to be visualized Open source software platforms for software development analytics Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 13 / 36
14. More information... http://jgbarah.gitbooks.io/evaluating-foss-projects/ Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 14 / 36
15. MetricsGrimoire Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 15 / 36
16. MetricsGrimoire: the current state of affairs Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 16 / 36
17. MetricsGrimoire: supported data sources Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 17 / 36
18. The Grimoire Development Dashboard http://projects.bitergia.com/opnfv/ Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 18 / 36
19. GrimoireLab Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 19 / 36
20. GrimoireLab: a complete redesign Perceval: basic data retrieval Arthur: retrieval orchestration ElasticSearch: storing data Python / Pandas scripts: enrich, analyze, customize the data Kibiter: Kibana fork to interact with the data http://grimoirelab.github.io http://blog.bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 20 / 36
21. GrimoireLab: Architecture git Bugzilla Gerrit Mbox Enrichment cSearch ter Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 21 / 36
22. Perceval: generic data retriever Library with common functionality One backend per software repository Written in Python3 Each backend produces “data items” Data items are commits, tickets, code review processes... Currently implemented backends: Git, Gerrit, Bugzilla, GitHub Issues, mbox, StackOverflow https://github.com/GrimoireLab/perceval Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 22 / 36
23. Perceval: example import perceval git_parser = perceval.backends.git.Git( uri=’http://github.com/torvalds/linux’, gitpath=’some_dir’ ) for item in git_parser.fetch(): print (item["Author"], item["CommitDate"]) Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 23 / 36
24. Arthur: retrieval orchestration Deals with collections of repositories Retrieval queues Can use a pool of nodes Written in Python3 Synchronization using MQ Short-term data persistence: Redis Incremental retrieval https://github.com/GrimoireLab/arthur Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 24 / 36
25. After Arthur: Data enrichment Extraction of first order metrics Computation of higher order metrics Data enrichment chains... ...by combining “processing units” Agnostic about data source / destination Specially tailored to Python/Pandas needs Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 25 / 36
26. ElasticSearch Long term data storage Raw and rich data Supports the data for dashboards Allows for flexible querying Allows for scale Data fed via REST API Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 26 / 36
27. Kibiter Fork of Kibana (the ElasticSearch dashboard) Some specific functionality Intention: convergence with upstream (proposing pull requests) Customization, branding, new widgets Flexible, powerful, easy drill-down https://github.com/GrimoireLab/kibiter Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 27 / 36
28. Preview Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 28 / 36
29. Preview: play with the new dashboards http://s.bitergia.com/db-fosdem16 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 29 / 36
30. Preview: analyzing a GitHub project https://github.com/grimoirelab/use_cases/tree/master/github Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 30 / 36
31. Preview: analyzing a GitHub project (2) https://github.com/grimoirelab/use_cases/tree/master/github Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 31 / 36
32. Preview: analyzing LibreOffice repositories https: //github.com/grimoirelab/use_cases/tree/master/documentfoundation Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 32 / 36
33. Pre-preview: GrimoireLab working system To be presented during OSCON http://flosscommunitymetrics.org Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 33 / 36
34. Stay tuned, more news to come Do you want to join us to develop GrimoireLab? https://blog.bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 34 / 36
35. License c 2016 Bitergia Some rights reserved. This presentation is distributed under the “Attribution-ShareAlike 3.0” license, by Creative Commons, available at http://creativecommons.org/licenses/by-sa/3.0/ Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 35 / 36
36. Credits “Man With Two Hats” Statue by Henk Visch, located in Otawa, Canada Picture by Lezumbalaberenjena in Wikimedia Commons License: Public domain https://commons.wikimedia.org/wiki/File: Man_With_Two_Hats_Ottawa_Statue_by_lezumbalaberenjena.jpg Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 36 / 36