Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GrimoireLab: A tool for Open Development Analytics

GrimoireLab: A tool for Open Development Analytics

Talk at Linux Foundation Collab Summit (Tahoe, CA, USA, March 30th 2016). Current state of GrimoireLab, and its relationship with Open Development Analytics.

Tweet

More Decks by Jesus M. Gonzalez-Barahona

Other Decks in Technology

Transcript

  1. GrimoireLab A toolset for Open Development Analytics Jesus M. Gonzalez-Barahona

    [email protected] @jgbarah Bitergia / LibreSoft (URJC) Linux Foundation Collaboration Summit Lake Tahoe (CA, USA), March 30th 2016 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 1 / 36
  2. Structure of the presentation 1 A bit of context 2

    A personal story 3 Open Development Analytics 4 MetricsGrimoire 5 GrimoireLab 6 Preview Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 2 / 36
  3. Me and my two hats Uni Rey Juan Carlos: LibreSoft

    research team Understanding free, open source software Data analytics approach Bitergia: From research to the real world Understanding software development Data analytics approach http://gsyc.es/~jgb Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 4 / 36
  4. The company The software development analytics company dashboards reports consultancy

    ... http://bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 5 / 36
  5. Key factors We could answer an interesting question, with data

    analytics (data, methodology) The software was FLOSS (sloccount) We could reproduce the study easily We could work on top of it easily We could apply new ideas We could improve the software We could automate everything Applying free, open source software principles to analytics Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 9 / 36
  6. Software development “the open source way” Common set of tools

    (diverse, but not that large): git / svn / hg Bugzilla / Jira / GitHub Gerrit Mailman / Gmane / StackOverflow IRC / Slack ... Similar processes: bug fixing coordination using tickets pre-merge code review general discussion in mailing lists support / meetings in IRC / Slack / StackOverflow ... Collection and analysis of data is possible Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 11 / 36
  7. From open development to open development analytics Information about code,

    community, development for open development projects can be retrieved, organized, analyzed Let’s publish analytics results & data Open Development Analytics: A new standard for transparency Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 12 / 36
  8. Getting data, making it useful But data lives in software

    development systems It has to be retrieved... ...from many different systems It has to be analyzed It has to be visualized Open source software platforms for software development analytics Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 13 / 36
  9. GrimoireLab: a complete redesign Perceval: basic data retrieval Arthur: retrieval

    orchestration ElasticSearch: storing data Python / Pandas scripts: enrich, analyze, customize the data Kibiter: Kibana fork to interact with the data http://grimoirelab.github.io http://blog.bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 20 / 36
  10. GrimoireLab: Architecture git Bugzilla Gerrit Mbox Enrichment cSearch ter Jesus

    Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 21 / 36
  11. Perceval: generic data retriever Library with common functionality One backend

    per software repository Written in Python3 Each backend produces “data items” Data items are commits, tickets, code review processes... Currently implemented backends: Git, Gerrit, Bugzilla, GitHub Issues, mbox, StackOverflow https://github.com/GrimoireLab/perceval Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 22 / 36
  12. Perceval: example import perceval git_parser = perceval.backends.git.Git( uri=’http://github.com/torvalds/linux’, gitpath=’some_dir’ )

    for item in git_parser.fetch(): print (item["Author"], item["CommitDate"]) Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 23 / 36
  13. Arthur: retrieval orchestration Deals with collections of repositories Retrieval queues

    Can use a pool of nodes Written in Python3 Synchronization using MQ Short-term data persistence: Redis Incremental retrieval https://github.com/GrimoireLab/arthur Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 24 / 36
  14. After Arthur: Data enrichment Extraction of first order metrics Computation

    of higher order metrics Data enrichment chains... ...by combining “processing units” Agnostic about data source / destination Specially tailored to Python/Pandas needs Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 25 / 36
  15. ElasticSearch Long term data storage Raw and rich data Supports

    the data for dashboards Allows for flexible querying Allows for scale Data fed via REST API Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 26 / 36
  16. Kibiter Fork of Kibana (the ElasticSearch dashboard) Some specific functionality

    Intention: convergence with upstream (proposing pull requests) Customization, branding, new widgets Flexible, powerful, easy drill-down https://github.com/GrimoireLab/kibiter Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 27 / 36
  17. Pre-preview: GrimoireLab working system To be presented during OSCON http://flosscommunitymetrics.org

    Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 33 / 36
  18. Stay tuned, more news to come Do you want to

    join us to develop GrimoireLab? https://blog.bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 34 / 36
  19. License c 2016 Bitergia Some rights reserved. This presentation is

    distributed under the “Attribution-ShareAlike 3.0” license, by Creative Commons, available at http://creativecommons.org/licenses/by-sa/3.0/ Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 35 / 36
  20. Credits “Man With Two Hats” Statue by Henk Visch, located

    in Otawa, Canada Picture by Lezumbalaberenjena in Wikimedia Commons License: Public domain https://commons.wikimedia.org/wiki/File: Man_With_Two_Hats_Ottawa_Statue_by_lezumbalaberenjena.jpg Jesus Gonzalez-Barahona (Bitergia) GrimoireLab March 2016 36 / 36