Data, data and data about your favourite community: The Grimoire Library

Transcript

1. Data, data and data about your favourite community The Grimoire

   Library Daniel Izquierdo-Cortazar dizquierdo@bitergia.com http://twitter.com/dizquierdo Bitergia FOSDEM, Brussels, February 1st, 2015 http://bitergia.com Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 1 / 16

2. c 2012-2015 Bitergia Some rights reserved. This presentation is distributed

   under the “Attribution-ShareAlike 3.0” license, by Creative Commons, available at http://creativecommons.org/licenses/by-sa/3.0/ Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 2 / 16

3. Why this talk? Massive data are produced by open source

   projects How can we take advantage of it? Is it useful to analyze my community? Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 3 / 16

4. GrimoireLib: goals Transparency db layer for Metrics Grimoire Reuse code:

   no need to create once and again the same queries Scalable and modular: a new metric is a new class https://github.com/VizGrimoire/GrimoireLib Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 4 / 16

5. GrimoireLib: other facts Based on Metrics Grimoire output (SQL ddbb)

   Have a look at http://bit.ly/fosdem-grimoire A bit of history: developed in R, migrated to Python Started in 2012 Team of 4/5 developers per month License: GPL v3 or later Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 5 / 16

6. Metrics and studies overview: source code and code review Source

   code (git, svn, hg, etc): usual ones: commits, authors, files, added/removed lines, branches, companies, etc not so usual: demographics, timezone, developers characterization Code Review (gerrit, github): merges, abandoned, submitted patchsets or changesets, people, companies, etc time to close, time waiting for the submitter or reviewer Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 6 / 16

7. Metrics and studies overview: communication channels: mailing lists, IRC, Q&A

   Mailing lists emails, people, companies, hot topics, time to first reply, emails initiating threads, those replying, unanswered posts, timezone analysis Question and answers (stackoverflow, askbot, discourse) top visited questions, labels, people, answers, comments, ... Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 7 / 16

8. Metrics and studies overview: ticketing systems: Bugzilla, Jira, Launchpad, ...

   Tickets: opened and closed tickets, efficiency, time to close tickets time to attend Other data sources: Wikis, Downloads, Releases, Apache logs Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 8 / 16

9. Available filters Filters: general: repository, company, domain, project, people source

   code: branch, module, file type, log message tickets: ticket type Examples: Commits (by company) (and by repo) (and by filetype) Time to close issues (by company) (and by tracker) Top companies (by project) Top developers (by project) (and by company) Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 9 / 16

10. Metrics main methods API: get agg: aggregated numbers get ts:

    evolutionary numbers get list: list of elements (eg authors) get trends: trends for a specific date during the last X days Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 10 / 16

11. How to Import the needed libraries: # Database access from

    vizgrimoire.metrics.query_builder import SCMQuery # Filters to apply from vizgrimoire.metrics.metrics_filter import MetricFilters # Let’s start playing with git activity metrics import vizgrimoire.metrics.scm_metrics as scm Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 11 / 16

12. How to Database access: # Instantiate database access # Playing

    with OpenStack source code database (MySQL) at # http://activity.openstack.org/dash/.../source_code.mysql.7z # Database named as openstack_source_code_fosdem2015 user = "root" password = "" source_code_db = "openstack_source_code_fosdem2015" identities_db = "openstack_source_code_fosdem2015" dbcon = SCMQuery(user, password, source_code_db, identities_db) Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 12 / 16

13. How to Instantiate filters: # Instantiate some filters to play

    with period = MetricFilters.PERIOD_MONTH startdate = "’2014-01-01’" enddate = "’2015-01-01’" # basic filter filters = MetricFilters(period, startdate, enddate) # company and repo filter filters_r = MetricFilters(period, startdate, enddate) filters_r.add_filter(MetricFilters.COMPANY, "Red Hat") filters_r.add_filter(MetricFilters.REPOSITORY, "nova.git") Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 13 / 16

14. How to Instantiate the metric you need # Retrieving data

    for each filter. # Let’s start with authors commits = scm.Commits(dbcon, filters) authors.get_agg() authors.get_ts() authors.get_list() authors.get_trends(filters.enddate, 7) Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 14 / 16

15. This is not the end You can use or contribute

    to *Grimoire Code and issues at: https://github.com/VizGrimoire/GrimoireLib IRC in Freenode at #metrics-grimoire Mailing list: https://lists.libresoft.es/ listinfo/metrics-grimoire Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 15 / 16

16. Data, data and data about your favourite community The Grimoire

    Library Daniel Izquierdo-Cortazar dizquierdo@bitergia.com http://twitter.com/dizquierdo Bitergia FOSDEM, Brussels, February 1st, 2015 http://speakerdeck.com/bitergia Daniel Izquierdo-Cortazar (Bitergia) Data, data and data about your favourite community FOSDEM 2015 16 / 16