Meet GrimoireLab: Making your life easier

Meet GrimoireLab: Making your life easier

Presentation at ICSME'18 (Madrid, Sept 25th 2018), tailored mainly to researchers. GrimoireLab is a Python toolset for software development analytics. It can deal with data from 20+ kinds of data sources (from git, GitHub and GitLab to Slack, mailing lists, Jenkins, Bugzilla, Jira or Gerrit) with a uniform Python API. It provides tools for identity management, data enrichment, visualization, and massive data retrieval.

B7081d0131ad47821467b8e81434cf7a?s=128

Jesus M. Gonzalez-Barahona

September 24, 2018
Tweet

Transcript

  1. chaoss.community Meet GrimoireLab Making your life easier ICSME 2018, Madrid

    (Spain), Sep 25 2018 Jesus M. Gonzalez-Barahona
  2. chaoss.community /me Like six years ago I was having coffees

    with the gang of Bitergia founders Involved in the company since then bitergia.com I work at Universidad Rey Juan Carlos... ...researching about software development gsyc.es/~jgb My two hats:
  3. /Bitergia Software Development Analytics for your peace of mind bitergia.com

  4. chaoss.community ... Produce integrated, open source software for analyzing software

    development …. chaoss.community
  5. chaoss.community When you’re collecting, preparing data for your next paper...

  6. chaoss.community When you’re collecting, preparing data for your next paper…

    use some help!
  7. chaoss.community GrimoireLab Software development analytics with free, open source software

    (a CHAOSS project) chaoss.github.io/grimoirelab chaoss.github.io/grimoirelab-tutorial
  8. chaoss.community Software development analytics opnfv.biterg.io chaoss.biterg.io

  9. chaoss.community GrimoireLab architecture

  10. chaoss.community Installing from pip $ python3 -m venv /tmp/gl $

    source /tmp/gl/bin/activate (gl) $ pip install --upgrade pip setuptools wheel (gl) $ pip install grimoirelab (gl) $ grimoirelab -v chaoss.github.io/grimoirelab-tutorial/basics/install.html
  11. chaoss.community Running from Docker images docker run -p 5601:5601 -p

    9200:9200 -v $(pwd)/credentials.cfg:/override.cfg -t grimoirelab/full credentials.cfg: [github] api-token = XXX
  12. chaoss.community Architecture: extraction

  13. chaoss.community Example: Perceval (gl) $ pip install perceval (gl) $

    perceval git https://github.com/grimoirelab/perceval (gl) $ perceval github grimoirelab perceval --sleep-for-rate -t XXXXX ...
  14. chaoss.community Example: Perceval from perceval.backends.core.git import Git repo_url = 'http://github.com/grimoirelab/perceval'

    repo_dir = '/tmp/perceval' repo = Git(uri=repo_url, gitpath=repo_dir) for commit in repo.fetch(): print(commit['data']['commit'])
  15. chaoss.community Producing a dashboard (assumes Elasticsearch at http://localhost:9200, Kibiter at

    http://localhost:5601 ) Prepare: mordred.cfg Prepare: projects.json Maybe: identities.yaml Maybe: menu.yaml (es) mordred -c mordred.cfg chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
  16. chaoss.community Using Perceval to compute metrics • Get items from

    the data source • Store them in a data structure • Compute metrics • You can also use Pandas, of course github.com/chaoss/wg-gmd/blob/master/examples/Code_ Commits.ipynb
  17. chaoss.community Architecture: enrichment

  18. chaoss.community Example: item in enriched index

  19. chaoss.community Computing metrics from enriched indexes • Connect to Elasticsearch

    indexes (produced with p2o, mordred) • Query them to get items of interest • Compute metrics • You can also use Pandas, of course chaoss.github.io/grimoirelab-tutorial/python/indexes.html
  20. chaoss.community Architecture: exploitation

  21. chaoss.community Example: producing a dashboard (es) $ pip install grimoire-mordred

    (es) $ mordred -c mordred.cfg (assumes Elasticsearch at http://localhost:9200, Kibiter at http://localhost:5601 ) Needs: mordred.cfg, projects.json, identities.yaml, menu.yaml chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
  22. chaoss.community GrimoireLab Software development analytics with free, open source software

    (a CHAOSS project) chaoss.github.io/grimoirelab chaoss.github.io/grimoirelab-tutorial
  23. chaoss.community License & Attributions “Pain Relief”, by Nick Youngson, Alpha

    Stock Images, CC by-sa 3.0 “Broken Glass Sadness Disappointment Can't Sad”, from Max Pixel, CC0 Slides licensed under Creative Commons ShareAlike 4.0 (CC by-sa 4.0) Copyright CHAOSS & Jesus M. Gonzalez-Barahona 2018