Meet GrimoireLab: Making your life easier

Presentation at ICSME'18 (Madrid, Sept 25th 2018), tailored mainly to researchers. GrimoireLab is a Python toolset for software development analytics. It can deal with data from 20+ kinds of data sources (from git, GitHub and GitLab to Slack, mailing lists, Jenkins, Bugzilla, Jira or Gerrit) with a uniform Python API. It provides tools for identity management, data enrichment, visualization, and massive data retrieval.

Transcript

1. chaoss.community Meet GrimoireLab Making your life easier ICSME 2018, Madrid (Spain), Sep 25 2018 Jesus M. Gonzalez-Barahona
2. chaoss.community /me Like six years ago I was having coffees with the gang of Bitergia founders Involved in the company since then bitergia.com I work at Universidad Rey Juan Carlos... ...researching about software development gsyc.es/~jgb My two hats:
3. /Bitergia Software Development Analytics for your peace of mind bitergia.com
4. chaoss.community ... Produce integrated, open source software for analyzing software development …. chaoss.community
5. chaoss.community When you’re collecting, preparing data for your next paper...
6. chaoss.community When you’re collecting, preparing data for your next paper… use some help!
7. chaoss.community GrimoireLab Software development analytics with free, open source software (a CHAOSS project) chaoss.github.io/grimoirelab chaoss.github.io/grimoirelab-tutorial
8. chaoss.community Software development analytics opnfv.biterg.io chaoss.biterg.io
9. chaoss.community GrimoireLab architecture
10. chaoss.community Installing from pip $ python3 -m venv /tmp/gl $ source /tmp/gl/bin/activate (gl) $ pip install --upgrade pip setuptools wheel (gl) $ pip install grimoirelab (gl) $ grimoirelab -v chaoss.github.io/grimoirelab-tutorial/basics/install.html
11. chaoss.community Running from Docker images docker run -p 5601:5601 -p 9200:9200 -v $(pwd)/credentials.cfg:/override.cfg -t grimoirelab/full credentials.cfg: [github] api-token = XXX
12. chaoss.community Architecture: extraction
13. chaoss.community Example: Perceval (gl) $ pip install perceval (gl) $ perceval git https://github.com/grimoirelab/perceval (gl) $ perceval github grimoirelab perceval --sleep-for-rate -t XXXXX ...
14. chaoss.community Example: Perceval from perceval.backends.core.git import Git repo_url = 'http://github.com/grimoirelab/perceval' repo_dir = '/tmp/perceval' repo = Git(uri=repo_url, gitpath=repo_dir) for commit in repo.fetch(): print(commit['data']['commit'])
15. chaoss.community Producing a dashboard (assumes Elasticsearch at http://localhost:9200, Kibiter at http://localhost:5601 ) Prepare: mordred.cfg Prepare: projects.json Maybe: identities.yaml Maybe: menu.yaml (es) mordred -c mordred.cfg chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
16. chaoss.community Using Perceval to compute metrics • Get items from the data source • Store them in a data structure • Compute metrics • You can also use Pandas, of course github.com/chaoss/wg-gmd/blob/master/examples/Code_ Commits.ipynb
17. chaoss.community Architecture: enrichment
18. chaoss.community Example: item in enriched index
19. chaoss.community Computing metrics from enriched indexes • Connect to Elasticsearch indexes (produced with p2o, mordred) • Query them to get items of interest • Compute metrics • You can also use Pandas, of course chaoss.github.io/grimoirelab-tutorial/python/indexes.html
20. chaoss.community Architecture: exploitation
21. chaoss.community Example: producing a dashboard (es) $ pip install grimoire-mordred (es) $ mordred -c mordred.cfg (assumes Elasticsearch at http://localhost:9200, Kibiter at http://localhost:5601 ) Needs: mordred.cfg, projects.json, identities.yaml, menu.yaml chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
22. chaoss.community GrimoireLab Software development analytics with free, open source software (a CHAOSS project) chaoss.github.io/grimoirelab chaoss.github.io/grimoirelab-tutorial
23. chaoss.community License & Attributions “Pain Relief”, by Nick Youngson, Alpha Stock Images, CC by-sa 3.0 “Broken Glass Sadness Disappointment Can't Sad”, from Max Pixel, CC0 Slides licensed under Creative Commons ShareAlike 4.0 (CC by-sa 4.0) Copyright CHAOSS & Jesus M. Gonzalez-Barahona 2018