Meet GrimoireLab: Making your life easier

Presentation at ICSME'18 (Madrid, Sept 25th 2018), tailored mainly to researchers. GrimoireLab is a Python toolset for software development analytics. It can deal with data from 20+ kinds of data sources (from git, GitHub and GitLab to Slack, mailing lists, Jenkins, Bugzilla, Jira or Gerrit) with a uniform Python API. It provides tools for identity management, data enrichment, visualization, and massive data retrieval.

Transcript

1. chaoss.community
   Meet
   GrimoireLab
   Making your life easier
   ICSME 2018, Madrid (Spain), Sep 25 2018
   Jesus M. Gonzalez-Barahona
   

   View Slide

2. chaoss.community
   /me
   Like six years ago I
   was having coffees
   with the gang of
   Bitergia founders
   Involved in the
   company since
   then
   bitergia.com
   I work at
   Universidad Rey
   Juan Carlos...
   ...researching about
   software
   development
   gsyc.es/~jgb
   My two hats:
   

   View Slide

3. /Bitergia Software Development Analytics
   for your peace of mind
   bitergia.com
   

   View Slide

4. chaoss.community
   ...
   Produce integrated, open source software
   for analyzing software development
   ….
   chaoss.community
   

   View Slide

5. chaoss.community
   When you’re collecting, preparing data
   for your next paper...
   

   View Slide

6. chaoss.community
   When you’re collecting, preparing data
   for your next paper… use some help!
   

   View Slide

7. chaoss.community
   GrimoireLab
   Software development analytics with
   free, open source software
   (a CHAOSS project)
   chaoss.github.io/grimoirelab
   chaoss.github.io/grimoirelab-tutorial
   

   View Slide

8. chaoss.community
   Software development analytics
   opnfv.biterg.io
   chaoss.biterg.io
   

   View Slide

9. chaoss.community
   GrimoireLab architecture
   

   View Slide

10. chaoss.community
    Installing from pip
    $ python3 -m venv /tmp/gl
    $ source /tmp/gl/bin/activate
    (gl) $ pip install --upgrade pip setuptools wheel
    (gl) $ pip install grimoirelab
    (gl) $ grimoirelab -v
    chaoss.github.io/grimoirelab-tutorial/basics/install.html
    

    View Slide

11. chaoss.community
    Running from Docker images
    docker run -p 5601:5601 -p 9200:9200
    -v $(pwd)/credentials.cfg:/override.cfg
    -t grimoirelab/full
    credentials.cfg:
    [github]
    api-token = XXX
    

    View Slide

12. chaoss.community
    Architecture: extraction
    

    View Slide

13. chaoss.community
    Example: Perceval
    (gl) $ pip install perceval
    (gl) $ perceval git https://github.com/grimoirelab/perceval
    (gl) $ perceval github grimoirelab perceval --sleep-for-rate
    -t XXXXX
    ...
    

    View Slide

14. chaoss.community
    Example: Perceval
    from perceval.backends.core.git import Git
    repo_url = 'http://github.com/grimoirelab/perceval'
    repo_dir = '/tmp/perceval'
    repo = Git(uri=repo_url, gitpath=repo_dir)
    for commit in repo.fetch():
    print(commit['data']['commit'])
    

    View Slide

15. chaoss.community
    Producing a dashboard
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Prepare: mordred.cfg
    Prepare: projects.json
    Maybe: identities.yaml
    Maybe: menu.yaml
    (es) mordred -c mordred.cfg
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
    

    View Slide

16. chaoss.community
    Using Perceval to compute metrics
    • Get items from the data source
    • Store them in a data structure
    • Compute metrics
    • You can also use Pandas, of course
    github.com/chaoss/wg-gmd/blob/master/examples/Code_
    Commits.ipynb
    

    View Slide

17. chaoss.community
    Architecture:
    enrichment
    

    View Slide

18. chaoss.community
    Example: item in enriched index
    

    View Slide

19. chaoss.community
    Computing metrics from enriched
    indexes
    • Connect to Elasticsearch indexes
    (produced with p2o, mordred)
    • Query them to get items of interest
    • Compute metrics
    • You can also use Pandas, of course
    chaoss.github.io/grimoirelab-tutorial/python/indexes.html
    

    View Slide

20. chaoss.community
    Architecture:
    exploitation
    

    View Slide

21. chaoss.community
    Example: producing a dashboard
    (es) $ pip install grimoire-mordred
    (es) $ mordred -c mordred.cfg
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Needs: mordred.cfg, projects.json, identities.yaml, menu.yaml
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html
    

    View Slide

22. chaoss.community
    GrimoireLab
    Software development analytics with
    free, open source software
    (a CHAOSS project)
    chaoss.github.io/grimoirelab
    chaoss.github.io/grimoirelab-tutorial
    

    View Slide

23. chaoss.community
    License & Attributions
    “Pain Relief”, by Nick Youngson,
    Alpha Stock Images, CC by-sa 3.0
    “Broken Glass Sadness Disappointment Can't Sad”,
    from Max Pixel, CC0
    Slides licensed under Creative Commons ShareAlike 4.0
    (CC by-sa 4.0)
    Copyright CHAOSS & Jesus M. Gonzalez-Barahona 2018
    

    View Slide