Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Meet GrimoireLab: Making your life easier

Meet GrimoireLab: Making your life easier

Presentation at ICSME'18 (Madrid, Sept 25th 2018), tailored mainly to researchers. GrimoireLab is a Python toolset for software development analytics. It can deal with data from 20+ kinds of data sources (from git, GitHub and GitLab to Slack, mailing lists, Jenkins, Bugzilla, Jira or Gerrit) with a uniform Python API. It provides tools for identity management, data enrichment, visualization, and massive data retrieval.

Jesus M. Gonzalez-Barahona

September 24, 2018
Tweet

More Decks by Jesus M. Gonzalez-Barahona

Other Decks in Technology

Transcript

  1. chaoss.community
    Meet
    GrimoireLab
    Making your life easier
    ICSME 2018, Madrid (Spain), Sep 25 2018
    Jesus M. Gonzalez-Barahona

    View Slide

  2. chaoss.community
    /me
    Like six years ago I
    was having coffees
    with the gang of
    Bitergia founders
    Involved in the
    company since
    then
    bitergia.com
    I work at
    Universidad Rey
    Juan Carlos...
    ...researching about
    software
    development
    gsyc.es/~jgb
    My two hats:

    View Slide

  3. /Bitergia Software Development Analytics
    for your peace of mind
    bitergia.com

    View Slide

  4. chaoss.community
    ...
    Produce integrated, open source software
    for analyzing software development
    ….
    chaoss.community

    View Slide

  5. chaoss.community
    When you’re collecting, preparing data
    for your next paper...

    View Slide

  6. chaoss.community
    When you’re collecting, preparing data
    for your next paper… use some help!

    View Slide

  7. chaoss.community
    GrimoireLab
    Software development analytics with
    free, open source software
    (a CHAOSS project)
    chaoss.github.io/grimoirelab
    chaoss.github.io/grimoirelab-tutorial

    View Slide

  8. chaoss.community
    Software development analytics
    opnfv.biterg.io
    chaoss.biterg.io

    View Slide

  9. chaoss.community
    GrimoireLab architecture

    View Slide

  10. chaoss.community
    Installing from pip
    $ python3 -m venv /tmp/gl
    $ source /tmp/gl/bin/activate
    (gl) $ pip install --upgrade pip setuptools wheel
    (gl) $ pip install grimoirelab
    (gl) $ grimoirelab -v
    chaoss.github.io/grimoirelab-tutorial/basics/install.html

    View Slide

  11. chaoss.community
    Running from Docker images
    docker run -p 5601:5601 -p 9200:9200
    -v $(pwd)/credentials.cfg:/override.cfg
    -t grimoirelab/full
    credentials.cfg:
    [github]
    api-token = XXX

    View Slide

  12. chaoss.community
    Architecture: extraction

    View Slide

  13. chaoss.community
    Example: Perceval
    (gl) $ pip install perceval
    (gl) $ perceval git https://github.com/grimoirelab/perceval
    (gl) $ perceval github grimoirelab perceval --sleep-for-rate
    -t XXXXX
    ...

    View Slide

  14. chaoss.community
    Example: Perceval
    from perceval.backends.core.git import Git
    repo_url = 'http://github.com/grimoirelab/perceval'
    repo_dir = '/tmp/perceval'
    repo = Git(uri=repo_url, gitpath=repo_dir)
    for commit in repo.fetch():
    print(commit['data']['commit'])

    View Slide

  15. chaoss.community
    Producing a dashboard
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Prepare: mordred.cfg
    Prepare: projects.json
    Maybe: identities.yaml
    Maybe: menu.yaml
    (es) mordred -c mordred.cfg
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html

    View Slide

  16. chaoss.community
    Using Perceval to compute metrics
    • Get items from the data source
    • Store them in a data structure
    • Compute metrics
    • You can also use Pandas, of course
    github.com/chaoss/wg-gmd/blob/master/examples/Code_
    Commits.ipynb

    View Slide

  17. chaoss.community
    Architecture:
    enrichment

    View Slide

  18. chaoss.community
    Example: item in enriched index

    View Slide

  19. chaoss.community
    Computing metrics from enriched
    indexes
    • Connect to Elasticsearch indexes
    (produced with p2o, mordred)
    • Query them to get items of interest
    • Compute metrics
    • You can also use Pandas, of course
    chaoss.github.io/grimoirelab-tutorial/python/indexes.html

    View Slide

  20. chaoss.community
    Architecture:
    exploitation

    View Slide

  21. chaoss.community
    Example: producing a dashboard
    (es) $ pip install grimoire-mordred
    (es) $ mordred -c mordred.cfg
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Needs: mordred.cfg, projects.json, identities.yaml, menu.yaml
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html

    View Slide

  22. chaoss.community
    GrimoireLab
    Software development analytics with
    free, open source software
    (a CHAOSS project)
    chaoss.github.io/grimoirelab
    chaoss.github.io/grimoirelab-tutorial

    View Slide

  23. chaoss.community
    License & Attributions
    “Pain Relief”, by Nick Youngson,
    Alpha Stock Images, CC by-sa 3.0
    “Broken Glass Sadness Disappointment Can't Sad”,
    from Max Pixel, CC0
    Slides licensed under Creative Commons ShareAlike 4.0
    (CC by-sa 4.0)
    Copyright CHAOSS & Jesus M. Gonzalez-Barahona 2018

    View Slide