Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Meet GrimoireLab: Making your life easier

Meet GrimoireLab: Making your life easier

Presentation at ICSME'18 (Madrid, Sept 25th 2018), tailored mainly to researchers. GrimoireLab is a Python toolset for software development analytics. It can deal with data from 20+ kinds of data sources (from git, GitHub and GitLab to Slack, mailing lists, Jenkins, Bugzilla, Jira or Gerrit) with a uniform Python API. It provides tools for identity management, data enrichment, visualization, and massive data retrieval.

Jesus M. Gonzalez-Barahona

September 24, 2018
Tweet

More Decks by Jesus M. Gonzalez-Barahona

Other Decks in Technology

Transcript

  1. chaoss.community
    Meet
    GrimoireLab
    Making your life easier
    ICSME 2018, Madrid (Spain), Sep 25 2018
    Jesus M. Gonzalez-Barahona

    View full-size slide

  2. chaoss.community
    /me
    Like six years ago I
    was having coffees
    with the gang of
    Bitergia founders
    Involved in the
    company since
    then
    bitergia.com
    I work at
    Universidad Rey
    Juan Carlos...
    ...researching about
    software
    development
    gsyc.es/~jgb
    My two hats:

    View full-size slide

  3. /Bitergia Software Development Analytics
    for your peace of mind
    bitergia.com

    View full-size slide

  4. chaoss.community
    ...
    Produce integrated, open source software
    for analyzing software development
    ….
    chaoss.community

    View full-size slide

  5. chaoss.community
    When you’re collecting, preparing data
    for your next paper...

    View full-size slide

  6. chaoss.community
    When you’re collecting, preparing data
    for your next paper… use some help!

    View full-size slide

  7. chaoss.community
    GrimoireLab
    Software development analytics with
    free, open source software
    (a CHAOSS project)
    chaoss.github.io/grimoirelab
    chaoss.github.io/grimoirelab-tutorial

    View full-size slide

  8. chaoss.community
    Software development analytics
    opnfv.biterg.io
    chaoss.biterg.io

    View full-size slide

  9. chaoss.community
    GrimoireLab architecture

    View full-size slide

  10. chaoss.community
    Installing from pip
    $ python3 -m venv /tmp/gl
    $ source /tmp/gl/bin/activate
    (gl) $ pip install --upgrade pip setuptools wheel
    (gl) $ pip install grimoirelab
    (gl) $ grimoirelab -v
    chaoss.github.io/grimoirelab-tutorial/basics/install.html

    View full-size slide

  11. chaoss.community
    Running from Docker images
    docker run -p 5601:5601 -p 9200:9200
    -v $(pwd)/credentials.cfg:/override.cfg
    -t grimoirelab/full
    credentials.cfg:
    [github]
    api-token = XXX

    View full-size slide

  12. chaoss.community
    Architecture: extraction

    View full-size slide

  13. chaoss.community
    Example: Perceval
    (gl) $ pip install perceval
    (gl) $ perceval git https://github.com/grimoirelab/perceval
    (gl) $ perceval github grimoirelab perceval --sleep-for-rate
    -t XXXXX
    ...

    View full-size slide

  14. chaoss.community
    Example: Perceval
    from perceval.backends.core.git import Git
    repo_url = 'http://github.com/grimoirelab/perceval'
    repo_dir = '/tmp/perceval'
    repo = Git(uri=repo_url, gitpath=repo_dir)
    for commit in repo.fetch():
    print(commit['data']['commit'])

    View full-size slide

  15. chaoss.community
    Producing a dashboard
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Prepare: mordred.cfg
    Prepare: projects.json
    Maybe: identities.yaml
    Maybe: menu.yaml
    (es) mordred -c mordred.cfg
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html

    View full-size slide

  16. chaoss.community
    Using Perceval to compute metrics
    • Get items from the data source
    • Store them in a data structure
    • Compute metrics
    • You can also use Pandas, of course
    github.com/chaoss/wg-gmd/blob/master/examples/Code_
    Commits.ipynb

    View full-size slide

  17. chaoss.community
    Architecture:
    enrichment

    View full-size slide

  18. chaoss.community
    Example: item in enriched index

    View full-size slide

  19. chaoss.community
    Computing metrics from enriched
    indexes
    • Connect to Elasticsearch indexes
    (produced with p2o, mordred)
    • Query them to get items of interest
    • Compute metrics
    • You can also use Pandas, of course
    chaoss.github.io/grimoirelab-tutorial/python/indexes.html

    View full-size slide

  20. chaoss.community
    Architecture:
    exploitation

    View full-size slide

  21. chaoss.community
    Example: producing a dashboard
    (es) $ pip install grimoire-mordred
    (es) $ mordred -c mordred.cfg
    (assumes Elasticsearch at http://localhost:9200,
    Kibiter at http://localhost:5601 )
    Needs: mordred.cfg, projects.json, identities.yaml, menu.yaml
    chaoss.github.io/grimoirelab-tutorial/sirmordred/dashboard.html

    View full-size slide

  22. chaoss.community
    GrimoireLab
    Software development analytics with
    free, open source software
    (a CHAOSS project)
    chaoss.github.io/grimoirelab
    chaoss.github.io/grimoirelab-tutorial

    View full-size slide

  23. chaoss.community
    License & Attributions
    “Pain Relief”, by Nick Youngson,
    Alpha Stock Images, CC by-sa 3.0
    “Broken Glass Sadness Disappointment Can't Sad”,
    from Max Pixel, CC0
    Slides licensed under Creative Commons ShareAlike 4.0
    (CC by-sa 4.0)
    Copyright CHAOSS & Jesus M. Gonzalez-Barahona 2018

    View full-size slide