GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Slides for FOSDEM 2017 talk at Python DevRoom about GrimoireLab

Transcript

1. GrimoireLab
   a Python toolset for software
   development analytics
   Jesus M. Gonzalez-Barahona (URJC)
   @jgbarah
   jgb at bitergia dot com
   speakerdeck.com/bitergia
   FOSDEM 2017, Python devroom
   Brussels (Belgium)
   February 5th 2017
   

   View Slide

2. Outline
   Some context
   The software doing the magic
   Let’s be practical
   Your turn
   

   View Slide

3. Some context
   

   View Slide

4. /me
   Like five years ago I
   was having coffees
   with the gang of
   Bitergia founders
   Involved in the
   company since then
   bitergia.com
   I work at
   Universidad Rey
   Juan Carlos...
   ...researching about
   software
   development
   gsyc.es/~jgb
   My two hats:
   

   View Slide

5. Building
   Software
   Development
   Dashboards
   cauldron.io
   

   View Slide

6. The software
   doing the magic
   

   View Slide

7. Architecture
   Original
   Data Sources
   Retrieval
   Perceval
   Enrichment
   GrimoireELK
   Visualization
   Kibiter
   (Kibana4/5)
   ElasticSearch
   GrimoireLab
   component
   

   View Slide

8. GrimoireLab
   grimoirelab.github.io
   

   View Slide

9. GrimoireLab
   grimoirelab.github.io
   

   View Slide

10. GrimoireLab
    Training
    Tutorial
    jgbarah.gitbooks.io/grimoirelab-training
    

    View Slide

11. Let’s be practical
    

    View Slide

12. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component
    

    View Slide

13. Retrieving
    data with
    Perceval
    In a Python3 environment…
    $ pip3 install perceval
    $ perceval git https://github.com/grimoirelab/perceval.git
    [2016-10-03 00:47:46,632] - Sir Perceval is on his quest.
    [2016-10-03 00:47:46,633] - Fetching commits:
    'https://github.com/grimoirelab/perceval.git' git repository from
    1970-01-01 00:00:00+00:00; all branches
    {
    "backend_name": "Git",
    "backend_version": "0.3.0",
    "data": {
    "Author": "Santiago Due\u00f1as
    ",
    "AuthorDate": "Tue Aug 18 18:08:27 2015 +0200",
    "Commit": "Santiago Due\u00f1as
    ",
    "CommitDate": "Tue Aug 18 18:08:27 2015 +0200",
    "commit":
    "dc78c254e464ff334892e0448a23e4cfbfc637a3",
    ….
    

    View Slide

14. Retrieving
    data with
    Perceval
    from Python
    from perceval.backends.core.github import GitHub
    # GitHub object for owner / repo, repo_dir for cloning
    repo = GitHub(owner=owner, repository=repo,
    api_token=args.token)
    # fetch all issues/pull requests
    for item in repo.fetch():
    if 'pull_request' in item['data']:
    kind = 'Pull request'
    else:
    kind = 'Issue'
    print(item['data']['number'], ':', kind)
    

    View Slide

15. Perceval
    Backends
    grimoirelab.github.io
    

    View Slide

16. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component
    

    View Slide

17. Creating
    indexes
    for a dashboard
    $ pip install grimoire-elk
    $ pip install grimoire-kidash
    $ p2o.py --enrich --index git_raw --index-enrich git \
    -e http://localhost:9200 --no_inc --debug \
    git https://github.com/grimoirelab/perceval.git
    $ kidash.py -e http://localhost:9200 \
    --import git-dashboard.json
    Results:
    Raw index: git_raw Kibana dashboard
    Enriched index: git
    

    View Slide

18. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component
    

    View Slide

19. Querying
    ElasticSearch
    API
    $ curl -XGET "http://elasticsearch_url/git/_search/?size=1&pretty"
    {...
    "hits" : {
    "total" : 407,
    "hits" : [ {
    "_index" : "commits",
    "_type" : "summary",
    "_id" : "AVfPp9Po5xUyv5saVPKU",
    "_score" : 1.0,
    "_source" : {
    "hash" : "d1253dd9876bb76e938a861acaceaae95241b46d",
    "commit" : "Santiago Dueñas ",
    "author" : "Santiago Dueñas ",
    "author_date" : "Wed Nov 18 10:59:52 2015 +0100",
    "files_no" : 3,
    "commit_date" : "Wed Nov 18 14:41:21 2015 +0100"
    }} ] } }
    

    View Slide

20. Python
    scripting
    from elasticsearch import Elasticsearch
    from elasticsearch_dsl import Search
    es = Elasticsearch(["elasticsearch_url”]
    s = Search(using=es, index=’git’)
    s = s.filter('range', files={'gt':0})
    s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)})
    s.aggs.metric('commits', 'cardinality', field='hash')
    s.aggs.bucket('histogram', 'date_histogram',
    field='author_date', interval='quarter')
    by_q = s.execute()
    for quarter in
    by_q.to_dict()['aggregations']['histogram']['buckets']:
    print("Unique commits for quarter starting on ",
    Quarter['key_as_string'], ": ", quarter['doc_count'])
    

    View Slide

21. More details:
    GrimoireLab
    Training
    Tutorial
    jgbarah.gitbooks.io/grimoirelab-training
    

    View Slide

22. Try it live:
    The Cauldron
    cauldron.io
    

    View Slide

23. Your turn
    Enjoy!
    http://grimoirelab.github.io
    https://jgbarah.gitbooks.io/grimoirelab-training
    http://caludron.io
    

    View Slide