Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Bitergia
PRO
February 05, 2017

GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Slides for FOSDEM 2017 talk at Python DevRoom about GrimoireLab

Bitergia
PRO

February 05, 2017
Tweet

More Decks by Bitergia

Other Decks in Technology

Transcript

  1. GrimoireLab
    a Python toolset for software
    development analytics
    Jesus M. Gonzalez-Barahona (URJC)
    @jgbarah
    jgb at bitergia dot com
    speakerdeck.com/bitergia
    FOSDEM 2017, Python devroom
    Brussels (Belgium)
    February 5th 2017

    View Slide

  2. Outline
    Some context
    The software doing the magic
    Let’s be practical
    Your turn

    View Slide

  3. Some context

    View Slide

  4. /me
    Like five years ago I
    was having coffees
    with the gang of
    Bitergia founders
    Involved in the
    company since then
    bitergia.com
    I work at
    Universidad Rey
    Juan Carlos...
    ...researching about
    software
    development
    gsyc.es/~jgb
    My two hats:

    View Slide

  5. Building
    Software
    Development
    Dashboards
    cauldron.io

    View Slide

  6. The software
    doing the magic

    View Slide

  7. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component

    View Slide

  8. GrimoireLab
    grimoirelab.github.io

    View Slide

  9. GrimoireLab
    grimoirelab.github.io

    View Slide

  10. GrimoireLab
    Training
    Tutorial
    jgbarah.gitbooks.io/grimoirelab-training

    View Slide

  11. Let’s be practical

    View Slide

  12. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component

    View Slide

  13. Retrieving
    data with
    Perceval
    In a Python3 environment…
    $ pip3 install perceval
    $ perceval git https://github.com/grimoirelab/perceval.git
    [2016-10-03 00:47:46,632] - Sir Perceval is on his quest.
    [2016-10-03 00:47:46,633] - Fetching commits:
    'https://github.com/grimoirelab/perceval.git' git repository from
    1970-01-01 00:00:00+00:00; all branches
    {
    "backend_name": "Git",
    "backend_version": "0.3.0",
    "data": {
    "Author": "Santiago Due\u00f1as
    ",
    "AuthorDate": "Tue Aug 18 18:08:27 2015 +0200",
    "Commit": "Santiago Due\u00f1as
    ",
    "CommitDate": "Tue Aug 18 18:08:27 2015 +0200",
    "commit":
    "dc78c254e464ff334892e0448a23e4cfbfc637a3",
    ….

    View Slide

  14. Retrieving
    data with
    Perceval
    from Python
    from perceval.backends.core.github import GitHub
    # GitHub object for owner / repo, repo_dir for cloning
    repo = GitHub(owner=owner, repository=repo,
    api_token=args.token)
    # fetch all issues/pull requests
    for item in repo.fetch():
    if 'pull_request' in item['data']:
    kind = 'Pull request'
    else:
    kind = 'Issue'
    print(item['data']['number'], ':', kind)

    View Slide

  15. Perceval
    Backends
    grimoirelab.github.io

    View Slide

  16. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component

    View Slide

  17. Creating
    indexes
    for a dashboard
    $ pip install grimoire-elk
    $ pip install grimoire-kidash
    $ p2o.py --enrich --index git_raw --index-enrich git \
    -e http://localhost:9200 --no_inc --debug \
    git https://github.com/grimoirelab/perceval.git
    $ kidash.py -e http://localhost:9200 \
    --import git-dashboard.json
    Results:
    Raw index: git_raw Kibana dashboard
    Enriched index: git

    View Slide

  18. Architecture
    Original
    Data Sources
    Retrieval
    Perceval
    Enrichment
    GrimoireELK
    Visualization
    Kibiter
    (Kibana4/5)
    ElasticSearch
    GrimoireLab
    component

    View Slide

  19. Querying
    ElasticSearch
    API
    $ curl -XGET "http://elasticsearch_url/git/_search/?size=1&pretty"
    {...
    "hits" : {
    "total" : 407,
    "hits" : [ {
    "_index" : "commits",
    "_type" : "summary",
    "_id" : "AVfPp9Po5xUyv5saVPKU",
    "_score" : 1.0,
    "_source" : {
    "hash" : "d1253dd9876bb76e938a861acaceaae95241b46d",
    "commit" : "Santiago Dueñas ",
    "author" : "Santiago Dueñas ",
    "author_date" : "Wed Nov 18 10:59:52 2015 +0100",
    "files_no" : 3,
    "commit_date" : "Wed Nov 18 14:41:21 2015 +0100"
    }} ] } }

    View Slide

  20. Python
    scripting
    from elasticsearch import Elasticsearch
    from elasticsearch_dsl import Search
    es = Elasticsearch(["elasticsearch_url”]
    s = Search(using=es, index=’git’)
    s = s.filter('range', files={'gt':0})
    s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)})
    s.aggs.metric('commits', 'cardinality', field='hash')
    s.aggs.bucket('histogram', 'date_histogram',
    field='author_date', interval='quarter')
    by_q = s.execute()
    for quarter in
    by_q.to_dict()['aggregations']['histogram']['buckets']:
    print("Unique commits for quarter starting on ",
    Quarter['key_as_string'], ": ", quarter['doc_count'])

    View Slide

  21. More details:
    GrimoireLab
    Training
    Tutorial
    jgbarah.gitbooks.io/grimoirelab-training

    View Slide

  22. Try it live:
    The Cauldron
    cauldron.io

    View Slide

  23. Your turn
    Enjoy!
    http://grimoirelab.github.io
    https://jgbarah.gitbooks.io/grimoirelab-training
    http://caludron.io

    View Slide