Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Bitergia
February 05, 2017

GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Slides for FOSDEM 2017 talk at Python DevRoom about GrimoireLab

Bitergia

February 05, 2017
Tweet

More Decks by Bitergia

Other Decks in Technology

Transcript

  1. GrimoireLab a Python toolset for software development analytics Jesus M.

    Gonzalez-Barahona (URJC) @jgbarah jgb at bitergia dot com speakerdeck.com/bitergia FOSDEM 2017, Python devroom Brussels (Belgium) February 5th 2017
  2. /me Like five years ago I was having coffees with

    the gang of Bitergia founders Involved in the company since then bitergia.com I work at Universidad Rey Juan Carlos... ...researching about software development gsyc.es/~jgb My two hats:
  3. Retrieving data with Perceval In a Python3 environment… $ pip3

    install perceval $ perceval git https://github.com/grimoirelab/perceval.git [2016-10-03 00:47:46,632] - Sir Perceval is on his quest. [2016-10-03 00:47:46,633] - Fetching commits: 'https://github.com/grimoirelab/perceval.git' git repository from 1970-01-01 00:00:00+00:00; all branches { "backend_name": "Git", "backend_version": "0.3.0", "data": { "Author": "Santiago Due\u00f1as <[email protected]>", "AuthorDate": "Tue Aug 18 18:08:27 2015 +0200", "Commit": "Santiago Due\u00f1as <[email protected]>", "CommitDate": "Tue Aug 18 18:08:27 2015 +0200", "commit": "dc78c254e464ff334892e0448a23e4cfbfc637a3", ….
  4. Retrieving data with Perceval from Python from perceval.backends.core.github import GitHub

    # GitHub object for owner / repo, repo_dir for cloning repo = GitHub(owner=owner, repository=repo, api_token=args.token) # fetch all issues/pull requests for item in repo.fetch(): if 'pull_request' in item['data']: kind = 'Pull request' else: kind = 'Issue' print(item['data']['number'], ':', kind)
  5. Creating indexes for a dashboard $ pip install grimoire-elk $

    pip install grimoire-kidash $ p2o.py --enrich --index git_raw --index-enrich git \ -e http://localhost:9200 --no_inc --debug \ git https://github.com/grimoirelab/perceval.git $ kidash.py -e http://localhost:9200 \ --import git-dashboard.json Results: Raw index: git_raw Kibana dashboard Enriched index: git
  6. Querying ElasticSearch API $ curl -XGET "http://elasticsearch_url/git/_search/?size=1&pretty" {... "hits" :

    { "total" : 407, "hits" : [ { "_index" : "commits", "_type" : "summary", "_id" : "AVfPp9Po5xUyv5saVPKU", "_score" : 1.0, "_source" : { "hash" : "d1253dd9876bb76e938a861acaceaae95241b46d", "commit" : "Santiago Dueñas <[email protected]>", "author" : "Santiago Dueñas <[email protected]>", "author_date" : "Wed Nov 18 10:59:52 2015 +0100", "files_no" : 3, "commit_date" : "Wed Nov 18 14:41:21 2015 +0100" }} ] } }
  7. Python scripting from elasticsearch import Elasticsearch from elasticsearch_dsl import Search

    es = Elasticsearch(["elasticsearch_url”] s = Search(using=es, index=’git’) s = s.filter('range', files={'gt':0}) s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) s.aggs.metric('commits', 'cardinality', field='hash') s.aggs.bucket('histogram', 'date_histogram', field='author_date', interval='quarter') by_q = s.execute() for quarter in by_q.to_dict()['aggregations']['histogram']['buckets']: print("Unique commits for quarter starting on ", Quarter['key_as_string'], ": ", quarter['doc_count'])