GrimoireLab, a Python toolset for software development analytics - FOSDEM 2017

Slides for FOSDEM 2017 talk at Python DevRoom about GrimoireLab

Transcript

1. GrimoireLab a Python toolset for software development analytics Jesus M. Gonzalez-Barahona (URJC) @jgbarah jgb at bitergia dot com speakerdeck.com/bitergia FOSDEM 2017, Python devroom Brussels (Belgium) February 5th 2017
2. Outline Some context The software doing the magic Let’s be practical Your turn
3. Some context
4. /me Like five years ago I was having coffees with the gang of Bitergia founders Involved in the company since then bitergia.com I work at Universidad Rey Juan Carlos... ...researching about software development gsyc.es/~jgb My two hats:
5. Building Software Development Dashboards cauldron.io
6. The software doing the magic
7. Architecture Original Data Sources Retrieval Perceval Enrichment GrimoireELK Visualization Kibiter (Kibana4/5) ElasticSearch GrimoireLab component
8. GrimoireLab grimoirelab.github.io
9. GrimoireLab grimoirelab.github.io
10. GrimoireLab Training Tutorial jgbarah.gitbooks.io/grimoirelab-training
11. Let’s be practical
12. Architecture Original Data Sources Retrieval Perceval Enrichment GrimoireELK Visualization Kibiter (Kibana4/5) ElasticSearch GrimoireLab component
13. Retrieving data with Perceval In a Python3 environment… $ pip3 install perceval $ perceval git https://github.com/grimoirelab/perceval.git [2016-10-03 00:47:46,632] - Sir Perceval is on his quest. [2016-10-03 00:47:46,633] - Fetching commits: 'https://github.com/grimoirelab/perceval.git' git repository from 1970-01-01 00:00:00+00:00; all branches { "backend_name": "Git", "backend_version": "0.3.0", "data": { "Author": "Santiago Due\u00f1as <sduenas@bitergia.com>", "AuthorDate": "Tue Aug 18 18:08:27 2015 +0200", "Commit": "Santiago Due\u00f1as <sduenas@bitergia.com>", "CommitDate": "Tue Aug 18 18:08:27 2015 +0200", "commit": "dc78c254e464ff334892e0448a23e4cfbfc637a3", ….
14. Retrieving data with Perceval from Python from perceval.backends.core.github import GitHub # GitHub object for owner / repo, repo_dir for cloning repo = GitHub(owner=owner, repository=repo, api_token=args.token) # fetch all issues/pull requests for item in repo.fetch(): if 'pull_request' in item['data']: kind = 'Pull request' else: kind = 'Issue' print(item['data']['number'], ':', kind)
15. Perceval Backends grimoirelab.github.io
16. Architecture Original Data Sources Retrieval Perceval Enrichment GrimoireELK Visualization Kibiter (Kibana4/5) ElasticSearch GrimoireLab component
17. Creating indexes for a dashboard $ pip install grimoire-elk $ pip install grimoire-kidash $ p2o.py --enrich --index git_raw --index-enrich git \ -e http://localhost:9200 --no_inc --debug \ git https://github.com/grimoirelab/perceval.git $ kidash.py -e http://localhost:9200 \ --import git-dashboard.json Results: Raw index: git_raw Kibana dashboard Enriched index: git
18. Architecture Original Data Sources Retrieval Perceval Enrichment GrimoireELK Visualization Kibiter (Kibana4/5) ElasticSearch GrimoireLab component
19. Querying ElasticSearch API $ curl -XGET "http://elasticsearch_url/git/_search/?size=1&pretty" {... "hits" : { "total" : 407, "hits" : [ { "_index" : "commits", "_type" : "summary", "_id" : "AVfPp9Po5xUyv5saVPKU", "_score" : 1.0, "_source" : { "hash" : "d1253dd9876bb76e938a861acaceaae95241b46d", "commit" : "Santiago Dueñas <sduenas@bitergia.com>", "author" : "Santiago Dueñas <sduenas@bitergia.com>", "author_date" : "Wed Nov 18 10:59:52 2015 +0100", "files_no" : 3, "commit_date" : "Wed Nov 18 14:41:21 2015 +0100" }} ] } }
20. Python scripting from elasticsearch import Elasticsearch from elasticsearch_dsl import Search es = Elasticsearch(["elasticsearch_url”] s = Search(using=es, index=’git’) s = s.filter('range', files={'gt':0}) s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) s.aggs.metric('commits', 'cardinality', field='hash') s.aggs.bucket('histogram', 'date_histogram', field='author_date', interval='quarter') by_q = s.execute() for quarter in by_q.to_dict()['aggregations']['histogram']['buckets']: print("Unique commits for quarter starting on ", Quarter['key_as_string'], ": ", quarter['doc_count'])
21. More details: GrimoireLab Training Tutorial jgbarah.gitbooks.io/grimoirelab-training
22. Try it live: The Cauldron cauldron.io
23. Your turn Enjoy! http://grimoirelab.github.io https://jgbarah.gitbooks.io/grimoirelab-training http://caludron.io