Soft dive into GrimoireLab. Twitter OSS workshop

Slides used for a soft dive into GrimoireLab during an internal workshop in Twitter HQ

Transcript

1. Deep Soft Dive Into GrimoireLab Manrique López @bitergia jsmanrique at bitergia.com https://speakerdeck.com/bitergia Twitter OSS internal workshop, Twitter Inc. offices, March 2018
2. /history/2001
3. /history/2007-2011
4. /history/2011-2014
5. /history/2011-2014
6. /history/2011-2014
7. /history/2007-2014
8. /history/2015 Grimoire Lab grimoirelab.github.io
9. /history/2017
10. /history/2017-... chaoss.github.io/grimoirelab
11. Architecture GrimoireLab components & technology
12. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash /what
13. /what Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash
14. /what Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash
15. /technology
16. /architecture Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash
17. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash redis ES SQL ES /architecture
18. /run $ docker run -p 127.0.0.1:5601:5601 \ -v $(pwd)/credentials.cfg:/mordred-override.cfg \ -t grimoirelab/full Credentials file: [github] api-token = XXX
19. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash redis ES SQL /gathering
20. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash ES SQL ES /enrichment
21. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash ES /consumption
22. /GrimoireLab Dashboard
23. Main components Have fun playing with them!
24. /before_start NOTE: To avoid installing components one by one, to speed up training: $ python3 -m venv /tmp/grimoirelab $ source /tmp/grimoirelab/bin/activate (grimoirelab) $ pip install --upgrade pip setuptools wheel ... (grimoirelab) $ pip install grimoire-mordred
25. Data gathering
26. /perceval Perceval github.com/chaoss/grimoirelab-perceval PyPI package: $ pip3 install perceval From sources: $ git clone https://github.com/chaoss/grimoire-perceval.git $ pip3 install -r requirements.txt $ python3 setup.py install Docker: https://hub.docker.com/r/grimoirelab/perceval/
27. /perceval From command line: $ perceval [-c <file>] [-g] <backend> [<args>] |--help | --version In your Python code: … from perceval.backends.core.<backend> import <Backend> … backend_repo = <Backend>(<params>) for item in backend_repo.fetch(): …
28. /perceval Backends: askbot Fetch questions and answers from Askbot site bugzilla Fetch bugs from a Bugzilla server bugzillarest Fetch bugs from a Bugzilla server (>=5.0) using its REST API confluence Fetch contents from a Confluence server discourse Fetch posts from Discourse site dockerhub Fetch repository data from Docker Hub site gerrit Fetch reviews from a Gerrit server git Fetch commits from Git github Fetch issues from GitHub gitlab Fetch issues from GitLab gmane Fetch messages from Gmane hyperkitty Fetch messages from a HyperKitty archiver jenkins Fetch builds from a Jenkins server jira Fetch issues from JIRA issue tracker mbox Fetch messages from MBox files mediawiki Fetch pages and revisions from a MediaWiki site meetup Fetch events from a Meetup group nntp Fetch articles from a NNTP news group phabricator Fetch tasks from a Phabricator site pipermail Fetch messages from a Pipermail archiver redmine Fetch issues from a Redmine server rss Fetch entries from a RSS feed server slack Fetch messages from a Slack channel stackexchange Fetch questions from StackExchange sites supybot Fetch messages from Supybot log files telegram Fetch messages from the Telegram server
29. /perceval Write your own backends! github.com/chaoss/grimoirelab-perceval/tree/master/perceval/backends/core … class <Backend>(Backend): … @metadata def fetch(self): … … class <Backend>Client: … class <Backend>Command(BackendCommand): …
30. /perceval
31. Data enrichment
32. /grimoirelab-elk Grimoirelab ELK github.com/chaoss/grimoirelab-elk $ p2o.py --enrich --index git_raw --index-enrich git \ -e http://localhost:9200 --no_inc --debug \ git https://github.com/chaoss/grimoirelab-perceval
33. /sortinghat Sorting Hat github.com/chaoss/grimoirelab-sortinghat Maintains an SQL database with identities that can be merged in the same unique identity. For each unique identity, a profile can be defined: name, email, and other data. Each unique identity can be related to one or more affiliations, for different time periods.
34. /sortinghat Manrique López <jsmanrique@bitergia.com> José Manrique López de la Fuente <jsmanrique@gmail.com> Manrique López <jsmanrique@gmail.com> jsmanrique jsmanrique@gmail.com jsmanrique@bitergia.com correo@jsmanrique.es jsmanrique jsmanrique@bitergia.com 02/2005 - 12/2010 CTIC 01/2010 - 12/2012 Andago 01/2013 - 06/2013 TapQuo 07/2013 - 12/2015 freelance (ASOLIF, CENATIC) 07/2013 - now Bitergia
35. /sortinghat github.com/chaoss/grimoirelab-sortinghat/blob/maste r/sortinghat/api.py … def add_identity(db, source, email=None, name=None, username=None, uuid=None): … def add_organization(db, organization): … def add_domain(db, organization, domain, is_top_domain=False, overwrite=False): … def add_enrollment(db, uuid, organization, from_date=None, to_date=None): … def edit_profile(db, uuid, **kwargs): … def merge_unique_identities(db, from_uuid, to_uuid): … def search_unique_identities(db, term, source=None): …
36. Data visualization
37. /visualization Kibiter github.com/chaoss/grimoirelab-kibiter Sigils github.com/chaoss/grimoirelab-sigils Kidash github.com/chaoss/grimoirelab-kidash
38. /visualization (grimoirelab) $ kidash --help usage: usage: kidash [options] Import or Export a Kibana Dashboard optional arguments: -h, --help show this help message and exit -e ELASTIC_URL, --elastic_url ELASTIC_URL Host with elastic search(default: http://127.0.0.1:9200) --elastic_url-enrich ELASTIC_URL_ENRICH Host with elastic search and enriched indexes --dashboard DASHBOARD Kibana dashboard id to export --export EXPORT_FILE file with the dashboard exported --import IMPORT_FILE file with the dashboard to be imported --kibana KIBANA_INDEX Kibana index name (.kibana default) --list list available dashboards -g, --debug --data-sources DATA_SOURCES [DATA_SOURCES ...] Data sources to be included --add-vis-studies Include visualizations for studies
39. /sigils github.com/chaoss/grimoirelab-sigils/tree/master/sch ema Data model used for each of the standard panels and indexes in GrimoireLab.
40. Playing with Data (bonus included) Data Science (sort of) for Community Managers
41. /dashboard twitteross.biterg.io
42. /dashboard twitteross.biterg.io/edit
43. /manuscripts Manuscripts github.com/chaoss/grimoirelab-manuscripts $ pip install manuscripts $ manuscripts -d /tmp/report-result \ -u http://twitteross:**********@twitteross.biterg.io/data \ -n TwitterOSS --data-sources git
44. /elasticsearch Python API elasticsearch-py.readthedocs.io elasticsearch-dsl.readthedocs.io Javascript API www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html
45. What’s next? New features coming
46. /hatstall
47. /bestiary
48. /kibiter6
49. /join #GrimoireLab in irc.freenode.net bit.ly/grimoirelab-mailman github.com/chaoss/grimoirelab-* (repositories being migrated from github.com/grimoirelab ) @GrimoireLab #GrimoireLab
50. Software Development Analytics for your peace of mind chaoss.github.io/grimoirelab chaoss.community jsmanrique@bitergia.com CEO at Bitergia