Soft dive into GrimoireLab. Twitter OSS workshop

Soft dive into GrimoireLab. Twitter OSS workshop

Slides used for a soft dive into GrimoireLab during an internal workshop in Twitter HQ

7dddc875546948b5b5094167c90dc10d?s=128

Bitergia

March 07, 2018
Tweet

Transcript

  1. Deep Soft Dive Into GrimoireLab Manrique López @bitergia jsmanrique at

    bitergia.com https://speakerdeck.com/bitergia Twitter OSS internal workshop, Twitter Inc. offices, March 2018
  2. /history/2001

  3. /history/2007-2011

  4. /history/2011-2014

  5. /history/2011-2014

  6. /history/2011-2014

  7. /history/2007-2014

  8. /history/2015 Grimoire Lab grimoirelab.github.io

  9. /history/2017

  10. /history/2017-... chaoss.github.io/grimoirelab

  11. Architecture GrimoireLab components & technology

  12. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash /what

  13. /what Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash

  14. /what Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash

  15. /technology

  16. /architecture Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash

  17. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash redis

    ES SQL ES /architecture
  18. /run $ docker run -p 127.0.0.1:5601:5601 \ -v $(pwd)/credentials.cfg:/mordred-override.cfg \

    -t grimoirelab/full Credentials file: [github] api-token = XXX
  19. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash redis

    ES SQL /gathering
  20. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash ES

    SQL ES /enrichment
  21. Mordred Perceval Arthur SortingHat Sigils Manuscripts Kibiter GELK Kidash ES

    /consumption
  22. /GrimoireLab Dashboard

  23. Main components Have fun playing with them!

  24. /before_start NOTE: To avoid installing components one by one, to

    speed up training: $ python3 -m venv /tmp/grimoirelab $ source /tmp/grimoirelab/bin/activate (grimoirelab) $ pip install --upgrade pip setuptools wheel ... (grimoirelab) $ pip install grimoire-mordred
  25. Data gathering

  26. /perceval Perceval github.com/chaoss/grimoirelab-perceval PyPI package: $ pip3 install perceval From

    sources: $ git clone https://github.com/chaoss/grimoire-perceval.git $ pip3 install -r requirements.txt $ python3 setup.py install Docker: https://hub.docker.com/r/grimoirelab/perceval/
  27. /perceval From command line: $ perceval [-c <file>] [-g] <backend>

    [<args>] |--help | --version In your Python code: … from perceval.backends.core.<backend> import <Backend> … backend_repo = <Backend>(<params>) for item in backend_repo.fetch(): …
  28. /perceval Backends: askbot Fetch questions and answers from Askbot site

    bugzilla Fetch bugs from a Bugzilla server bugzillarest Fetch bugs from a Bugzilla server (>=5.0) using its REST API confluence Fetch contents from a Confluence server discourse Fetch posts from Discourse site dockerhub Fetch repository data from Docker Hub site gerrit Fetch reviews from a Gerrit server git Fetch commits from Git github Fetch issues from GitHub gitlab Fetch issues from GitLab gmane Fetch messages from Gmane hyperkitty Fetch messages from a HyperKitty archiver jenkins Fetch builds from a Jenkins server jira Fetch issues from JIRA issue tracker mbox Fetch messages from MBox files mediawiki Fetch pages and revisions from a MediaWiki site meetup Fetch events from a Meetup group nntp Fetch articles from a NNTP news group phabricator Fetch tasks from a Phabricator site pipermail Fetch messages from a Pipermail archiver redmine Fetch issues from a Redmine server rss Fetch entries from a RSS feed server slack Fetch messages from a Slack channel stackexchange Fetch questions from StackExchange sites supybot Fetch messages from Supybot log files telegram Fetch messages from the Telegram server
  29. /perceval Write your own backends! github.com/chaoss/grimoirelab-perceval/tree/master/perceval/backends/core … class <Backend>(Backend): …

    @metadata def fetch(self): … … class <Backend>Client: … class <Backend>Command(BackendCommand): …
  30. /perceval

  31. Data enrichment

  32. /grimoirelab-elk Grimoirelab ELK github.com/chaoss/grimoirelab-elk $ p2o.py --enrich --index git_raw --index-enrich

    git \ -e http://localhost:9200 --no_inc --debug \ git https://github.com/chaoss/grimoirelab-perceval
  33. /sortinghat Sorting Hat github.com/chaoss/grimoirelab-sortinghat Maintains an SQL database with identities

    that can be merged in the same unique identity. For each unique identity, a profile can be defined: name, email, and other data. Each unique identity can be related to one or more affiliations, for different time periods.
  34. /sortinghat Manrique López <jsmanrique@bitergia.com> José Manrique López de la Fuente

    <jsmanrique@gmail.com> Manrique López <jsmanrique@gmail.com> jsmanrique jsmanrique@gmail.com jsmanrique@bitergia.com correo@jsmanrique.es jsmanrique jsmanrique@bitergia.com 02/2005 - 12/2010 CTIC 01/2010 - 12/2012 Andago 01/2013 - 06/2013 TapQuo 07/2013 - 12/2015 freelance (ASOLIF, CENATIC) 07/2013 - now Bitergia
  35. /sortinghat github.com/chaoss/grimoirelab-sortinghat/blob/maste r/sortinghat/api.py … def add_identity(db, source, email=None, name=None, username=None,

    uuid=None): … def add_organization(db, organization): … def add_domain(db, organization, domain, is_top_domain=False, overwrite=False): … def add_enrollment(db, uuid, organization, from_date=None, to_date=None): … def edit_profile(db, uuid, **kwargs): … def merge_unique_identities(db, from_uuid, to_uuid): … def search_unique_identities(db, term, source=None): …
  36. Data visualization

  37. /visualization Kibiter github.com/chaoss/grimoirelab-kibiter Sigils github.com/chaoss/grimoirelab-sigils Kidash github.com/chaoss/grimoirelab-kidash

  38. /visualization (grimoirelab) $ kidash --help usage: usage: kidash [options] Import

    or Export a Kibana Dashboard optional arguments: -h, --help show this help message and exit -e ELASTIC_URL, --elastic_url ELASTIC_URL Host with elastic search(default: http://127.0.0.1:9200) --elastic_url-enrich ELASTIC_URL_ENRICH Host with elastic search and enriched indexes --dashboard DASHBOARD Kibana dashboard id to export --export EXPORT_FILE file with the dashboard exported --import IMPORT_FILE file with the dashboard to be imported --kibana KIBANA_INDEX Kibana index name (.kibana default) --list list available dashboards -g, --debug --data-sources DATA_SOURCES [DATA_SOURCES ...] Data sources to be included --add-vis-studies Include visualizations for studies
  39. /sigils github.com/chaoss/grimoirelab-sigils/tree/master/sch ema Data model used for each of the

    standard panels and indexes in GrimoireLab.
  40. Playing with Data (bonus included) Data Science (sort of) for

    Community Managers
  41. /dashboard twitteross.biterg.io

  42. /dashboard twitteross.biterg.io/edit

  43. /manuscripts Manuscripts github.com/chaoss/grimoirelab-manuscripts $ pip install manuscripts $ manuscripts -d

    /tmp/report-result \ -u http://twitteross:**********@twitteross.biterg.io/data \ -n TwitterOSS --data-sources git
  44. /elasticsearch Python API elasticsearch-py.readthedocs.io elasticsearch-dsl.readthedocs.io Javascript API www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html

  45. What’s next? New features coming

  46. /hatstall

  47. /bestiary

  48. /kibiter6

  49. /join #GrimoireLab in irc.freenode.net bit.ly/grimoirelab-mailman github.com/chaoss/grimoirelab-* (repositories being migrated from

    github.com/grimoirelab ) @GrimoireLab #GrimoireLab
  50. Software Development Analytics for your peace of mind chaoss.github.io/grimoirelab chaoss.community

    jsmanrique@bitergia.com CEO at Bitergia