GitHub Data and Insights

Originally presented at Collab Summit 2016, this talk covers the use of GHTorrent to gather and analyze public repo and community data from GitHub. We talk about using Azure Data Lake as well as how you can set up this infrastructure yourself.

Transcript

1. GitHub Insights Understanding Open Source @jeffmcaffer Microsoft Open Source Programs Office
2. Microsoft and open source
3. Microsoft repo trends
4. Business insights
5. Operational insights
6. Project health: Pull request performance
7. User Geolocation
8. Factors affecting Pull Request merge
9. Approach Data for the masses
10. GitHub by the numbers (Early 2016)
11. Approach http://ghtorrent.org
12. How does it work? http://api.github.com/events
13. Example event (condensed) https://api.github.com/users/Cephei https://api.github.com/repos/PowerDMS/Owin.Scim https://api.github.com/repos/PowerDMS/Owin.Scim/commits/c751014f634d73e0b72f78a53c8cf137888b309d https://api.github.com/orgs/PowerDMS
14. Entities
15. GHTorrent by the numbers
16. Using the data
17. Using the data: Hosted http://ghtorrent.org
18. Using the data: Self-service https://github.com/ghtorrent/ghtorrent-webhook
19. Using the data: Azure Data Lake
20. Wrap-up
21. Privacy
22. Futures
23. Questions