A tale of two datasets

Presentation given at the 2013 ICSM panel on Open Access

1. A tale of two datasets Georgios Gousios TU Delft
2. open access
6. Software Quality Observatory for OSS
8. 50k LOC!
9.  750 OSS repositories, SVN, bugs, emails 1.5GB processed data dump
10. demo.sqo-oss.org
11. SQO-OSS facts • 6 partners • ~30 publications • 4 PhDs funded • press releases on project releases • rated excellent by the EC
12. 1 external user 2 external publications
14. 
15.  
16.   GHTorrent
17.   GHTorrent
18.  mirror event stream
19. <<event>> PushEvent <<api>> /users/:user ensure_user <<api>> /repos/:user/:repo/ ensure_repo <<api>> /repos/:user/:repo/commits ensure_commits ensure_user <<api>> /:user/:repo/sha ensure_commit ensure_user <<api>> /users/:user/ followers ensure_followers <<api>> /repos/:user/:repo/ commits/:sha/comments ensure_commit_comments <<api>> /users/:user/orgs ensure_orgs <<api>> /orgs/:org/teams ensure_teams recursive dependency retrieval
20. relational database
21. repositories users organizations issues /users/:user /user/repos /repos/:user/:repo/issues /orgs/:org {    "type":  "User",    "public_gists":  0,    "login":  "gousiosg",    "followers":  8,    "name":  "Georgios  Gousios",    "public_repos":  4,    "created_at":  ...,    "id":  386172,    "following":  4, } { . . . NoSQL database as cache
22. periodic dumps of DBs online
23. query DBs online
24. GHTorrent facts • 2.0 TB in MongoDB, 40GB in MySQL • 1 developer • 3 papers • advertised on social media • 1.5 years
25. 5 external users 3 external papers MSR14 challenge dataset
26. why the difference?
27. but Github is hot!
28. but Github is hot!
29. but Github is hot! so was SourceForge, Gnome, KDE etc
30. but Github is hot! so was SourceForge, Gnome, KDE etc the Github Archive project offers a subset of the data in an easier to query format
31. SQO-OSS GHTorrent Tools pluggable platform ruby library and cmd-line tools Data initially none, then raw repos raw + processed Documentation mostly programming mostly data formats Released after 2 years immediately Attracted first user 2 years 2 months
32. open source research
33. aim for lean and mean
34. infrastructures and platforms are overrated
35. open now trumps open when it’s done