A tale of two datasets

Transcript

1. A tale of two datasets Georgios Gousios TU Delft

2. open access

3. None
4. None
5. None

6. Software Quality Observatory for OSS

7. None

8. 50k LOC!

9.  750 OSS repositories, SVN, bugs, emails 1.5GB processed data

   dump

10. demo.sqo-oss.org

11. SQO-OSS facts • 6 partners • ~30 publications • 4

    PhDs funded • press releases on project releases • rated excellent by the EC

12. 1 external user 2 external publications

13. None

14. 

15.  

16.   GHTorrent

17.   GHTorrent

18.  mirror event stream

19. <<event>> PushEvent <<api>> /users/:user ensure_user <<api>> /repos/:user/:repo/ ensure_repo <<api>> /repos/:user/:repo/commits

    ensure_commits ensure_user <<api>> /:user/:repo/sha ensure_commit ensure_user <<api>> /users/:user/ followers ensure_followers <<api>> /repos/:user/:repo/ commits/:sha/comments ensure_commit_comments <<api>> /users/:user/orgs ensure_orgs <<api>> /orgs/:org/teams ensure_teams recursive dependency retrieval

20. relational database

21. repositories users organizations issues /users/:user /user/repos /repos/:user/:repo/issues /orgs/:org {  

     "type":  "User",    "public_gists":  0,    "login":  "gousiosg",    "followers":  8,    "name":  "Georgios  Gousios",    "public_repos":  4,    "created_at":  ...,    "id":  386172,    "following":  4, } { . . . NoSQL database as cache

22. periodic dumps of DBs online

23. query DBs online

24. GHTorrent facts • 2.0 TB in MongoDB, 40GB in MySQL

    • 1 developer • 3 papers • advertised on social media • 1.5 years

25. 5 external users 3 external papers MSR14 challenge dataset

26. why the difference?

27. but Github is hot!

28. but Github is hot!

29. but Github is hot! so was SourceForge, Gnome, KDE etc

30. but Github is hot! so was SourceForge, Gnome, KDE etc

    the Github Archive project offers a subset of the data in an easier to query format

31. SQO-OSS GHTorrent Tools pluggable platform ruby library and cmd-line tools

    Data initially none, then raw repos raw + processed Documentation mostly programming mostly data formats Released after 2 years immediately Attracted first user 2 years 2 months

32. open source research

33. aim for lean and mean

34. infrastructures and platforms are overrated

35. open now trumps open when it’s done