Slide 1
Slide 1 text
A tale of two datasets
Georgios Gousios
TU Delft
Slide 2
Slide 2 text
open access
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Software Quality Observatory for OSS
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
50k
LOC!
Slide 9
Slide 9 text
750 OSS repositories, SVN, bugs, emails
1.5GB processed data dump
Slide 10
Slide 10 text
demo.sqo-oss.org
Slide 11
Slide 11 text
SQO-OSS facts
• 6 partners
• ~30 publications
• 4 PhDs funded
• press releases on project releases
• rated excellent by the EC
Slide 12
Slide 12 text
1 external user
2 external publications
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Slide 15
Slide 15 text
Slide 16
Slide 16 text
GHTorrent
Slide 17
Slide 17 text
GHTorrent
Slide 18
Slide 18 text
mirror event stream
Slide 19
Slide 19 text
<>
PushEvent
<>
/users/:user
ensure_user
<>
/repos/:user/:repo/
ensure_repo
<>
/repos/:user/:repo/commits
ensure_commits
ensure_user
<>
/:user/:repo/sha
ensure_commit
ensure_user
<>
/users/:user/
followers
ensure_followers
<>
/repos/:user/:repo/
commits/:sha/comments
ensure_commit_comments
<>
/users/:user/orgs
ensure_orgs
<>
/orgs/:org/teams
ensure_teams
recursive dependency retrieval
Slide 20
Slide 20 text
relational database
Slide 21
Slide 21 text
repositories
users
organizations
issues
/users/:user
/user/repos
/repos/:user/:repo/issues
/orgs/:org
{
"type":
"User",
"public_gists":
0,
"login":
"gousiosg",
"followers":
8,
"name":
"Georgios
Gousios",
"public_repos":
4,
"created_at":
...,
"id":
386172,
"following":
4,
}
{
.
.
.
NoSQL database as cache
Slide 22
Slide 22 text
periodic dumps of DBs online
Slide 23
Slide 23 text
query DBs online
Slide 24
Slide 24 text
GHTorrent facts
• 2.0 TB in MongoDB, 40GB in MySQL
• 1 developer
• 3 papers
• advertised on social media
• 1.5 years
Slide 25
Slide 25 text
5 external users
3 external papers
MSR14 challenge dataset
Slide 26
Slide 26 text
why the difference?
Slide 27
Slide 27 text
but Github is hot!
Slide 28
Slide 28 text
but Github is hot!
Slide 29
Slide 29 text
but Github is hot!
so was SourceForge, Gnome, KDE etc
Slide 30
Slide 30 text
but Github is hot!
so was SourceForge, Gnome, KDE etc
the Github Archive project offers a subset
of the data in an easier to query format
Slide 31
Slide 31 text
SQO-OSS GHTorrent
Tools pluggable platform
ruby library and
cmd-line tools
Data
initially none, then
raw repos
raw + processed
Documentation
mostly
programming
mostly data
formats
Released after 2 years immediately
Attracted first
user
2 years 2 months
Slide 32
Slide 32 text
open source research
Slide 33
Slide 33 text
aim for lean and mean
Slide 34
Slide 34 text
infrastructures and
platforms are overrated
Slide 35
Slide 35 text
open now trumps
open when it’s done