Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A tale of two datasets
Search
Georgios Gousios
September 25, 2013
Technology
0
490
A tale of two datasets
Presentation given at the 2013 ICSM panel on Open Access
Georgios Gousios
September 25, 2013
Tweet
Share
More Decks by Georgios Gousios
See All by Georgios Gousios
NLP + SE = ❤️
gousiosg
0
290
The troubles of modern dependency management and what to do about them
gousiosg
0
540
Mining Repositories with Apache Spark
gousiosg
0
660
My adventures with open everything
gousiosg
0
300
Structure and Evolution of Package Dependency Networks
gousiosg
0
780
Mining Github for fun and profit
gousiosg
9
63k
GitHub Insights: Understanding Open Source
gousiosg
0
370
Work Practices and Challenges in Pull-Based Development: The Contributor’s Perspective
gousiosg
0
920
Big Data in Software Engineering panel and Privacy: Should we care?
gousiosg
0
290
Other Decks in Technology
See All in Technology
RSCの時代にReactとフレームワークの境界を探る
uhyo
9
2.4k
Nstockの一人目エンジニアが 3年間かけて向き合ってきた セキュリティのこととこれから〜あれから半年〜
yo41sawada
0
190
サンドボックス技術でAI利活用を促進する
koh_naga
0
160
20250903_1つのAWSアカウントに複数システムがある環境におけるアクセス制御をABACで実現.pdf
yhana
2
290
iPhone Eye Tracking機能から学ぶやさしいアクセシビリティ
fujiyamaorange
0
350
衝突して強くなる! BLUE GIANTと アジャイルチームの共通点とは ― いきいきと活気に満ちたグルーヴあるチームを作るコツ ― / BLUE GIANT and Agile Teams
naitosatoshi
0
290
AIエージェントの活用に重要な「MCP (Model Context Protocol)」とは何か
masayamoriofficial
0
250
Obsidian応用活用術
onikun94
0
230
LLM翻訳ツールの開発と海外のお客様対応等への社内導入事例
gree_tech
PRO
0
450
Webアクセシビリティ入門
recruitengineers
PRO
3
1.5k
モダンフロントエンド 開発研修
recruitengineers
PRO
10
6.3k
Grafana Meetup Japan Vol. 6
kaedemalu
1
200
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Optimizing for Happiness
mojombo
379
70k
A designer walks into a library…
pauljervisheath
207
24k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Bash Introduction
62gerente
614
210k
Producing Creativity
orderedlist
PRO
347
40k
Art, The Web, and Tiny UX
lynnandtonic
302
21k
Code Review Best Practice
trishagee
70
19k
4 Signs Your Business is Dying
shpigford
184
22k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Agile that works and the tools we love
rasmusluckow
330
21k
Transcript
A tale of two datasets Georgios Gousios TU Delft
open access
None
None
None
Software Quality Observatory for OSS
None
50k LOC!
750 OSS repositories, SVN, bugs, emails 1.5GB processed data
dump
demo.sqo-oss.org
SQO-OSS facts • 6 partners • ~30 publications • 4
PhDs funded • press releases on project releases • rated excellent by the EC
1 external user 2 external publications
None
GHTorrent
GHTorrent
mirror event stream
<<event>> PushEvent <<api>> /users/:user ensure_user <<api>> /repos/:user/:repo/ ensure_repo <<api>> /repos/:user/:repo/commits
ensure_commits ensure_user <<api>> /:user/:repo/sha ensure_commit ensure_user <<api>> /users/:user/ followers ensure_followers <<api>> /repos/:user/:repo/ commits/:sha/comments ensure_commit_comments <<api>> /users/:user/orgs ensure_orgs <<api>> /orgs/:org/teams ensure_teams recursive dependency retrieval
relational database
repositories users organizations issues /users/:user /user/repos /repos/:user/:repo/issues /orgs/:org {
"type": "User", "public_gists": 0, "login": "gousiosg", "followers": 8, "name": "Georgios Gousios", "public_repos": 4, "created_at": ..., "id": 386172, "following": 4, } { . . . NoSQL database as cache
periodic dumps of DBs online
query DBs online
GHTorrent facts • 2.0 TB in MongoDB, 40GB in MySQL
• 1 developer • 3 papers • advertised on social media • 1.5 years
5 external users 3 external papers MSR14 challenge dataset
why the difference?
but Github is hot!
but Github is hot!
but Github is hot! so was SourceForge, Gnome, KDE etc
but Github is hot! so was SourceForge, Gnome, KDE etc
the Github Archive project offers a subset of the data in an easier to query format
SQO-OSS GHTorrent Tools pluggable platform ruby library and cmd-line tools
Data initially none, then raw repos raw + processed Documentation mostly programming mostly data formats Released after 2 years immediately Attracted first user 2 years 2 months
open source research
aim for lean and mean
infrastructures and platforms are overrated
open now trumps open when it’s done