Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Platforms for Data Science
Search
Deepak Singh
October 01, 2011
Technology
3
190
Platforms for Data Science
Talk given at the "Computing on the Brink" series
Deepak Singh
October 01, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
93
FGED Keynote
mndoci
3
89
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Intel Theater Presentation @ SC11
mndoci
6
170
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Other Decks in Technology
See All in Technology
Grafana MCP serverでなんかし隊 / Try Grafana MCP server
kohbis
0
230
Google I/O 2025 Keynote & Developer Keynote Overview
yanzm
0
100
CloudBruteによる外部からのS3バケットの探索・公開の発見について / 20250605 Kumiko Henmi
shift_evolve
3
360
libsyncrpcってなに?
uhyo
0
250
Cursor Meetup Tokyo
iamshunta
5
1.5k
dbt Cloudの新機能を紹介!データエンジニアリングの民主化:GUIで操作、SQLで管理する新時代のdbt Cloud
sagara
0
130
Agent Development Kit によるエージェント開発入門
enakai00
18
2.6k
Applied NLP in the Age of Generative AI: Future-Proof Strategies for Banking and Finance
inesmontani
PRO
0
220
ソフトウェアテストのAI活用_ver1.20
fumisuke
0
220
FASTと向き合うことで見えた、大規模アジャイルの難しさと楽しさ
wooootack
0
250
20250514_未経験から Fintech実務参画まで。学生エンジニアの挑戦録
hideto1008
0
880
JavaのMCPサーバーで体験するAIエージェントの世界
tatsuya1bm
1
210
Featured
See All Featured
Site-Speed That Sticks
csswizardry
9
620
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Mobile First: as difficult as doing things right
swwweet
223
9.6k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Designing for humans not robots
tammielis
253
25k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
Embracing the Ebb and Flow
colly
85
4.7k
Statistics for Hackers
jakevdp
799
220k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
123
52k
Making the Leap to Tech Lead
cromwellryan
134
9.3k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
Transcript
There is no magic There is only awesome D e
e p a k S i n g h Platforms for data science
bioinformatics image: Ethan Hein
3
collection
curation
analysis
what’s the big deal?
None
Source: http://www.nature.com/news/specials/bigdata/index.html
Image: Yael Fitzpatrick (AAAS)
Image: Yael Fitzpatrick (AAAS)
lots of data
lots of people
lots of places
constant change
we want to make our data more effective
versioning
provenance
filter
aggregate
extend
mashup
human interfaces
None
image: Leo Reynolds
hard problem
really hard problem
so how do get there?
information platforms
Image: Drew Conway
dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise
of the data scientist, Beautiful Data
the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent
Systems, 24, 8-12 (2009)
accept all data formats
evolve APIs
beyond databases and the data warehouse
data as a programmable resource
data is a royal garden
compute is a fungible commodity
optimizing the most valuable resource
compute, storage, workflows, memory, transmission, algorithms, cost, …
people Credit: Pieter Musterd a CC-BY-NC-ND license
Image: Chris Dagdigian
my bias
cloud services
distributed systems
scale
global
consumption models
on-demand
what is the value of your data?
None
None
Credit: Angel Pizzaro, U. Penn
mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
None
Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/
None
None
30,472 cores
$1279/hr
http://cloudbiolinux.org/
http://usegalaxy.org/cloud
in summary
large scale data requires a rethink
data architecture
compute architecture
distributed, programmable infrastructure
cloud services
remove constraints
can we build data science platforms?
there is no magic there is only awesome
deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood&
Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license