Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Platforms for Data Science
Search
Deepak Singh
October 01, 2011
Technology
3
180
Platforms for Data Science
Talk given at the "Computing on the Brink" series
Deepak Singh
October 01, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
92
Platforms for scientific data analysis
mndoci
3
79
FGED Keynote
mndoci
3
68
Open Mic Science - May 7, 2012
mndoci
4
1.2k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
240
A Platform for Data Science
mndoci
6
13k
Intel Theater Presentation @ SC11
mndoci
6
160
Talk at West Coast Association of Shared Directors meeting
mndoci
3
140
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
99
Other Decks in Technology
See All in Technology
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
37k
BPStudyの200回を中心にIT業界を振り返る。そしてこれから
haru860
3
410
データベース03: 関係データモデル
trycycle
0
100
MapLibreとAmazon Location Service
dayjournal
1
190
Documentação de Produtos: Artefatos essenciais na prática
rigolon
1
160
20分で完全に理解するGrafanaダッシュボード
hamadakoji
5
910
社内アプリで Cloudflare D1を プロダクト運用してみた体験談(Tokyo)
haochenx
0
120
M&A戦略を支えるデータマネジメント (MIDAS Tech Study #16 GENDA Komiyama)
kommy339
1
120
uvを使ってストレスフリーな Python開発をしよう!
r74tech
0
100
止まらないLinuxシステムを構築する_高信頼性クラスタ入門
koedoyoshida
2
470
生成AIの変革の時代に、直近1年で直面した課題とその解決策
ktc_wada
0
700
Cypress or Playwright?
rainerhahnekamp
0
170
Featured
See All Featured
Building Flexible Design Systems
yeseniaperezcruz
320
37k
Designing for humans not robots
tammielis
247
25k
Happy Clients
brianwarren
92
6.4k
Designing Experiences People Love
moore
136
23k
Unsuck your backbone
ammeep
663
57k
Making Projects Easy
brettharned
109
5.5k
Agile that works and the tools we love
rasmusluckow
325
20k
A better future with KSS
kneath
231
16k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
660
120k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
For a Future-Friendly Web
brad_frost
172
9k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
245
20k
Transcript
There is no magic There is only awesome D e
e p a k S i n g h Platforms for data science
bioinformatics image: Ethan Hein
3
collection
curation
analysis
what’s the big deal?
None
Source: http://www.nature.com/news/specials/bigdata/index.html
Image: Yael Fitzpatrick (AAAS)
Image: Yael Fitzpatrick (AAAS)
lots of data
lots of people
lots of places
constant change
we want to make our data more effective
versioning
provenance
filter
aggregate
extend
mashup
human interfaces
None
image: Leo Reynolds
hard problem
really hard problem
so how do get there?
information platforms
Image: Drew Conway
dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise
of the data scientist, Beautiful Data
the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent
Systems, 24, 8-12 (2009)
accept all data formats
evolve APIs
beyond databases and the data warehouse
data as a programmable resource
data is a royal garden
compute is a fungible commodity
optimizing the most valuable resource
compute, storage, workflows, memory, transmission, algorithms, cost, …
people Credit: Pieter Musterd a CC-BY-NC-ND license
Image: Chris Dagdigian
my bias
cloud services
distributed systems
scale
global
consumption models
on-demand
what is the value of your data?
None
None
Credit: Angel Pizzaro, U. Penn
mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
None
Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/
None
None
30,472 cores
$1279/hr
http://cloudbiolinux.org/
http://usegalaxy.org/cloud
in summary
large scale data requires a rethink
data architecture
compute architecture
distributed, programmable infrastructure
cloud services
remove constraints
can we build data science platforms?
there is no magic there is only awesome
[email protected]
Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood&
Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license