Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beating State-of-the-art By -10000% @ CIDR Gong...
Search
Reynold Xin
January 07, 2013
Research
1
130
Beating State-of-the-art By -10000% @ CIDR Gong Show
I gave a 5-min Gong Show talk at CIDR on my experience with Spark, Shark, and GraphX.
Reynold Xin
January 07, 2013
Tweet
Share
More Decks by Reynold Xin
See All by Reynold Xin
(Berkeley CS186 guest lecture) Big Data Analytics Systems: What Goes Around Comes Around
rxin
12
1.9k
Interface Design for Spark Community
rxin
12
1.3k
Spark Committer Night meetup @ NYC
rxin
1
110
Apache Spark: Unified Platform for Big Data
rxin
1
210
Advanced Spark @ Spark Summit 2014
rxin
4
310
Apache Spark: Easier and Faster Big Data
rxin
2
270
GraphX at Spark User Meetup
rxin
0
140
Shark SIGMOD research deck
rxin
2
490
The Spark Ecosystem: Fast and Expressive Big Data Analytics in Scala @ Scala Days 2013
rxin
3
690
Other Decks in Research
See All in Research
Data-centric AI勉強会 「ロボットにおけるData-centric AI」
haraduka
0
480
Weekly AI Agents News! 12月号 プロダクト/ニュースのアーカイブ
masatoto
0
330
PhD Defence: Considering Temporal and Contextual Information for Lexical Semantic Change Detection
a1da4
0
130
論文紹介: COSMO: A Large-Scale E-commerce Common Sense Knowledge Generation and Serving System at Amazon (SIGMOD 2024)
ynakano
1
410
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.7k
Large Vision Language Model (LVLM) に関する最新知見まとめ (Part 1)
onely7
24
6k
Whoisの闇
hirachan
3
310
The Economics of Platforms 輪読会 第1章
tomonatu8
0
150
コミュニティドライブプロジェクト
smartfukushilab1
0
220
アプリケーションから知るモデルマージ
maguro27
0
260
大規模言語モデルを用いたニュースデータのセンチメント判定モデルの開発および実体経済センチメントインデックスの構成
nomamist
1
130
セミコン地域における総合交通戦略
trafficbrain
0
120
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
68
4.6k
Site-Speed That Sticks
csswizardry
4
420
Scaling GitHub
holman
459
140k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7.1k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
650
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
1.1k
How to train your dragon (web standard)
notwaldorf
91
5.9k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
GitHub's CSS Performance
jonrohan
1030
460k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
Measuring & Analyzing Core Web Vitals
bluesmoon
6
260
Transcript
Beating State-of-the-art By -10000% Reynold Xin, AMPLab, UC Berkeley with
help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
Beating State-of-the-art By -10000% NOT A TYPO Reynold Xin, AMPLab,
UC Berkeley with help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
MapReduce deterministic, idempotent tasks fault-tolerance elasticity resource sharing
“The bar for open source software is at historical low.”
“The bar for open source software is at historical low.”
i.e. “This is the right time to do grad school.”
iterative machine learning OLAP strong temporal locality
Does in-memory computation help in petabyte-scale warehouses?
Does in-memory computation help in petabyte-scale warehouses? YES
Spark How to do in-memory computation efficiently in a fault-tolerant
way?
Shark How to do SQL query processing efficiently in “MapReduce”
style SQL on top of Spark Hive compatible (UDF, Type, InputFormat, Metadata)
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.”
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.” i.e. “You should’ve come to grad school 2 years earlier.”
Shark in-memory columnar store dynamic query re-optimization and a lot
of engineering...
Query 1 Query 2 Log Regress 0 20 40 60
80 100 120 110 94 64 0.96 1 0.7 Runtime (seconds) on a 100-node EC2 cluster Shark/Spark Hive/Hadoop
iterative machine learning SQL query processing
iterative machine learning SQL query processing graph computation
GraphLab on Spark
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. Not bad for a day of work!
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. but I later found out that it is still 10X slower than the latest version of GraphLab :(
A lot of open questions for fault- tolerant, distributed graph
computation. “MapReduce”? Data partitioning? Fault-tolerance? Asynchrony?
iterative machine learning www.spark-project.org SQL query processing shark.cs.berkeley.edu graph computation
www.wait-another-year.com