Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Large scale graph processing with apache giraph
Search
André Kelpe
May 23, 2012
Programming
2
5.4k
Large scale graph processing with apache giraph
André Kelpe
May 23, 2012
Tweet
Share
More Decks by André Kelpe
See All by André Kelpe
Cascading 3 and beyond
fs111
0
160
The Cascading (big) data application framework
fs111
1
230
SELECT ALL THE THINGS - Cascading Lingual, ANSI SQL for Apache Hadoop
fs111
0
200
A whirlwind tour through Lingual: ANSI SQL for Apache Hadoop
fs111
1
190
Tor for everyone!
fs111
0
180
Other Decks in Programming
See All in Programming
イベントソーシングによってインピーダンスミスマッチから解放された話
tkawae
1
310
eBPF Updates (March 2025)
kentatada
0
120
Go1.24で testing.B.Loopが爆誕
kuro_kurorrr
0
140
Node.js, Deno, Bun 最新動向とその所感について
yosuke_furukawa
PRO
6
3k
Devinのメモリ活用の学びを自社サービスにどう組み込むか?
itarutomy
0
1.3k
보일러플레이트 코드가 진짜 나쁜 건가요?
gaeun5744
0
360
2025/3/18 サービスの成長で生じる幅広いパフォーマンスの問題を、 AIで手軽に解決する
shirahama_x
0
150
Going Structural with Named Tuples
bishabosha
0
140
Functional APIから再考するLangGraphを使う理由
os1ma
4
640
Denoでフロントエンド開発 2025年春版 / Frontend Development with Deno (Spring 2025)
petamoriken
1
1.3k
Django for Data Science (Boston Python Meetup, March 2025)
wsvincent
0
190
OpenTelemetryを活用したObservability入門 / Introduction to Observability with OpenTelemetry
seike460
PRO
0
190
Featured
See All Featured
Building an army of robots
kneath
304
45k
Raft: Consensus for Rubyists
vanstee
137
6.8k
Typedesign – Prime Four
hannesfritz
41
2.6k
GitHub's CSS Performance
jonrohan
1030
460k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Embracing the Ebb and Flow
colly
84
4.6k
Building Flexible Design Systems
yeseniaperezcruz
328
38k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
12k
Into the Great Unknown - MozCon
thekraken
35
1.7k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
490
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
8
690
Large-scale JavaScript Application Architecture
addyosmani
511
110k
Transcript
Large scale graph processing with apache giraph André Kelpe @fs111
http://kel.pe
graphs 101
vertices and edges
v2 v5 v4 v7 v3 v8 v6 v1 v9 v8
v10 simple graph
graphs are everywhere road network, the www, social graphs etc.
graphs can be huge
google knows!
Pregel
Pregel by google Describes graph processing approach based on BSP
(Bulk Synchronous Parallel)
pro-tip: search for „pregel_paper.pdf“ on github ;-)
Properties of Pregel batch-oriented, scalable, fault tolerant processing of graphs
It is not a graph database It is a processing
framework
BSP vertex centric processing in so called supersteps
BSP vertices send messages to each other
BSP synchronization points between supersteps
execution of superstep S Each vertex processes messages generated in
S-1 and send messages to be processed in S+1 and determines to halt.
None
apache giraph
giraph Loose implementation of Pregel ideas on top of Hadoop
M/R coming from yahoo
apache giraph http://incubator.apache.org/giraph/
giraph avoid overhead of classic M/R process but reuse existing
infrastructure
giraph simple map jobs in master worker setup. coordination via
zookeeper. messaging via own RPC protocol. in memory processing. custom input and output formats.
current status version 0.1 released compatible with a multitude of
hadoop versions (we use CDH3 at work) still lots of things to do, join the fun!
the APIs the APIs
Vertex-API /** *@param <I> vertex id * @param <V> vertex
data * @param <E> edge data * @param <M> message data */ class BasicVertex<I extends WritableComparable, V extends Writable, E extends Writable, M extends Writable> void compute(Iterator<M> msgIterator); void sendMsg(I id, M msg); void voteToHalt();
Shortest path example https://cwiki.apache.org/confl uence/display/GIRAPH/Shorte st+Paths+Example
v2 v5 v4 v7 v3 v8 v6 v1 v9 v8
v10 simple graph
private boolean isSource() { return (getVertexId().get() == getContext().getConfiguration().getLong(SOURCE_ID, SOURCE_ID_DEFAULT)); }
@Override public void compute(Iterator<DoubleWritable> msgIterator) { if (getSuperstep() == 0) { setVertexValue(new DoubleWritable(Double.MAX_VALUE)); } double minDist = isSource() ? 0d : Double.MAX_VALUE; while (msgIterator.hasNext()) { minDist = Math.min(minDist, msgIterator.next().get()); } if (minDist < getVertexValue().get()) { setVertexValue(new DoubleWritable(minDist)); for (Edge<LongWritable, FloatWritable> edge : getOutEdgeMap().values()) { sendMsg(edge.getDestVertexId(), new DoubleWritable(minDist + edge.getEdgeValue().get())); } } voteToHalt(); }
GiraphJob job = new GiraphJob(getConf(), getClass().getName()); job.setVertexClass(SimpleShortestPathVertex.class); job.setVertexInputFormatClass(SimpleShortestPathsVertexInputFormat.class); job.setVertexOutputFormatClass( SimpleShortestPathsVertexOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(„/foo/bar/baz“)); FileOutputFormat.setOutputPath(job, new Path(„/foo/bar/quux“)); job.getConfiguration().setLong(SimpleShortestPathsVertex.SOURCE_ID, Long.parseLong(argArray[2])); job.setWorkerConfiguration(minWorkers, maxWorkers), 100.0f); GiraphJob
see also http://incubator.apache.org/giraph/ https://cwiki.apache.org/confluence/displ ay/GIRAPH/Shortest+Paths+Example http://googleresearch.blogspot.com/2009/ 06/large-scale-graph-computing-at- google.html
Thanks! Questions?