Large scale graph processing with apache giraph

Large scale graph processing with apache giraph André Kelpe @fs111

http://kel.pe

graphs 101

vertices and edges

v2 v5 v4 v7 v3 v8 v6 v1 v9 v8

v10 simple graph

graphs are everywhere road network, the www, social graphs etc.

graphs can be huge

google knows!

Pregel

Pregel by google Describes graph processing approach based on BSP

(Bulk Synchronous Parallel)

pro-tip: search for „pregel_paper.pdf“ on github ;-)

Properties of Pregel batch-oriented, scalable, fault tolerant processing of graphs

It is not a graph database It is a processing

framework

BSP vertex centric processing in so called supersteps

BSP vertices send messages to each other

BSP synchronization points between supersteps

execution of superstep S Each vertex processes messages generated in

S-1 and send messages to be processed in S+1 and determines to halt.

None

apache giraph

giraph Loose implementation of Pregel ideas on top of Hadoop

M/R coming from yahoo

apache giraph http://incubator.apache.org/giraph/

giraph avoid overhead of classic M/R process but reuse existing

infrastructure

giraph simple map jobs in master worker setup. coordination via

zookeeper. messaging via own RPC protocol. in memory processing. custom input and output formats.

current status version 0.1 released compatible with a multitude of

hadoop versions (we use CDH3 at work) still lots of things to do, join the fun!

the APIs the APIs

Vertex-API /** *@param <I> vertex id * @param <V> vertex

data * @param <E> edge data * @param <M> message data */ class BasicVertex<I extends WritableComparable, V extends Writable, E extends Writable, M extends Writable> void compute(Iterator<M> msgIterator); void sendMsg(I id, M msg); void voteToHalt();

Shortest path example https://cwiki.apache.org/confl uence/display/GIRAPH/Shorte st+Paths+Example

v2 v5 v4 v7 v3 v8 v6 v1 v9 v8

v10 simple graph

private boolean isSource() { return (getVertexId().get() == getContext().getConfiguration().getLong(SOURCE_ID, SOURCE_ID_DEFAULT)); }

@Override public void compute(Iterator<DoubleWritable> msgIterator) { if (getSuperstep() == 0) { setVertexValue(new DoubleWritable(Double.MAX_VALUE)); } double minDist = isSource() ? 0d : Double.MAX_VALUE; while (msgIterator.hasNext()) { minDist = Math.min(minDist, msgIterator.next().get()); } if (minDist < getVertexValue().get()) { setVertexValue(new DoubleWritable(minDist)); for (Edge<LongWritable, FloatWritable> edge : getOutEdgeMap().values()) { sendMsg(edge.getDestVertexId(), new DoubleWritable(minDist + edge.getEdgeValue().get())); } } voteToHalt(); }

GiraphJob job = new GiraphJob(getConf(), getClass().getName()); job.setVertexClass(SimpleShortestPathVertex.class); job.setVertexInputFormatClass(SimpleShortestPathsVertexInputFormat.class); job.setVertexOutputFormatClass( SimpleShortestPathsVertexOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(„/foo/bar/baz“)); FileOutputFormat.setOutputPath(job, new Path(„/foo/bar/quux“)); job.getConfiguration().setLong(SimpleShortestPathsVertex.SOURCE_ID, Long.parseLong(argArray[2])); job.setWorkerConfiguration(minWorkers, maxWorkers), 100.0f); GiraphJob

see also http://incubator.apache.org/giraph/ https://cwiki.apache.org/confluence/displ ay/GIRAPH/Shortest+Paths+Example http://googleresearch.blogspot.com/2009/ 06/large-scale-graph-computing-at- google.html

Thanks! Questions?