Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large scale graph processing with apache giraph

Large scale graph processing with apache giraph


André Kelpe

May 23, 2012


  1. Large scale graph processing with apache giraph André Kelpe @fs111

  2. graphs 101

  3. vertices and edges

  4. v2 v5 v4 v7 v3 v8 v6 v1 v9 v8

    v10 simple graph
  5. graphs are everywhere road network, the www, social graphs etc.

  6. graphs can be huge

  7. google knows!

  8. Pregel

  9. Pregel by google Describes graph processing approach based on BSP

    (Bulk Synchronous Parallel)
  10. pro-tip: search for „pregel_paper.pdf“ on github ;-)

  11. Properties of Pregel batch-oriented, scalable, fault tolerant processing of graphs

  12. It is not a graph database It is a processing

  13. BSP vertex centric processing in so called supersteps

  14. BSP vertices send messages to each other

  15. BSP synchronization points between supersteps

  16. execution of superstep S Each vertex processes messages generated in

    S-1 and send messages to be processed in S+1 and determines to halt.
  17. None
  18. apache giraph

  19. giraph Loose implementation of Pregel ideas on top of Hadoop

    M/R coming from yahoo
  20. apache giraph http://incubator.apache.org/giraph/

  21. giraph avoid overhead of classic M/R process but reuse existing

  22. giraph simple map jobs in master worker setup. coordination via

    zookeeper. messaging via own RPC protocol. in memory processing. custom input and output formats.
  23. current status version 0.1 released compatible with a multitude of

    hadoop versions (we use CDH3 at work) still lots of things to do, join the fun!
  24. the APIs the APIs

  25. Vertex-API /** *@param <I> vertex id * @param <V> vertex

    data * @param <E> edge data * @param <M> message data */ class BasicVertex<I extends WritableComparable, V extends Writable, E extends Writable, M extends Writable> void compute(Iterator<M> msgIterator); void sendMsg(I id, M msg); void voteToHalt();
  26. Shortest path example https://cwiki.apache.org/confl uence/display/GIRAPH/Shorte st+Paths+Example

  27. v2 v5 v4 v7 v3 v8 v6 v1 v9 v8

    v10 simple graph
  28. private boolean isSource() { return (getVertexId().get() == getContext().getConfiguration().getLong(SOURCE_ID, SOURCE_ID_DEFAULT)); }

    @Override public void compute(Iterator<DoubleWritable> msgIterator) { if (getSuperstep() == 0) { setVertexValue(new DoubleWritable(Double.MAX_VALUE)); } double minDist = isSource() ? 0d : Double.MAX_VALUE; while (msgIterator.hasNext()) { minDist = Math.min(minDist, msgIterator.next().get()); } if (minDist < getVertexValue().get()) { setVertexValue(new DoubleWritable(minDist)); for (Edge<LongWritable, FloatWritable> edge : getOutEdgeMap().values()) { sendMsg(edge.getDestVertexId(), new DoubleWritable(minDist + edge.getEdgeValue().get())); } } voteToHalt(); }
  29. GiraphJob job = new GiraphJob(getConf(), getClass().getName()); job.setVertexClass(SimpleShortestPathVertex.class); job.setVertexInputFormatClass(SimpleShortestPathsVertexInputFormat.class); job.setVertexOutputFormatClass( SimpleShortestPathsVertexOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path(„/foo/bar/baz“)); FileOutputFormat.setOutputPath(job, new Path(„/foo/bar/quux“)); job.getConfiguration().setLong(SimpleShortestPathsVertex.SOURCE_ID, Long.parseLong(argArray[2])); job.setWorkerConfiguration(minWorkers, maxWorkers), 100.0f); GiraphJob
  30. see also http://incubator.apache.org/giraph/ https://cwiki.apache.org/confluence/displ ay/GIRAPH/Shortest+Paths+Example http://googleresearch.blogspot.com/2009/ 06/large-scale-graph-computing-at- google.html

  31. Thanks! Questions?