usages • GraphX introduction • Pregel • Code example The main focus will be the programming model The code is available at: https://github.com/andreaiacono/TalkGraphX
run of MapReduce reads from disk (e.g. HDFS) the initial data, computes the results and then stores them on disk; since most algorithms on graphs are iterative, this means that for every iteration the whole data must be read and written from/to disk. It's better to use a distributed dataflow framework
system built on top of Apache Spark “Graph processing systems represent graph structured data as a property graph, which associates user-defined properties with each vertex and edge.” “The Spark storage abstraction called Resilient Distributed Datasets (RDDs) enables applications to keep data in memory, which is essential for iterative graph algorithms.” “RDDs permit user-defined data partitioning, and the execution engine can exploit this to co-partition RDDs and co-schedule tasks to avoid data movement. This is essential for encoding partitioned graphs.” Excerpt from GraphX: Graph Processing in a Distributed Dataflow Framework https://amplab.cs.berkeley.edu/wp-content/uploads/2014/09/graphx.pdf
designed by Google (https://kowshik.github.io/JPregel/pregel_paper.pdf) It consists of a sequence of supersteps until termination. In each superstep, every vertex can: • modify its state or the one of any of its neighbours • receive the messages sent to it during the previous superstep • send messages to its neighbours (that will be received in next superstep) • vote to halt When a node votes to halt, it goes to inactive state; if in a later superstep it receives a message, the framework will awake it changing its state to active. When all the nodes have voted to halt, the computation stops; otherwise it can be set a maximum number of iteration. Edges don't have any computation. When writing algorithms, you have to think as a vertex.
uses three functions for implementing Pregel: • vprog: the vertex program computed for each vertex that receives the incoming message and computes a new vertex value • sendMsg: the function used for sending messages to other vertices • mergeMsg: a function that takes two incoming messages and merges them into a single message Unlike Google's Pregel, GraphX implementation of Pregel: • leave the message construction out of the vertex-program, so to have a more efficient distributed execution • permits access to both vertices attributes of an edge while building the messages • contraints sending messages to graph structure (only to neighbours)
algorithms that: • respect the neighborhood structure GraphX is NOT well suited for algorithms that: • need iteration among distant vertices • change the structure of the graph When to use GraphX
List[City]()), maxIterations = Int.MaxValue, activeDirection = EdgeDirection.Out // the direction of edges on which to run `sendMsg` ) ( // vprog (returns the new vertex attribute for this vertex) (vertexId, currentVertexAttr, newVertexAttr) => if (currentVertexAttr.distance <= newVertexAttr.distance) currentVertexAttr else newVertexAttr, // sendMsg (sends a new VertexAttribute to the neighbours) edgeTriplet => if (edgeTriplet.srcAttr.distance < (edgeTriplet.dstAttr.distance - edgeTriplet.attr)) { Iterator(( edgeTriplet.dstId, new VertexAttribute( edgeTriplet.dstAttr.cityName, edgeTriplet.srcAttr.distance + edgeTriplet.attr, edgeTriplet.srcAttr.path :+ new City(edgeTriplet.dstAttr.cityName, edgeTriplet.dstId ) ))) else Iterator.empty }, // mergeMsg (collapses all the incoming messages for a vertex – two at a time - into one) (attribute1, attribute2) => if (attribute1.distance < attribute2.distance) attribute1 else attribute2 ) Shortest path code sample (2/2)
andreaiacono
hollycummins
biacco42
harunatsujita
ivargrimstad
satoshi7190
manfredsteyer
akikogoto
euxn23
seike460
furukawayasuto
junk0612
qnighy
kneath
tammyeverts
thekraken
phodgson
swwweet
brianwarren
csswizardry
eileencodes
lynnandtonic
davidbonilla
shlominoach
quramy
