Edgelixir: Distributed Graph Processing in Elixir

Distributed graph processing systems are an important part of the modern data toolkit. Graph structures are a great way to solve a lot of problems, and distributed systems let us scale our problem size. This talk walks through how some of Elixir's features (including OTP, macros, and functional programming itself) can come together to simplify writing a distributed graph system. The result is Edgelixir; an early, work-in-progress, Pregel-based graph framework.

Transcript

1. Distributed graph processing in Elixir Edgelixir: Nathan Lapierre @n_lap
2. Goals for this talk • Learning more about Elixir, OTP, and the Erlang VM • I don’t assume any special background knowledge • Using Elixir for distributed data processing (“Big Data”) • Working with graphs in Elixir
3. Talk Outline • Why Graphs? • Distributed graph processing • Pregel model and existing frameworks • Edgelixir • Why Elixir makes a lot of sense! • How to use Edgelixir • Dive into Edgelixir’s code • Summary & Questions/Feedback
4. 0 10 20 30 40 50 60 70 80 90 100 4 3 2 1 % Prepared Weeks Remaining Graph: Presentation preparedness vs. weeks until conference Graph: New conference friends So we’re on the same page… Terms:Vertex (node), Edge (arc), weight, (un)directed
5. Why Graphs? • We can model a lot of problems as graphs • Social networks • Web graph • Science • Constraint satisfaction • A lot more… • We have a lot of well understood machinery to apply to graphs • Topology, distance, coloring , centrality, clustering… • This makes solving problems with graphs very intuitive • … and satisfying!
6. Distributed Graph Processing • Our graphs can easily be too big to process on one computer • Too large (RAM/Disk-bound) • Too slow (CPU-bound) • Need distributed processing to scale computing power with our problem size • Multicore • Multi-computer • Graphs aren’t as easy as other workloads to distribute • Less obvious how to divide the work while keeping communication overhead low
7. Pregel • Malewicz et al. (2010) • Graph distribution by message passing • Goals: • Many algorithms implemented in one system • “Think like a vertex” programming • Fault tolerant • Scalable Figure from Giraph Getting Started Docs
8. Open Source Pregel-based Frameworks Most popular: • Apache Giraph • JVM-based, Java • Built on Hadoop MapReduce • GraphX • JVM-based, Scala • Built on Apache Spark • Others… Both are: • Mature projects used in production at scale • A little bit terrifying • Thousands of LoCs • Complicated to deploy and configure
9. Pregel in Elixir • The Erlang VM, and the Elixir language/standard library have everything we need to implement Pregel out-of-the-box • Erlang OTP has decades of distributed computing knowledge baked in • Let’s build a concise Pregel-like Elixir package using the most appropriate Elixir features
10. The Toolbox • :digraph • ETS-backed mutable graph storage in Erlang standard library • :global • Globally name registration across cluster • Caveat: scalability issues, OTP team working on it • GenServer • Message passing and state • Behaviours and Protocols • Graph input/output extensibility
11. Edgelixir Inputs: A graph, compute/3 Output: A graph Graph persistence (:digraph/ETS) Edgelixir OTP app Node N Superstep Supervisor (GenServer) Vertex compute 1 Vertex compute V …
12. EdgelixirExample • mix.exs
13. EdgelixirExample • config.exs
15. Running EdgelixirExample • system.config [{kernel, [ {sync_nodes_optional, ['node1@127.0.0.1', 'node2@127.0.0.1']}, {sync_nodes_timeout, 10000} ] }]. • Running: iex --name node1@127.0.0.1 --erl "-config sys.config" -S mix
16. Reading the Graph •Graph data is loaded and parsed • Edgelixir.GraphSource, Edgelixir.GraphFormat •Vertices/edges are partitioned • Edgelixir.GraphPartition •If the vertex/edge belongs on the current node, store in node’s :digraph
17. Superstep Supervisor: Distributed Sync Barrier • Nothing exactly like this built into OTP • (:global locks might work) • Can build a simple one with GenServer • Assumption: no new nodes are coming online after starting • GenServer state holds # of nodes that have hit the barrier so far • Compares to # of nodes in the cluster
18. Superstep Supervisor: Message passing • Messages come from compute/3 • Edgelixir.Superstep.message_neighbours/2 • Checks node partition during send • Sends only to necessary node • Messages are stored in GenServer state • Messages are passed to compute/3 during each superstep
19. Vertex Processes •For each local vertex, just spawn compute/3 (Process.spawn/4) •Let the VM handle scheduling
20. Future Work •Aggregation •Easy: GenServer or Agent •Other behaviour functions •Dangling edges, etc •Better fault tolerance
21. Github, Hex, docs: edgelixir.net Questions & Feedback @n_lap nathan@lapierre.ca