Distributed Graph Processing with Scala and Akka (SVSS '13)

Distributed Graph Processing with Scala and Akka (SVSS '13)

In recent years, the boom of online social networks such as Facebook and Twitter have presented several interesting problems, especially in regards to their massive underlying graph structures. With such a large and rich dataset, it is clearly beneficial to leverage these graphs to power features like friendship recommendation. However, dealing with so much data in a scalable manner is difficult, and considerable amounts of engineering and research efforts have gone into solving this problem, manifesting into systems such as Pregel and graph databases (e.g. Neo4J).

As a research assistant during the school year at UC Santa Barbara, I have implemented a distributed graph processing system for use on the lab cluster. The system is designed specifically for trivially parallelizable graph algorithms (which most algorithms I've run in the lab are). Lots of code will be shown, both of the system and of applications written using the system. I will also talk briefly about what I have planned for future iterations of the system.

Fb8e986500c5059b2a6c0b2184bb0faf?s=128

Adelbert Chang

August 03, 2013
Tweet

Transcript

  1. 3.

    About Me •4th year student @ UC Santa Barbara •BS/MS

    Computer Science Saturday, August 3, 13
  2. 4.

    About Me •4th year student @ UC Santa Barbara •BS/MS

    Computer Science •Research Assistant •Large-scale graph mining and modeling •Cluster Computing Saturday, August 3, 13
  3. 5.

    About Me •4th year student @ UC Santa Barbara •BS/MS

    Computer Science •Research Assistant •Large-scale graph mining and modeling •Cluster Computing •Engineering Analytics Intern @ Box Saturday, August 3, 13
  4. 6.

    About Me •4th year student @ UC Santa Barbara •BS/MS

    Computer Science •Research Assistant •Large-scale graph mining and modeling •Cluster Computing •Engineering Analytics Intern @ Box •Scala since January 2012 Saturday, August 3, 13
  5. 14.

    Motivation •Many of our algorithms are embarassingly parallel •Pregel model

    is good, but too heavy for us Saturday, August 3, 13
  6. 15.

    Motivation •Many of our algorithms are embarassingly parallel •Pregel model

    is good, but too heavy for us •Example: Shortest path •Split work on nodes •Run BFS, return a Map[Int, Int] Saturday, August 3, 13
  7. 17.
  8. 18.

    Context + Assumptions •Studying large-scale static graphs, typically those found

    in online social networks •Cluster of around 30 machines Saturday, August 3, 13
  9. 19.

    Context + Assumptions •Studying large-scale static graphs, typically those found

    in online social networks •Cluster of around 30 machines •Cluster shares a file system Saturday, August 3, 13
  10. 20.

    Context + Assumptions •Studying large-scale static graphs, typically those found

    in online social networks •Cluster of around 30 machines •Cluster shares a file system •Graphs are large, but can fit into machine machine memory Saturday, August 3, 13
  11. 21.

    Context + Assumptions •Studying large-scale static graphs, typically those found

    in online social networks •Cluster of around 30 machines •Cluster shares a file system •Graphs are large, but can fit into machine machine memory •We want “raw” results dumped straight to disk Saturday, August 3, 13
  12. 25.

    User Requirements •Users should •Not have to interact with Akka

    •Only need to define the algorithm and the input Saturday, August 3, 13
  13. 26.

    User Requirements •Users should •Not have to interact with Akka

    •Only need to define the algorithm and the input •Be able to put an upper bound on number of threads per machine Saturday, August 3, 13
  14. 29.
  15. 30.

    System Requirements •The system should •Be easy to deploy without

    any cluster setup •Be fault tolerant Saturday, August 3, 13
  16. 31.

    System Requirements •The system should •Be easy to deploy without

    any cluster setup •Be fault tolerant •Be elastic Saturday, August 3, 13
  17. 32.

    System Requirements •The system should •Be easy to deploy without

    any cluster setup •Be fault tolerant •Be elastic •Graph should be loaded locally Saturday, August 3, 13
  18. 33.

    System Requirements •The system should •Be easy to deploy without

    any cluster setup •Be fault tolerant •Be elastic •Graph should be loaded locally •Clean up and shut itself down afterwards Saturday, August 3, 13
  19. 40.

    Inspiration •We want a balancing dispatcher for remoting •Proxy mailbox

    is backed by a number of Actors Saturday, August 3, 13
  20. 41.

    Inspiration •We want a balancing dispatcher for remoting •Proxy mailbox

    is backed by a number of Actors •Messages are sent to a proxy mailbox Saturday, August 3, 13
  21. 42.

    Inspiration •We want a balancing dispatcher for remoting •Proxy mailbox

    is backed by a number of Actors •Messages are sent to a proxy mailbox •Messages distributed to idle Actors Saturday, August 3, 13
  22. 46.

    Solution •Design the system to act similarly to a balancing

    dispatcher •A single Actor (Master) represents the dispatcher Saturday, August 3, 13
  23. 47.

    Solution •Design the system to act similarly to a balancing

    dispatcher •A single Actor (Master) represents the dispatcher •Each remote Actor (Worker) has it’s own mailbox Saturday, August 3, 13
  24. 48.

    Solution •Design the system to act similarly to a balancing

    dispatcher •A single Actor (Master) represents the dispatcher •Each remote Actor (Worker) has it’s own mailbox •Workers report to Masters when idle Saturday, August 3, 13
  25. 50.

    Design Decision •Akka is capable of both remote lookup and

    remote deployment Saturday, August 3, 13
  26. 51.

    Design Decision •Akka is capable of both remote lookup and

    remote deployment •Remote Deployment Saturday, August 3, 13
  27. 52.

    Design Decision •Akka is capable of both remote lookup and

    remote deployment •Remote Deployment •Master becomes connected to Worker automatically Saturday, August 3, 13
  28. 53.

    Design Decision •Akka is capable of both remote lookup and

    remote deployment •Remote Deployment •Master becomes connected to Worker automatically •Remote lookup Saturday, August 3, 13
  29. 54.

    Design Decision •Akka is capable of both remote lookup and

    remote deployment •Remote Deployment •Master becomes connected to Worker automatically •Remote lookup •Workers can be added/killed at runtime Saturday, August 3, 13
  30. 89.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    DoAlgorithm Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  31. 90.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkIsReady Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  32. 91.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkerRequestsWork Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  33. 92.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkToBeDone Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  34. 93.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  35. 94.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    HandleResult Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  36. 95.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkComplete Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  37. 96.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkIsDone Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13
  38. 97.

    Application Application Sabre Master ResultHandler Sabre.execute() system.actorOf system.actorOf Worker Worker

    WorkIsDone Worker Worker Worker Application Sabre Master ResultHandler Worker Worker Saturday, August 3, 13