Graph Algorithms: Predict Real-World Behavior

Learn how graph algorithms can help you predict real-world behavior and why an averages approach fails to describe group dynamics. Get an overview of which algorithms to apply for various types of optimal paths, influence in a network, and community detection and see use cases that span across industries including recommendations, resiliency planning, fraud prevention, and traffic engineering/routing (such as IP and call). From this session, you will gain the knowledge to recognize whether you have a graph analytics problem and how you can get started.

Transcript

1. Graph Algorithms: 
 Predict Real-World Behavior Groups and Routes and Flows - Oh My! 
 Jennifer Reif
2. Jennifer Reif Software Engineer, Neo4j - Developer - Blogger - Conference speaker Love my cat, coffee, and traveling :) jennifer.reif@neo4j.com @jmhreif Graph Algorithms The Right Tool for Real-World Networks
3. 3 Photo: US National Archives
4. Averages 
 Will Fail You
5. Do You Have a Graph Analytics Problem? Forecast Behavior & Prescribe Action
6. Do You Have a Graph Analytics Problem? Propagation Pathways Forecast Behavior & Prescribe Action
7. Do You Have a Graph Analytics Problem? Flow & Dynamics Propagation Pathways Forecast Behavior & Prescribe Action
8. Do You Have a Graph Analytics Problem? Requires Understanding Relationships and Structures Flow & Dynamics Interactions & Resiliency Propagation Pathways Forecast Behavior & Prescribe Action
9. Averages Aren’t Reality Nodes Relationships Average Distribution
 - Random - “There is No Network in Nature that we know of that would be described by the Random network model.” 
 –Albert-László Barabási
10. Averages Aren’t Reality Nodes Relationships Average Distribution
 - Random - Nodes Relationships Power Law Distribution - Scale-Free -
11. Most nodes have the same number of links No highly connected nodes - Scale-Free - - Small World - And You’ll Never See the Structures Nodes Relationships Average Distribution
 - Random -
12. Graph Algorithms
 Extract Structure and Infer Behavior Source: “Communities, modules and large-scale structure in networks“ - Mark Newman Source: “Hierarchical structure and the prediction of missing links in networks”; ”Structure and inference in annotated networks” - A. Clauset, C. Moore, and M.E.J. Newman.
13. Graph Algorithms
 Extract Structure and Infer Behavior Source: “Communities, modules and large-scale structure in networks“ - Mark Newman Source: “Hierarchical structure and the prediction of missing links in networks”; ”Structure and inference in annotated networks” - A. Clauset, C. Moore, and M.E.J. Newman.
14. Understand & Predict
15. Graph Algorithms
16. Graph Algorithms Finds the optimal path or evaluates route availability and quality Pathfinding 
 & Search
17. Graph Algorithms Finds the optimal path or evaluates route availability and quality Pathfinding 
 & Search Determines the importance of distinct nodes in the network Centrality
18. Graph Algorithms Finds the optimal path or evaluates route availability and quality Pathfinding 
 & Search Determines the importance of distinct nodes in the network Centrality Evaluates how a group is clustered or partitioned Community Detection
19. • Single-Source Shortest Path ◦ Calculates “shortest” path between a node and all other nodes Algorithms - Pathfinding & Search • All-Pairs Shortest Path ◦ Finds all shortest paths between all nodes
20. • Single-Source Shortest Path ◦ Calculates path between a node and all other nodes Algorithms - Pathfinding & Search • All-Pairs Shortest Path ◦ Calculates shortest path group with all shortest paths between nodes • Minimum Weight Spanning Tree ◦ Calculates the path with the smallest value for visiting all nodes Least Cost Routing
21. • Single-Source Shortest Path ◦ Calculates path between a node and all other nodes Algorithms - Pathfinding & Search • All-Pairs Shortest Path ◦ Calculates shortest path group with all shortest paths between nodes • Minimum Weight Spanning Tree ◦ Calculates the path with the smallest value for visiting all nodes Least Cost Routing
22. • Single-Source Shortest Path ◦ Calculates path between a node and all other nodes Algorithms - Pathfinding & Search • All-Pairs Shortest Path ◦ Calculates shortest path group with all shortest paths between nodes • Minimum Weight Spanning Tree ◦ Calculates the path with the smallest value for visiting all nodes
23. • Parallel Breadth-First Search & Depth-First Search ◦ Traverses tree structure by exploring nearest neighbors (BFS) or down each branch (DFS) • Single-Source Shortest Path ◦ Calculates path between a node and all other nodes Algorithms - Pathfinding & Search • All-Pairs Shortest Path ◦ Calculates shortest path group with all shortest paths between nodes • Minimum Weight Spanning Tree ◦ Calculates the path with the smallest value for visiting all nodes
24. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths)
25. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths)
26. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths)
27. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths)
28. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Closeness ◦ Which nodes are able to reach entire group the fastest
29. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Closeness ◦ Which nodes are able to reach entire group the fastest
30. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Closeness ◦ Which nodes are able to reach entire group the fastest • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths) Source: Maven 7
31. Algorithms - Centralities • PageRank ◦ Which nodes have the most overall influence • Closeness ◦ Which nodes are able to reach entire group the fastest • Betweenness ◦ Which nodes are the bridges between different clusters (most shortest paths) • Degree ◦ The number of connections in/out of a node
32. Source: Network Science - Barabasi
33. Source: Network Science - Barabasi
34. Understanding Influence Source: “Robustness of the European power grids under intentional attack.” - R.V. Sole, M. Rosas-Casals, B. Corominas-Murtra, and S. Valverde. Source: “Network Science” - Barabasi
35. Understanding Influence Source: “Robustness of the European power grids under intentional attack.” - R.V. Sole, M. Rosas-Casals, B. Corominas-Murtra, and S. Valverde. Source: “Network Science” - Barabasi Preventing 
 Cascading Failures with 
 4 Nodes Removed
36. Algorithms – Community Detection • Label Propagation ◦ Spreads labels based on neighbors to infer clusters
37. Algorithms – Community Detection • Label Propagation ◦ Spreads labels based on neighbors to infer clusters • Union Find / Weakly Connected Components ◦ Finds groups of nodes that all have a path to each other
 • Strongly Connected Components ◦ Finds groups of nodes that are all connected 
 to each other following the 
 direction of relationships
38. Algorithms – Community Detection • Label Propagation ◦ Spreads labels based on neighbors to infer clusters • Union Find / Weakly Connected Components ◦ Finds groups of nodes that all have a path to each other
 • Strongly Connected Components ◦ Finds groups of nodes that are all connected 
 to each other following the 
 direction of relationships • Louvain Modularity ◦ Measures the presumed accuracy of community grouping Source: “Fast unfolding of communities in large networks” – Blondel, Guillaume, Lambiotte, Lefebvre
39. Algorithms – Community Detection • Label Propagation ◦ Spreads labels based on neighbors to infer clusters • Union Find / Weakly Connected Components ◦ Finds groups of nodes that all have a path to each other
 • Strongly Connected Components ◦ Finds groups of nodes that are all connected 
 to each other following the 
 direction of relationships • Louvain Modularity ◦ Measures the presumed accuracy of community grouping • Triangle-Count & Clustering Coefficient ◦ Measures the degree that nodes tend to cluster together
40. 18 Graph Algorithms Apply to All Real-World Networks Where You Need to Predict Complex Interactions Anti Money Laundering Recommendations Terrorist Networks
41. 18 Graph Algorithms Apply to All Real-World Networks Where You Need to Predict Complex Interactions Anti Money Laundering Recommendations Terrorist Networks Credit-Checks Fraud Prevention Cybersecurity Network Design PoS Profitability Alternate Routing Urban Resource Placement Theory Generation ML Feature Extraction Disease Spread Rippling Travel / Logistic Delays Drug Gene- Targeting
42. Graphs are one of the Unifying Themes of computer science . . . 
 
 That so many different structures
 can be modeled using a single formalism
 is a Source of Great Power 
 to the educated programmer.” 
 
 - Steven S. Skiena “
43. What inspired us?
44. Many Moving Parts! Example Workflow Pipeline based on John Swain’s Twitter Analysis Twitter 
 Streaming API Python Tweet Collection 
 (includes user data) Rabbit MQ MongoDB Neo4j R Scripts
 -Graph Stats -Community Detection MySQL Graph .gra phml Tableau Graph Visualization Moved from Twitter Search API to Streaming API Replaced Python Twitter libraries (Tweepy) with raw API calls Streaming tweets in message queue Full tweets and user data stored in MongoDB Built graph for analysis in Neo4j from tweets persisted in MongoDB Analysis in R iGraph libraries for algorithms Some text analysis e.g. LDA topics Results published in MySQL for Tableau Graphml for import to Gephi with stats precalculated
45. Our Goal is to Simplify Twitter 
 Streaming API Python Tweet Collection 
 (includes user data) Rabbit MQ MongoDB Neo4j R Scripts
 -Graph Stats -Community Detection MySQL Graph .gra phml Tableau Graph Visualization Example Workflow Pipeline based on John Swain’s Twitter Analysis
46. Graph Algorithms with Neo4j: Accessing & Executing
47. Neo4j 
 Native Graph Database Analytics Integrations Cypher Query Language Wide Range of
 APOC Procedures Optimized 
 Graph Algorithms
48. 1. Call as Cypher procedure 2. Pass in specification (Label, Prop, Query) and configuration 3. ~.stream variant returns (a lot) of results
 CALL algo.<name>.stream('Label','TYPE',{conf})
 YIELD nodeId, score
 4. non-stream variant writes results to graph returns statistics
 CALL algo.<name>('Label','TYPE',{conf}) How To…
49. Pass in Cypher statement for node- and relationship-lists.
 
 CALL algo.<name>(
 'MATCH ... RETURN id(n)',
 'MATCH (n)-->(m) 
 RETURN id(n) as source, 
 id(m) as target', {graph:'cypher'}) Cypher Projection
50. Game of Thrones • 800 nodes • 400 relationships Demo Time!
52. Neo4j Sandbox
53. Neo4j Desktop
54. Game of Thrones • 800 nodes • 400 relationships • Sandbox: Yelp Business Graph • 5m nodes • 17m relationships • GitHub: https://github.com/neo4j- contrib/neo4j-data-science- yelp/blob/master/notebooks/ neo4j_yelp_00_data_load.ipynb Neo4j Community Graph • 280k nodes • 1.4m relationships • GitHub: https://github.com/community- graph/documentation Browser: http://138.197.15.1:7474 username: "all" pwd: “readonly” DBPedia • 11m nodes • 116m relationships • Coming soon…. Datasets :play data_science
55. • Slack: neo4j.com/slack at #neo4j-graph-algorithms • YouTube: youtube.com/neo4j • GitHub: github.com/neo4j-contrib/neo4j-graph-algorithms • Whitepaper: neo4j.com/graph-analytics jennifer.reif@neo4j.com @jmhreif
56. Thank You! 
 Questions? 33 jennifer.reif@neo4j.com @jmhreif