GRAIL: A Scalable Index for Reachability Queries in Very Large Graphs

90-minute presentation for the Advanced topics in Data Management course at KAUST, on "GRAIL: A Scalable Index for Reachability Queries in Very Large Graphs" by Hilmi Yıldırım et al, published in VLDBJ '11.

1. GRAIL: A Scalable Index For Reachability Queries in Large Graphs
2. Given a directed graph , , is → ?
3. Reachability in a directed graph → reachability in a DAG
4. Approaches Hop Labeling Path based Interval Labeling
5. Interval Labeling on Trees = [ , ] Tree T , Desired labeling property: ⊆ ⇒ → Candidates: Min-post labeling Pre-post labeling
6. Min-post Interval Labeling on Trees = [ , ] Post-order rank of u Minimum post-order rank of all the nodes in the sub-tree rooted at u Linear time and space
7. Min-post Interval Labeling on Trees = [ , ] Post-order rank of u Minimum post-order rank of all the nodes in the sub-tree rooted at u Linear time and space ⊆ ⇒ →
8. Min-post Interval Labeling on DAGS Do not visit a node than once, keep the post-order rank of the first visit. ⊆ ⇒ ↛ = [ , ] Post-order rank of u Minimum post-order rank of all the nodes in the sub-tree rooted at u ⊆ ⇏ →
9. Min-post Interval Labeling on DAGS ⊆ ⇒ ↛ = [ , ] Assume → , = [ , ] = [ , ]
10. GRIPP Key Ideas Apply interval labeling to the tree. Reachable(u, v): may trigger multiple containment queries If containment is satisfied, return true If containment is not satisfied For all non-tree edges (x, y), x is a descendant of u, If Reachable(y, v), return true Return false // out of non-tree edges Query time: O(|E| - |V|) = O(t), for t non-tree edges
11. GRAIL Key Ideas Apply post-order interval labeling to the DAG. Hypothesis: Most of the reachability information is captured by interval labeling Reachable(u, v): If containment is not satisfied, return false. No false negatives If containment is satisfied, do something. May have false positives Can we decrease the number of false positives? Yes, compute many intervals. How? How do we handle false positives?
12. Computing The Index (Intervals) d intervals O(d(n+m)) time O(d|V|) space
13. Computing The Index (Intervals) Random traversal Fixed reverse pairs Bottom up* Heuristic (guided)* During the ith traversal, select the node with the most exceptions in the previous i – 1 traversals Too expensive Use some heuristics to select nodes with many possible exceptions Time complexity: O(d(n+m) + dn(Plog(P)), P = maximum out degree
14. Computing The Index (Intervals) Heuristic (guided)* During the ith traversal, select the node with the most exceptions in the previous i – 1 traversals Too expensive Maximum Volume Maximum minimum interval Maximum adjusted volume Maximum adjusted minimum interval
15. Answering Queries Else If ⊆ If ⊆ return ↛ • Exception Lists, or • Pruned DFS
16. Pruned DFS
17. Exception Lists
18. Computing Exception Lists Direct exceptions Indirect exceptions
19. Computing Exception Lists Direct exceptions (first dimension) Interval query: O(logN + K) O(nP(logn + X)) P: maximum out degree X: maximum number of exceptions
20. Computing Exception Lists Indirect exceptions (first dimension) O(nXP2) P: maximum out degree X: maximum number of exceptions
21. Computing Exception Lists Subsequent dimensions O(nXP’) P’: maximum in degree, X: maximum number of exceptions
22. Computing Exception Lists Direct exceptions Indirect exceptions O(nXP(logn + X + P)) P: maximum in/out degree X: maximum number of exceptions
23. Optimizations Topological sorting Search strategies: DFS, BFS, BBFS Positive cut filter
24. Optimizations: Positive cut filter
25. Datasets
26. Index Construction Time (ms): Small Sparse
27. Index Size (#entries): Small Sparse
28. Query Time (ms): Small Sparse
29. Index Construction Time (ms): Small Dense
30. Index Size (#entries): Small Dense
31. Query Time (ms): Small Dense
32. Index Construction Time (ms) and Size: Large Real
33. Query Time (ms): Large Real
34. Baseline Methods: Random Queries
35. Baseline Methods: Positive Queries
36. GRAIL Optimizations
37. GRAIL Query Times for Positive Queries
38. Effect of Exceptions
39. Query Times for different number of traversals
40. Query Times for different number of traversals
41. Query Times for different Traversals
42. Number of exceptions remaining for different traversals