ScalaDays 2013: How Kiama helps with Hadoop computing

This talk gives a brief overview of the BigData tooling landscape and where Scoobi, a distributed collection Scala library for Hadoop, stands. Then it shows what are the challenges in translating Scoobi abstractions to Hadoop constructs and how Scala, as programming language, and Kiama (http://code.google.com/kiama), as a graph-processing library, can be leveraged to support this translation. In particular:
- How rewriting rules based on partial functions are a very succinct way to pre-process the computation graph and to optimise it.
- How attribute grammars can be used to implement general graph traversal algorithms

Transcript

1. A walk down the beach: How Kiama helps with Hadoop computing ScalaDays
2. What’s in there for you? Big data iama
3. Big data
4. Big data tools Storm
5. Scoobi
6. A collection-like DSL String! (Int, Int)! (Int, Iterable[Int])! (Int, Int)!
7. Hadoop
8. Hadoop / Java
9. Scoobi
10. What is Scoobi? A compiler!
11. Hadoop
12. A framework Mapper Mapper Mapper Mapper Mapper Mapper Mapper Mapper Reducer Reducer “Hello”! “World”! (“Hello”, ! [1, 1, 1])! (“World”, ! [1, 1, 1, 1])! (“Hello”, 3)! (“World”, 4)! combiner (“World”, 1)! (“Hello”, 1)! “Hello”! “World”! “World”! “World”! “Hello”!
13. Scoobi => Hadoop
14. Step  one:  create  nodes   Easy :-) Load   ParallelDo   GroupByKey   Combine   Materialise  
15. Step  one:  create  nodes   Load   ParallelDo   GroupByKey   Combine   Materialise  
16. Scoobi => Hadoop
17. Transla=on?   Load   ParallelDo   GroupByKey   Combine   Materialise   Mapper   Mapper   Mapper   Mapper   Mapper   Mapper   Mapper   Mapper   Reducer   Reducer   (“Hello”, ! [1, 1, 1])! (“World”, ! [1, 1, 1, 1])! (“Hello”, 3)! (“World”, 4)! combiner   (“World”, 1)! (“Hello”, 1)!
18. Step  two:  simplify   Node  0   ParallelDo  1   ParallelDo  2   Node  3   Node  0   fused   ParallelDo   Node  3   A => B! B => C! A => C!
19. Kiama
20. Kiama  tree  rewri=ng   pattern matching partial function rule combinators
21. Kiama  tree  
22. Kiama  tree   team! person1! person2! Eric! G.! Ben! M.! Torreborre! Lever! number1(xxx xxx xxx)! number1(xxx xxx xxx)!
23. Kiama  strategies  
24. Kiama  strategies   “Family” combinators parent! child1! child3! child2!
25. Kiama  strategies   “Control” combinators s1 ‘or’ s2! s1 ‘and’ s2! s1 ‘non-deterministic or’ s2! if s1 then s2 else s3!
26. Kiama  strategies   try something! iterate! succeed if s fails! “Control” combinators
27. Kiama  strategies   Traversals from the top of the tree to the leaves!
28. Kiama  strategies   Traversals from the leaves to the top of the tree! start from bottom, stop as soon as succeed!
29. Kiama  strategies   Example Capitalize all last names in the team!
30. Kiama  strategies   Scoobi examples All Scoobi strategies!
31. Kiama  strategies   Scoobi examples Truncate the graph!
32. Kiama  strategies,  warnings   Infinite loops! Can create semantically invalid trees Equality is used everywhere. Define equals and hashCode wisely => use unique ids on nodes The new tree is a copy of the old one. Traceability? Node attributes might be traversable!
33. Kiama:  rewri=ng   Fusion rule for a DAG ParallelDo  1   Load  1   ParallelDo  2   uses(pd1).size <= 1 ! ParallelDo  3   Dataflow direction! Graph direction!
34. Kiama:  rewri=ng   Rewriting duplicates! PD1   Load  1   PD2   PD3   Dataflow direction! Graph direction! PD1’   Load  1’   PD2’   PD3’   PD1’   Load  1’   Root   Root   => use the MemoRewriter!
35. Scala
36. PaJern  matching  /  implicits  /  case  classes   Doesn’t play well with pattern matching and rewriting!
37. Scoobi => Hadoop
38. Step  three:  define  MapReduce  jobs   Load   ParallelDo  1   GroupByKey   Combine   A! (K, V)! (K, Iterable[V])! (K, V)! ParallelDo  2   B! Mapper   Reducer   combiner   MapReduce job!
39. Step  three:  jobs  dependencies   Load   ParallelDo  1   GroupByKey1   ParallelDo  2   GroupByKey2   ParallelDo  3   Mscr1 depends on Mscr2! Find dependencies? Mscr1 Mscr2
40. Graph  algorithm:  layering   Output1   Output3   1. selection! 2. longest path to leaves! 3. group by longest path! Output2   Longest path?
41. Kiama
42. Kiama:  aJribute  grammars   node:  AJributable   child1   attribute! child2   children! parent! Attribute definition! Function! Values are memoised!
43. Kiama:  aJribute  grammars   Output  1   Output  3   Output  2   leaves! leaves! leaves!
44. Kiama:  aJribute  grammars   Output  1   Output  3   Output  2   longest = 1! longest = 2! longest = 2! leaves! leaves! leaves!
45. Kiama:  aJribute  grammars   Longest path to a node
46. Kiama:  aJribute  grammars   Layers all nodes in the tree! group by longest path! join with longest path! return nodes only!
47. Scoobi => Hadoop
48. Step  four:  channels  defini=on   Load   ParallelDo  1   ParallelDo  2   GroupByKey1   ParallelDo  4   Output channel! ParallelDo  3   GroupByKey2   ParallelDo  5   Input channel! Output channel! Mapper   Reducer   tag1! tag2!
49. Kiama
50. Cycles n1   n2   n3   co-parent! co-parent! Graph direction! n4   n5   co-parent! Kiama:  aJribute  grammars  
51. Kiama:  aJribute  grammars   Transitive uses n2   n3   n5   children! root! n1   n3  
52. Kiama:  aJribute  grammars   Transitive uses usesTable:! ! n1 -> Set()! n2 -> Set(n1)! n3 -> Set(n1)! n4 -> Set(n2)! n5 -> Set(n2, n3)! ! Scalaz FTW!! n2   n4   n5   n1   n3  
53. Scala
54. Scalaz   Update of a Map[K, Collection[V]] 1. if the key doesn’t exist, add a new key with an empty collection! 1.1 add the new element to the new collection! 2. Otherwise add the new element to the existing collection!
55. Scalaz   Monoids! Can add 2 maps together! Because can add to a Set!
56. Kiama
57. Kiama:  aJribute  grammars   Mutation :-( Step 1! n1   n2   n3   children! parent! Step 2! n0   n1   n2   n3   children! parent! n4   Also for “attr”!
58. Kiama:  aJribute  grammars   Directed graphs != trees ParallelDo   Materialise1   Load1   Load2   Materialise2   Graph direction! Dataflow direction! Where to start? Root  
59. Kiama:  rewri=ng  +  aJribute  grammars   Combine  1   Load  1   PD2   PD3   Dataflow direction! Graph direction! Relations change! Root   PD1’   Load  1   PD2   PD3   Root   parent?!
60. Scoobi
61. Step  five:  execu=on   Layer1     Layer2     sink   value   mscr1   mscr2   mscr3   mscr4   mscr5  
62. Recap Big data iama
63. Recap It is possible to abstract the Hadoop “map-reduce” paradigm to a “distributed collections” one Scala is a flexible language to represent those computations Kiama helps tremendously for: •  modifying the graph •  computing nodes properties
64. Thank you!
65. PS You like the Aussie way of life + Scala? => Apply at NICTA!