Cascalog

Short demo talk on Cascalog on Hadoop UG in Munich

Transcript

1. Cascalog Hassle-free MapReduce that matches your scale Thursday, May 23, 13
2. Thursday, May 23, 13
3. Setting expecations •This is not a guide •And not a tutorial •Doesn’t claim to be complete •Mostly to give you an idea •And encourage you to explore further Thursday, May 23, 13
4. How much time do you spend on writing logic that framework should take care of? Thursday, May 23, 13
5. How easy is it to debug your map/reduce aggragation? Thursday, May 23, 13
6. Hadoop + Java composable, but too vebrose Pig, Hive too concrete, lack of abstraction and composition Thursday, May 23, 13
7. Thursday, May 23, 13
8. • Clear, declarative syntax • Inner and outer joins • Aggregators • Functions • Subqueries, composition • Sorting • Performant Thursday, May 23, 13
9. Casca-WHAT? • Built on top of Hadoop (MapReduce) • Cascading (tuples, workflows, job execution) • Written in Clojure • Datalog (logic programming) Thursday, May 23, 13
10. Abstract evrthn! Thursday, May 23, 13
11. Source where data pours from Thursday, May 23, 13
12. Pipe that data flows through Thursday, May 23, 13
13. Filter that makes sure that only good stuff goes through Thursday, May 23, 13
14. Tuple they actually flow Thursday, May 23, 13
15. Thursday, May 23, 13
16. Query anatomy Thursday, May 23, 13
17. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30)) Thursday, May 23, 13
18. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30)) Output Thursday, May 23, 13
19. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30)) output vars Thursday, May 23, 13
20. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30)) Input input vars Thursday, May 23, 13
21. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30)) Logic/aggregations Thursday, May 23, 13
22. Sources and sinks • HDFS (go figure) • Cassandra • MongoDB • SQL data sources • File system • Memory sources Thursday, May 23, 13
23. (?<- (stdout) [?person] (age ?person 25)) Exact match of second element in a tuple Thursday, May 23, 13
24. (defn younger-than? [limit age] (< age limit)) (?<- (stdout) [?person ?age] (age ?person ?age) (younger-than? 32 ?age)) Predicate match, fn call Predicate Thursday, May 23, 13
25. (?<- (stdout) [?person ?count] (follows ?person _) (c/count ?count)) Aggregation Thursday, May 23, 13
26. SHOWTIME! Thursday, May 23, 13
27. Benefits •Query language is same as application language •Subqueries, reusability •Ad-hoc querying •Cascading underneath, so taps for all DBs work •Reuse application logic •Text editor integration Thursday, May 23, 13
28. @ifesdjeen (twitter/github) Thursday, May 23, 13