EuroClojure 2012: Introduction to Cascalog

Cascalog is a data processing library for Clojure. Cascalog mixes functional and logic programming. It makes data processing code very concise, easier to grasp and reason about.

Being just a Clojure library, all of Clojures’ features are just a keystroke away – no need to learn another custom language (like Pig or Hive). This also allows for powerful abstraction and composition capabilities.

This is the slide deck of my presentation at EuroClojure 2012 (http://euroclojure.com/2012). The slides are mainly about my motivations for using Cascalog. The library itself was presented in a live coding environment. As soon as the recording is available, I'll add a link to it here.

Keywords: Hadoop, Clojure, Cascalog, Pig, Hive, Logic Programming

Transcript

1. Introduction to Cascalog Stefan Hübner, Nokia Berlin EuroClojure 2012
2. Nokia Maps • Address & POI Search • Category Search • Nearby Recommendations
3. Hadoop • Batch Processing • (Very) Large Scale • Distributed Filesystem • Parallel Computation • Fault-Tolerant
4. Hadoop • Batch Processing • (Very) Large Scale • Distributed Filesystem • Parallel Computation • Fault-Tolerant
5. Hadoop • Batch Processing • (Very) Large Scale • Distributed Filesystem • Parallel Computation • Fault-Tolerant
6. Hadoop MapReduce API • Tedious and verbose • Hard to test • Hard to refactor
7. • Tedious and verbose • Hard to test • Hard to refactor Hadoop MapReduce API
8. Pig and Hive • Define their own query language • Custom operations in Java, Python, ... • Non-intuitive integration
9. (())
10. Star Trek I - "The Motion Picture", Paramount Pictures V'Gr's question
11. Star Trek I - "The Motion Picture", Paramount Pictures "Ist das wirklich alles? Ist da sonst gar nichts mehr?"
12. Cascalog Cascalog Cascading Hadoop Abstraction Variables and logic Tuples, data workflows Key/value pairs, simple aggregation slide (c) Nathan Marz, reproduced with permission
13. Queries (<- ; defines a query [?person] ; output variables (age ?person ?age) ; generator with two variables (< ?age 30)) ; filter
14. Queries (<- [?person] (age ?person ?age) ; generator with two variables (< ?age 30)) ; filter Predicates
15. Predicates • Functions • Filters • Aggregators • Generators
17. Thank you! Stefan Hübner, @sthuebner http://knowyourmeme.com/memes/cereal-guy