AppsFlyer presenting: Cascalog, MapReduce for The Code Craftsman

This presentation describe AppsFlyer's work with Hadoop in the Clojure production environment.

Transcript

1. Cascalog
2. AppsFlyer We use Clojure Used python previously Mobile app conversion tracking
3. Challenges User Retention Cohort analysis ~0.5TB of data collected daily
4. What is Cascalog? A Clojure DSL for writing Hadoop jobs On top of Cascading
5. Clojure Modern LISP for the JVM
6. Why Cascalog? We already know and love Clojure Same tools - test in the REPL Custom operations are ordinary functions (no UDFs)
7. Why Cascalog? Cascalog is Datalog Fits our use cases well Simple (once you know it )
8. Relations and Tuples Relation 1 Relation 2 t 1 t 2 t 3 t 4 t 1 t 2 t 3 t 4 Relational Model
9. Query Anatomy
10. Query Anatomy Order does not matter Grouping is implicit
11. Why Cascalog? Composition
12. Why Cascalog? Composition Joins are implicit
13. Generators Cascalog taps Hadoop and local file systems Clojure sequences [["alice" 28] ["bob" 33] ["chris" 40] ["david" 25] ["emily" 25] ["george" 31]] Cascalog queries Defined using <-
14. Demo
15. The AppsFlyer flow Kafka Secor AWS S3 Data Collection Continuously saves Kafka topics to HDFS/S3 according to a scheme
16. The AppsFlyer flow Data Processing Lemur Spins up Hadoop cluster Submit steps Data processing (using cascalog) Export processed data (using Apache Sqoop) Postgresql 1 2 1 2
17. Questions ?