Clojure in AppsFlyer - a Java.il talk

1. Clojure in AppsFlyer A few () to rule them all...
2. AppsFlyer Who?! • The World's leading Mobile Measurement Platform • Founded in 2011 by Reshef Mann and Oren Kaniel • Just completed Round B funding - total of $28M • Processing 2.5B daily events (it was 1.9B just 2 months ago and 250M at the start of 2014!) • 13 people in the development team (we were just 6 people 12 months ago!)
4. The Catalyst • A single python service which queried Google's Bigquery for a couple of hundred thousands of records “chokes” on memory usage. • The possible solutions: – Multi processing (no real thread solution because of the GIL) – C implementation and python interop – Pypy – Different programming language
5. The Selected Solution • A different programming language • Selected language must be: – Functional – Supports concurrency – Good eco-system • Possible candidates: Erlang, Haskell, OCaml, Scala, F# on mono, Clojure
6. WHY FUNCTIONAL?!
7. Why Functional def send_via_http(url, data): data['c'] = 3 http.post(url, json.dumps(data)) def send_via_kafka(topic, partition, data): data['a'] = 11 producer.send(topic, partition, json.dumps(data)) data = {'a': 1, 'b': 2} http_sender = partial(send_via_http, 'http://www.appsflyer.com') kafka_sender = partial(send_via_kafka, 'dummy_topic', 'partition_1') send_fns = [http_sender, kafka_sender] map(lambda send_fn: send_fn(data), send_fns)
8. Why Clojure • JVM • Repl based development • Enforces use of FP paradigm more strictly than Scala (we really want that JVM) • Dynamic • Sequence based processing capabilities really fit in the visualized data flow of AppsFlyer • Easy and common Java interop
9. Clojure Intro (Short Demo)
10. The Dangers • Performance: – Dynamic types – Immutability • Need to understand most of the underlying mechanism in order to write code that performs above average – Lazy evaluation and the (count xs) conundrum • Good Clojure code is fast as, at most, good Java code
11. The Initial Hardships • Lisp can be a bit intimidating (brackets hell) – A good IDE support with Rainbow Parentheses and Paredit really helps • Functional way of thought takes a bit getting used to – State!!! (modeling) • Learning the eco-system of both Java and Clojure was a lot (at first)
12. Tool Support • IDE's: – Emacs (with Clojure Mode) – Eclipse (with Counterclockwise) – IntelliJ (with Cursive or La Clojure) – Light Table – Nightcode – Vim (with Fireplace)
13. Debugging DUH?!
14. Leiningen • Build tool • Dependency resolver tool (Maven on steroids) • Development enrichment tool: – nRepl built in – Development profiles – Helper libs usability
15. How it Looks Like Today? • Each service is a single “uberjar” • No application servers/services • Small, isolated services with async communication • Docker container per service • Dedicated machines for DB clusters – no docker (just plain old Amazon virtualization)
17. The Future? • General utility lib • General enriched component lib • Mesos • Marathon • Who knows?! (Rust, Nim)
18. We're Hiring! jobs@appsflyer.com
19. Debate!
20. Persistent Data Structures • All Clojure data structures are persistent • Collections maintain performance guarantees: – New + Old version of collections are available after change – New versions are not full copies – Thread safe, iteration safe • Sequences replace traditional lists – All Clojure and Java collections can be made into seqs
21. Persistent Data Structure Performance • O(log32 N) ~= O(1) A B C D E F G H I J K L M N' A’ C’ G’ N