Clojure Reducers

Transcript

1. We'll take care of it. Personally. Clojure Reducers Philipp Schirmacher | innoQ
2. © 2014 innoQ Deutschland GmbH http://upload.wikimedia.org/wikip
3. © 2014 innoQ Deutschland GmbH Clojure http://upload.wikimedia.org/wikipedia/en/1/1a/Clo A practical Lisp variant for the JVM Functional programming Dynamic Typing Full-featured macro system Bi-directional Java interop Immutable data structures
4. © 2014 innoQ Deutschland GmbH Lisp??
5. © 2014 innoQ Deutschland GmbH Lots of irritating silly parentheses
6. © 2014 innoQ Deutschland GmbH clojure environment Clojuresque (Gradle) Leiningen
7. © 2013 innoQ Deutschland GmbH http://www.tbray.org/ongoing/When/200x/2008/09/25/-big/R0010774.jpg.html Rich Hickey
8. © 2014 innoQ Deutschland GmbH clojure crash course
9. © 2014 innoQ Deutschland GmbH generic data types
10. © 2014 innoQ Deutschland GmbH {:name "Clojure" :features [:functional :jvm :parens] :creator "Rich Hickey" :stable-version {:number "1.5.1" :release "2013/03/10"}}
11. © 2014 innoQ Deutschland GmbH functions
12. © 2014 innoQ Deutschland GmbH (:city {:name "innoQ" :city "Monheim"}) > "Monheim" (map inc [1 2 3]) > (2 3 4) (+ 1 2) > 3
13. © 2014 innoQ Deutschland GmbH (reduce + 0 [1 2 3]) > 6 (filter odd? [1 2 3 4 5]) > (1 3 5) (map inc [1 2 3]) > (2 3 4)
14. © 2014 innoQ Deutschland GmbH public interface ISeq extends IPersistentCollection { Object first(); // called 'rest' in clojure ISeq next(); ISeq cons(Object o); } https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/ISeq.java
15. © 2014 innoQ Deutschland GmbH lazy sequences
16. © 2014 innoQ Deutschland GmbH (iterate inc 1) > (1 2 3 4...)
17. [1 function1] © 2014 innoQ Deutschland GmbH (iterate inc 1) [1 [2 [3 function3]]] [1 [2 function2]] [1 [2 [3 [4 function4]]]] ...
18. © 2013 innoQ Deutschland GmbH http://cdn4.spiegel.de/images/image-380967-custom-igec.jpg
19. © 2014 innoQ Deutschland GmbH (defn process [revenue-data] (->> revenue-data (filter in-relevant-region?) (map enhance) (reduce determine-key-figures {})))
20. © 2014 innoQ Deutschland GmbH (defn process [nums] (->> nums (filter odd?) (map inc) (reduce + 0))) (time (process (range 1000000))) ;; "Elapsed time: 155.562 msecs"
21. © 2014 innoQ Deutschland GmbH why is it so slow!?
22. map lter © 2014 innoQ Deutschland GmbH reduce (0 1 2 3 4 5 6 7 8...) (2 4 6...)
23. [0 1 2 3 4 5 6 7 8 9 10...999999] © 2014 innoQ Deutschland GmbH reduce
24. © 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000))) (defn proc-with-reduce [nums] (reduce (fn [acc n] (if (odd? n) (+ acc (inc n)) acc)) 0 nums)) (time (proc-with-reduce numbers-vec)) ;; "Elapsed time: 50.353 msecs"
25. © 2014 innoQ Deutschland GmbH reducers
26. © 2014 innoQ Deutschland GmbH (ns reducersexample (:require [clojure.core.reducers :as r])) (time (proc-with-reducers numbers-vec)) ;; "Elapsed time: 53.784 msecs" (defn proc-with-reducers [nums] (->> nums (r/filter odd?) (r/map inc) (reduce + 0)))
27. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1 2 3])) mapresult > #<reducers$folder$reify__1147 clojure.core.reducers$folder$reify__1147@149544d> (reduce + 0 mapresult) > 9 (reduce conj [] mapresult) > [2 3 4]
28. © 2014 innoQ Deutschland GmbH (defprotocol CollReduce "Protocol for collection types that can implement reduce faster than first/next recursion. Called by clojure.core/reduce. Baseline implementation defined in terms of Iterable." (coll-reduce [this reducef] [this reducef init])) https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/protocols.clj
29. © 2014 innoQ Deutschland GmbH (reduce + 0 [1 2 3]) > 6 If you reduce me I’ll iterate over my elements!
30. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1 2 3])) (reduce + 0 mapresult) > 9 If you reduce me I’ll transform your reducing function and delegate to [1 2 3]! (reduce (fn [acc element] (+ acc (inc element))) 0 [1 2 3])
31. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce [this reducef init] (coll-reduce [1 2 3] ((mapping-with inc) reducef) init)))
32. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [f] (fn [reducef] (fn [acc element] (reducef acc (f element)))))
33. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [inc] (fn [+] (fn [acc element] (+ acc (inc element)))))
34. © 2014 innoQ Deutschland GmbH (defn lazy-seq-map [f coll] (lazy-seq (if (empty? coll) coll (cons (f (first coll)) (lazy-seq-map f (rest coll))))))
35. © 2014 innoQ Deutschland GmbH separation of concerns
36. © 2014 innoQ Deutschland GmbH [0 1 2 3 4 5 6 7 8] [0 1 2] [3 4 5] [6 7 8] 3 12 21 15 36 split reduce combine combine
37. © 2014 innoQ Deutschland GmbH [0 1 2 3 4 5 6 7 8] [0 1 2] [3 4 5] [6 7 8] 3 12 21 33 36 split reduce combine combine
38. © 2014 innoQ Deutschland GmbH (doc r/fold) ------------------------- clojure.core.reducers/fold ([reducef coll] [combinef reducef coll] [n combinef reducef coll]) Reduces a collection using a (potentially parallel) reduce-combine strategy. The collection is partitioned into groups of approximately n (default 512), each of which is reduced with reducef (with a seed value obtained by calling (combinef) with no arguments). The results of these reductions are then reduced with combinef (default reducef). combinef must be associative, and, when called with no arguments, (combinef) must produce its identity element. These operations may be performed in parallel, but the results will preserve order.
39. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1 2 3])) (reduce + mapresult) > 9 (r/fold + mapresult) > 9
40. © 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000))) (defn proc-with-fold [nums] (->> nums (r/filter odd?) (r/map inc) (r/fold +))) ;; 2 cores (no hyper-threading) (time (proc-with-fold numbers-vec)) ;; "Elapsed time: 28.138 msecs"
41. © 2014 innoQ Deutschland GmbH implementation time lazy sequences 156ms reduce by hand 50ms reducers 54ms reducers with fold 28ms
42. © 2014 innoQ Deutschland GmbH (defn count-words [text] (let [words (clojure.string/split text #"\s+")] (reduce (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) {} words)))
43. © 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words (clojure.string/split text #"\s+")] (r/fold (fn ([acc1 acc2] (merge-with + acc1 acc2)) ([] {})) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))
44. © 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words (clojure.string/split text #"\s+")] (r/fold (r/monoid (partial merge-with +) hash-map) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))
45. © 2014 innoQ Deutschland GmbH summary ‣ laziness is a good default ‣ reducers are an optimization ‣ separation of concerns ‣ fast because of fork/join ‣ fork/join with normal clojure vectors
46. Thank You! We'll take care of it. Personally. © 2014 innoQ Deutschland GmbH Philipp Schirmacher philipp.schirmacher@innoq.com