Save 37% off PRO during our Black Friday Sale! »

Clojure Reducers

A0335428a9bc84ac981640853ca3a7eb?s=47 pschirmacher
May 06, 2014
210

Clojure Reducers

A0335428a9bc84ac981640853ca3a7eb?s=128

pschirmacher

May 06, 2014
Tweet

Transcript

  1. We'll take care of it. Personally. Clojure Reducers Philipp Schirmacher

    | innoQ
  2. © 2014 innoQ Deutschland GmbH http://upload.wikimedia.org/wikip

  3. © 2014 innoQ Deutschland GmbH Clojure http://upload.wikimedia.org/wikipedia/en/1/1a/Clo A practical Lisp

    variant for the JVM Functional programming Dynamic Typing Full-featured macro system Bi-directional Java interop Immutable data structures
  4. © 2014 innoQ Deutschland GmbH Lisp??

  5. © 2014 innoQ Deutschland GmbH Lots of irritating silly parentheses

  6. © 2014 innoQ Deutschland GmbH clojure environment Clojuresque (Gradle) Leiningen

  7. © 2013 innoQ Deutschland GmbH http://www.tbray.org/ongoing/When/200x/2008/09/25/-big/R0010774.jpg.html Rich Hickey

  8. © 2014 innoQ Deutschland GmbH clojure crash course

  9. © 2014 innoQ Deutschland GmbH generic data types

  10. © 2014 innoQ Deutschland GmbH {:name "Clojure" :features [:functional :jvm

    :parens] :creator "Rich Hickey" :stable-version {:number "1.5.1" :release "2013/03/10"}}
  11. © 2014 innoQ Deutschland GmbH functions

  12. © 2014 innoQ Deutschland GmbH (:city {:name "innoQ" :city "Monheim"})

    > "Monheim" (map inc [1 2 3]) > (2 3 4) (+ 1 2) > 3
  13. © 2014 innoQ Deutschland GmbH (reduce + 0 [1 2

    3]) > 6 (filter odd? [1 2 3 4 5]) > (1 3 5) (map inc [1 2 3]) > (2 3 4)
  14. © 2014 innoQ Deutschland GmbH public interface ISeq extends IPersistentCollection

    { Object first(); // called 'rest' in clojure ISeq next(); ISeq cons(Object o); } https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/ISeq.java
  15. © 2014 innoQ Deutschland GmbH lazy sequences

  16. © 2014 innoQ Deutschland GmbH (iterate inc 1) > (1

    2 3 4...)
  17. [1 function1] © 2014 innoQ Deutschland GmbH (iterate inc 1)

    [1 [2 [3 function3]]] [1 [2 function2]] [1 [2 [3 [4 function4]]]] ...
  18. © 2013 innoQ Deutschland GmbH http://cdn4.spiegel.de/images/image-380967-custom-igec.jpg

  19. © 2014 innoQ Deutschland GmbH (defn process [revenue-data] (->> revenue-data

    (filter in-relevant-region?) (map enhance) (reduce determine-key-figures {})))
  20. © 2014 innoQ Deutschland GmbH (defn process [nums] (->> nums

    (filter odd?) (map inc) (reduce + 0))) (time (process (range 1000000))) ;; "Elapsed time: 155.562 msecs"
  21. © 2014 innoQ Deutschland GmbH why is it so slow!?

  22. map lter © 2014 innoQ Deutschland GmbH reduce (0 1

    2 3 4 5 6 7 8...) (2 4 6...)
  23. [0 1 2 3 4 5 6 7 8 9

    10...999999] © 2014 innoQ Deutschland GmbH reduce
  24. © 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000)))

    (defn proc-with-reduce [nums] (reduce (fn [acc n] (if (odd? n) (+ acc (inc n)) acc)) 0 nums)) (time (proc-with-reduce numbers-vec)) ;; "Elapsed time: 50.353 msecs"
  25. © 2014 innoQ Deutschland GmbH reducers

  26. © 2014 innoQ Deutschland GmbH (ns reducersexample (:require [clojure.core.reducers :as

    r])) (time (proc-with-reducers numbers-vec)) ;; "Elapsed time: 53.784 msecs" (defn proc-with-reducers [nums] (->> nums (r/filter odd?) (r/map inc) (reduce + 0)))
  27. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1

    2 3])) mapresult > #<reducers$folder$reify__1147 clojure.core.reducers$folder$reify__1147@149544d> (reduce + 0 mapresult) > 9 (reduce conj [] mapresult) > [2 3 4]
  28. © 2014 innoQ Deutschland GmbH (defprotocol CollReduce "Protocol for collection

    types that can implement reduce faster than first/next recursion. Called by clojure.core/reduce. Baseline implementation defined in terms of Iterable." (coll-reduce [this reducef] [this reducef init])) https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/protocols.clj
  29. © 2014 innoQ Deutschland GmbH (reduce + 0 [1 2

    3]) > 6 If you reduce me I’ll iterate over my elements!
  30. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1

    2 3])) (reduce + 0 mapresult) > 9 If you reduce me I’ll transform your reducing function and delegate to [1 2 3]! (reduce (fn [acc element] (+ acc (inc element))) 0 [1 2 3])
  31. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce

    [this reducef init] (coll-reduce [1 2 3] ((mapping-with inc) reducef) init)))
  32. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce

    [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [f] (fn [reducef] (fn [acc element] (reducef acc (f element)))))
  33. © 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce

    [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [inc] (fn [+] (fn [acc element] (+ acc (inc element)))))
  34. © 2014 innoQ Deutschland GmbH (defn lazy-seq-map [f coll] (lazy-seq

    (if (empty? coll) coll (cons (f (first coll)) (lazy-seq-map f (rest coll))))))
  35. © 2014 innoQ Deutschland GmbH separation of concerns

  36. © 2014 innoQ Deutschland GmbH [0 1 2 3 4

    5 6 7 8] [0 1 2] [3 4 5] [6 7 8] 3 12 21 15 36 split reduce combine combine
  37. © 2014 innoQ Deutschland GmbH [0 1 2 3 4

    5 6 7 8] [0 1 2] [3 4 5] [6 7 8] 3 12 21 33 36 split reduce combine combine
  38. © 2014 innoQ Deutschland GmbH (doc r/fold) ------------------------- clojure.core.reducers/fold ([reducef

    coll] [combinef reducef coll] [n combinef reducef coll]) Reduces a collection using a (potentially parallel) reduce-combine strategy. The collection is partitioned into groups of approximately n (default 512), each of which is reduced with reducef (with a seed value obtained by calling (combinef) with no arguments). The results of these reductions are then reduced with combinef (default reducef). combinef must be associative, and, when called with no arguments, (combinef) must produce its identity element. These operations may be performed in parallel, but the results will preserve order.
  39. © 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1

    2 3])) (reduce + mapresult) > 9 (r/fold + mapresult) > 9
  40. © 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000)))

    (defn proc-with-fold [nums] (->> nums (r/filter odd?) (r/map inc) (r/fold +))) ;; 2 cores (no hyper-threading) (time (proc-with-fold numbers-vec)) ;; "Elapsed time: 28.138 msecs"
  41. © 2014 innoQ Deutschland GmbH implementation time lazy sequences 156ms

    reduce by hand 50ms reducers 54ms reducers with fold 28ms
  42. © 2014 innoQ Deutschland GmbH (defn count-words [text] (let [words

    (clojure.string/split text #"\s+")] (reduce (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) {} words)))
  43. © 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words

    (clojure.string/split text #"\s+")] (r/fold (fn ([acc1 acc2] (merge-with + acc1 acc2)) ([] {})) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))
  44. © 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words

    (clojure.string/split text #"\s+")] (r/fold (r/monoid (partial merge-with +) hash-map) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))
  45. © 2014 innoQ Deutschland GmbH summary ‣ laziness is a

    good default ‣ reducers are an optimization ‣ separation of concerns ‣ fast because of fork/join ‣ fork/join with normal clojure vectors
  46. Thank You! We'll take care of it. Personally. © 2014

    innoQ Deutschland GmbH Philipp Schirmacher philipp.schirmacher@innoq.com