Clojure Reducers - Speaker Deck

Slide 1

Slide 1 text

We'll take care of it. Personally. Clojure Reducers Philipp Schirmacher | innoQ

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2014 innoQ Deutschland GmbH Clojure http://upload.wikimedia.org/wikipedia/en/1/1a/Clo A practical Lisp variant for the JVM Functional programming Dynamic Typing Full-featured macro system Bi-directional Java interop Immutable data structures

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© 2014 innoQ Deutschland GmbH public interface ISeq extends IPersistentCollection { Object first(); // called 'rest' in clojure ISeq next(); ISeq cons(Object o); } https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/ISeq.java

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000))) (defn proc-with-reduce [nums] (reduce (fn [acc n] (if (odd? n) (+ acc (inc n)) acc)) 0 nums)) (time (proc-with-reduce numbers-vec)) ;; "Elapsed time: 50.353 msecs"

Slide 25

Slide 25 text

Slide 26

Slide 26 text

© 2014 innoQ Deutschland GmbH (ns reducersexample (:require [clojure.core.reducers :as r])) (time (proc-with-reducers numbers-vec)) ;; "Elapsed time: 53.784 msecs" (defn proc-with-reducers [nums] (->> nums (r/filter odd?) (r/map inc) (reduce + 0)))

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2014 innoQ Deutschland GmbH (defprotocol CollReduce "Protocol for collection types that can implement reduce faster than first/next recursion. Called by clojure.core/reduce. Baseline implementation defined in terms of Iterable." (coll-reduce [this reducef] [this reducef init])) https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/protocols.clj

Slide 29

Slide 29 text

Slide 30

Slide 30 text

© 2014 innoQ Deutschland GmbH (def mapresult (r/map inc [1 2 3])) (reduce + 0 mapresult) > 9 If you reduce me I’ll transform your reducing function and delegate to [1 2 3]! (reduce (fn [acc element] (+ acc (inc element))) 0 [1 2 3])

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [f] (fn [reducef] (fn [acc element] (reducef acc (f element)))))

Slide 33

Slide 33 text

© 2014 innoQ Deutschland GmbH ;; mapresult (reify CollReduce (coll-reduce [this + 0] (coll-reduce [1 2 3] ((mapping-with inc) +) 0))) (defn mapping-with [inc] (fn [+] (fn [acc element] (+ acc (inc element)))))

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

© 2014 innoQ Deutschland GmbH (doc r/fold) ------------------------- clojure.core.reducers/fold ([reducef coll] [combinef reducef coll] [n combinef reducef coll]) Reduces a collection using a (potentially parallel) reduce-combine strategy. The collection is partitioned into groups of approximately n (default 512), each of which is reduced with reducef (with a seed value obtained by calling (combinef) with no arguments). The results of these reductions are then reduced with combinef (default reducef). combinef must be associative, and, when called with no arguments, (combinef) must produce its identity element. These operations may be performed in parallel, but the results will preserve order.

Slide 39

Slide 39 text

Slide 40

Slide 40 text

© 2014 innoQ Deutschland GmbH (def numbers-vec (vec (range 1000000))) (defn proc-with-fold [nums] (->> nums (r/filter odd?) (r/map inc) (r/fold +))) ;; 2 cores (no hyper-threading) (time (proc-with-fold numbers-vec)) ;; "Elapsed time: 28.138 msecs"

Slide 41

Slide 41 text

Slide 42

Slide 42 text

© 2014 innoQ Deutschland GmbH (defn count-words [text] (let [words (clojure.string/split text #"\s+")] (reduce (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) {} words)))

Slide 43

Slide 43 text

© 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words (clojure.string/split text #"\s+")] (r/fold (fn ([acc1 acc2] (merge-with + acc1 acc2)) ([] {})) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))

Slide 44

Slide 44 text

© 2014 innoQ Deutschland GmbH (defn parallel-count-words [text] (let [words (clojure.string/split text #"\s+")] (r/fold (r/monoid (partial merge-with +) hash-map) (fn [acc word] (let [current (get acc word 0)] (assoc acc word (inc current)))) words)))

Slide 45

Slide 45 text

Slide 46

Slide 46 text