Clojure - A Sweetspot for Analytics

EuroClojure 2015 Talk Slides:

Clojure is getting more and more traction, and more people use it for all kinds of backend processing. During last 3 years us in ClojureWerkz concentrated on making lives of backend developers simple. Today, Clojure is one of the best choices for Analytics and Data-Driven Backends.

I'll tell you about our motivation, experiences and our success story, how we made a data processing backend, currently pushing millions of messages per second, how Clojure made our development cycles and time-to-production shorter, lives of our devs better, and made our customers happier.

About the speaker: Alex is working on making backends for analytics and data processing. He's been involved with Clojure since 2011, and co-created ClojureWerkz, is actively involved in development and maintenance of many Clojure libraries. Spends most of free time reading Math and Probability Theory textbooks, figuring out how things work.

Transcript

2. Sweetspot for Analytics
4. ClojureWerkz 35+ high-quality Clojure libraries User reports from all over the world 20+ active contributors We value documentation
5. Why Clojure is so awesome?
6. Overtone isn’t just for music
7. Minority Report with Clojure REPL
8. Great for Math™
9. Talk Expectations
10. What is even Analytics
11. Getting something out of nothing
12. Problems to solve understand the shape of data in a pace that suits your business
13. Problems to solve classify and predict the outcomes reliably
14. Problems to solve model and understand your chances
15. Challenges
16. Clojure Shine
17. Anglican Is what you open when they say “what are the odds”
19. Anglican Probabilistic Programming DSL/Language Clojure Macros at their very best
20. Coin flips (let [data (->> #(sample (flip 0.5)) repeatedly (take 50) (map #(if % "heads" "tails")) frequencies)] (plot/bar-chart (keys data) (vals data))) Anglican
21. Coin Flips Coin flips: fair coin, few trials
22. Coin Flips Coin flips: rigged coin, many trials
23. Coin Flips Coin flips: fair coin, many trials
24. Simulating Cassandra Cluster Sizes disclaimer: model is simplified and numbers are made up! 1 node can handle 10K requests Latency is normally distributed with mean of 20ms standard deviation of 5ms “Extra” requests add overhead exponentially Anglican
25. (def base-requests (* 10 1000)) (defquery cluster-latency [n write-rate] (let [per-node (/ write-rate n) overhead (/ 1000 (if (> per-node base-requests) (- per-node base-requests) 1))] (predict :latency (+ (sample (exponential overhead)) (sample (normal 20 5)))))) disclaimer: model is simplified and numbers are made up! Simulating Cassandra Cluster Sizes
26. 5 Nodes / 50K requests per second Simulating Cassandra Cluster Sizes
27. 5 Nodes / 500K requests per second Simulating Cassandra Cluster Sizes
28. 30 Nodes / 500K Requests Simulating Cassandra Cluster Sizes
29. Subset of Clojure, compiled into CPS-style fns Stackless language Built-in memoisation DSL for building sampling fns for distributions Anglican
30. Statistiker En statistiker er en person som jobber innen faget statistikk.
31. Implementing gaussian Naïve Bayes Algorithm
32. Implementing Naïve Bayes Algorithm
33. P(blue)= Number of Blue Total number of objects P(red)= Number of Red Total number of objects
34. P(X | blue)= Number of Blue near X Total number of blue P(X | red)= Number of Red near X Total number of Red
35. P(blue)= Number of Blue Total number of objects P(red)= Number of Blue Total number of objects Model (prior) (defn make-model [train-data] (let [total (->> train-data vals (map count) (reduce +))] (for [[k v] train-data] [k {:p (/ (count v) total) :evidence (->> v transpose (map (fn [v] {:mean (mean v) :variance (variance v)})))}])))
36. Classifier (posterior) (defn posterior-prob [point variance mean] (* (/ 1 (sqrt (* 2 pi variance))) (exp (/ (* -1 (pow (- point mean) 2)) (* 2 variance))))) (map (fn [point {:keys [mean variance]}] (posterior-prob point variance mean)) model) P(X | blue)= Number of Blue near X Total number of blue P(X | red)= Number of Red near X Total number of Red
37. P(X | blue)= Number of Blue near X Total number of blue P(X | red)= Number of Red near X Total number of Red
38. Implementing Linear Regression with Gradient Descent
39. Linear Regression with Gradient Descent (s/defrecord GradientProblem [^{:s ObjectiveFunction} objective-fn ^{:s ObjectiveFunctionGradient} objective-fn-gradient])
40. Linear Regression: Objective Function Basically, the distance between predicted and actual Y: (objective-function (fn [intercept slope] (let [f (line intercept slope) res (->> points (map (fn [[x y]] (sqr (- y (f x))))) (reduce +))] res)))
41. Linear Regression: Objective Function Basically, the distance between predicted and actual Y: (objective-function-gradient (let [factors (->> points (map butlast) (map #(cons 1 %))) y (map last points)] (fn [& point] (let [xT (matrix/transpose factors) m! (matrix/inverse (matrix/dot xT factors)) b (matrix/dot xT y)] (ops/- (matrix/mmul m! b) point)))))
42. Linear Regression: Objective Function
43. Bunch of JVM libraries available clojure.matrix is great clojure.match greatly helps with algos Clojure fns are easy to test With immutable DSs nothing goes wrong Experience Report
44. Balagan When `update-in` and `get-in` is not enough
46. Nested data structures Map-inside-vector-inside-map Straightforward query language Balagan
47. [:* :* even? :c] Balagan
48. :* wildcard matches Custom key matchers (essentially any fn) `get-in`-like key matches Balagan
49. (b/select {:a {:b [{:c 1} {:c 2} {:c 3}]}} [:* :* even? :c]) ;; => [1 3] Balagan
50. Walking-transformations Match paths and apply updates Balagan
51. (b/update {:a {:b [1 2 3]}} [:a :b :*] inc) ;; => {:a {:b [2 3 4]}} Balagan
52. Operations on complex data structures DSL for querying and transforming Balagan
53. Inspired by Enlive Extensible walkers for custom data structs Clojure/ClojureScript enabled via cljx Balagan
54. Meltdown Stream Processing with a long half-life
56. Data processing pipeline Performant Tuneable Multiple backends (Disruptor, Queues, sync…) CEP/EEP for everyone Meltdown
57. Base API (reactor/on <selector> <event handler>) (reactor/notify <key> <event) Enqueue Handle
58. filter* map* batch* reduce* group* consume* Base API
59. (reactor/on ($ “a”) filter* map* batch* reduce* (reactor/on ($ “b”) (reactor/on ($ “c”) (reactor/on ($ “a”) ) ) ) ) “Named” streams: DSLs
60. Reduce boilerplate for processing topologies Implicit wiring between occurring parts No changes to the base API Attach parts of the stream for better composition DSLs
61. (streams/stream ($ “a”) filter* map* batch* reduce* ) “Anonymous” streams: DSLs
62. On-premise streams Per-entity streams Decouple data processing pipelines Avoid hash lookups within sync operations Parallelize Maintain streams independently DSLs
63. (streams/lazy-stream ($ #“a”) filter* map* batch* reduce* ) “Lazy” streams: DSLs
64. DSLs: Macros Powerful way to hide the “wiring” No changes to API Completely different handling logic Eager, delayed, wired, etc… streams
65. Max-out single-box performance Pluggable back-end Anonymous, lazy, named streams DSLs
66. Buffy the byte buffer slayer
67. byte buffer
68. Buffy Composeable binary protocols Partial deserialisation Named access to serialised parts
69. (spec :my-field-1 (int32-type) :my-field-2 (string-type 10)) Create a spec out of parts 0 4 14 +------------+-------------------------+ | my-field-1 | my-field-2 | | (int) | (10 string) | +------------+-------------------------+ Memory Layout Buffy
70. (let [s (spec :int-field (int32-type) :string-field (string-type 10)) buf (compose-buffer s)] (set-field buf :int-field 101) (get-field buf :int-field)) ;; => 101 Use spec to access fields like in a map Buffy
71. (let [s (spec :first-field (int32-type) :second-field (string-type 10) :third-field (boolean-type)) buf (compose-buffer spec)] (set-fields buf {:first-field 101 :second-field "string" :third-field true}) (decompose buf)) ;; => {:third-field true :second-field "string" :first-field 101} Or decode them all together Buffy
72. (composite-type (int32-type) (string-type 10)) Composite (tuple) types Buffy DSLs
73. (repeated-type (string-type 10) 5) Array/vector types Buffy DSLs
74. (repeated-type (composite-type (int32-type) (string-type 10)) 5) And recursion! Buffy DSLs
75. (def dynamic-string (frame-type (frame-encoder [value] length (short-type) (count value) string (string-type (count value)) value) (frame-decoder [buffer offset] length (short-type) string (string-type (read length buffer offset))) second)) Dynamic types: netstrings Buffy DSLs
76. Protocols help to abstract a notion of Data Type Data Types are extendable! Macros for creating the custom decoders Lessons Learned
77. Conclusions
78. @ifesdjeen