Introduction to Clojure, the hard way

1. Fabulous adventures with Clojure Friday, October 26, 12
2. It’s all about data Friday, October 26, 12
3. Code is data Friday, October 26, 12
4. Data is data Friday, October 26, 12
5. Let’s set expectations Friday, October 26, 12
6. • sane development environment, integrated with a language • ability to develop and evaluate things on the fly • ease maintenance production • scalability (doh) • integration with well-known / existing tools • good, stable runtime • runtime debugging toolkit • performance What are you looking for? Friday, October 26, 12
7. • Leiningen, Clojure build tool https://github.com/technomancy/leiningen • repl just run `lein repl` in your console • emacs (it’s written in lisp, you know?) excellent nrepl and (older thing) clojure-swank, paredit-mode, formatting • nrepl get to the guts of runtime • VisualVm and the rest of Java goodness like jmap, jstack, jstat etc etc etc Sane development environment Friday, October 26, 12
8. • JMX java management extensions • nrepl • JVM configuration/tuning • No shared mutable state • No locks • Pure functions -> same input -> same output • Easy composition and pipelining Easy maintenance in production Friday, October 26, 12
9. • Baseline performance • Utilization of all processor resources anyone can eat out all the memory, right? • Painless parallel execution • Memory efficiency as far as it is possible with lisps, of course. in my mind that was mostly about structural sharing & persistent DS • JIT / inlining Scalability You keep using that word. I do not think it means what you think it means. (voice in my head) Friday, October 26, 12
10. • Very, very good Java interop options • Most of libraries are either ported to Clojure, or use Java library underneath • Your favorite Java application container / web server • Jar packaging / public / private Maven repositories • Not to replace, but to complement your existing JVM toolkit Integration Friday, October 26, 12
11. let’s build a small, but very useful thing Friday, October 26, 12
12. before we start, a couple of concepts... Friday, October 26, 12
13. immutable, persistent data structures Friday, October 26, 12
14. not about disk persistency Friday, October 26, 12
15. {:a 1} Friday, October 26, 12
16. {:a 1} Friday, October 26, 12
17. {:a 1 :b 2} {:a 1 :c 3} Friday, October 26, 12
18. (def a ‘(1 2 3)) 3 a 2 1 Friday, October 26, 12
19. (def a ‘(1 2 3)) (def b (cons 4 a)) 3 a 2 1 b 4 Friday, October 26, 12
20. (def a ‘(1 2 3)) (def b (cons 4 a)) (def c (cons 5 (rest a)) 3 a 2 1 b 4 c 5 Friday, October 26, 12
21. • doesn’t have anything to do with disk persistence • efficient creation of “modified” versions • structural sharing • inherently thread & iteration-safe will not mutate in a middle • immutable modification yields new coll • composite Persistent DSs? Friday, October 26, 12
22. STM Friday, October 26, 12
23. (def a (atom {})) a {} @reference value Friday, October 26, 12
24. (def a (atom {})) (swap! a assoc :b 1) a {} @reference value {:b 1} Friday, October 26, 12
25. (def a (atom {})) (swap! a assoc :b 1) a {} @reference value {:b 1} Friday, October 26, 12
26. (def a (atom {})) (swap! a assoc :b 1) a {} @reference value {:b 1} Friday, October 26, 12
27. (def a (atom {})) Declaration Initial value Friday, October 26, 12
28. (swap! a assoc :b 1) attributes for it Mutating func Friday, October 26, 12
29. (def a (atom {})) (swap! a assoc :b 1) (swap! a assoc :c 2) a @reference {:b 1} Friday, October 26, 12
30. (def a (atom {})) (swap! a assoc :b 1) (swap! a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Friday, October 26, 12
31. (def a (atom {})) (swap! a assoc :b 1) (swap! a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Structural sharing Friday, October 26, 12
32. (def a (atom {})) (swap! a assoc :b 1) (swap! a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Friday, October 26, 12
33. • reference, not value • refs / agents / atoms / vars • transactional memory access yes, with retries • no user locks, no deadlocks • mutation through pure function • coordinated • readers can read value at any point in time STM?.. Friday, October 26, 12
34. destructuring Friday, October 26, 12
35. (let [[first-item second-item third-item :as all-items] [1 2 3]] (println (format "First item: %d" first-item)) (println (format "Second item: %d" second-item)) (println (format "Third item: %d" third-item)) (println (format "All items: %s" all-items))) ;; First item: 1 ;; Second item: 2 ;; Third item: 3 ;; All items: [1 2 3] Friday, October 26, 12
36. let’s see it all in action! Friday, October 26, 12
37. production monitoring with Clojure Friday, October 26, 12
38. events are stored in memory, in atom Friday, October 26, 12
39. (defonce events (atom {})) Friday, October 26, 12
40. obtaining a store Friday, October 26, 12
41. ;; Get application identifier (defn get-locator ([app env] (keyword (str app "_" env))) ;; (get-locator “my-cool-app” “production”) ;; => :my-cool-app_production (defn get-or-create-store ([locator] (if-let [store (get @events locator)] ;; If the store does exists return reference to it store (do ;; If the store does not exist yet, create it (swap! events assoc locator (ref [])) (get @events locator)))) ([app env] (get-or-create-store (get-locator app env))) Friday, October 26, 12
42. event format Friday, October 26, 12
43. { ;; Md5 hash of an event, that uniquely identifies it :md5 "cd0e351d2eefdf0f79e0b55a0efe543b", ;; Arbitrary additional info :additional_info {:execution_time 0.5469999, :url “http://mysite.com/page0” }, ;; Event Type identifier (404, exception, page load time) :type "page_load", ;; Dispatcher host name :hostname "dc0-web01", ;; Time when event was received :received_at ..., ;; Tags assigned to the event :tags ["metrics" "performance"] } Friday, October 26, 12
44. adding an event Friday, October 26, 12
45. (defn add [m app env] (let [store (get-or-create-store app env)] ;; Start a transaction (dosync ;; It doesn’t really matter in which order we receive ;; events, as we receive them very often. So let’s use ;; commute. (conj) is commutative, so we will get (almost) ;; same result no matter what was the order of the received ;; events (commute store conj m)))) Friday, October 26, 12
46. launch a listener thread Friday, October 26, 12
47. (.start (Thread. ^Runnable (fn [] (udp/connect {:port 1234 :host "0.0.0.0"}))))) Java Thread Type hint Friday, October 26, 12
48. and let it run for a bit... Friday, October 26, 12
49. now, we’ve got some production data Friday, October 26, 12
50. what do we do with it?.. Friday, October 26, 12
51. let’s get stats! Friday, October 26, 12
52. Which hosts generate events? Friday, October 26, 12
53. (map (fn [[k v]] (distinct (map (fn [event] (:host event)) @v))) @events) Iterate through collection, generates lazy seq Destructuring Destructuring Friday, October 26, 12
54. (map (fn [[k v]] (distinct (map (fn [event] (:host event)) @v))) @events) k is now a locator :my-cool-app_production ... v is now a reference to events collection Friday, October 26, 12
55. (map (fn [[k v]] (distinct (map (fn [event] (:host event)) @v))) @events) take :host from every event Friday, October 26, 12
56. (map (fn [[k v]] (distinct (map (fn [event] (:host event)) @v))) @events) now, only distinct :host values from what we’ve got Friday, October 26, 12
57. (map (fn [[k v]] (distinct (map (fn [event] (:host event)) @v))) @events) both events hash is in atom per- application/environment vector is in ref, we need to dereference them Friday, October 26, 12
58. results? Friday, October 26, 12
59. (("dc01-web01" "dc01-web02") ("dc01-worker01" "dc01-worker02")) Friday, October 26, 12
60. now, let’s get an overview of types... Friday, October 26, 12
61. (defn- apply-to-values [map function] (into {} (for [[key value] map] [key (function value)]))) Friday, October 26, 12
62. (zipmap (keys @events) (map (fn [[locator c]] (let [collection @c types-grouped (group-by #(:type %) collection)] (apply-to-values types-grouped (fn [app-with-type] {:count (count all-with-type)})))) @events)) v group all the events by type Friday, October 26, 12
63. {"exception": [{:type "exception" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... } {:type "exception" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... }] "404": [{:type "404" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... } {:type "404" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... }]} group all the events by type Friday, October 26, 12
64. (zipmap (keys @events) (map (fn [[locator c]] (let [collection @c types-grouped (group-by #(:type %) collection)] (apply-to-values types-grouped (fn [app-with-type] {:count (count all-with-type)})))) @events)) apply fn to all values of a hash Friday, October 26, 12
65. {:my_webapp_production {"404" {:count 25540}, "page_load" {:count 4557}, "exception" {:count 473}} :rests_api_production {"stats" {:count 264}}, :my_webapp_staging {"404" {:count 2344}}, :worker {"memory_usage" {:count 17006}}} results! Friday, October 26, 12
66. average page load time?.. Friday, October 26, 12
67. no problem! Friday, October 26, 12
68. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body] `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) declare a dynamic variable. thread and scope local. Friday, October 26, 12
69. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body] `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) macro that will filter the collection for us, and put results for execution in binding Friday, October 26, 12
70. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body] `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) body will be evaluated in scope of binding so we could operate the filtered collection without referencing it elsewhere Friday, October 26, 12
71. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body] `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) filter function is applied to all entries, and only those that match will make it to *collection* Friday, October 26, 12
72. (with-coll-filtered "my-web-app" "production" #(= "page_load" (:type %)) (let [page-load-times (map (fn [i] (get-in i [:additional_info :execution_time])) *collection*)] (/ (apply + page-load-times) (count page-load-times)))) filter by page_load type get only execution times sum! count average Friday, October 26, 12
73. enter Clojure Friday, October 26, 12
74. Friday, October 26, 12
75. Werkz Friday, October 26, 12
76. It just werkz Friday, October 26, 12
77. Monger MongoDB client for a more civilized age: friendly, flexible and with batteries included https://github.com/michaelklishin/monger http://clojuremongodb.info/ Friday, October 26, 12
78. Friday, October 26, 12
79. Friday, October 26, 12
80. • Clojure is concise, performant and productive • Amazing for processing (big) data • Big set of available tools • Tools are easy-to-use, next-level • Will change the way you write programs • Benefits ahead Takeaway Friday, October 26, 12
81. Also by ClojureWerkz Langohr Feature-rich Clojure RabbitMQ client that embraces AMQP 0.9.1 Model https://github.com/michaelklishin/langohr Friday, October 26, 12
82. Also by ClojureWerkz Welle An expressive Clojure client for Riak https://github.com/michaelklishin/welle http://clojureriak.info/ Friday, October 26, 12
83. Also by ClojureWerkz Neocons A feature rich idiomatic Clojure client for the Neo4J REST API https://github.com/michaelklishin/neocons http://clojureneo4j.info/ Friday, October 26, 12
84. Also by ClojureWerkz Elastisch Clojure client for ElasticSearch https://github.com/clojurewerkz/elastisch Friday, October 26, 12
85. Also by ClojureWerkz Quartzite A powerful scheduling library for Clojure* https://github.com/michaelklishin/quartzite http://clojurequartz.info/ *now comes with Quartz-Mongodb and Quartzite-REST Friday, October 26, 12
86. Friday, October 26, 12
87. ifesdjeen clojurewerkz / / michaelklishin / Also, you should follow Friday, October 26, 12