Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Clojure, the hard way

αλεx π
October 26, 2012
640

Introduction to Clojure, the hard way

αλεx π

October 26, 2012
Tweet

Transcript

  1. • sane development environment, integrated with a language • ability

    to develop and evaluate things on the fly • ease maintenance production • scalability (doh) • integration with well-known / existing tools • good, stable runtime • runtime debugging toolkit • performance What are you looking for? Friday, October 26, 12
  2. • Leiningen, Clojure build tool https://github.com/technomancy/leiningen • repl just run

    `lein repl` in your console • emacs (it’s written in lisp, you know?) excellent nrepl and (older thing) clojure-swank, paredit-mode, formatting • nrepl get to the guts of runtime • VisualVm and the rest of Java goodness like jmap, jstack, jstat etc etc etc Sane development environment Friday, October 26, 12
  3. • JMX java management extensions • nrepl • JVM configuration/tuning

    • No shared mutable state • No locks • Pure functions -> same input -> same output • Easy composition and pipelining Easy maintenance in production Friday, October 26, 12
  4. • Baseline performance • Utilization of all processor resources anyone

    can eat out all the memory, right? • Painless parallel execution • Memory efficiency as far as it is possible with lisps, of course. in my mind that was mostly about structural sharing & persistent DS • JIT / inlining Scalability You keep using that word. I do not think it means what you think it means. (voice in my head) Friday, October 26, 12
  5. • Very, very good Java interop options • Most of

    libraries are either ported to Clojure, or use Java library underneath • Your favorite Java application container / web server • Jar packaging / public / private Maven repositories • Not to replace, but to complement your existing JVM toolkit Integration Friday, October 26, 12
  6. (def a ‘(1 2 3)) (def b (cons 4 a))

    3 a 2 1 b 4 Friday, October 26, 12
  7. (def a ‘(1 2 3)) (def b (cons 4 a))

    (def c (cons 5 (rest a)) 3 a 2 1 b 4 c 5 Friday, October 26, 12
  8. • doesn’t have anything to do with disk persistence •

    efficient creation of “modified” versions • structural sharing • inherently thread & iteration-safe will not mutate in a middle • immutable modification yields new coll • composite Persistent DSs? Friday, October 26, 12
  9. (def a (atom {})) (swap! a assoc :b 1) a

    {} @reference value {:b 1} Friday, October 26, 12
  10. (def a (atom {})) (swap! a assoc :b 1) a

    {} @reference value {:b 1} Friday, October 26, 12
  11. (def a (atom {})) (swap! a assoc :b 1) a

    {} @reference value {:b 1} Friday, October 26, 12
  12. (def a (atom {})) (swap! a assoc :b 1) (swap!

    a assoc :c 2) a @reference {:b 1} Friday, October 26, 12
  13. (def a (atom {})) (swap! a assoc :b 1) (swap!

    a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Friday, October 26, 12
  14. (def a (atom {})) (swap! a assoc :b 1) (swap!

    a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Structural sharing Friday, October 26, 12
  15. (def a (atom {})) (swap! a assoc :b 1) (swap!

    a assoc :c 2) a @reference {:b 1} {:b 1 :c 2 } Friday, October 26, 12
  16. • reference, not value • refs / agents / atoms

    / vars • transactional memory access yes, with retries • no user locks, no deadlocks • mutation through pure function • coordinated • readers can read value at any point in time STM?.. Friday, October 26, 12
  17. (let [[first-item second-item third-item :as all-items] [1 2 3]] (println

    (format "First item: %d" first-item)) (println (format "Second item: %d" second-item)) (println (format "Third item: %d" third-item)) (println (format "All items: %s" all-items))) ;; First item: 1 ;; Second item: 2 ;; Third item: 3 ;; All items: [1 2 3] Friday, October 26, 12
  18. ;; Get application identifier (defn get-locator ([app env] (keyword (str

    app "_" env))) ;; (get-locator “my-cool-app” “production”) ;; => :my-cool-app_production (defn get-or-create-store ([locator] (if-let [store (get @events locator)] ;; If the store does exists return reference to it store (do ;; If the store does not exist yet, create it (swap! events assoc locator (ref [])) (get @events locator)))) ([app env] (get-or-create-store (get-locator app env))) Friday, October 26, 12
  19. { ;; Md5 hash of an event, that uniquely identifies

    it :md5 "cd0e351d2eefdf0f79e0b55a0efe543b", ;; Arbitrary additional info :additional_info {:execution_time 0.5469999, :url “http://mysite.com/page0” }, ;; Event Type identifier (404, exception, page load time) :type "page_load", ;; Dispatcher host name :hostname "dc0-web01", ;; Time when event was received :received_at ..., ;; Tags assigned to the event :tags ["metrics" "performance"] } Friday, October 26, 12
  20. (defn add [m app env] (let [store (get-or-create-store app env)]

    ;; Start a transaction (dosync ;; It doesn’t really matter in which order we receive ;; events, as we receive them very often. So let’s use ;; commute. (conj) is commutative, so we will get (almost) ;; same result no matter what was the order of the received ;; events (commute store conj m)))) Friday, October 26, 12
  21. (map (fn [[k v]] (distinct (map (fn [event] (:host event))

    @v))) @events) Iterate through collection, generates lazy seq Destructuring Destructuring Friday, October 26, 12
  22. (map (fn [[k v]] (distinct (map (fn [event] (:host event))

    @v))) @events) k is now a locator :my-cool-app_production ... v is now a reference to events collection Friday, October 26, 12
  23. (map (fn [[k v]] (distinct (map (fn [event] (:host event))

    @v))) @events) take :host from every event Friday, October 26, 12
  24. (map (fn [[k v]] (distinct (map (fn [event] (:host event))

    @v))) @events) now, only distinct :host values from what we’ve got Friday, October 26, 12
  25. (map (fn [[k v]] (distinct (map (fn [event] (:host event))

    @v))) @events) both events hash is in atom per- application/environment vector is in ref, we need to dereference them Friday, October 26, 12
  26. (defn- apply-to-values [map function] (into {} (for [[key value] map]

    [key (function value)]))) Friday, October 26, 12
  27. (zipmap (keys @events) (map (fn [[locator c]] (let [collection @c

    types-grouped (group-by #(:type %) collection)] (apply-to-values types-grouped (fn [app-with-type] {:count (count all-with-type)})))) @events)) v group all the events by type Friday, October 26, 12
  28. {"exception": [{:type "exception" :md5 ... :additional_info ... :hostname ... :received_at

    ... :tags ... } {:type "exception" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... }] "404": [{:type "404" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... } {:type "404" :md5 ... :additional_info ... :hostname ... :received_at ... :tags ... }]} group all the events by type Friday, October 26, 12
  29. (zipmap (keys @events) (map (fn [[locator c]] (let [collection @c

    types-grouped (group-by #(:type %) collection)] (apply-to-values types-grouped (fn [app-with-type] {:count (count all-with-type)})))) @events)) apply fn to all values of a hash Friday, October 26, 12
  30. {:my_webapp_production {"404" {:count 25540}, "page_load" {:count 4557}, "exception" {:count 473}}

    :rests_api_production {"stats" {:count 264}}, :my_webapp_staging {"404" {:count 2344}}, :worker {"memory_usage" {:count 17006}}} results! Friday, October 26, 12
  31. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body]

    `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) declare a dynamic variable. thread and scope local. Friday, October 26, 12
  32. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body]

    `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) macro that will filter the collection for us, and put results for execution in binding Friday, October 26, 12
  33. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body]

    `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) body will be evaluated in scope of binding so we could operate the filtered collection without referencing it elsewhere Friday, October 26, 12
  34. (declare ^:dynamic *collection*) (defmacro with-coll-filtered [app env flt & body]

    `(let [store# (deref (get-or-create-store ~app ~env))] (binding [*collection* (filter ~flt store#)] ~@body))) filter function is applied to all entries, and only those that match will make it to *collection* Friday, October 26, 12
  35. (with-coll-filtered "my-web-app" "production" #(= "page_load" (:type %)) (let [page-load-times (map

    (fn [i] (get-in i [:additional_info :execution_time])) *collection*)] (/ (apply + page-load-times) (count page-load-times)))) filter by page_load type get only execution times sum! count average Friday, October 26, 12
  36. Monger MongoDB client for a more civilized age: friendly, flexible

    and with batteries included https://github.com/michaelklishin/monger http://clojuremongodb.info/ Friday, October 26, 12
  37. • Clojure is concise, performant and productive • Amazing for

    processing (big) data • Big set of available tools • Tools are easy-to-use, next-level • Will change the way you write programs • Benefits ahead Takeaway Friday, October 26, 12
  38. Also by ClojureWerkz Langohr Feature-rich Clojure RabbitMQ client that embraces

    AMQP 0.9.1 Model https://github.com/michaelklishin/langohr Friday, October 26, 12
  39. Also by ClojureWerkz Welle An expressive Clojure client for Riak

    https://github.com/michaelklishin/welle http://clojureriak.info/ Friday, October 26, 12
  40. Also by ClojureWerkz Neocons A feature rich idiomatic Clojure client

    for the Neo4J REST API https://github.com/michaelklishin/neocons http://clojureneo4j.info/ Friday, October 26, 12
  41. Also by ClojureWerkz Quartzite A powerful scheduling library for Clojure*

    https://github.com/michaelklishin/quartzite http://clojurequartz.info/ *now comes with Quartz-Mongodb and Quartzite-REST Friday, October 26, 12