Tweet
Tweet

Slide 1

Slide 1 text

@ Alexey Kachayev, Ivan Kryvoruchko | November, 2017

Slide 2

Slide 2 text

Who We Are • 3k+ mobile apps for events • Content management system • Social network for attendees • Leads retrieval mobile application • Marketing automation platform • Event registration system (in development)

Slide 3

Slide 3 text

In Figures • 9K+ events with 500K+ speakers • 3.7M+ mobile devices • 1.7M+ users shared 1.4M posts with 11.3M+ likes • Mobile API with 19K req/min at peak • 118 repos and 25+ services (micro!) • 4B+ messages in Kafka (weekly) • 360M+ documents in ElasticSearch

Slide 4

Slide 4 text

What Do We Use Clojure For? • For everything • No, seriously

Slide 5

Slide 5 text

Elaborate? • Application servers • Data pipeline • ETL jobs • Internal tools • UI • Scheduling, talking to databases, data manipulation, … you name it

Slide 6

Slide 6 text

Clojure at Attendify • Main & default language • 4+ years in production • 12 engineers (and hiring) • CLOC: 112K+ Clojure, 78K+ ClojureScript • It’s still a technology, not magic

Slide 7

Slide 7 text

Why Do We Use Clojure? • Because we like it • No, seriously

Slide 8

Slide 8 text

Elaborate? • Clojure = ideas + access to Java libraries and JVM - immutability, epochal time model, atoms, transformation chains, EDN, uniform domain modeling, declarative approach, runtime polymorphism with custom dispatch etc • It’s easier to use Clojure ideas in Clojure - no, seriously - “haskell in python” << “python” & “haskell in python” << “haskell” - btw, “java in clojure” >> “java” - discipline & convention don’t work, language core does

Slide 9

Slide 9 text

Agenda • Libraries & ecosystem • Project setup & dev tools • Servers & microservices • HTTP • GraphQL • Data: PG, Kafka and others • Errors handling: why do you need to • Property based testing

Slide 10

Slide 10 text

Libraries & ecosystem • A lot of stuff - Clojars, Clojure Toolbox, ClojureWerkz • You still have Java - There are a lot of wrappers, but when it’s necessary… - You still have tooling: GC logs, OOM analyzers, profilers and more • You still work with Maven - lein, wagon, checkouts • Lisp-style ecosystem - A lot of libraries per each task (including not-very-production-ready)

Slide 11

Slide 11 text

Project Setup • Configuration is always tricky - environ library, .lein-env-dev files • components - Manage (stateful) world around you - Show code dependencies • dev/user.clj - With (go) and (reset) • nREPL - Try ultra (plugin for lein)

Slide 12

Slide 12 text

Project Setup: component (defrecord CSVExporter [] component/Lifecycle (start [component] (if (:executor component) component (assoc component :executor (e/fixed-thread-executor threads-count)))) (stop [component] (when-let [executor (:executor component)] (.shutdownNow executor)) (assoc component :executor nil)))

Slide 13

Slide 13 text

Project Setup: component (defrecord CSVExporter [] component/Lifecycle (start [component] (if (:executor component) component (assoc component :executor (e/fixed-thread-executor threads-count)))) (stop [component] (when-let [executor (:executor component)] (.shutdownNow executor)) (assoc component :executor nil))) prepare cleanup

Slide 14

Slide 14 text

Project Setup: component (component/system-map … :csv-exporter (component/using (export/new-csv-exporter {:immediate-export-limit csv-immediate-export-limit :immediate-export-timeout csv-immediate-export-timeout :threads-count csv-export-threads-count}) [:s3-client :email-notifier :profiles-search-handler :elastic-client :settings-event-store]) … ) (e/export csv-exporter account-id other-params)

Slide 15

Slide 15 text

Project Setup: component (component/system-map … :csv-exporter (component/using (export/new-csv-exporter {:immediate-export-limit csv-immediate-export-limit :immediate-export-timeout csv-immediate-export-timeout :threads-count csv-export-threads-count}) [:s3-client :email-notifier :profiles-search-handler :elastic-client :settings-event-store]) … ) (e/export csv-exporter account-id other-params) component config deps convention

Slide 16

Slide 16 text

Project Setup: REPL $ lein repl nREPL server started on port 63248 on host 127.0.0.1 - nrepl://127.0.0.1:63248 user=> (go) :started user=> (keys system) (:mailchimp-client :app-component :kafka-producer …) user=> (:s3-client system) #augustine.s3.S3Client{:url …} user=> (reset) :reloading (…) :resumed user=>

Slide 17

Slide 17 text

Project Setup: REPL $ lein repl nREPL server started on port 63248 on host 127.0.0.1 - nrepl://127.0.0.1:63248 user=> (go) :started user=> (keys system) (:mailchimp-client :app-component :kafka-producer …) user=> (:s3-client system) #augustine.s3.S3Client{:url …} user=> (reset) :reloading (…) :resumed user=> try this once

Slide 18

Slide 18 text

Servers & Microservices • RPC and GraphQL, not REST - JSON-RPC for clients and servers, nippy for inter-servers communication • aleph: smart wrapper for Netty - With ring & compojure (migrating to aleph only) • manifold, not core.async - Futures & chaining, executors, scheduling • Own library to define/use RPC - There are a few open-source libraries, like slacker and castra, tho’

Slide 19

Slide 19 text

Servers & Microservices (defservice “audience.fetchTimeline" :allowed-auth #{:builder} :validate-params {:profileId s/Str (s/optional-key :includeTypes) [s/Str] (s/optional-key :pageSize) s/Int} :response-format Timeline (fn [{:keys [timeline-handler]} {:keys [profileId]}] (timeline/timeline-entries …)))

Slide 20

Slide 20 text

Servers & Microservices (defservice “audience.fetchTimeline" :allowed-auth #{:builder} :validate-params {:profileId s/Str (s/optional-key :includeTypes) [s/Str] (s/optional-key :pageSize) s/Int} :response-format Timeline (fn [{:keys [timeline-handler]} {:keys [profileId]}] (timeline/timeline-entries …))) macro method name component(s) call params schema enforced check when :debug?

Slide 21

Slide 21 text

Servers & Microservices (rpc/call http-client {:method “audience.fetchTimeline” :params {:profileId “42”}})

Slide 22

Slide 22 text

Servers & Microservices (rpc/call http-client {:method “audience.fetchTimeline” :params {:profileId “42”} :codec :nippy :idempotent? true})

Slide 23

Slide 23 text

Servers & Microservices returns manifold.deferred component with aleph.http (rpc/call http-client {:method “audience.fetchTimeline” :params {:profileId “42”} :codec :nippy :idempotent? true}) retries policy request/response serializer

Slide 24

Slide 24 text

Servers & Microservices • defmulti rpc/execute • defservice compiles down to defmethod • ring route to call rpc/execute • ring handlers to take care about - Logging - Errors handling & reporting - Serialization format (JSON by default)

Slide 25

Slide 25 text

Servers & Microservices (defservice “audience.fetchTimeline" :allowed-auth #{:builder} :validate-params {:profileId s/Str (s/optional-key :includeTypes) [s/Str] (s/optional-key :pageSize) s/Int} :response-format Timeline :dispatch-on rpc/execute (fn [{:keys [timeline-handler]} {:keys [profileId]}] (timeline/timeline-entries …))) :dispatch-on rpc/execute “magic”

Slide 26

Slide 26 text

manifold || core.async • core.async is opinionated - a beautiful idea fighting an ugly reality - pretending everything is fine • manifold forces you to make decisions - a reality is ugly, find a way to deal with it - good luck, at least you know the truth!

Slide 27

Slide 27 text

manifold || core.async • core.async is opinionated - a beautiful idea fighting an ugly reality - pretending everything is fine • manifold forces you to make decisions - a reality is ugly, find a way to deal with it - good luck, at least you know the truth! More: https://github.com/ztellman/manifold/blob/master/docs/rationale.md

Slide 28

Slide 28 text

manifold || core.async • Concurrency is (still) hard • Implementing a server you (still) care about - Capacity planning - Interruptions and termination logic - Exceptions - Timeouts - Tail latency during spikes - Etc etc etc

Slide 29

Slide 29 text

HTTP with Aleph • Hundreds of millions HTTP requests daily - It’s hard, no matter what technology you use • “Upgraded” aleph server: - backpressure with no “waiters”, using j.u.c.Semaphore - custom threads pool for RPC tasks execution - a lot of (smaller) threads pools not to mess up with the server - “manual” j.u.c.RejectedExecutionException handlers instead of built-in from aleph - graceful shutdown

Slide 30

Slide 30 text

HTTP with Aleph: Own Wrapper (let [rpc-thread-count (atom 0) rpc-thread-factory (e/thread-factory #(format "augustine-rpc-%s-%d" project (swap! rpc-thread-count inc)) (deliver (promise) nil)) ;; unbounded rpc-executor (e/utilization-executor 0.9 512 {:thread-factory rpc-thread-factory}) semaphore (Semaphore. (or max-concurrent-requests 1024))]

Slide 31

Slide 31 text

HTTP with Aleph: Own Wrapper (let [rpc-thread-count (atom 0) rpc-thread-factory (e/thread-factory #(format "augustine-rpc-%s-%d" project (swap! rpc-thread-count inc)) (deliver (promise) nil)) ;; unbounded rpc-executor (e/utilization-executor 0.9 512 {:thread-factory rpc-thread-factory}) semaphore (Semaphore. (or max-concurrent-requests 1024))] to use our names separate executor (not default) pretty hard control

Slide 32

Slide 32 text

HTTP with Aleph: Own Wrapper (let [rpc-thread-count (atom 0) rpc-thread-factory (e/thread-factory #(format "augustine-rpc-%s-%d" project (swap! rpc-thread-count inc)) (deliver (promise) nil)) ;; unbounded rpc-executor (e/utilization-executor 0.9 512 {:thread-factory rpc-thread-factory}) semaphore (Semaphore. (or max-concurrent-requests 1024))] to use our names separate executor (not default) pretty hard control (deliver (promise) nil))

Slide 33

Slide 33 text

HTTP with Aleph

Slide 34

Slide 34 text

HTTP with Aleph • Using aleph client - Moving on with a default executor is (generally) a bad idea - But we had been doing this for 2+ years tho’ • “Upgraded” aleph client: - Accepting > 1 hosts to deal with servers failures - Retries policy that understands backpressure control, idempotency flags and connection errors - multipart/form-data encoding (back-ported)

Slide 35

Slide 35 text

HTTP with Aleph • Good HTTP client is harder than you might think - Connections pooling, dealing with keep-alive, embracing server-side interruptions - Unexpected server’s behavior (who reads RFC anyways?) - A lot of different (!) timeouts (connection, request sending, response first byte, response headers, chunks, body etc) - Essential complexity from a library point of view

Slide 36

Slide 36 text

HTTP with Aleph ¯\_(ツ)_/¯

Slide 37

Slide 37 text

HTTP with Aleph ¯\_(ツ)_/¯ (╯°□°)╯︵ ┻━┻

Slide 38

Slide 38 text

HTTP with Aleph ¯\_(ツ)_/¯ (╯°□°)╯︵ ┻━┻ Code: https://github.com/ztellman/aleph/blob/ 2ba3484ead3a8443667d3a3caddf9739c6841d9c/src/aleph/http.clj

Slide 39

Slide 39 text

GraphQL • Query language for APIs • Using it for Attendify Leads mobile application • Own wrapper on top of graphql-java • Everything is data ¯\_(ツ)_/¯

Slide 40

Slide 40 text

GraphQL • There are lacinia, graphql-clj and others • Libraries solve least of your problems: parsing & validating requests • Key problem is dealing with a client-side flexibility and loading data • Lucky we: most queries are compiled to SQL • Ideally: use muse to figure out a way to fetch data

Slide 41

Slide 41 text

Data • korma & hikari to work with PostgreSQL - Not really happy • Java API for Kafka producers/consumers - Own wrapper, but there are a few open-sourced, like kinsky & clj-kafka • carmine to work with Redis - Used to use built-in tasks/jobs queue, not now • Event sourcing with our own library - There are a few open-source libraries, like rill and cqrs-server, tho’

Slide 42

Slide 42 text

Data: Korma (In Theory) (defentity applications (table “application”) (pk :id) (transform #(update % :name capitalize))) (defn select-app [{:keys [conn]} app-id] (-> (select applications (db conn) (fields :name :icon) (where {:id app-id})) first))

Slide 43

Slide 43 text

Data: Korma (In Practice) (defn fetch-profile-with-sources* [db apikey id] (->> (models/execute-query db [(format "SELECT ps.*, ss.integration_id as source_integration_id, ss.remote_id as source_remote_id, ss.payload as source_payload, ss.is_deactivated as source_is_deactivated FROM %s ps LEFT OUTER JOIN %s ss on (ps.id = ss.profile_id) WHERE ps.apikey = ? AND ps.id = ?" (tables/route tables/profile-tbl apikey) (tables/route tables/profile-source-tbl apikey)) [apikey id]] {:raw-results? true}) (functor/fmap (fn [profiles] (->> profiles utils/aggregate-profiles (map transform-profile) first (#(dissoc % :password)))))))

Slide 44

Slide 44 text

Data: Korma (In Practice) (defn fetch-profile-with-sources* [db apikey id] (->> (models/execute-query db [(format "SELECT ps.*, ss.integration_id as source_integration_id, ss.remote_id as source_remote_id, ss.payload as source_payload, ss.is_deactivated as source_is_deactivated FROM %s ps LEFT OUTER JOIN %s ss on (ps.id = ss.profile_id) WHERE ps.apikey = ? AND ps.id = ?" (tables/route tables/profile-tbl apikey) (tables/route tables/profile-source-tbl apikey)) [apikey id]] {:raw-results? true}) (functor/fmap (fn [profiles] (->> profiles utils/aggregate-profiles (map transform-profile) first (#(dissoc % :password))))))) query, not DSL component not entities either manual transform partitioning

Slide 45

Slide 45 text

Data: Korma (In Practice) • Connections pooling • Sophisticated queries • Serialization handling (array, hstore, jsonb) • Transactions, retries, rollbacks • java.sql.SQLException • Tables partitioning

Slide 46

Slide 46 text

Data: Kafka Consumer • Internal library • Manage threads, subscribe/unsubscribe • Handle serialization • Exceptions, errors • Commit offsets

Slide 47

Slide 47 text

Data: Kafka Consumer (component/using (kafka/new-whisper {:name "pulse-audience-latest" :topic “attendify.s.a.v1" :on-message (partial pulse-audience/notify! pulse-audience-expiration-threshold pulse-audience-cache-ttl-ms) :threads 32 :kafka-config (assoc kafka-config :offset-reset "latest")}) [:redis-connector :db])

Slide 48

Slide 48 text

Data: Kafka Consumer (component/using (kafka/new-whisper {:name "pulse-audience-latest" :topic “attendify.s.a.v1" :on-message (partial pulse-audience/notify! pulse-audience-expiration-threshold pulse-audience-cache-ttl-ms) :threads 32 :kafka-config (assoc kafka-config :offset-reset "latest")}) [:redis-connector :db]) internal library sub/unsub consumers called on each message passed as a first arg

Slide 49

Slide 49 text

Event Sourcing: Own Library (def event-store-config {:initial-state {:segments []} :journal (event-sourcing/map->SyncPartitionedSQLJournal {:snapshot-entity settings-snapshot-entity :journal-entity settings-journal-entity}) :commands [{:cmd/name "create-segment" :cmd/schema {:name s/Str :query s/Str :createdBy s/Str} :cmd/init (fn [state cmd] (e/right (assoc cmd :id (next-id)))) :cmd/fold (fn [state cmd] (e/right (update state :segments conj cmd)))} ]})

Slide 50

Slide 50 text

Event Sourcing: Own Library (def event-store-config {:initial-state {:segments []} :journal (event-sourcing/map->SyncPartitionedSQLJournal {:snapshot-entity settings-snapshot-entity :journal-entity settings-journal-entity}) :commands [{:cmd/name "create-segment" :cmd/schema {:name s/Str :query s/Str :createdBy s/Str} :cmd/init (fn [state cmd] (e/right (assoc cmd :id (next-id)))) :cmd/fold (fn [state cmd] (e/right (update state :segments conj cmd)))} ]}) “updating” state prepare command validate params how to persist “the beginning” errors handling

Slide 51

Slide 51 text

Property Based Testing • “Official” org.clojure/test.check library • A lot of “supporting” libraries

Slide 52

Slide 52 text

Property Based Testing (defn gen-entity [n] (gen/fmap (fn [[id rev entity]] (assoc entity :id id :rev rev)) (gen/tuple gen-maybe-flake gen-maybe-flake (gen/resize n (gen/map gen/keyword gen/string-alphanumeric))))) (…) (defspec resolving-invalid-map-siblings 100 (prop/for-all [items (gen/such-that (fn [items] (< 1 (count (set (map :id items))))) (gen/not-empty (gen/vector (gen-entity 5))))] (either/left? (merge-items default-resolver items))))

Slide 53

Slide 53 text

Property Based Testing (defn gen-entity [n] (gen/fmap (fn [[id rev entity]] (assoc entity :id id :rev rev)) (gen/tuple gen-maybe-flake gen-maybe-flake (gen/resize n (gen/map gen/keyword gen/string-alphanumeric))))) (…) (defspec resolving-invalid-map-siblings 100 (prop/for-all [items (gen/such-that (fn [items] (< 1 (count (map :id items)))) (gen/not-empty (gen/vector (gen-entity 5))))] (either/left? (merge-items default-resolver items)))) describe properties define generators combining provided gens

Slide 54

Slide 54 text

Errors Handling • Most of your production code - No, seriously • No good story here - Clojure gives us exceptions and nil, only • Own library for either - cats seems to be the standard here, but no one cares about monads, right? • Typing your data, not your code - schema instead of spec

Slide 55

Slide 55 text

Errors Handling (In Practice) (->> (pg-profile/fetch-profile-by-email* db' apikey email) (either/bind #(check-profile-nil kp apikey token email %)) (either/bind #(check-profile-deleted kp apikey token email %)) (either/bind #(check-profile-claimed-or-registered kp apikey token email %)) (either/bind #(check-valid-password kp apikey token email % password)) (either/bind #(deactivate-old-sessions db' apikey % token)) (either/bind (fn [profile] (let [session-id (or session-id (generate-session-id)) now (Timestamp/from (Instant/now))] (—>> (insert-session db' …) (either/fmap (fn [session] {…})))))))

Slide 56

Slide 56 text

Errors Handling (In Practice) (->> (pg-profile/fetch-profile-by-email* db' apikey email) (either/bind #(check-profile-nil kp apikey token email %)) (either/bind #(check-profile-deleted kp apikey token email %)) (either/bind #(check-profile-claimed-or-registered kp apikey token email %)) (either/bind #(check-valid-password kp apikey token email % password)) (either/bind #(deactivate-old-sessions db' apikey % token)) (either/bind (fn [profile] (let [session-id (or session-id (generate-session-id)) now (Timestamp/from (Instant/now))] (—>> (insert-session db' …) (either/fmap (fn [session] {…}))))))) errors handling

Slide 57

Slide 57 text

Errors Handling (In Practice) (lete [profiles (fetch-access-keys-by-events db apikey restricted-event-ids) labels (labels/fetch-labels {:db db} apikey {:type "access-key"}) events-access-keys (->> labels (filter #(some (set (:events %)) restricted-event-ids)) (map :id)) profile-ids-blacklist (->> profiles (remove #(some events-access-keys (vec (.getArray (:access_keys %))))) (map :id))] (perform-fetching-by-events db apikey event-ids profile-ids-blacklist))

Slide 58

Slide 58 text

Errors Handling (In Practice) (lete [profiles (fetch-access-keys-by-events db apikey restricted-event-ids) labels (labels/fetch-labels {:db db} apikey {:type "access-key"}) events-access-keys (->> labels (filter #(some (set (:events %)) restricted-event-ids)) (map :id)) profile-ids-blacklist (->> profiles (remove #(some events-access-keys (vec (.getArray (:access_keys %))))) (map :id))] (perform-fetching-by-events db apikey event-ids profile-ids-blacklist)) macro deferred[either[a’]] deferred[either[a’]] a’

Slide 59

Slide 59 text

Dynamic Typing (In Theory) (defn new-app-from-opts [{:keys [environ private? project custom-handler exec-fn request-log-level request-log-transform-fn max-concurrent-requests await-termination-ms] :or {private? true project "Attendify" exec-fn execute request-log-level :debug request-log-transform-fn identity max-concurrent-requests 16384}}] (map->AppComponent {:environ environ …)

Slide 60

Slide 60 text

Dynamic Typing (In Theory) (defn new-app-from-opts [{:keys [environ private? project custom-handler exec-fn request-log-level request-log-transform-fn max-concurrent-requests await-termination-ms] :or {private? true project "Attendify" exec-fn execute request-log-level :debug request-log-transform-fn identity max-concurrent-requests 16384}}] (map->AppComponent {:environ environ …) “defaults”

Slide 61

Slide 61 text

Dynamic Typing (In Practice) (new-app-from-opts {:environ {} :private? false :project "Riverside" :exec-fn rpc/execute :max-conucrrent-requests 1024 :await-termination-ms 2000})

Slide 62

Slide 62 text

Dynamic Typing (In Practice) (new-app-from-opts {:environ {} :private? false :project "Riverside" :exec-fn rpc/execute :max-conucrrent-requests 1024 :await-termination-ms 2000}) 1. How many concurrent requests are allowed?

Slide 63

Slide 63 text

Dynamic Typing (In Practice) (new-app-from-opts {:environ {} :private? false :project "Riverside" :exec-fn rpc/execute :max-conucrrent-requests 1024 :await-termination-ms 2000}) 1. How many concurrent requests are allowed? 2. When will we notice???

Slide 64

Slide 64 text

Dynamic Typing (In Practice) (defn new-app-from-opts [{:keys [environ private? project custom-handler exec-fn request-log-level request-log-transform-fn max-concurrent-requests await-termination-ms] :or {private? true project "Attendify" exec-fn execute request-log-level :debug request-log-transform-fn identity max-concurrent-requests 16384} :as opts}] (schema.core/validate datatypes/AppOptions opts) (…))

Slide 65

Slide 65 text

Dynamic Typing (In Practice) (defn new-app-from-opts [{:keys [environ private? project custom-handler exec-fn request-log-level request-log-transform-fn max-concurrent-requests await-termination-ms] :or {private? true project "Attendify" exec-fn execute request-log-level :debug request-log-transform-fn identity max-concurrent-requests 16384} :as opts}] (schema.core/validate datatypes/AppOptions opts) (…)) (schema.core/validate datatypes/AppOptions opts) close keys space (by default)

Slide 66

Slide 66 text

Errors Handling • schema: close keys space by default • clojure.spec: open keys space by design • ¯\_(ツ)_/¯

Slide 67

Slide 67 text

We’re hiring #_:) More info: https://attendify.workable.com/j/6A2EBF73B6

Slide 68

Slide 68 text

Q&A