Clojure At Attendify

Transcript

1. Clojure at Attendify Dmytro Morozov, December 2019
3. In Figures 80K events 8M mobile devices 4M attendees shared 6.5M posts with 17M likes Mobile API with ~20K req/min at peak 118 repos with ~40 services ~600 msg/sec in Kafka 800M documents in ElasticSearch
4. Attendify’s Recognition
5. Scala Erlang Go Haskell Python Kotlin Clojure From www.lego.com
6. What Do We Use Clojure For? Core Attendify Services Data Pipeline UI (ClojureScript)
7. tonsky/rum for reactive UI juxt/bidi for Routing Flux-like state management (check out clj-commons/citrus) ClojureScript
8. So, Why Clojure? Because We Like It ¯\_(ツ)_/¯
9. Current State In production since 2013 Main & default language Currently 11 engineers 50 repositories LOC: 195K Clojure, 133K ClojureScript Hiring Clojurists is hard
10. Agenda 1. Data Validation 2. Working with SQL 3. Dealing with Errors 4. RPC API 5. Java Interop
11. How to Stay Sane Without Static Types?
12. Your Service Your Service When to validate? DB UI Service Message Broker
13. Libraries for Validating Data clojure/core.spec metosin/malli plumatic/schema - metosin/schema-tools - kitapps/schema-refined ❤
14. Introducing schema
15. (require '[schema.core :as s]) (def Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [{:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status TicketStatus}]})
16. (def TicketStatus (s/enum :open :closed)) (def Ticket {:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status TicketStatus}) (def Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [Ticket]}) (def CreateTicketRequest (dissoc Ticket :id))
17. (s/check Ticket {:id 42 :title "Early Bird" :text "Some randomness" :quantity 100 :status :skipped}) "=> {:id (not (instance? java.util.UUID 42)), :description missing-required-key, :status (not ("#{:open :closed} :skipped)), :text disallowed-key}
18. Limitations of Schema Regular functions don’t work well with Schema
19. (s/defschema Address {:street s/Str (s/optional-key :city) s/Str (s/required-key :country) {:name s/Str}}) ";; where's my city? (select-keys Address [:street :city]) ; {:street java.lang.String} ";; where's my city? (s/explain (select-keys AddressWithoutCountry [:street :city])) ";; "=> {:street Str} (s/check AddressWithoutCountry {:street "Volodymyrska St", :city "Kyiv"}) ";; "=> {:city disallowed-key}
20. Limitations of Schema Schema predicates are not first-class citizens
21. (def User ("-> {:id FlakeId :email s/Str :created-at s/Str} (s/constrained (fn [{:keys [id created-at]}] (validate id created-at))))) (def CreateUserReq (dissoc User :id :created-at)) (s/validate CreateUserReq {:email "alice@attendify.com"}) ";; Value does not match schema: ";; {:id missing-required-key, ";; :created-at missing-required-key}
22. “If you have expectations (of others) that aren't being met, those expectations are your own responsibility. You are responsible for your own needs. If you want things, make them.” -Rich Hickey, Open Source is Not About You
23. schema-refined
24. Predicates ";; "manually" with refined and predicates (def LatCoord (r/refined double (r/OpenClosedInterval -90.0 90.0))) (def LngCoord (r/OpenClosedIntervalOf double -180.0 180.0)) ";; Product type using a simple map (def GeoPoint {:lat LatCoord :lng LngCoord}) ";; using built-in types (def Route (r/BoundedListOf GeoPoint 2 50))
25. (def InZurich {:lat (r/refined double (r/OpenInterval 47.34 47.39)) :lng (r/refined double (r/OpenInterval 8.51 8.57))}) (def InRome {:lat (r/refined double (r/OpenInterval 41.87 41.93)) :lng (r/refined double (r/OpenInterval 12.46 12.51))})
26. Predicates … Compose ";; you can use schemas as predicates ";; First/Last are good examples of predicate “generics" (def RouteFromZurich (r/refined Route (r/First InZurich))) (def RouteToRome (r/refined Route (r/Last InRome))) ";; And, Or, Not, On (def RouteFromZurichToRome (r/refined Route (r/And (r/First InZurich) (r/Last InRome))))
27. Sum Types (def EmptyScrollableList {:items (s/eq []) :totalCount (s/eq 0) :hasNext (s/eq false)}) (defn NonEmptyScrollableListOf [dt] {:items [dt] :totalCount s/PosInt :hasNext s/Bool}) (defn ScrollableListOf [dt] (r/dispatch-on :totalCount 0 EmptyScrollableList :else (NonEmptyScrollableListOf dt)))
28. Product Types (def -FreeTicket (r/Struct :id r/NonEmptyStr :type (s/eq "free") :title r/NonEmptyStr :quantity (r/OpenIntervalOf int 1 1e4) :description (s/maybe r/NonEmptyStr) :status (s/enum :open :closed))) (def FreeTicket (r/guard -FreeTicket '(:quantity :status) enough-sits-when-open)) ";; #<StructMap {:description (constrained Str should-not-be-blank) ";; :type (eq "free") ";; :title (constrained Str should-not-be-blank) ";; :status (enum :open :closed) ";; :id java.lang.String ";; :quantity (constrained int should-be-bounded-by-range-given)} ";; Guarded with ";; enough-sits-when-open over '(:quantity :status)>
29. Product Types (def -PaidTicket (assoc FreeTicket :type (s/eq "paid") :priceInCents r/PositiveInt :taxes [Tax] :fees (s/enum :absorb :pass))) (def PaidTicket (r/guard -PaidTicket '(:taxes :fees) pass-tax-included)) ";; #<StructMap {""...} ";; Guarded with ";; enough-sits-when-open over '(:quantity :status) ";; pass-tax-included over '(:taxes :fees)>
30. TLDR Non-defined data shape is a brittle assumption Focus on modelling schemas for input and output Readability of schemas matters Types = Documentation
31. Working with SQL
32. JDBC (defn fetch-profile-with-sources* [db apikey id] (""->> (db/execute-query db [(format "SELECT ps.*, ss.integration_id as source_integration_id, ss.remote_id as source_remote_id, ss.payload as source_payload, ss.is_deactivated as source_is_deactivated, events.event_ids FROM %s ps LEFT OUTER JOIN %s ss on (ps.id = ss.profile_id) LEFT OUTER JOIN (SELECT att.profile_id, att.status, array_agg(att.event_id) as event_ids FROM %s att WHERE att.status IN(?, ?) AND att.profile_id = ? AND att.apikey = ? GROUP BY att.profile_id, att.status) events WHERE ps.apikey = ? AND ps.id = ?" (db/route tables/profile-tbl apikey) (db/route tables/profile-source-tbl apikey) (db/route tables/attendee-tbl apikey)) [(tables/attendee-status-str"->int "joined") (tables/attendee-status-str"->int "prejoined") id apikey apikey id]] {:raw-results? true}) (either/fmap -format-profile-with-sources))) Component Query, not DSL Either Partitioning
33. HugSQL (def fetch-messages* (dbutils/db-fn "SELECT m.id, jsonb_agg(distinct m) messages, jsonb_agg(distinct l) likes, max(l.rev) likes_max_rev, jsonb_agg(distinct r) replies, max(r.rev) replies_max_rev, jsonb_agg(distinct i) inappropriate_reports FROM :i:message-table m LEFT JOIN :i:like-table l ON l.message_id = m.id LEFT JOIN :i:reply-table r ON r.message_id = m.id LEFT JOIN :i:inappropriate-table i ON i.message_id = m.id WHERE m.event = :event --~ (when (:last-id params) \"AND m.id < :last-id\") --~ (when (:owner-id params) \"AND m.owner_id = :owner-id\") --~ (when (:types params) \"AND m.type IN (:v*:types)\") GROUP BY m.id ORDER BY m.id DESC LIMIT :page-size" {:tables {:message-table message-table :like-table like-table :reply-table reply-table :inappropriate-table inappropriate-table} :key-fn :apikey})) Partition Key Template conditions
34. SQL in the Wild Connections pooling Sophisticated queries Serialization (array, jsonb) Transactions, retries, rollbacks
35. SQL in the Wild Error handling Tables partitioning Event Sourcing
36. Error Handling
37. Error Handling Most of your production code No common approach Either for errors, not Exceptions
38. Example (defn check-email-not-used [db apikey profile email] (""->> (fetch-profile-by-email db apikey email) (either/bind (fn [other-profile] (if (some? other-profile) (either/left (errors/error "::errors/email-already-used)) (either/right profile))))))
39. Real World Example (defn create-session [db apikey email token password] (""->> (profile/fetch-profile-by-email* db apikey email) (either/bind "#(check-profile-nil apikey token email %)) (either/bind "#(check-profile-deleted apikey token email %)) (either/bind "#(check-profile-claimed apikey token email %)) (either/bind "#(check-valid-password apikey token email % password)) (either/bind "#(deactivate-old-sessions db apikey % token)) (either/bind (fn [profile] ""...)))) Error Handling
40. Real World Example (require '[cats.core :as m]) ("-> (fetch-user-by-id user-id) (m/bind enrich-user-profile)) IllegalArgumentException No implementation of method: :-mbind of protocol: #'cats.protocols/Monad found for class: cats.builtin$reify"__49991 clojure.core/-cache-protocol-fn (core_deftype.clj:583)
41. Monad in a monad (require [manifold.deferred :as d]) ";; Deferred (Either Email) (defn retrieve-account-email [stripe-client account-id] (d/chain' (retrieve stripe-client "accounts" account-id) (partial either/fmap :email)))
43. TLDR Either allows for composable error handling Monads are hard to use with dynamic typing Use exceptions for fatal errors
44. Doing RPC at Attendify
45. RPC RPC and GraphQL, not REST Aleph for network communication Manifold for concurrency Our own abstraction to define RPC methods
46. Aleph Not opinionated Based on Netty Asynchronous High performant Provides nice abstraction for concurrency
47. RPC API (defservice "audience.fetchTimelinePage" :allowed-auth "#{:builder} :validate-params {:scrollCursor s/Str} :response-format Timeline (fn [{:keys [timeline-handler accountId]} {:keys [scrollCursor]}] (timeline/timeline-entries …)))) Macro RPC Method Name Schema Request Params Components Returns manifold.deferred
48. RPC vs REST
49. Wrappers and Why You Don’t Need Them
50. Problems Opinionated Now you have 2 problems Doesn’t give a full control Slow dev cycle (>1 contributors)
51. Just use Java * Clojure has good Java interop Most Java APIs are good enough Own abstractions allow for tighter control * Use wrappers when Java API is not composable
52. What Clojure Has Taught Us There’s Still Joy In Programming Developing with REPL is powerful Start with simple Design for Errors Embrace Experiments, But Stay Pragmatic
53. Want to write Clojure? Join us at Attendify!
54. THANKS