Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Clojure At Attendify

Clojure At Attendify

Dmytro Morozov

December 07, 2019
Tweet

Other Decks in Programming

Transcript

  1. In Figures 80K events 8M mobile devices 4M attendees shared

    6.5M posts with 17M likes Mobile API with ~20K req/min at peak 118 repos with ~40 services ~600 msg/sec in Kafka 800M documents in ElasticSearch
  2. Current State In production since 2013 Main & default language

    Currently 11 engineers 50 repositories LOC: 195K Clojure, 133K ClojureScript Hiring Clojurists is hard
  3. Agenda 1. Data Validation 2. Working with SQL 3. Dealing

    with Errors 4. RPC API 5. Java Interop
  4. (require '[schema.core :as s]) (def Event {:id s/Uuid :name s/Str

    :online? s/Bool :sits s/Num :tickets [{:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status TicketStatus}]})
  5. (def TicketStatus (s/enum :open :closed)) (def Ticket {:id s/Uuid :title

    s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status TicketStatus}) (def Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [Ticket]}) (def CreateTicketRequest (dissoc Ticket :id))
  6. (s/check Ticket {:id 42 :title "Early Bird" :text "Some randomness"

    :quantity 100 :status :skipped}) "=> {:id (not (instance? java.util.UUID 42)), :description missing-required-key, :status (not ("#{:open :closed} :skipped)), :text disallowed-key}
  7. (s/defschema Address {:street s/Str (s/optional-key :city) s/Str (s/required-key :country) {:name

    s/Str}}) ";; where's my city? (select-keys Address [:street :city]) ; {:street java.lang.String} ";; where's my city? (s/explain (select-keys AddressWithoutCountry [:street :city])) ";; "=> {:street Str} (s/check AddressWithoutCountry {:street "Volodymyrska St", :city "Kyiv"}) ";; "=> {:city disallowed-key}
  8. (def User ("-> {:id FlakeId :email s/Str :created-at s/Str} (s/constrained

    (fn [{:keys [id created-at]}] (validate id created-at))))) (def CreateUserReq (dissoc User :id :created-at)) (s/validate CreateUserReq {:email "[email protected]"}) ";; Value does not match schema: ";; {:id missing-required-key, ";; :created-at missing-required-key}
  9. “If you have expectations (of others) that aren't being met,

    those expectations are your own responsibility. You are responsible for your own needs. If you want things, make them.” -Rich Hickey, Open Source is Not About You
  10. Predicates ";; "manually" with refined and predicates (def LatCoord (r/refined

    double (r/OpenClosedInterval -90.0 90.0))) (def LngCoord (r/OpenClosedIntervalOf double -180.0 180.0)) ";; Product type using a simple map (def GeoPoint {:lat LatCoord :lng LngCoord}) ";; using built-in types (def Route (r/BoundedListOf GeoPoint 2 50))
  11. (def InZurich {:lat (r/refined double (r/OpenInterval 47.34 47.39)) :lng (r/refined

    double (r/OpenInterval 8.51 8.57))}) (def InRome {:lat (r/refined double (r/OpenInterval 41.87 41.93)) :lng (r/refined double (r/OpenInterval 12.46 12.51))})
  12. Predicates … Compose ";; you can use schemas as predicates

    ";; First/Last are good examples of predicate “generics" (def RouteFromZurich (r/refined Route (r/First InZurich))) (def RouteToRome (r/refined Route (r/Last InRome))) ";; And, Or, Not, On (def RouteFromZurichToRome (r/refined Route (r/And (r/First InZurich) (r/Last InRome))))
  13. Sum Types (def EmptyScrollableList {:items (s/eq []) :totalCount (s/eq 0)

    :hasNext (s/eq false)}) (defn NonEmptyScrollableListOf [dt] {:items [dt] :totalCount s/PosInt :hasNext s/Bool}) (defn ScrollableListOf [dt] (r/dispatch-on :totalCount 0 EmptyScrollableList :else (NonEmptyScrollableListOf dt)))
  14. Product Types (def -FreeTicket (r/Struct :id r/NonEmptyStr :type (s/eq "free")

    :title r/NonEmptyStr :quantity (r/OpenIntervalOf int 1 1e4) :description (s/maybe r/NonEmptyStr) :status (s/enum :open :closed))) (def FreeTicket (r/guard -FreeTicket '(:quantity :status) enough-sits-when-open)) ";; #<StructMap {:description (constrained Str should-not-be-blank) ";; :type (eq "free") ";; :title (constrained Str should-not-be-blank) ";; :status (enum :open :closed) ";; :id java.lang.String ";; :quantity (constrained int should-be-bounded-by-range-given)} ";; Guarded with ";; enough-sits-when-open over '(:quantity :status)>
  15. Product Types (def -PaidTicket (assoc FreeTicket :type (s/eq "paid") :priceInCents

    r/PositiveInt :taxes [Tax] :fees (s/enum :absorb :pass))) (def PaidTicket (r/guard -PaidTicket '(:taxes :fees) pass-tax-included)) ";; #<StructMap {""...} ";; Guarded with ";; enough-sits-when-open over '(:quantity :status) ";; pass-tax-included over '(:taxes :fees)>
  16. TLDR Non-defined data shape is a brittle assumption Focus on

    modelling schemas for input and output Readability of schemas matters Types = Documentation
  17. JDBC (defn fetch-profile-with-sources* [db apikey id] (""->> (db/execute-query db [(format

    "SELECT ps.*, ss.integration_id as source_integration_id, ss.remote_id as source_remote_id, ss.payload as source_payload, ss.is_deactivated as source_is_deactivated, events.event_ids FROM %s ps LEFT OUTER JOIN %s ss on (ps.id = ss.profile_id) LEFT OUTER JOIN (SELECT att.profile_id, att.status, array_agg(att.event_id) as event_ids FROM %s att WHERE att.status IN(?, ?) AND att.profile_id = ? AND att.apikey = ? GROUP BY att.profile_id, att.status) events WHERE ps.apikey = ? AND ps.id = ?" (db/route tables/profile-tbl apikey) (db/route tables/profile-source-tbl apikey) (db/route tables/attendee-tbl apikey)) [(tables/attendee-status-str"->int "joined") (tables/attendee-status-str"->int "prejoined") id apikey apikey id]] {:raw-results? true}) (either/fmap -format-profile-with-sources))) Component Query, not DSL Either Partitioning
  18. HugSQL (def fetch-messages* (dbutils/db-fn "SELECT m.id, jsonb_agg(distinct m) messages, jsonb_agg(distinct

    l) likes, max(l.rev) likes_max_rev, jsonb_agg(distinct r) replies, max(r.rev) replies_max_rev, jsonb_agg(distinct i) inappropriate_reports FROM :i:message-table m LEFT JOIN :i:like-table l ON l.message_id = m.id LEFT JOIN :i:reply-table r ON r.message_id = m.id LEFT JOIN :i:inappropriate-table i ON i.message_id = m.id WHERE m.event = :event --~ (when (:last-id params) \"AND m.id < :last-id\") --~ (when (:owner-id params) \"AND m.owner_id = :owner-id\") --~ (when (:types params) \"AND m.type IN (:v*:types)\") GROUP BY m.id ORDER BY m.id DESC LIMIT :page-size" {:tables {:message-table message-table :like-table like-table :reply-table reply-table :inappropriate-table inappropriate-table} :key-fn :apikey})) Partition Key Template conditions
  19. Example (defn check-email-not-used [db apikey profile email] (""->> (fetch-profile-by-email db

    apikey email) (either/bind (fn [other-profile] (if (some? other-profile) (either/left (errors/error "::errors/email-already-used)) (either/right profile))))))
  20. Real World Example (defn create-session [db apikey email token password]

    (""->> (profile/fetch-profile-by-email* db apikey email) (either/bind "#(check-profile-nil apikey token email %)) (either/bind "#(check-profile-deleted apikey token email %)) (either/bind "#(check-profile-claimed apikey token email %)) (either/bind "#(check-valid-password apikey token email % password)) (either/bind "#(deactivate-old-sessions db apikey % token)) (either/bind (fn [profile] ""...)))) Error Handling
  21. Real World Example (require '[cats.core :as m]) ("-> (fetch-user-by-id user-id)

    (m/bind enrich-user-profile)) IllegalArgumentException No implementation of method: :-mbind of protocol: #'cats.protocols/Monad found for class: cats.builtin$reify"__49991 clojure.core/-cache-protocol-fn (core_deftype.clj:583)
  22. Monad in a monad (require [manifold.deferred :as d]) ";; Deferred

    (Either Email) (defn retrieve-account-email [stripe-client account-id] (d/chain' (retrieve stripe-client "accounts" account-id) (partial either/fmap :email)))
  23. TLDR Either allows for composable error handling Monads are hard

    to use with dynamic typing Use exceptions for fatal errors
  24. RPC RPC and GraphQL, not REST Aleph for network communication

    Manifold for concurrency Our own abstraction to define RPC methods
  25. RPC API (defservice "audience.fetchTimelinePage" :allowed-auth "#{:builder} :validate-params {:scrollCursor s/Str} :response-format

    Timeline (fn [{:keys [timeline-handler accountId]} {:keys [scrollCursor]}] (timeline/timeline-entries …)))) Macro RPC Method Name Schema Request Params Components Returns manifold.deferred
  26. Problems Opinionated Now you have 2 problems Doesn’t give a

    full control Slow dev cycle (>1 contributors)
  27. Just use Java * Clojure has good Java interop Most

    Java APIs are good enough Own abstractions allow for tighter control * Use wrappers when Java API is not composable
  28. What Clojure Has Taught Us There’s Still Joy In Programming

    Developing with REPL is powerful Start with simple Design for Errors Embrace Experiments, But Stay Pragmatic