Keep Your Data Safe With Refined Types

Slide 1

Slide 1 text

Keep Your Data Safe With Reﬁned Types Doing Clojure, Sleeping Well ™ Oleksii Kachaiev, @kachayev

Slide 2

Slide 2 text

@Me • CTO at Attendify • 5+ years with Clojure in production • Creator of Muse | Aleph & Netty contributor • More: protocols, algebras, Haskell, Idris • @kachayev on Twitter & Github

Slide 3

Slide 3 text

What Am I Even Talking About?

Slide 4

Slide 4 text

"No Types" ™ In The Wild I don't like the term "dynamic language" But you all know what I mean Almost no compile-time correctness guarantees

Slide 5

Slide 5 text

"No Types" ™ In The Wild You still can do a lot and go really far Less data structures requires less checks, right? Kinda "banned" topic by the community

Slide 6

Slide 6 text

What Can Possibly Go Wrong?™

Slide 7

Slide 7 text

U Y No Types? We still need some kind of "types" • to model data in advance • to validate your data Otherwise you'll mess something up quickly

Slide 8

Slide 8 text

a choice between “you want to take your pain up front or gradually over time” — Clojure the Devil…is in the detail

Slide 9

Slide 9 text

Any data-intense application is built around the model that's being implemented in a dynamically typed language remains informally deﬁned and requires the number of prays quadratic to the number of non-deﬁned data types. At some point supporting such a system becomes indistinguishable from magic.

Slide 10

Slide 10 text

I call this "Harry-from- Hogwarts isomorphism". — Oleksii Kachaiev

Slide 11

Slide 11 text

To Take From This Talk • non-deﬁned data shape = someone's assumption • simple when designing, impossible when operating • typing data with Int and String doesn't help a lot • you don't have to type data transformations • (as long as input & output are covered)

Slide 12

Slide 12 text

When To Validate? • RPC request comes in • RPC response comes out • Reading from & writing to DB (disks, caches etc) • Reading from & writing to Kafka (queues, logs etc) • And more!

Slide 13

Slide 13 text

Introducing schema

Slide 14

Slide 14 text

(require '[schema.core :as s]) (def Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [{:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status (s/enum :open :closed)}]})

Slide 15

Slide 15 text

Are Those Even "Types"? Checking things in runtime opens a lot of doors! Idris built on the idea that types are values Same goes for your runtime It's just data! You can operate it as you need.

Slide 16

Slide 16 text

(def TicketStatus (s/enum :open :closed)) (def Ticket {:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status TicketStatus}) (def Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [Ticket]}) (def CreateTicketRequest (dissoc Ticket :id))

Slide 17

Slide 17 text

(s/check Ticket {:id 42 :title "Early Bird" :text "Some randomness" :quantity 100 :status :skipped}) ;; => {:id (not (instance? java.util.UUID 42)) :description missing-required-key :status (not (#{:open :closed} :skipped)) :text disallowed-key}

Slide 18

Slide 18 text

How Safe We Are Now?

Slide 19

Slide 19 text

Our Goals Are • Readability and Soundness (harder than it seems) • Being as precise as we can • Avoid as many bugs as possible • Provide clean and useful error messages • Keep serialization and business logic separated

Slide 20

Slide 20 text

Story #1 Fantastic nil And Where To Find It (shameless spoiler: everywhere)

Slide 21

Slide 21 text

How To Deﬁne Optional? #1: s/maybe (def TicketDescription {:description (s/maybe s/Str)}) ;; works {:description nil} {:description "A lot of free places!"} ;; doesn't {:description 1457} {:text "Really a lot!"}

Slide 22

Slide 22 text

How To Deﬁne Optional? #1: s/maybe Looks good! It's "type-safe" and it's functional! (functor, wheeee) But it's still error-prone ☹

Slide 23

Slide 23 text

How To Deﬁne Optional? #1: s/maybe (let [draft (read-from-db db-conn ticket-id)] (rpc/call "createFreeTicket" {:id ticket-id :title (:title draft) :description (sanitize-html (:decsription draft)) :price default-price :quantity 100}))

Slide 24

Slide 24 text

Clojure is the nil-tolerant language. You will miss something. Sooner or later.

Slide 25

Slide 25 text

Tolerant-Reader They Say ;; clojure.spec has "open keys space" design ;; meaning unknown keys are OKay ;; combine with nil-able values (rpc/call "createFreeTicket" {:id ticket-id :title (:title draft) :decsription (sanitize-html (:description draft)) :price default-price :quantity 100}) ;; welcome to data hell ;; :trollface:

Slide 26

Slide 26 text

How To Deﬁne Optional? #2: optional-key (def TicketDescription {(s/optional-key :description) s/Str}) ;; now you have more work to do (cond-> ticket (do-i-have-description?) (assoc :description "This would be amazing!")) (harder to mess up, but still... )

Slide 27

Slide 27 text

How To Deﬁne Optional? #3: All The Above! (def TicketDescription {(s/optional-key :description) (s/maybe s/Str)}) ;; hmm... now you can pass whatever you want!

Slide 28

Slide 28 text

How To Deﬁne Optional? #3: All The Above! Your API users will beg you for this! It's so super ﬂexible! Please, just don't.

Slide 29

Slide 29 text

How To Deﬁne Optional? Being "type-safe" is not a goal Being "functional" is not a goal Our goal is to reduce number of errors

Slide 30

Slide 30 text

Deﬁne Optional With Own "Void" ;; domain specific voided value (def UnlimitedPurchase {:limited? (s/eq false)}) (def PurchaseLimit {:limited? (s/eq true) :limit s/Num}) ;; generic voided value (def NoTicketDescription {:description {:nothing (s/eq true)}}) (def TicketDescription {:description {:just s/Str}})

Slide 31

Slide 31 text

Story #2 Staying Precise With Constraints

Slide 32

Slide 32 text

Be Precise! (def PositiveInt (s/constrained s/Int pos? 'should-be-positive)) (def NonEmptyStr (s/constrained s/Str #(not (clojure.string/blank? %)) 'should-not-be-blank))

Slide 33

Slide 33 text

Combine Things! (defn BoundedListOf [dt left right] (s/constrained [dt] #(<= left (count %) right) 'collection-length-should-conform-boundaries)) {:id s/Uuid :name NonEmptyStr :online? s/Bool :sits PositiveInt :tickets (BoundedListOf Ticket 1 25)}

Slide 34

Slide 34 text

Express Business Rules (def -Event {:id s/Uuid :name s/Str :online? s/Bool :sits s/Num :tickets [Ticket]}) (def Event (-> -Event (s/constrained (fn [{:keys [sits tickets]}] (>= sits (apply + (map :quantity tickets)))) 'tickets-quantities-should-not-exceed-sits-count) (s/constrained ...)))

Slide 35

Slide 35 text

Story #3 Sum Types ↠ Conditionals

Slide 36

Slide 36 text

Sum Types data Result a b = Ok a | Error b enum Result { Ok(T), Error(E), } type result('good, 'bad) = | Ok('good) | Error('bad);

Slide 37

Slide 37 text

Sum Types In Clojure (defn Result [ok error] (s/either {:ok ok} {:error error})) WARN: Deprecated!

Slide 38

Slide 38 text

Sum Types In Clojure (personal opinion) schema is designed for validation, not modeling No "difference by construction" Easy to mess up

Slide 39

Slide 39 text

s/conditional Instead

Slide 40

Slide 40 text

Just Specify Discriminator (defn Result [ok error] (s/conditional #(contains? % :ok) {:ok ok} #(contains? % :error) {:error error})) (better, but still... ! )

Slide 41

Slide 41 text

Example #2.1 Free or Paid?

Slide 42

Slide 42 text

This Is Bad :( (def Ticket {:id Id :type (s/enum "free" "paid") :name NonEmptyStr :quantity (TypedRange int 1 1e4) :description (Maybe NonEmptyStr) (s/optional-key :priceInCents) PositiveInt (s/optional-key :taxes) [Tax] (s/optional-key :fees) (s/enum :absorb :pass) :status (e/enum :open :closed)})

Slide 43

Slide 43 text

Way Better! (def FreeTicket {:id Id :type (s/eq "free") :title NonEmptyStr :quantity (TypedRange int 1 1e4) :description (Maybe NonEmptyStr) :status (e/enum :open :closed)}) (def PaidTicket (assoc FreeTicket :type (s/eq "paid") :priceInCents PositiveInt :taxes [Tax] :fees (s/enum :absorb :pass)))

Slide 44

Slide 44 text

After Cosmetic Changes... (def Ticket (s/conditional #(= "free" (:type %)) FreeTicket #(= "paid" (:type %)) PaidTicket)) turned into (def Ticket (dispatch-on :type "free" FreeTicket "paid" PaidTicket))

Slide 45

Slide 45 text

Example #2.2 Scroll API

Slide 46

Slide 46 text

(def EmptyScrollableList {:items (s/eq []) :totalCount (s/eq 0) :hasNext (s/eq false) :hasPrev (s/eq false) :nextPageCursor (s/eq nil) :prevPageCursor (s/eq nil)}) (defn NonEmptyScrollableList [dt] (dispatch-on (juxt :hasNext :hasPrev) [false false] (SinglePage dt) [true false] (FirstPage dt) [false true] (LastPage dt) [true true] (ScrollableListSlice dt))) (defn ScrollableList [dt] (dispatch-on :totalCount 0 EmptyScrollableList :else (NonEmptyScrollableList dt)))

Slide 47

Slide 47 text

Reﬁne All The Types!

Slide 48

Slide 48 text

How Are We Doing So Far?

Slide 49

Slide 49 text

(def -Ticket {:id s/Uuid :title s/Str :description (s/maybe s/Str) :quantity s/Num (s/optional-key :price) s/Num :status (s/enum :open :closed)}) (def Ticket (s/constrained -Ticket (fn [{:keys [quantity status]}] (or (= :closed status) (< 0 quantity))))) (def CreateTicketRequest (dissoc Ticket :id :status))

Slide 50

Slide 50 text

So Far So Good? (s/check CreateTicketRequest {:title "Works?" :description "Probably not :(" :quantity 10}) ;;=> {:id missing-required-key, :status missing-required-key} ;; but why? (class -Ticket) ;;=> clojure.lang.PersistentArrayMap (class Ticket) ;;=> schema.core.Constrained

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

[com.attendify/schema-reﬁned "0.3.0-alpha4"]

Slide 53

Slide 53 text

schema-reﬁned • https://github.com/KitApps/schema-reﬁned • schema on steroids (a lot of them) • refined: constrained on steroids • Struct: product types (maps) on steroids • StructDispatch: conditional on steroids

Slide 54

Slide 54 text

Very Motivational Examples

Slide 55

Slide 55 text

Predicates ;; "manually" with refined and predicates (def LatCoord (r/refined double (r/OpenClosedInterval -90.0 90.0))) ;; the same using built-in types ;; (or functions to create types from other types, a.k.a. generics) (def LngCoord (r/OpenClosedIntervalOf double -180.0 180.0)) ;; Product type using a simple map (def GeoPoint {:lat LatCoord :lng LngCoord}) ;; using built-in types (def Route (r/BoundedListOf GeoPoint 2 50))

Slide 56

Slide 56 text

Now What? (def input [{:lat 48.8529 :lng 2.3499} {:lat 51.5085 :lng -0.0762} {:lat 40.0086 :lng 28.9802}]) ;; Route now is a valid schema, ;; so you can use it as any other schema (schema/check Route input)

Slide 57

Slide 57 text

Predicates... More! (def InZurich {:lat (r/refined double (r/OpenInterval 47.34 47.39)) :lng (r/refined double (r/OpenInterval 8.51 8.57))}) (def InRome {:lat (r/refined double (r/OpenInterval 41.87 41.93)) :lng (r/refined double (r/OpenInterval 12.46 12.51))})

Slide 58

Slide 58 text

Predicates... Compose! ;; you can use schemas as predicates ;; First/Last are good examples of predicate "generics" (def RouteFromZurich (r/refined Route (r/First InZurich))) (def RouteToRome (r/refined Route (r/Last InRome))) ;; And, Or, Not, On (def RouteFromZurichToRome (r/refined Route (r/And (r/First InZurich) (r/Last InRome))))

Slide 59

Slide 59 text

Predicates... Compose! ;; or even more (def FromZurichToRome (r/And (r/First InZurich) (r/Last InRome))) (defn LessNHops [n] (r/BoundedSize 2 (+ 2 n))) (def RouteFromZurichToRomeWithLess3Hops (r/refined Route (r/And FromZurichToRome (LessNHops n))))

Slide 60

Slide 60 text

Readability Matters. A Lot ;; following the rule ;; {v: T | P(v)} (def Coord (refined double (OpenClosedInterval -180.0 180.0))) ;; #Refined{v: double | v ∈ (-180.0, 180.0]} (def QuickRoute (BoundedListOf double 2 4)) ;; #Refined{v: [double] | (count v) ∈ [2, 4]} (refined [double] (Rest (OpenInterval 0 1))) ;; #Refined{v: [double] | ∀v' ∊ (rest v): v' ∈ (0, 1)}

Slide 61

Slide 61 text

Products Sums Guards

Slide 62

Slide 62 text

(def -FreeTicket (Struct :id Id :type (s/eq "free") :title NonEmptyStr :quantity (OpenIntervalOf 1 1e4) :description (s/maybe NonEmptyStr) :status (s/enum :open :closed))) (def FreeTicket (guard -FreeTicket '(:quantity :status) enough-sits-when-open)) ;; #

Slide 63

Slide 63 text

Carry Guards Carefully (def -PaidTicket (assoc FreeTicket :type (s/eq "paid") :priceInCents PositiveInt :taxes [Tax] :fees (s/enum :absorb :pass))) (def PaidTicket (guard -PaidTicket '(:taxes :fees) pass-tax-included)) ;; #

Slide 64

Slide 64 text

Respectful Sum Type (def Ticket (StructDispatch :type "free" FreeTicket "paid" PaidTicket)) ;; # {...} ;; paid => {...}> (def CreateTicketRequest (dissoc Ticket :id :status)) ;; this works!

Slide 65

Slide 65 text

Track Guards Applicability (dissoc PaidTicket :status) ;; # ;; (only one guard left)

Slide 66

Slide 66 text

Prevent Self-Shooting

Slide 67

Slide 67 text

Catch Modeling Issues In Advance (def CreateFreeTicket (dissoc Ticket :type)) ;; CompilerException java.lang.IllegalArgumentException: ;; You are trying to dissoc key ':type' ;; that is used in dispatch function. ;; Even thought it's doable theoretically, ;; we are kindly encourage you ;; to avoid such kind of manipulations. ;; Otherwise it's gonna be a mess. ;; , compiling:(form-init467997445288647843.clj:1:23)

Slide 68

Slide 68 text

Philosophical Extending type (assoc, merge etc) is simpler • by implementation • to catch mentally We still fully support reduction (dissoc) "Request" types are a perfect use case

Slide 69

Slide 69 text

What's Inside StructMap ;; potemkin's helper creates type that acts like a map (def-map-type StructMap [data ;; <- key/value pairs by themself guards ;; <- guards appended mta] ;; <- meta information (meta [_] ...) (with-meta [_ m] ...) (keys [_] ...) (assoc [_ k v] ...) (dissoc [_ k] ...) (get [_ k default-value] ...))

Slide 70

Slide 70 text

What's Inside (def-map-type StructDispatchMap [keys-slice downstream-slice ;; <- keys slices collected from options dispatch-fn options guards updates ;; <- delayed assoc, dissoc operations mta] ...)

Slide 71

Slide 71 text

Protocol To Deal With Guards (defprotocol Guardable (append-guard [this guard]) (get-guards [this]))

Slide 72

Slide 72 text

Put Everything Together (extend-type StructMap ;; same for StructDispatch Guardable (append-guard [^StructMap this guard]) (get-guards [^StructMap this]) s/Schema (spec [this] this) (explain [^StructMap this]) schema-spec/CoreSpec (subschemas [^StructMap this]) (checker [^StructMap this params])) (defmethod print-method StructMap ;; same for StructDispatch [^StructMap struct ^java.io.Writer writer])

Slide 73

Slide 73 text

So What?..

Slide 74

Slide 74 text

What Do We Have Now? • Less tests (way-way-way less) • Less bugs (way-way-way less) • More conﬁdence • Better sleep

Slide 75

Slide 75 text

Next...

Slide 76

Slide 76 text

Error Messages • Clean & friendly errors are hard • You should invest a lot from the very beginning • .. just this to make it happen • Context sensitivity is super useful • Machines and human craves for different messages

Slide 77

Slide 77 text

More Features • Separated "business" and "serialization" logic • Catch more "unreasonable" predicates • Support for "generics" • ... functions are not always the best ﬁt

Slide 78

Slide 78 text

Thanks! q&a please