Thinking in Properties - Speaker Deck

Thinking in Properties

by Susan Potter

Slide 1

Slide 1 text

Thinking in Properties And Beyond (Testing) Susan Potter 2020-08-01

Slide 2

Slide 2 text

Intro

Slide 3

Slide 3 text

Intro finger $(whoami) • Introduced to QuickCheck in Erlang ~2010 • Adopted Haskell’s QuickCheck, Hedgehog, and ScalaCheck at work • ”Testing” in production, thinking in properties, 4 years Susan Potter Thinking in Properties 2020-08-01 1 / 41

Slide 4

Slide 4 text

Intro Agenda • An Origin Story (with code) • Mental Models (above the code) • Beyond Testing (illustrations) Susan Potter Thinking in Properties 2020-08-01 2 / 41

Slide 5

Slide 5 text

An Origin Story

Slide 6

Slide 6 text

An Origin Story Discovering Superpowers Susan Potter Thinking in Properties 2020-08-01 3 / 41

Slide 7

Slide 7 text

An Origin Story Discovering Superpowers Explore domain with types data List a = EmptyList | Cons a (List a) type Pred a = (a -> Bool) type Comp a = (a -> a -> Ordering) sortBy :: Comp a -> List a -> List a filter :: Pred a -> List a -> List a reverse :: List a -> List a last, first :: List a -> Maybe a Susan Potter Thinking in Properties 2020-08-01 3 / 41

Slide 8

Slide 8 text

An Origin Story Discovering Superpowers Explore domain with usage examples -- | Reverse the elements of a list -- >>> reverse (Cons 1 (Cons 2 (Cons 3 EmptyList))) → -- Cons 3 (Cons 2 (Cons 1 EmptyList)) -- -- >>> reverse EmptyList -- EmptyList reverse :: List a -> List a Susan Potter Thinking in Properties 2020-08-01 4 / 41

Slide 9

Slide 9 text

An Origin Story Discovering Superpowers Encode examples as re-runnable tests describe "Lib.reverse" $ do it "returns [5,4,3,2,1] given [1,2,3,4,5]" $ do reverse [1,2,3,4,5] `shouldBe` [5,4,3,2,1] it "returns empty list given empty list" $ do reverse [] `shouldBe` [] Susan Potter Thinking in Properties 2020-08-01 5 / 41

Slide 10

Slide 10 text

An Origin Story Discovering Superpowers Rinse & Repeat Susan Potter Thinking in Properties 2020-08-01 6 / 41

Slide 11

Slide 11 text

An Origin Story Discovering Superpowers Continuous Learning Figure 1: Schedule for Erlang Factory SF 2011 where my mind was blown Susan Potter Thinking in Properties 2020-08-01 7 / 41

Slide 12

Slide 12 text

An Origin Story Harnessing Newly Found Superpowers Characteristics of property-based testing Where we have: • generators that produce random ”arbitrary” values for inputs • general rules that hold without knowing inputs upfront • shrinking of failed values • test runs assert rule multiple times using new generated values Related terms: generative testing, fuzz testing (or ”fuzzing”) Susan Potter Thinking in Properties 2020-08-01 8 / 41

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

An Origin Story Harnessing Newly Found Superpowers One General Rule: Round-tripping (two rides gets us back home) Examples: -- Assumes x is encodable and decodable roundtrip0 = \x -> decode (encode x) == Just x roundtrip1 = \x -> decodeBase64 (encodeBase64 x) == Right x Counter-examples: • sha256 is one way! Susan Potter Thinking in Properties 2020-08-01 9 / 41

Slide 17

Slide 17 text

An Origin Story Harnessing Newly Found Superpowers Introducing generators To make these ideas concrete we will be using hedgehog with these imports: import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range import Control.Monad (replicateM) import Data.Function (($), (.)) Hedgehog integrates shrinking with generation. We will not discuss this difference to QuickCheck but read Well Typed’s blog post about this here. Susan Potter Thinking in Properties 2020-08-01 10 / 41

Slide 18

Slide 18 text

An Origin Story Harnessing Newly Found Superpowers Primitive generators by example >>> replicateM 25 $ Gen.sample Gen.lower "okohcpxrkfunkmwnqujnnhxkg" >>> let currencies = [ "USD", "JPY", "EUR", "CHF", "CNY" ] >>> replicateM 5 $ Gen.sample $ Gen.element currencies ["USD","CNY","USD","JPY","USD"] >>> replicateM 5 $ Gen.sample $ Gen.choice [ Gen.ascii, Gen.unicode ] → ['f', 'c', 'j', '\1068213', '<'] Susan Potter Thinking in Properties 2020-08-01 11 / 41

Slide 19

Slide 19 text

An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 1/2 Suppose our domain looks like this: import Data.Word (Word8, Word16) type W16 = Word16 data IP = IPv4 Word8 Word8 Word8 Word8 | IPv6 W16 W16 W16 W16 W16 W16 W16 W16 Susan Potter Thinking in Properties 2020-08-01 12 / 41

Slide 20

Slide 20 text

An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 2/2 genW8 = Gen.word8 Range.constantBounded genW16 = Gen.word16 Range.constantBounded genIPv4 = IPv4 <$> genW8 <*> genW8 <*> genW8 <*> genW8 genIPv6 = IPv6 <$> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 genAnyIP = Gen.choice [ genIPv4, genIPv6 ] sampleIPs n = replicateM n (Gen.sample genAnyIP) Susan Potter Thinking in Properties 2020-08-01 13 / 41

Slide 21

Slide 21 text

An Origin Story Harnessing Newly Found Superpowers Sampling generated domain data >>> sampleIPs 3 [ "136.59.149.200" , "338d:2397:f612:e036:b27c:2298:4db8:b933" , "5.38.65.204" ] Susan Potter Thinking in Properties 2020-08-01 14 / 41

Slide 22

Slide 22 text

An Origin Story Harnessing Newly Found Superpowers Writing Our First Property! genList :: MonadGen m => m a -> m [a] genList = Gen.list (Range.linear 0 1000) genInt = Gen.int (Range.linear 0 100000) -- "round-tripping" property prop_reverse_reverse = property $ do xs <- forAll $ genList genInt Lib.reverse (Lib.reverse xs) === xs Susan Potter Thinking in Properties 2020-08-01 15 / 41

Slide 23

Slide 23 text

An Origin Story Harnessing Newly Found Superpowers Reviewing Our First Property! Questions about prop_reverse_reverse: • Does it assert anything about reverse ’s speciﬁcation? • Do callers of reverse need to exploit ”round-tripping”? • Does an implementation exist that typechecks yet fails this property? • Are we generating interesting data given the operation’s type? • Are we resigned to function-level property testing? Susan Potter Thinking in Properties 2020-08-01 16 / 41

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

An Origin Story Harnessing Newly Found Superpowers Quick Review: Example-based tests over time t ~ 0 → t → ∞ Quick ⌣ →  / ⌢ Coverage ? →  / ⌢ Repeatable ⌣ →  / ⌢ Documents usage  →  / ⌢ Documents contract ⌢ → ⌢ Effort  →  / ⌢ • can measure coverage • ﬁxtures provide test data • interesting ﬁxtures brittle • over time tends to ⌢ Susan Potter Thinking in Properties 2020-08-01 17 / 41

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

An Origin Story Harnessing Newly Found Superpowers Quick Review: Property-based tests initially t ~ 0 Quick ⌣ Coverage ⌣ Repeatable ⌣ Documents usage  Documents contract ⌣ /  Effort ? • Can we measure coverage ? • We need to maintain generators instead of ﬁxtures! • Not constrained by imagination! ⌣ • Am I smart enough to think up relevant and meaningful properties ? Susan Potter Thinking in Properties 2020-08-01 18 / 41

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Mental Models

Slide 39

Slide 39 text

Mental Models Deriving Properties: Algebraic laws Law What might it be good for Idempotency e.g. event log effects, REST APIs, conﬁg effects Associativity e.g. map/reduce distribution Commutativity e.g. map/reduce local parallelism Distributivity e.g. stable or performant rewrite rules Identity element e.g. for invariance at that value for operation Round-tripping e.g. encoding/decoding Absorption e.g. boolean algebra rewrites Transitivity e.g. dependency closures Susan Potter Thinking in Properties 2020-08-01 19 / 41

Slide 40

Slide 40 text

Mental Models Algebraic laws: Idempotency (running 1+ times yields same result) Examples: idem0 = \x -> abs x == abs (abs x) idem2 = \x -> toUpper s == toUpper (toUpper s) • curl -XPUT https://foo.bar/resource/123 -d baz=qux Counter-example: • curl -XPOST https://foo.bar/resource -d baz=qux Algebra: Given x ∈ A and f ∈ (A → A) and f(x) = f(f(x)) then f is idempotent. Susan Potter Thinking in Properties 2020-08-01 20 / 41

Slide 41

Slide 41 text

Mental Models Algebraic laws: Associativity (brackets to the left or right) Examples: assoc0 = \x y z -> (x + y) + z == x + (y + z) assoc1 = \x y z -> (x ++ y) ++ z == x ++ (y ++ z) Counter-example: • (x − y) − z = x − (y − z) Algebra: Given x, y, z ∈ A, ⊕ ∈ (A → A → A) and (x ⊕ y) ⊕ z = x ⊕ (y ⊕ z) then ⊕ is associative. Susan Potter Thinking in Properties 2020-08-01 21 / 41

Slide 42

Slide 42 text

Mental Models Algebraic laws: Commutativity (any order will do) Examples: comm0 = \x y -> x + y == y + x comm1 = \x y -> x * y == y * x Counter-example: • x + +y = y + +x • x − y = y − x Algebra: Given x, y ∈ A, ⊕ ∈ (A → A → A) and $x ⊕ y = y ⊕x then ⊕ is commutative. Susan Potter Thinking in Properties 2020-08-01 22 / 41

Slide 43

Slide 43 text

Mental Models Algebraic laws: Distributivity (one operation over another) Examples: dist0 = \x y z -> x*(y + z) == x*y + x*z Counter-example: • x + (y ∗ z) = x + y ∗ x + z where x = 1, y = 2, z = 3 1 + (2 ∗ 3) = 1 + 2 ∗ 1 + 3 ⇒ 7 = 6 Algebra: Given x, y, z ∈ A and ⊕, ⊗ ∈ (A → A → A) then ⊗ is distributive over ⊕ when x ⊗ (y ⊕ z) = x ⊗ y ⊕ x ⊗ z Susan Potter Thinking in Properties 2020-08-01 23 / 41

Slide 44

Slide 44 text

Mental Models Algebraic laws: Identity element Examples: identity0 = \x -> 0 + x == x identity1 = \x -> False || x == x identity2 = \s -> "" ++ s == s Counter-example: -NonEmpty does not have an identity element Algebra: ∃e ∈ A, ∀a ∈ A, e ⊕ a = a then e is the identity element in A. Susan Potter Thinking in Properties 2020-08-01 24 / 41

Slide 45

Slide 45 text

Mental Models Algebraic laws: Absorption Examples: absorption0 = \a b -> (a || (a && b)) == a absorption1 = \a b -> (a && (a || b)) == a Algebra: Given ∧, ∨ ∈ (A → A → A) and a, b ∈ A then when a ∧ (a ∨ b) = a = a ∨ (a ∧ b) Susan Potter Thinking in Properties 2020-08-01 25 / 41

Slide 46

Slide 46 text

Mental Models Deriving Properties: Relational laws -- implicitly expect sort and last to be correct prop_max_is_last_of_sort = property $ do xs <- forAll $ genList Gen.ascii Just (max xs) === last (sort xs) prop_last_is_first_of_reversed = property $ do xs <- forAll $ genList Gen.unicode last xs === first (reverse xs) Susan Potter Thinking in Properties 2020-08-01 26 / 41

Slide 47

Slide 47 text

Mental Models Deriving Properties: Abstraction laws Using hedgehog-classes package we can check our typeclass instances according to the abstraction laws: import Hedgehog import Hedgehog.Classes import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range investmentPortfolioSemigroup = lawsCheck (semigroupLaws genInvestmentPortfolio) portfolioFoldable = lawsCheck (foldableLaws genPortfolio) Susan Potter Thinking in Properties 2020-08-01 27 / 41

Slide 48

Slide 48 text

Mental Models Deriving Properties: Reﬂections, Rotations, Distortions prop_rotated_colors_same = property $ do img <- forAll $ genImage colors (rotate90 img) === colors img • normalizing audio shouldn’t change time length • reversing a list shouldn’t change length Susan Potter Thinking in Properties 2020-08-01 28 / 41

Slide 49

Slide 49 text

Mental Models Deriving Properties: Informal model checking Sometimes you can model the basic state machine of a system simply: • model of interesting parts of stateful system • not exhaustive • thinking in state machine models • generate sequence or call graph of commands • assert pre- and post-conditions or invariants • careful you don’t write a second implementation of the SUT just to test it! Susan Potter Thinking in Properties 2020-08-01 29 / 41

Slide 50

Slide 50 text

Mental Models Deriving Properties: Legacy oracles Replacing legacy systems: • bind to old lib as oracle • assert new rewritten library matches oracle for same inputs • good for e.g. calculation engines or data pipelines • might need large build engineering effort Susan Potter Thinking in Properties 2020-08-01 30 / 41

Slide 51

Slide 51 text

Mental Models Deriving Properties: Does not barf Wrapping lower-level code via FFI: • gaps between foreign input or output types and native types • runtime exceptions thrown for some input values (inform design) • sanity checking FFI wrapping Susan Potter Thinking in Properties 2020-08-01 31 / 41

Slide 52

Slide 52 text

Mental Models Deriving Properties: Metamorphic relations • Running against SUT twice with possibly different inputs • A relation exists between those inputs • Assert a relation exists between the outputs of those system runs An example across inputs and outputs, but the relation between inputs and outputs can be different: x, y ∈ Inputs, x ≤ y, x′ = SUT(x), y′ = SUT(y) then x′ ≤ y′ Susan Potter Thinking in Properties 2020-08-01 32 / 41

Slide 53

Slide 53 text

Mental Models Deriving Properties: Metamorphic relation patterns • Input equivalence • Shufﬂing • Conjunctive conditions • Disjunctive conditions • Disjoint partitions • Complete partitions • Partition difference Susan Potter Thinking in Properties 2020-08-01 33 / 41

Slide 54

Slide 54 text

Mental Models Deriving Properties: Heckle Yourself! • mutation testing • alter your code until your tests fail • if no tests fail, throw your tests out (curation) • question your assumptions Susan Potter Thinking in Properties 2020-08-01 34 / 41

Slide 55

Slide 55 text

Beyond testing

Slide 56

Slide 56 text

Beyond testing Properties of Delivery Pipelines Property: Source consistency Ensuring fast-forward only ”merges”: main() { local -r mergeBase="$(git merge-base HEAD origin/deploy)" local -r deployHead="$(git rev-parse origin/deploy)" test "${mergeBase}" = "${deployHead}" } set -e main # should exit with 0 for success Susan Potter Thinking in Properties 2020-08-01 35 / 41

Slide 57

Slide 57 text

Beyond testing Stateful Migrations (in production) Properties: pre and post conditions and invariants between migration phases • moving a stateful cluster from one datacenter to another • upgrading Elastic search into a new cluster • online schema migrations of large tables with binlog syncing and atomic rename (e.g. MySQL) Susan Potter Thinking in Properties 2020-08-01 36 / 41

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Beyond testing System Monitoring Property: Connectedness! Given public name for service name: • resolve name to A records (IPs) • ∀ IPs should negotiate TLS handshake • ∀ IPs should make HTTP request with Host Susan Potter Thinking in Properties 2020-08-01 37 / 41

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Beyond testing Production Data Checks Sometimes your generators don’t generate data you see in production! Legacy systems exist with no property-based testing toolchain! • Structured logging can record inputs and results; validate OOB • Run property checks against production inputs and outputs in Haskell :) Susan Potter Thinking in Properties 2020-08-01 38 / 41

Slide 64

Slide 64 text

Wrapping Up

Slide 65

Slide 65 text

Wrapping Up In Closing • Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Wrapping Up Questions? GitHub @mbbx6spp LinkedIn /in/susanpotter Twitter @SusanPotter Web Personal site Consulting Training Thank you for listening! Susan Potter Thinking in Properties 2020-08-01 40 / 41

Slide 68

Slide 68 text

Wrapping Up Credits • Photo by Elias Castillo on Unsplash • Photo by Juan Rumimpunu on Unsplash • Photo by LinkedIn Sales Navigator on Unsplash • Photo by Leonardo Sanches on Unsplash • Photo by Mélissa Jeanty on Unsplash • Photo by Chris Liverani on Unsplash • Photo by Damir Spanic on Unsplash • Photo by Serrah Galos on Unsplash • Photo by Sergey Zolkin on Unsplash • Photo by Roman Mager on Unsplash • Photo by Miguel Ibáñez on Unsplash • Photo by Science in HD on Unsplash • Photo by Steve Douglas on Unsplash • Photo by Natalie Parham on Unsplash Susan Potter Thinking in Properties 2020-08-01 41 / 41