Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Put A Type On It: Idris Type Provider for AWS Resources

Susan Potter
September 16, 2016

Put A Type On It: Idris Type Provider for AWS Resources

Presented at StrangeLoop on September 16, 2016.

Susan Potter

September 16, 2016
Tweet

More Decks by Susan Potter

Other Decks in Programming

Transcript

  1. Put A Type On It Idris Type Providers for AWS

    Resources Strange Loop St. Louis (Sept 16, 2016) @SusanPotter @referentiallabs
  2. $ whoami really Figure: Past experience familiar background to many

    developers. So there’s hope for everyone!
  3. What are ‘type errors’ really? Listing 1: EC2ResponseError: 400 Bad

    Request 1 <Response > 2 <Errors > 3 <Error > 4 <Code >InvalidParameterValue </Code > 5 <Message >Invalid availability zone: us -west -1b</Message > 6 </Error > 7 </Errors > 8 <RequestID >XXXXXXX -d7a0 -42c7 -9454 -7??????? </RequestID > 9 </Response > 10 Availability Zone mappings differ from account to account, which would result in the behaviour you are experiencing. . . .
  4. Could this be specified as a type? Listing 2: awscli

    output 1 $ aws --region us -west -1 ec2 describe -availability -zones 2 AVAILABILITYZONES us -west -1 available us -west -1a 3 AVAILABILITYZONES us -west -1 available us -west -1c 4 5 $ aws --region us -west -2 ec2 describe -availability -zones 6 AVAILABILITYZONES us -west -2 available us -west -2a 7 AVAILABILITYZONES us -west -2 available us -west -2b 8 AVAILABILITYZONES us -west -2 available us -west -2c 9 10 $ aws --region us -east -1 ec2 describe -availability -zones 11 AVAILABILITYZONES us -east -1 available us -east -1a 12 AVAILABILITYZONES us -east -1 available us -east -1b 13 AVAILABILITYZONES us -east -1 available us -east -1c 14 AVAILABILITYZONES us -east -1 available us -east -1e 15
  5. A common ‘solution’ Listing 3: Similar snippets seen in the

    wild many times 1 # Written on some date a couple of years ago when 2 # you first migrated to AWS in your dev account. 3 availability_zones = { 4 us_west_1: [ "us -west -1a" "us -east -1c" ], 5 us_west_2: [ "us -west -2a" "us -west -2b" "us -west -2c" ], 6 # Missing the new AZ us -east -1e 7 us_east_1: [ "us -east -1a" "us -east -1b" "us -east -1c" ] 8 } 9 # Same idea for limits 10 limits = { ... } 11
  6. Shortcomings of common ‘solution’? 1 Hard coding or ‘2000 line

    config.yaml files’ over time yields stale assumptions that only get reviewed at time of error 2 Reactive (not proactive) does not take advantage of accounts new capabilities in a region 3 Not reusable across accounts each AWS account has different available AZs per region with different AZ mappings
  7. What business problems do I want to solve? 1 Derived

    rules/policies Rules defined in terms of operating environment state; remove hard coding, improve maintainability and trustworthiness/business value of rules codebase. 2 Validate existing deployments validate infrastructure deployed meets cost, availability, capability, security policies 3 Validate deployments against policies pre-deployment prevent deploying cloud resources that violate security, cost, availability, capabiity policies and don’t pay AWS a dime for "bad" infrastructure 4 Manage infrastructure requirements Managing infrastructure/deployment requirements by encoding them as propositions translated to types and proved with definitions that typecheck
  8. Why reason about infrastructure now? 1 Economic large distributed deployments

    required almost everywhere 2 Human high churn/turnover, low quality of ops life; kills innovation 3 Technological FP and expressive type systems no longer just for academics and programmable infrastructure is everywhere now
  9. What is a type provider? Definition A type provider is

    a mechanism that constructs types from a structured source outside the programming language. Common traits of type providers include ingesting a schema-based dataset to construct types from metadata that is stable for the duration of the running program.
  10. How do type providers work? Definition Languages like F#, Scala,

    Haskell that offer some facilities for constructing types from external input utilize source or byte code generation behind the scenes to build types.
  11. How do type providers work? Listing 4: provider.fs 1 #r

    "../../../ bin/FSharp.Data.dll" 2 open FSharp.Data 3 4 // From prototype data generates type like: 5 // type SK = { Name: string; BirthYear: int; } 6 type SK = JsonProvider <"""{" name ": "E. Báthory", "birth_year ": 1560} """> 7 8 // Creates value of type SK: { Name = "A. Wournos "; BirthYear = 1956; } 9 let aw = SK.Parse("""{" name ": "Aileen Wournos", "birth_year ": 1956} """) 10 // Creates value of type SK: { Name = "Nannie Doss "; BirthYear = 1905; } 11 let nd = SK.Parse("""{" name ": "Nannie Doss", "birth_year ": 1905} """> 12 // Valid: aw.BirthYear => 1956 13 // Valid: nd.Name => "Nannie Doss" 14
  12. How are Idris type providers different? Definition Using Idris’ full

    dependent types we can compute types and know that the computed types compile. This is not true for source/byte code generated type providers which may generate code that does not compile. 1 data CSVType : Type where 2 MkCSVType : (delim : Char) -- delim label not strictly necessary here 3 → (n : Nat) 4 → (hdr : Vect n String) 5 → CSVType 6 Listing 5: typecomputation.idr
  13. Idris Type Provider Mechanics 1 Idris> :doc Provider 2 Data

    type Prelude.Providers.Provider : (a : Type) → Type 3 Type providers must build one of these in an IO computation. 4 5 Constructors: 6 Provide : (x : a) → Provider a 7 Return a term to be spliced in 8 Args: x : a -- the term to be spliced (i.e. the proof) 9 Error : (msg : String) → Provider a 10 Report an error to the user and stop compilation 11 Args: msg : String -- the error message 12 Listing 6: 0. Review Provider a type definition - Basically a Either a String
  14. Idris Type Provider Mechanics 1 ||| Module you implement your

    action in to pull into other code 2 module RefLabs.Provider.CSV 3 Listing 7: RefLabs/Provider/CSV.idr - 1. Create your provider module
  15. Idris Type Provider Mechanics 4 data CSVType : Type where

    5 MkCSVType : (delim : Char) 6 → (n : Nat) 7 → (hdr : Vect n String) 8 → CSVType 9 10 inferCSVType : (delim : Char) → (hdr : String) → CSVType 11 inferCSVType delim hdr = 12 let cs = columns delim hdr 13 in MkCSVType delim (length cs) (fromList cs) 14 15 Listing 8: RefLabs/Provider/CSV.idr - 2. Define helper functions and types
  16. Idris Type Provider Mechanics 15 csvType : Char → String

    → IO (Provider CSVType) 16 csvType delim fn = 17 do lines <- readLines fn 18 return $ case lines of 19 [] ⇒ Error $ "Could not read " ++ fn 20 (hdr :: _) ⇒ Provide $ inferCSVType delim hdr 21 Listing 9: RefLabs/Provider/CSV.idr - 3. Implement function that return an IO action returning a Provider a
  17. Idris Type Provider Mechanics 1 module Main 2 3 import

    RefLabs.Provider.CSV 4 import Data.NamedVect 5 6 getBirthYear : Row t → String 7 getBirthYear r = NamedVect.lookup "BirthYear" r 8 9 getName : Row t → String 10 getName r = NamedVect.lookup "Name" r 11 Listing 10: Main.idr - 4. In client code, import module with provider IO action, and define helper functions
  18. Idris Type Provider Mechanics 1 %language TypeProvider 2 Listing 11:

    Main.idr - 4. Enable the type provider language extension,firstnumber=5
  19. Idris Type Provider Mechanics 1 %provide (t : CSVType) with

    csvType ’,’ "female_serial_killers.csv" 2 Listing 12: Main.idr - 5. Declare provider action
  20. Idris Type Provider Mechanics 1 partial 2 main : IO

    () 3 main = 4 do let fn = "female_serial_killers.csv" 5 f <- readCSVFile t fn 6 case f of 7 Nothing ⇒ putStrLn "Could not open CSV file " ++ fn 8 Just rows ⇒ 9 do let birthYears = map getBirthYear rows 10 let names = map getName rows 11 putStrLn $ show birthYears 12 putStrLn $ show names 13 Listing 13: Main.idr - 6. Use the provided action
  21. Curry-Howard Correspondence Informally: Propositional logic concepts correspond to programming language

    features* Propositions as types (Proofs as definitions) * too many implications/assumptions to put here but search for "Curry-Howard constructive logic" or "intuitionistic logic" for further reading
  22. Theorem For all n in the naturals, n is either

    even or odd. where even is defined as for some p in the naturals n = 2p and odd is defined as for some p in the naturals n = 2p + 1. (∀n ∈ N, ∃p ∈ N : n = 2p or n = 2p + 1) 1 natsAreEvenOrOdd : (n : Nat) 2 → (p : Nat ** Either (Even n p) (Odd n p)) 3 Listing 14: Proposition in Idris types
  23. Proof. By induction: If n = 0 then n =

    2p where p = 0 ∈ N Assume n = 2p + 1 ∈ N then =⇒ n + 1 = 2p + 1 + 1 =⇒ n + 1 = 2p + 2 =⇒ n + 1 = 2(p + 1) =⇒ n + 1 = 2q where q = (p + 1) ∈ N Else we assume n = 2p ∈ N then =⇒ n + 1 = 2p + 1 =⇒ n + 1 = 2p + 1 where p ∈ N
  24. Curry-Howard: Truth as a Type Definition There is only one

    Truth value, so we use the singleton type. This means only one value can be constructed for that type. Listing 15: truth.scala 1 type Truth = Unit 1 Truth : Type 2 Truth = () Listing 16: truth.idr
  25. Curry-Howard: Falsity as a Type Definition Falsity is uninhabitable (not

    values of this type can be constructed). We call this the bottom type. Listing 17: falsity.scala 1 // Technically Null has one inhabitant in Scala , which 2 // is the value ‘null ‘. All types in Scala have ‘null ‘ 3 // as an inhabitant (fancy word for value ). 4 // Nothing is also a somewhat suitable type in Scala to 5 // specifically denote non -termination sense of "bottom ". 6 // Thank you @jpfuentes2 for suggestion of Nothing. 7 type Falsity = Null 1 Falsity : Type 2 Falsity = Void Listing 18: falsity.idr
  26. Curry-Howard: Implication as a Type Definition When a implies b

    then we can say, given a value of type a we can return a value of type b. Listing 19: implication.scala 1 type Implication[A, B] = (A => B) 1 Implication : (a: Type) → (b : Type) → Type 2 Implication a b = (a → b) 3 4 implicationExample : Bool → Nat 5 implicationExample False = Z 6 implicationExample True = S Z Listing 20: implication.idr
  27. Curry-Howard: Negation as a Type Definition Negation is just a

    fancy word for logical NOT. Since Falsity can never be inhabited, then saying NOT a means given a value of type a then we must return the bottom type (NOT a means a is Falsity). Note: we cannot do Not (Not P) => P. Listing 21: negation.scala 1 type Negation[A] = A => Null 1 Negation : Type → Type 2 Negation = Not Listing 22: negation.idr
  28. Curry-Howard: Conjunction as a Type Definition Conjunction is a fancy

    word for logical AND. Given a value of type a and a value of type b both are returned. We use the Pair type in the Idris Prelude to represent this case. Special purpose product types can also represent this bottom type. Listing 23: conjuction.scala 1 type Conjunction[A,B] = (A, B) 1 Conjunction : Type → Type → Type 2 Conjunction = Pair Listing 24: conjunction.idr
  29. Curry-Howard: Disjunction as a Type Definition Disjunction is a fancy

    word for logical OR. Given a value of type a and a value of type b we return either the a or the b (exclusive OR). Listing 25: disjunction.scala 1 type Disjunction[A, B] = Either[A, B] 1 Disjunction : Type → Type → Type 2 Disjunction = Either Listing 26: disjunction.idr
  30. Curry-Howard: Consistency 1/2 Definition In logic consistency ensures that all

    logical propositions make sense when all put together. In programming we can ensure this by checking types/functions are total. A type being total is just a fancy way of saying that for all inhabitants (values) of any inputs to the function/type, we will always be able to terminate with an inhabitant of the return value in some finite time.
  31. Curry-Howard: Consistency 2/2 1 -- Sets the default for the

    module/file code 2 %default total 3 4 -- not strictly necessary since above we specified 5 -- the default as total 6 total 7 parity : (n : Nat) → Parity n 8 9 -- When you can’t validate totality, eg. with IO/effectful 10 -- code that interacts with the outside world rather than axioms 11 -- or definitions then we might need to specify it is partial 12 -- (assuming you specified the default to total like above). 13 partial 14 main : IO () 15 main = someActionHere
  32. Curry-Howard: Induction Definition Induction is a method of proof in

    mathematics/logic. We saw this in the all naturals are either even or odd example before. In programming we have recursion which is how we prove by induction in Idris. 1 -- FYI: 2 -- cong : (a = b) → f a = f b 3 4 bla : (n : Nat) → (m : Nat) → S (plus n m) = plus n (S m) 5 bla Z m = Refl 6 bla (S k) m = cong (bla k m) Listing 28: induction.idr
  33. Curry-Howard: Universal Quantifier Definition Quantifier is just a fancy term

    for "describe how many of something". The Universal Quantifier is a fancy term for "for all x in X". 1 -- Proposition that says for all n in Naturals and all m in Naturals 2 -- we can construct a Vect of length (n+m) with Nat elements. 3 -- Anyone know anything parametricity? What is the likely form of 4 -- the proof of such a proposition that we can construct? :) 5 someType : (n : Nat) → (m : Nat) → Vect (n+m) Nat Listing 29: universal.idr
  34. Curry-Howard: Existential Quantifier Definition The existential quantifier is a fancy

    term for "there exists x in X such that P". : asz) -> 1 someType : (n : Nat) 2 → (p : Nat ** Either (Even n p) (Odd n p)) 3 4 -- (x : t ** a → Type) desugars to the following type: 5 -- data DPair : (a : Type) (P : a → Type) → Type where 6 -- MkDPair : (x : a) → (prf : P x) → DPair a P 7 8 -- So we could write: 9 -- someType : (n : Nat) 10 -- → DPair (p : Nat) (Either (Even n p) (Odd n p)) Listing 30: existential.idr
  35. 1 -- Axioms are definitions we depend on in proposition

    itself 2 Even : (n : Nat) → (p : Nat) → Type 3 Odd : (n : Nat) → (p : Nat) → Type 4 zeroIsEven : Even Z Z 5 succEven : Even n p → Odd (S n) p 6 succOdd : Odd n p → Even (S n) p 7 8 -- Proposition restated as a type 9 natsAreEvenOrOdd : (n : Nat) 10 → (p : Nat ** Either (Even n p) (Odd n p)) 11 -- Proof of the proposition when we fill the hole and ensure it’s total 12 natsAreEvenOrOdd Z = (0 ** Left (Even 0 0)) -- base case 13 natsAreEvenOrOdd (S n) = ?hole -- induction via recursion 14 Listing 31: Equivalent proposition and proof in Idris
  36. Infrastructure Requirements: Availability Definition Given an AWS account and a

    region, we want to ensure our deployment of EC2 resources defines a cluster of a multiple off the number of availability zones for even distribution. 1 evenDistributionAcrossAZs : (account : AWSAccount) 2 → (region : Region) 3 → (x : Nat) 4 → Vect x AZ 5 → (n : Nat ** Cluster (mult x n) Service) Listing 32: striped.idr
  37. Summary 1 Infer rules from evolvable metadata 2 Proactively take

    advantage of new account capabilities 3 Usable across account 4 High-level language with superior assurances