Put A Type On It Idris Type Providers for AWS Resources Strange Loop St. Louis (Sept 16, 2016) @SusanPotter @referentiallabs 

$ whoami Figure: Building services and infrastructures that can be reasoned about. . . to varying degrees. 

$ whoami really Figure: Past experience familiar background to many developers. So there’s hope for everyone! 

Figure: Questions welcome during session plus Q&A at end. 

What are ‘type errors’ really? Listing 1: EC2ResponseError: 400 Bad Request 1 2 3 4 InvalidParameterValue 5 Invalid availability zone: us -west -1b 6 7 8 XXXXXXX -d7a0 -42c7 -9454 -7??????? 9 10 Availability Zone mappings differ from account to account, which would result in the behaviour you are experiencing. . . . 

Could this be specified as a type? Listing 2: awscli output 1 $ aws --region us -west -1 ec2 describe -availability -zones 2 AVAILABILITYZONES us -west -1 available us -west -1a 3 AVAILABILITYZONES us -west -1 available us -west -1c 4 5 $ aws --region us -west -2 ec2 describe -availability -zones 6 AVAILABILITYZONES us -west -2 available us -west -2a 7 AVAILABILITYZONES us -west -2 available us -west -2b 8 AVAILABILITYZONES us -west -2 available us -west -2c 9 10 $ aws --region us -east -1 ec2 describe -availability -zones 11 AVAILABILITYZONES us -east -1 available us -east -1a 12 AVAILABILITYZONES us -east -1 available us -east -1b 13 AVAILABILITYZONES us -east -1 available us -east -1c 14 AVAILABILITYZONES us -east -1 available us -east -1e 15 

A common ‘solution’ Listing 3: Similar snippets seen in the wild many times 1 # Written on some date a couple of years ago when 2 # you first migrated to AWS in your dev account. 3 availability_zones = { 4 us_west_1: [ "us -west -1a" "us -east -1c" ], 5 us_west_2: [ "us -west -2a" "us -west -2b" "us -west -2c" ], 6 # Missing the new AZ us -east -1e 7 us_east_1: [ "us -east -1a" "us -east -1b" "us -east -1c" ] 8 } 9 # Same idea for limits 10 limits = { ... } 11 

Shortcomings of common ‘solution’? 1 Hard coding or ‘2000 line config.yaml files’ over time yields stale assumptions that only get reviewed at time of error 2 Reactive (not proactive) does not take advantage of accounts new capabilities in a region 3 Not reusable across accounts each AWS account has different available AZs per region with different AZ mappings 

What business problems do I want to solve? 1 Derived rules/policies Rules defined in terms of operating environment state; remove hard coding, improve maintainability and trustworthiness/business value of rules codebase. 2 Validate existing deployments validate infrastructure deployed meets cost, availability, capability, security policies 3 Validate deployments against policies pre-deployment prevent deploying cloud resources that violate security, cost, availability, capabiity policies and don’t pay AWS a dime for "bad" infrastructure 4 Manage infrastructure requirements Managing infrastructure/deployment requirements by encoding them as propositions translated to types and proved with definitions that typecheck 

Why reason about infrastructure now? 1 Economic large distributed deployments required almost everywhere 2 Human high churn/turnover, low quality of ops life; kills innovation 3 Technological FP and expressive type systems no longer just for academics and programmable infrastructure is everywhere now 

What is a type provider? Definition A type provider is a mechanism that constructs types from a structured source outside the programming language. Common traits of type providers include ingesting a schema-based dataset to construct types from metadata that is stable for the duration of the running program. 

How do type providers work? Definition Languages like F#, Scala, Haskell that offer some facilities for constructing types from external input utilize source or byte code generation behind the scenes to build types. 

How do type providers work? Listing 4: provider.fs 1 #r "../../../ bin/FSharp.Data.dll" 2 open FSharp.Data 3 4 // From prototype data generates type like: 5 // type SK = { Name: string; BirthYear: int; } 6 type SK = JsonProvider <"""{" name ": "E. Báthory", "birth_year ": 1560} """> 7 8 // Creates value of type SK: { Name = "A. Wournos "; BirthYear = 1956; } 9 let aw = SK.Parse("""{" name ": "Aileen Wournos", "birth_year ": 1956} """) 10 // Creates value of type SK: { Name = "Nannie Doss "; BirthYear = 1905; } 11 let nd = SK.Parse("""{" name ": "Nannie Doss", "birth_year ": 1905} """> 12 // Valid: aw.BirthYear => 1956 13 // Valid: nd.Name => "Nannie Doss" 14 

How are Idris type providers different? Definition Using Idris’ full dependent types we can compute types and know that the computed types compile. This is not true for source/byte code generated type providers which may generate code that does not compile. 1 data CSVType : Type where 2 MkCSVType : (delim : Char) -- delim label not strictly necessary here 3 → (n : Nat) 4 → (hdr : Vect n String) 5 → CSVType 6 Listing 5: typecomputation.idr 

Figure: Still awake? 

Idris Type Provider Mechanics 1 Idris> :doc Provider 2 Data type Prelude.Providers.Provider : (a : Type) → Type 3 Type providers must build one of these in an IO computation. 4 5 Constructors: 6 Provide : (x : a) → Provider a 7 Return a term to be spliced in 8 Args: x : a -- the term to be spliced (i.e. the proof) 9 Error : (msg : String) → Provider a 10 Report an error to the user and stop compilation 11 Args: msg : String -- the error message 12 Listing 6: 0. Review Provider a type definition - Basically a Either a String 

Idris Type Provider Mechanics 1 ||| Module you implement your action in to pull into other code 2 module RefLabs.Provider.CSV 3 Listing 7: RefLabs/Provider/CSV.idr - 1. Create your provider module 

Idris Type Provider Mechanics 4 data CSVType : Type where 5 MkCSVType : (delim : Char) 6 → (n : Nat) 7 → (hdr : Vect n String) 8 → CSVType 9 10 inferCSVType : (delim : Char) → (hdr : String) → CSVType 11 inferCSVType delim hdr = 12 let cs = columns delim hdr 13 in MkCSVType delim (length cs) (fromList cs) 14 15 Listing 8: RefLabs/Provider/CSV.idr - 2. Define helper functions and types 

Idris Type Provider Mechanics 15 csvType : Char → String → IO (Provider CSVType) 16 csvType delim fn = 17 do lines <- readLines fn 18 return $ case lines of 19 [] ⇒ Error $ "Could not read " ++ fn 20 (hdr :: _) ⇒ Provide $ inferCSVType delim hdr 21 Listing 9: RefLabs/Provider/CSV.idr - 3. Implement function that return an IO action returning a Provider a 

Idris Type Provider Mechanics 1 module Main 2 3 import RefLabs.Provider.CSV 4 import Data.NamedVect 5 6 getBirthYear : Row t → String 7 getBirthYear r = NamedVect.lookup "BirthYear" r 8 9 getName : Row t → String 10 getName r = NamedVect.lookup "Name" r 11 Listing 10: Main.idr - 4. In client code, import module with provider IO action, and define helper functions 

Idris Type Provider Mechanics 1 %language TypeProvider 2 Listing 11: Main.idr - 4. Enable the type provider language extension,firstnumber=5 

Idris Type Provider Mechanics 1 %provide (t : CSVType) with csvType ’,’ "female_serial_killers.csv" 2 Listing 12: Main.idr - 5. Declare provider action 

Idris Type Provider Mechanics 1 partial 2 main : IO () 3 main = 4 do let fn = "female_serial_killers.csv" 5 f <- readCSVFile t fn 6 case f of 7 Nothing ⇒ putStrLn "Could not open CSV file " ++ fn 8 Just rows ⇒ 9 do let birthYears = map getBirthYear rows 10 let names = map getName rows 11 putStrLn $ show birthYears 12 putStrLn $ show names 13 Listing 13: Main.idr - 6. Use the provided action 

Figure: Questions? 

Curry-Howard Correspondence Informally: Propositional logic concepts correspond to programming language features* Propositions as types (Proofs as definitions) * too many implications/assumptions to put here but search for "Curry-Howard constructive logic" or "intuitionistic logic" for further reading 

Theorem For all n in the naturals, n is either even or odd. where even is defined as for some p in the naturals n = 2p and odd is defined as for some p in the naturals n = 2p + 1. (∀n ∈ N, ∃p ∈ N : n = 2p or n = 2p + 1) 1 natsAreEvenOrOdd : (n : Nat) 2 → (p : Nat ** Either (Even n p) (Odd n p)) 3 Listing 14: Proposition in Idris types 

Proof. By induction: If n = 0 then n = 2p where p = 0 ∈ N Assume n = 2p + 1 ∈ N then =⇒ n + 1 = 2p + 1 + 1 =⇒ n + 1 = 2p + 2 =⇒ n + 1 = 2(p + 1) =⇒ n + 1 = 2q where q = (p + 1) ∈ N Else we assume n = 2p ∈ N then =⇒ n + 1 = 2p + 1 =⇒ n + 1 = 2p + 1 where p ∈ N 

Curry-Howard: Truth as a Type Definition There is only one Truth value, so we use the singleton type. This means only one value can be constructed for that type. Listing 15: truth.scala 1 type Truth = Unit 1 Truth : Type 2 Truth = () Listing 16: truth.idr 

Curry-Howard: Falsity as a Type Definition Falsity is uninhabitable (not values of this type can be constructed). We call this the bottom type. Listing 17: falsity.scala 1 // Technically Null has one inhabitant in Scala , which 2 // is the value ‘null ‘. All types in Scala have ‘null ‘ 3 // as an inhabitant (fancy word for value ). 4 // Nothing is also a somewhat suitable type in Scala to 5 // specifically denote non -termination sense of "bottom ". 6 // Thank you @jpfuentes2 for suggestion of Nothing. 7 type Falsity = Null 1 Falsity : Type 2 Falsity = Void Listing 18: falsity.idr 

Curry-Howard: Implication as a Type Definition When a implies b then we can say, given a value of type a we can return a value of type b. Listing 19: implication.scala 1 type Implication[A, B] = (A => B) 1 Implication : (a: Type) → (b : Type) → Type 2 Implication a b = (a → b) 3 4 implicationExample : Bool → Nat 5 implicationExample False = Z 6 implicationExample True = S Z Listing 20: implication.idr 

Curry-Howard: Negation as a Type Definition Negation is just a fancy word for logical NOT. Since Falsity can never be inhabited, then saying NOT a means given a value of type a then we must return the bottom type (NOT a means a is Falsity). Note: we cannot do Not (Not P) => P. Listing 21: negation.scala 1 type Negation[A] = A => Null 1 Negation : Type → Type 2 Negation = Not Listing 22: negation.idr 

Curry-Howard: Conjunction as a Type Definition Conjunction is a fancy word for logical AND. Given a value of type a and a value of type b both are returned. We use the Pair type in the Idris Prelude to represent this case. Special purpose product types can also represent this bottom type. Listing 23: conjuction.scala 1 type Conjunction[A,B] = (A, B) 1 Conjunction : Type → Type → Type 2 Conjunction = Pair Listing 24: conjunction.idr 

Curry-Howard: Disjunction as a Type Definition Disjunction is a fancy word for logical OR. Given a value of type a and a value of type b we return either the a or the b (exclusive OR). Listing 25: disjunction.scala 1 type Disjunction[A, B] = Either[A, B] 1 Disjunction : Type → Type → Type 2 Disjunction = Either Listing 26: disjunction.idr 

Curry-Howard: Consistency 1/2 Definition In logic consistency ensures that all logical propositions make sense when all put together. In programming we can ensure this by checking types/functions are total. A type being total is just a fancy way of saying that for all inhabitants (values) of any inputs to the function/type, we will always be able to terminate with an inhabitant of the return value in some finite time. 

Curry-Howard: Consistency 2/2 1 -- Sets the default for the module/file code 2 %default total 3 4 -- not strictly necessary since above we specified 5 -- the default as total 6 total 7 parity : (n : Nat) → Parity n 8 9 -- When you can’t validate totality, eg. with IO/effectful 10 -- code that interacts with the outside world rather than axioms 11 -- or definitions then we might need to specify it is partial 12 -- (assuming you specified the default to total like above). 13 partial 14 main : IO () 15 main = someActionHere 

Curry-Howard: Induction Definition Induction is a method of proof in mathematics/logic. We saw this in the all naturals are either even or odd example before. In programming we have recursion which is how we prove by induction in Idris. 1 -- FYI: 2 -- cong : (a = b) → f a = f b 3 4 bla : (n : Nat) → (m : Nat) → S (plus n m) = plus n (S m) 5 bla Z m = Refl 6 bla (S k) m = cong (bla k m) Listing 28: induction.idr 

Curry-Howard: Universal Quantifier Definition Quantifier is just a fancy term for "describe how many of something". The Universal Quantifier is a fancy term for "for all x in X". 1 -- Proposition that says for all n in Naturals and all m in Naturals 2 -- we can construct a Vect of length (n+m) with Nat elements. 3 -- Anyone know anything parametricity? What is the likely form of 4 -- the proof of such a proposition that we can construct? :) 5 someType : (n : Nat) → (m : Nat) → Vect (n+m) Nat Listing 29: universal.idr 

Curry-Howard: Existential Quantifier Definition The existential quantifier is a fancy term for "there exists x in X such that P". : asz) -> 1 someType : (n : Nat) 2 → (p : Nat ** Either (Even n p) (Odd n p)) 3 4 -- (x : t ** a → Type) desugars to the following type: 5 -- data DPair : (a : Type) (P : a → Type) → Type where 6 -- MkDPair : (x : a) → (prf : P x) → DPair a P 7 8 -- So we could write: 9 -- someType : (n : Nat) 10 -- → DPair (p : Nat) (Either (Even n p) (Odd n p)) Listing 30: existential.idr 

1 -- Axioms are definitions we depend on in proposition itself 2 Even : (n : Nat) → (p : Nat) → Type 3 Odd : (n : Nat) → (p : Nat) → Type 4 zeroIsEven : Even Z Z 5 succEven : Even n p → Odd (S n) p 6 succOdd : Odd n p → Even (S n) p 7 8 -- Proposition restated as a type 9 natsAreEvenOrOdd : (n : Nat) 10 → (p : Nat ** Either (Even n p) (Odd n p)) 11 -- Proof of the proposition when we fill the hole and ensure it’s total 12 natsAreEvenOrOdd Z = (0 ** Left (Even 0 0)) -- base case 13 natsAreEvenOrOdd (S n) = ?hole -- induction via recursion 14 Listing 31: Equivalent proposition and proof in Idris 

Figure: Too much math? 

Infrastructure Requirements: Availability Definition Given an AWS account and a region, we want to ensure our deployment of EC2 resources defines a cluster of a multiple off the number of availability zones for even distribution. 1 evenDistributionAcrossAZs : (account : AWSAccount) 2 → (region : Region) 3 → (x : Nat) 4 → Vect x AZ 5 → (n : Nat ** Cluster (mult x n) Service) Listing 32: striped.idr 

Summary 1 Infer rules from evolvable metadata 2 Proactively take advantage of new account capabilities 3 Usable across account 4 High-level language with superior assurances 

Questions Figure: Heckle me @SusanPotter later too. 

Questions Figure: https://github.com/referentiallabs/idris-type-providers