Tweet
Tweet

Slide 1

Slide 1 text

A presentation by @stuherbert
 for @GanbaroDigital JSON Schema For Validating API Requests

Slide 2

Slide 2 text

Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since 1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart

Slide 3

Slide 3 text

Follow me I do tweet a lot about non-tech stuff though :) @stuherbert

Slide 4

Slide 4 text

@GanbaroDigital ?? ?? Have you used JSON Schema?

Slide 5

Slide 5 text

@GanbaroDigital Why JSON Schema?

Slide 6

Slide 6 text

@GanbaroDigital I've seen a lot of hand-written API validation code.

Slide 7

Slide 7 text

@GanbaroDigital Hand-written input validation code has issues.

Slide 8

Slide 8 text

@GanbaroDigital Handwritten Validation
 Suffers From ... • Duplication • Coverage Gaps

Slide 9

Slide 9 text

@GanbaroDigital ?? ?? Can JSON Schema offer a different way? That's what I've been looking into.

Slide 10

Slide 10 text

@GanbaroDigital This is just my experience over the last 18 months.

Slide 11

Slide 11 text

@GanbaroDigital I'm here to learn from your experience too.

Slide 12

Slide 12 text

@GanbaroDigital API Request Anatomy

Slide 13

Slide 13 text

@GanbaroDigital HTTP API Request

Slide 14

Slide 14 text

@GanbaroDigital HTTP API Request HTTP Verb (Method)

Slide 15

Slide 15 text

@GanbaroDigital e.g. GET, POST, PUT, or HEAD

Slide 16

Slide 16 text

@GanbaroDigital HTTP API Request Query Path (Route) HTTP Verb (Method)

Slide 17

Slide 17 text

@GanbaroDigital e.g. /team/chelsea/+all_players

Slide 18

Slide 18 text

@GanbaroDigital HTTP API Request Query Path (Route) Query Params (Filters) HTTP Verb (Method)

Slide 19

Slide 19 text

@GanbaroDigital e.g. /team/chelsea/+all_players ?season_start=2017

Slide 20

Slide 20 text

@GanbaroDigital e.g. /team/chelsea/+all_players ?season_start=2017

Slide 21

Slide 21 text

@GanbaroDigital HTTP API Request HTTP Headers (Metadata) Query Path (Route) Query Params (Filters) HTTP Verb (Method)

Slide 22

Slide 22 text

@GanbaroDigital e.g. Content-Type: application/json

Slide 23

Slide 23 text

@GanbaroDigital HTTP API Request HTTP Headers (Metadata) Request Body (Payload) Query Path (Route) Query Params (Filters) HTTP Verb (Method)

Slide 24

Slide 24 text

@GanbaroDigital HTTP API Request HTTP Headers (Metadata) Request Body (Payload) Query Path (Route) Query Params (Filters) HTTP Verb (Method)

Slide 25

Slide 25 text

@GanbaroDigital Not every API request can include a Request Body.

Slide 26

Slide 26 text

@GanbaroDigital HTTP Requests w/out Bodies • GET • HEAD

Slide 27

Slide 27 text

@GanbaroDigital HTTP Requests w/ Bodies • POST • PUT • PATCH

Slide 28

Slide 28 text

@GanbaroDigital Request bodies are typically* used to change the data stored by your system.

Slide 29

Slide 29 text

@GanbaroDigital “ Request bodies are user input. They must be validated before they can be safely used.

Slide 30

Slide 30 text

@GanbaroDigital API Request API Response

Slide 31

Slide 31 text

@GanbaroDigital Action API Request API Response

Slide 32

Slide 32 text

@GanbaroDigital Action Filter Input Escape Output API Request API Response

Slide 33

Slide 33 text

@GanbaroDigital Action API Request API Response Filter Input Escape Output View / Template Controller Model / Controller

Slide 34

Slide 34 text

@GanbaroDigital Where does JSON schema help with this?

Slide 35

Slide 35 text

@GanbaroDigital Action Filter Input Escape Output API Request API Response

Slide 36

Slide 36 text

@GanbaroDigital Action Validate Format Filter Input Escape Output API Request API Response

Slide 37

Slide 37 text

@GanbaroDigital Action Validate Format Filter Input Validate Format Escape Output API Request API Response

Slide 38

Slide 38 text

@GanbaroDigital Action Validate Format Filter Input Validate Format Escape Output JSON Schema JSON Schema View / Template JSON Schema API Request API Response Model / Controller

Slide 39

Slide 39 text

@GanbaroDigital “ If your API accepts JSON payloads JSON schema can be part of your input validation process.

Slide 40

Slide 40 text

@GanbaroDigital “ If your API returns JSON payloads JSON schema can be part of your quality assurance process.

Slide 41

Slide 41 text

@GanbaroDigital “ If your API client 
 sends or receives JSON payloads JSON schema can be part of your robustness process.

Slide 42

Slide 42 text

@GanbaroDigital Introducing JSON Schema

Slide 43

Slide 43 text

@GanbaroDigital “ JSON Schema are JSON documents that describe other JSON documents

Slide 44

Slide 44 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#" }

Slide 45

Slide 45 text

@GanbaroDigital $schema tells your validator which version of JSON Schema to use

Slide 46

Slide 46 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema" }

Slide 47

Slide 47 text

@GanbaroDigital $id is used for re-using one JSON schema inside another

Slide 48

Slide 48 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema" }

Slide 49

Slide 49 text

@GanbaroDigital Validators do not expect the JSON Schema to exist at the given URI. Neither should you.

Slide 50

Slide 50 text

@GanbaroDigital Simple Schema

Slide 51

Slide 51 text

@GanbaroDigital A simple JSON Schema describes one data structure.

Slide 52

Slide 52 text

@GanbaroDigital Every data structure has a type.

Slide 53

Slide 53 text

@GanbaroDigital • array • boolean • integer* • null • number • object • string

Slide 54

Slide 54 text

@GanbaroDigital * 'integer' is a special form of 'number' because Javascript doesn't support separate integer and float types

Slide 55

Slide 55 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "string" }

Slide 56

Slide 56 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "object" }

Slide 57

Slide 57 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "array" }

Slide 58

Slide 58 text

@GanbaroDigital Every type has its own set of constraints.

Slide 59

Slide 59 text

@GanbaroDigital http://json-schema.org/latest/json-schema- validation.html#rfc.section.6

Slide 60

Slide 60 text

@GanbaroDigital “ In the spec, they're called "validation keywords". It's easier to design a JSON schema if you think of them as 'constraints'.

Slide 61

Slide 61 text

@GanbaroDigital “ JSON schema constraints are used to reject input, not accept input.

Slide 62

Slide 62 text

@GanbaroDigital Incoming Data Is Only Rejected If ... 1. You define a constraint 2. ... that the incoming data falls foul of

Slide 63

Slide 63 text

@GanbaroDigital Worked Example

Slide 64

Slide 64 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 65

Slide 65 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#" }

Slide 66

Slide 66 text

@GanbaroDigital If possible, always use the latest draft of JSON Schema.

Slide 67

Slide 67 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

Slide 68

Slide 68 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

Slide 69

Slide 69 text

@GanbaroDigital Use domains that you actually own. Don't be an ass and use someone else's.

Slide 70

Slide 70 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

Slide 71

Slide 71 text

@GanbaroDigital The query path is a namespace. Use it to leave space for the undiscovered country.

Slide 72

Slide 72 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

Slide 73

Slide 73 text

@GanbaroDigital Be kind to future you. Use descriptive names in $id.

Slide 74

Slide 74 text

@GanbaroDigital JSON schema design follows a simple pattern. Start at the edge, then drill down.

Slide 75

Slide 75 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 76

Slide 76 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 77

Slide 77 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object" }

Slide 78

Slide 78 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { } }

Slide 79

Slide 79 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 80

Slide 80 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { "uid": { "type": "integer" } } }

Slide 81

Slide 81 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 82

Slide 82 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 83

Slide 83 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Accept

Slide 84

Slide 84 text

@GanbaroDigital { "uid": "one thousand and one", "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 85

Slide 85 text

@GanbaroDigital { "uid": "one thousand and one", "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Reject

Slide 86

Slide 86 text

@GanbaroDigital ?? ?? Remember what I said before, about constraints and rejection?

Slide 87

Slide 87 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Accept

Slide 88

Slide 88 text

@GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 89

Slide 89 text

@GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Accept

Slide 90

Slide 90 text

@GanbaroDigital We haven't defined a constraint to reject negative values. So, they pass validation. Even though they're invalid.

Slide 91

Slide 91 text

@GanbaroDigital “JSON Schema does not understand the context. Only constraints.

Slide 92

Slide 92 text

@GanbaroDigital http://json-schema.org/latest/json-schema- validation.html#rfc.section.6

Slide 93

Slide 93 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { "uid": { "type": "integer" } } }

Slide 94

Slide 94 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 95

Slide 95 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Accept

Slide 96

Slide 96 text

@GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Reject

Slide 97

Slide 97 text

@GanbaroDigital We've defined constraints for a UNIX user ID. Wouldn't it be great if we could split that out, for future reuse?

Slide 98

Slide 98 text

@GanbaroDigital Compound Schema

Slide 99

Slide 99 text

@GanbaroDigital A compound JSON Schema is a single JSON file that describes multiple data structures.

Slide 100

Slide 100 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 101

Slide 101 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": { "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 102

Slide 102 text

@GanbaroDigital The constraints on a property are actually a nested JSON schema!

Slide 103

Slide 103 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix" }

Slide 104

Slide 104 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { } }

Slide 105

Slide 105 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 106

Slide 106 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 107

Slide 107 text

@GanbaroDigital $id in our subschema defines a URI #fragment This gets concatenated to the $id at the top of the file.

Slide 108

Slide 108 text

@GanbaroDigital The full $id is now: http://ganbarodigital.com/schemas/unix#uid

Slide 109

Slide 109 text

@GanbaroDigital Other approaches to subschema IDs are available. This one works everywhere, and doesn't break people's brains.

Slide 110

Slide 110 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 111

Slide 111 text

@GanbaroDigital I make these property names the same as the $id fragment just for consistency.

Slide 112

Slide 112 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 113

Slide 113 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 114

Slide 114 text

@GanbaroDigital Use $ref to pull in a JSON schema from somewhere else.

Slide 115

Slide 115 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }

Slide 116

Slide 116 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 }, "user-record": { "$id": "#user-record", "type": "object", "properties": { "uid": { "$ref": "#uid" } } } } }

Slide 117

Slide 117 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": { "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 }, "user-record": { "$id": "#user-record", "type": "object", "properties": { "uid": { "$ref": "#uid" } } } } }

Slide 118

Slide 118 text

@GanbaroDigital Using JSON Schema

Slide 119

Slide 119 text

@GanbaroDigital From PHP

Slide 120

Slide 120 text

@GanbaroDigital composer require justinrainbow/json-schema

Slide 121

Slide 121 text

@GanbaroDigital Watch out for packages that go beyond the spec. You can't use those schemas anywhere else.

Slide 122

Slide 122 text

@GanbaroDigital From Javascript

Slide 123

Slide 123 text

@GanbaroDigital npm install --save ajv

Slide 124

Slide 124 text

@GanbaroDigital Lessons Learned In The Field

Slide 125

Slide 125 text

@GanbaroDigital JSON

Slide 126

Slide 126 text

@GanbaroDigital No comments!!!

Slide 127

Slide 127 text

@GanbaroDigital 'Annotation keywords' have been added to the spec to help.

Slide 128

Slide 128 text

@GanbaroDigital JSON Schema Isn’t Stable

Slide 129

Slide 129 text

@GanbaroDigital At the time of writing, the latest is Draft 07.

Slide 130

Slide 130 text

@GanbaroDigital The basics feel like they have stabilised.

Slide 131

Slide 131 text

@GanbaroDigital I'm expecting upcoming drafts to experiment with difficult problems around schema reuse.

Slide 132

Slide 132 text

@GanbaroDigital My gut feel is that it's going to take several more drafts before they nail it.

Slide 133

Slide 133 text

@GanbaroDigital Not all drafts are created equal.

Slide 134

Slide 134 text

@GanbaroDigital Not all packages have kept up with the drafts. Some have been abandoned.

Slide 135

Slide 135 text

@GanbaroDigital $ref Resolving

Slide 136

Slide 136 text

@GanbaroDigital Validators do not expect the JSON Schema to exist at the given URI. Neither should you.

Slide 137

Slide 137 text

@GanbaroDigital You need a validator package that delegates JSON schema loading back to YOU.

Slide 138

Slide 138 text

@GanbaroDigital Regex Compatibility

Slide 139

Slide 139 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 140

Slide 140 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 141

Slide 141 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "username": { "$id": "#username", "type": "string", "pattern": "^[a-z_][a-z0-9-_]{0,31}$" } } }

Slide 142

Slide 142 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "username": { "$id": "#username", "type": "string", "pattern": "^[a-z_][a-z0-9-_]{0,31}$" } } }

Slide 143

Slide 143 text

@GanbaroDigital The spec says all regexes should conform with ECMA 262.

Slide 144

Slide 144 text

@GanbaroDigital • PHP implementations use industry-standard PCRE regular expressions • Javascript implementations all use ECMA-262? • What do Java / .NET / etc etc use?

Slide 145

Slide 145 text

@GanbaroDigital Keep your regexes simple to make them portable. Oh, and anchor them.

Slide 146

Slide 146 text

@GanbaroDigital Not For Correctness Validation

Slide 147

Slide 147 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 148

Slide 148 text

@GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }

Slide 149

Slide 149 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "gid": { "$id": "#gid", "$ref": "#uid" }, "groups": { "$id": "#groups", "type": "array", "items": { "$ref": "#gid" }, "additionalItems": false, "minItems": 1 } } }

Slide 150

Slide 150 text

@GanbaroDigital ?? ?? What does that schema do?

Slide 151

Slide 151 text

@GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "gid": { "$id": "#gid", "$ref": "#uid" }, "groups": { "$id": "#groups", "type": "array", "items": { "$ref": "#gid" }, "additionalItems": false, "minItems": 1 } } }

Slide 152

Slide 152 text

@GanbaroDigital That schema only ensures that "groups" is an array and each entry in that array is formatted as a group ID.

Slide 153

Slide 153 text

@GanbaroDigital It does not, and cannot ensure that each group ID: actually exists ... is suitable for the user ... ... or any other validation that belongs in your business logic.

Slide 154

Slide 154 text

@GanbaroDigital “ You cannot delegate business logic into JSON Schema.

Slide 155

Slide 155 text

@GanbaroDigital Benefits of Using JSON Schema

Slide 156

Slide 156 text

@GanbaroDigital Handwritten Validation
 Suffers From ... • Duplication • Coverage Gaps

Slide 157

Slide 157 text

@GanbaroDigital Separate out common / generic schemas into their own compound schema doc. Then reuse them!

Slide 158

Slide 158 text

@GanbaroDigital Making your payloads work with JSON Schema results in better payload design.

Slide 159

Slide 159 text

@GanbaroDigital “ If it's hard to write a JSON Schema for, it'll be hard for an API user to deal with too.

Slide 160

Slide 160 text

@GanbaroDigital Use application/vnd+ as API content types. Then you can delegate JSON Schema validation to a middleware component.

Slide 161

Slide 161 text

@GanbaroDigital

Slide 162

Slide 162 text

Thank You Any Questions? A presentation by @stuherbert
 for @GanbaroDigital