Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JSON Schema For Validating API Requests

JSON Schema For Validating API Requests

When Stuart needed to build an API-first app for a client, he wanted to find a better way to do input validation. He'd just come off a quality audit role on a large-scale project where he'd seen first-hand how it just isn't feasible to expect anyone to achieve 100% input validation through entirely manual means.

That experience, and that search, led him to JSON Schema.

In this talk, Stuart will introduce you to JSON Schema. He'll show you the essential points of writing a schema, and how to use it both from PHP and from JavaScript. Finally, he'll discuss the current state of the JSON schema draft, and share the compatibility issues he's seen over the last couple of years.

2c1dc90ff7bf69097a151677624777d2?s=128

Stuart Herbert

July 02, 2018
Tweet

Transcript

  1. A presentation by @stuherbert
 for @GanbaroDigital JSON Schema For Validating

    API Requests
  2. Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since

    1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart
  3. Follow me I do tweet a lot about non-tech stuff

    though :) @stuherbert
  4. @GanbaroDigital ?? ?? Have you used JSON Schema?

  5. @GanbaroDigital Why JSON Schema?

  6. @GanbaroDigital I've seen a lot of hand-written API validation code.

  7. @GanbaroDigital Hand-written input validation code has issues.

  8. @GanbaroDigital Handwritten Validation
 Suffers From ... • Duplication • Coverage

    Gaps
  9. @GanbaroDigital ?? ?? Can JSON Schema offer a different way?

    That's what I've been looking into.
  10. @GanbaroDigital This is just my experience over the last 18

    months.
  11. @GanbaroDigital I'm here to learn from your experience too.

  12. @GanbaroDigital API Request Anatomy

  13. @GanbaroDigital HTTP API Request

  14. @GanbaroDigital HTTP API Request HTTP Verb (Method)

  15. @GanbaroDigital e.g. GET, POST, PUT, or HEAD

  16. @GanbaroDigital HTTP API Request Query Path (Route) HTTP Verb (Method)

  17. @GanbaroDigital e.g. /team/chelsea/+all_players

  18. @GanbaroDigital HTTP API Request Query Path (Route) Query Params (Filters)

    HTTP Verb (Method)
  19. @GanbaroDigital e.g. /team/chelsea/+all_players ?season_start=2017

  20. @GanbaroDigital e.g. /team/chelsea/+all_players ?season_start=2017

  21. @GanbaroDigital HTTP API Request HTTP Headers (Metadata) Query Path (Route)

    Query Params (Filters) HTTP Verb (Method)
  22. @GanbaroDigital e.g. Content-Type: application/json

  23. @GanbaroDigital HTTP API Request HTTP Headers (Metadata) Request Body (Payload)

    Query Path (Route) Query Params (Filters) HTTP Verb (Method)
  24. @GanbaroDigital HTTP API Request HTTP Headers (Metadata) Request Body (Payload)

    Query Path (Route) Query Params (Filters) HTTP Verb (Method)
  25. @GanbaroDigital Not every API request can include a Request Body.

  26. @GanbaroDigital HTTP Requests w/out Bodies • GET • HEAD

  27. @GanbaroDigital HTTP Requests w/ Bodies • POST • PUT •

    PATCH
  28. @GanbaroDigital Request bodies are typically* used to change the data

    stored by your system.
  29. @GanbaroDigital “ Request bodies are user input. They must be

    validated before they can be safely used.
  30. @GanbaroDigital API Request API Response

  31. @GanbaroDigital Action API Request API Response

  32. @GanbaroDigital Action Filter Input Escape Output API Request API Response

  33. @GanbaroDigital Action API Request API Response Filter Input Escape Output

    View / Template Controller Model / Controller
  34. @GanbaroDigital Where does JSON schema help with this?

  35. @GanbaroDigital Action Filter Input Escape Output API Request API Response

  36. @GanbaroDigital Action Validate Format Filter Input Escape Output API Request

    API Response
  37. @GanbaroDigital Action Validate Format Filter Input Validate Format Escape Output

    API Request API Response
  38. @GanbaroDigital Action Validate Format Filter Input Validate Format Escape Output

    JSON Schema JSON Schema View / Template JSON Schema API Request API Response Model / Controller
  39. @GanbaroDigital “ If your API accepts JSON payloads JSON schema

    can be part of your input validation process.
  40. @GanbaroDigital “ If your API returns JSON payloads JSON schema

    can be part of your quality assurance process.
  41. @GanbaroDigital “ If your API client 
 sends or receives

    JSON payloads JSON schema can be part of your robustness process.
  42. @GanbaroDigital Introducing JSON Schema

  43. @GanbaroDigital “ JSON Schema are JSON documents that describe other

    JSON documents
  44. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#" }

  45. @GanbaroDigital $schema tells your validator which version of JSON Schema

    to use
  46. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema" }

  47. @GanbaroDigital $id is used for re-using one JSON schema inside

    another
  48. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema" }

  49. @GanbaroDigital Validators do not expect the JSON Schema to exist

    at the given URI. Neither should you.
  50. @GanbaroDigital Simple Schema

  51. @GanbaroDigital A simple JSON Schema describes one data structure.

  52. @GanbaroDigital Every data structure has a type.

  53. @GanbaroDigital • array • boolean • integer* • null •

    number • object • string
  54. @GanbaroDigital * 'integer' is a special form of 'number' because

    Javascript doesn't support separate integer and float types
  55. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "string" }

  56. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "object" }

  57. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/my-schema", "type": "array" }

  58. @GanbaroDigital Every type has its own set of constraints.

  59. @GanbaroDigital http://json-schema.org/latest/json-schema- validation.html#rfc.section.6

  60. @GanbaroDigital “ In the spec, they're called "validation keywords". It's

    easier to design a JSON schema if you think of them as 'constraints'.
  61. @GanbaroDigital “ JSON schema constraints are used to reject input,

    not accept input.
  62. @GanbaroDigital Incoming Data Is Only Rejected If ... 1. You

    define a constraint 2. ... that the incoming data falls foul of
  63. @GanbaroDigital Worked Example

  64. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  65. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#" }

  66. @GanbaroDigital If possible, always use the latest draft of JSON

    Schema.
  67. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

  68. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

  69. @GanbaroDigital Use domains that you actually own. Don't be an

    ass and use someone else's.
  70. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

  71. @GanbaroDigital The query path is a namespace. Use it to

    leave space for the undiscovered country.
  72. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record" }

  73. @GanbaroDigital Be kind to future you. Use descriptive names in

    $id.
  74. @GanbaroDigital JSON schema design follows a simple pattern. Start at

    the edge, then drill down.
  75. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  76. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  77. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object" }

  78. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    } }
  79. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  80. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    "uid": { "type": "integer" } } }
  81. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  82. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  83. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" } Accept
  84. @GanbaroDigital { "uid": "one thousand and one", "username": "stuart", "password":

    "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" }
  85. @GanbaroDigital { "uid": "one thousand and one", "username": "stuart", "password":

    "totally-insecure", "groups": [ 1, 1001 ], "description": "Stuart Herbert" } Reject
  86. @GanbaroDigital ?? ?? Remember what I said before, about constraints

    and rejection?
  87. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" } Accept
  88. @GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  89. @GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" } Accept
  90. @GanbaroDigital We haven't defined a constraint to reject negative values.

    So, they pass validation. Even though they're invalid.
  91. @GanbaroDigital “JSON Schema does not understand the context. Only constraints.

  92. @GanbaroDigital http://json-schema.org/latest/json-schema- validation.html#rfc.section.6

  93. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    "uid": { "type": "integer" } } }
  94. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  95. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" } Accept
  96. @GanbaroDigital { "uid": -9999, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" } Reject
  97. @GanbaroDigital We've defined constraints for a UNIX user ID. Wouldn't

    it be great if we could split that out, for future reuse?
  98. @GanbaroDigital Compound Schema

  99. @GanbaroDigital A compound JSON Schema is a single JSON file

    that describes multiple data structures.
  100. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  101. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/user-record", "type": "object", "properties": {

    "uid": { "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  102. @GanbaroDigital The constraints on a property are actually a nested

    JSON schema!
  103. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix" }

  104. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { } }

  105. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  106. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  107. @GanbaroDigital $id in our subschema defines a URI #fragment This

    gets concatenated to the $id at the top of the file.
  108. @GanbaroDigital The full $id is now: http://ganbarodigital.com/schemas/unix#uid

  109. @GanbaroDigital Other approaches to subschema IDs are available. This one

    works everywhere, and doesn't break people's brains.
  110. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  111. @GanbaroDigital I make these property names the same as the

    $id fragment just for consistency.
  112. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  113. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  114. @GanbaroDigital Use $ref to pull in a JSON schema from

    somewhere else.
  115. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 } } }
  116. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 }, "user-record": { "$id": "#user-record", "type": "object", "properties": { "uid": { "$ref": "#uid" } } } } }
  117. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "uid": {

    "$id": "#uid", "type": "integer", "minimum": 1, "maximum": 4294967294 }, "user-record": { "$id": "#user-record", "type": "object", "properties": { "uid": { "$ref": "#uid" } } } } }
  118. @GanbaroDigital Using JSON Schema

  119. @GanbaroDigital From PHP

  120. @GanbaroDigital composer require justinrainbow/json-schema

  121. @GanbaroDigital Watch out for packages that go beyond the spec.

    You can't use those schemas anywhere else.
  122. @GanbaroDigital From Javascript

  123. @GanbaroDigital npm install --save ajv

  124. @GanbaroDigital Lessons Learned In The Field

  125. @GanbaroDigital JSON

  126. @GanbaroDigital No comments!!!

  127. @GanbaroDigital 'Annotation keywords' have been added to the spec to

    help.
  128. @GanbaroDigital JSON Schema Isn’t Stable

  129. @GanbaroDigital At the time of writing, the latest is Draft

    07.
  130. @GanbaroDigital The basics feel like they have stabilised.

  131. @GanbaroDigital I'm expecting upcoming drafts to experiment with difficult problems

    around schema reuse.
  132. @GanbaroDigital My gut feel is that it's going to take

    several more drafts before they nail it.
  133. @GanbaroDigital Not all drafts are created equal.

  134. @GanbaroDigital Not all packages have kept up with the drafts.

    Some have been abandoned.
  135. @GanbaroDigital $ref Resolving

  136. @GanbaroDigital Validators do not expect the JSON Schema to exist

    at the given URI. Neither should you.
  137. @GanbaroDigital You need a validator package that delegates JSON schema

    loading back to YOU.
  138. @GanbaroDigital Regex Compatibility

  139. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  140. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  141. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "username": {

    "$id": "#username", "type": "string", "pattern": "^[a-z_][a-z0-9-_]{0,31}$" } } }
  142. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "username": {

    "$id": "#username", "type": "string", "pattern": "^[a-z_][a-z0-9-_]{0,31}$" } } }
  143. @GanbaroDigital The spec says all regexes should conform with ECMA

    262.
  144. @GanbaroDigital • PHP implementations use industry-standard PCRE regular expressions •

    Javascript implementations all use ECMA-262? • What do Java / .NET / etc etc use?
  145. @GanbaroDigital Keep your regexes simple to make them portable. Oh,

    and anchor them.
  146. @GanbaroDigital Not For Correctness Validation

  147. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  148. @GanbaroDigital { "uid": 1001, "username": "stuart", "password": "totally-insecure", "groups": [

    1, 1001 ], "description": "Stuart Herbert" }
  149. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "gid": {

    "$id": "#gid", "$ref": "#uid" }, "groups": { "$id": "#groups", "type": "array", "items": { "$ref": "#gid" }, "additionalItems": false, "minItems": 1 } } }
  150. @GanbaroDigital ?? ?? What does that schema do?

  151. @GanbaroDigital { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://ganbarodigital.com/schemas/unix", "definitions": { "gid": {

    "$id": "#gid", "$ref": "#uid" }, "groups": { "$id": "#groups", "type": "array", "items": { "$ref": "#gid" }, "additionalItems": false, "minItems": 1 } } }
  152. @GanbaroDigital That schema only ensures that "groups" is an array

    and each entry in that array is formatted as a group ID.
  153. @GanbaroDigital It does not, and cannot ensure that each group

    ID: actually exists ... is suitable for the user ... ... or any other validation that belongs in your business logic.
  154. @GanbaroDigital “ You cannot delegate business logic into JSON Schema.

  155. @GanbaroDigital Benefits of Using JSON Schema

  156. @GanbaroDigital Handwritten Validation
 Suffers From ... • Duplication • Coverage

    Gaps
  157. @GanbaroDigital Separate out common / generic schemas into their own

    compound schema doc. Then reuse them!
  158. @GanbaroDigital Making your payloads work with JSON Schema results in

    better payload design.
  159. @GanbaroDigital “ If it's hard to write a JSON Schema

    for, it'll be hard for an API user to deal with too.
  160. @GanbaroDigital Use application/vnd+<schema> as API content types. Then you can

    delegate JSON Schema validation to a middleware component.
  161. @GanbaroDigital

  162. Thank You Any Questions? A presentation by @stuherbert
 for @GanbaroDigital