Tweet
Tweet

Slide 1

Slide 1 text

Juan Cruz Viotti, Founder at Sourcemeta Applying Software Engineering Practices to Schemas The JSON-Schema- fi rst approach to API speci fi cations

Slide 2

Slide 2 text

Founder / Consultant at Sourcemeta • TSC member of JSON Schema • O’Reilly author on the topic of JSON Schema for data science • Award-winning research at the University of Oxford on JSON Schema • Author of various JSON Schema tooling, such as LearnJSONSchema.com and AlterSchema About me: Juan Cruz Viotti

Slide 3

Slide 3 text

API Speci fi cations Human/Machine readable descriptions of APIs • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability

Slide 4

Slide 4 text

API Speci fi cations Human/Machine readable descriptions of APIs Source: Postman 2023 State of API Report • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability

Slide 5

Slide 5 text

API Speci fi cations Human/Machine readable descriptions of APIs Source: Postman 2023 State of API Report • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability Game Changing!

Slide 6

Slide 6 text

Are you using any of these?

Slide 7

Slide 7 text

The OpenAPI Journey: Episode 1 The glamorous life of an API developer

Slide 8

Slide 8 text

The OpenAPI Journey: Episode 1 A declarative de fi nition of my main API endpoint in just 68 lines of YAML The glamorous life of an API developer

Slide 9

Slide 9 text

The OpenAPI Journey: Episode 2 The good life of an API developer

Slide 10

Slide 10 text

The OpenAPI Journey: Episode 2 CRUD operations of my most important resource is just 249 lines of YAML The good life of an API developer

Slide 11

Slide 11 text

The OpenAPI Journey: Episode 3 The decent life of an API developer

Slide 12

Slide 12 text

The OpenAPI Journey: Episode 3 A sample real-world declarative de fi nition of an API in just 806 lines of YAML The decent life of an API developer

Slide 13

Slide 13 text

The OpenAPI Journey: Final Episode The “unless-there-is-tooling-I-will-switch-careers” API developer

Slide 14

Slide 14 text

The OpenAPI Journey: Final Episode A (great!) real-world production-ready declarative de fi nition of an API in JUST 143943 lines of YAML that GitHub will refuse to preview The “unless-there-is-tooling-I-will-switch-careers” API developer

Slide 15

Slide 15 text

The OpenAPI Journey: Final Episode A (great!) real-world production-ready declarative de fi nition of an API in JUST 143943 lines of YAML that GitHub will refuse to preview The “unless-there-is-tooling-I-will-switch-careers” API developer (At companies like Stripe, there indeed seems to be proprietary tooling for generating/managing this)

Slide 16

Slide 16 text

>80% of this is JSON Schema Either as re-usable components or inlined in path de fi nitions KEY OBSERVATION

Slide 17

Slide 17 text

>80% of this is JSON Schema Either as re-usable components or inlined in path de fi nitions KEY OBSERVATION

Slide 18

Slide 18 text

All of these speci fi cations are in a way wrappers around JSON Schema TAKING IT ONE STEP FURTHER…

Slide 19

Slide 19 text

All of these speci fi cations are in a way wrappers around JSON Schema TAKING IT ONE STEP FURTHER… JSON SCHEMA ALL THE WAY

Slide 20

Slide 20 text

If you want to improve your API speci fi cations…

Slide 21

Slide 21 text

If you want to improve your API speci fi cations… The best thing you can do is Improve your JSON Schemas

Slide 22

Slide 22 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience!

Slide 23

Slide 23 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation

Slide 24

Slide 24 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from

Slide 25

Slide 25 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all

Slide 26

Slide 26 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc

Slide 27

Slide 27 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc Schemas are overly complicated from what they are supposed to match (bad practices, etc)

Slide 28

Slide 28 text

The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc Schemas are overly complicated from what they are supposed to match (bad practices, etc) Relying on non-fully-compliant JSON Schema implementations

Slide 29

Slide 29 text

The JSON Schema fi rst approach to API speci fi cations

Slide 30

Slide 30 text

The JSON Schema fi rst approach to API speci fi cations TLDR; Just treat you schemas as code. You already know all of this!

Slide 31

Slide 31 text

The JSON Schema fi rst approach to API speci fi cations TLDR; Just treat you schemas as code. You already know all of this!

Slide 32

Slide 32 text

Surely you don’t write all your projects as single huge code fi les

Slide 33

Slide 33 text

Surely you don’t write all your projects as single huge code fi les What about your OpenAPIs?

Slide 34

Slide 34 text

Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1

Slide 35

Slide 35 text

Extract your JSON Schemas as individual fi les on a GitHub repo • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! The JSON Schema fi rst approach: Step #1

Slide 36

Slide 36 text

Extract your JSON Schemas as individual fi les on a GitHub repo • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! • And you can share the same schemas with more than one API speci fi cation without copy-pasting (yay!) The JSON Schema fi rst approach: Step #1

Slide 37

Slide 37 text

Extract your JSON Schemas as individual fi les on a GitHub repo A “fundamental of getting your schema house in order” Kin Lane, the API Evangelist • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! • And you can share the same schemas with more than one API speci fi cation without copy-pasting (yay!) The JSON Schema fi rst approach: Step #1

Slide 38

Slide 38 text

Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1

Slide 39

Slide 39 text

Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1 https://wellshapedwords.com/posts/split- fi les-to-save-time/ There are various EXISTING tools to “unbundle” an OpenAPI and extract its schemas as separate fi les

Slide 40

Slide 40 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2

Slide 41

Slide 41 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 https://github.com/sourcemeta/jsonschema Shameless plug: You may enjoy my own CLI, as it was speci fi cally created with these use cases in mind

Slide 42

Slide 42 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Shameless plug: You may enjoy my own CLI, as it was speci fi cally created with these use cases in mind https://github.com/sourcemeta/jsonschema

Slide 43

Slide 43 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant!

Slide 44

Slide 44 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant!

Slide 45

Slide 45 text

Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant! Because of it, many developers inadvertently create bad schemas

Slide 46

Slide 46 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3

Slide 47

Slide 47 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 This is analogous to checking if your code actually compiles

Slide 48

Slide 48 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 This is analogous to checking if your code actually compiles

Slide 49

Slide 49 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3

Slide 50

Slide 50 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems

Slide 51

Slide 51 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems

Slide 52

Slide 52 text

Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems Sounds obvious, but you would be surprised at how many people upgrade their schemas by just bumping $schema without taking a look at anything else

Slide 53

Slide 53 text

Are you using any code formatters? Like prettier, rustfmt, gofmt, etc

Slide 54

Slide 54 text

Are you using any code formatters? Like prettier, rustfmt, gofmt, etc Same for your JSON Schemas, right?

Slide 55

Slide 55 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4

Slide 56

Slide 56 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4

Slide 57

Slide 57 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4

Slide 58

Slide 58 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Dialect fi rst, so we know how to read the rest

Slide 59

Slide 59 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Schema-wide metadata at the top

Slide 60

Slide 60 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Type information fi rst, if any

Slide 61

Slide 61 text

Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Type-speci fi c constraints last

Slide 62

Slide 62 text

Are you using any code linters at work?

Slide 63

Slide 63 text

Are you using any code linters at work? You are linting your JSON Schemas too, right?

Slide 64

Slide 64 text

Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5

Slide 65

Slide 65 text

Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing

Slide 66

Slide 66 text

Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing

Slide 67

Slide 67 text

Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing For most rules, you can do: jsonschema lint —fix schema.json

Slide 68

Slide 68 text

Do you write automated tests for your code?

Slide 69

Slide 69 text

Do you write automated tests for your code? You are testing your JSON Schemas too, right?

Slide 70

Slide 70 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 71

Slide 71 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 72

Slide 72 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 73

Slide 73 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 74

Slide 74 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 75

Slide 75 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 76

Slide 76 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 77

Slide 77 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 78

Slide 78 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6

Slide 79

Slide 79 text

Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6 The syntax of my test runner is intentionally inspired by the of fi cial JSON Schema Test Suite

Slide 80

Slide 80 text

Just like you would do with your code, run all of this on CI/CD!

Slide 81

Slide 81 text

We provide an easy GitHub Actions integration

Slide 82

Slide 82 text

Make sure your schemas “compile”

Slide 83

Slide 83 text

Enforce a common readable style

Slide 84

Slide 84 text

Catch obvious issues and avoid bad practices

Slide 85

Slide 85 text

Make sure the schemas actually do what you intend

Slide 86

Slide 86 text

https://github.com/sourcemeta/jsonschema JSON Schema CLI Thanks a lot!