apidays Paris 2024 - Applying Software Engineering Practices to Schemas - Juan Cruz Viotti, Sourcemeta

Applying Software Engineering Practices to Schemas
Juan Cruz Viotti, Director at Sourcemeta

apidays Paris 2024 - The Future API Stack for Mass Innovation
December 3 - 5, 2024

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. Juan Cruz Viotti, Founder at Sourcemeta Applying Software Engineering Practices to Schemas The JSON-Schema- fi rst approach to API speci fi cations
2. Founder / Consultant at Sourcemeta • TSC member of JSON Schema • O’Reilly author on the topic of JSON Schema for data science • Award-winning research at the University of Oxford on JSON Schema • Author of various JSON Schema tooling, such as LearnJSONSchema.com and AlterSchema About me: Juan Cruz Viotti
3. API Speci fi cations Human/Machine readable descriptions of APIs • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability
4. API Speci fi cations Human/Machine readable descriptions of APIs Source: Postman 2023 State of API Report • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability
5. API Speci fi cations Human/Machine readable descriptions of APIs Source: Postman 2023 State of API Report • Single source of truth • Standardised documentation • Code generation • Improved testing / validation • Increased discoverability Game Changing!
6. Are you using any of these?
7. The OpenAPI Journey: Episode 1 The glamorous life of an API developer
8. The OpenAPI Journey: Episode 1 A declarative de fi nition of my main API endpoint in just 68 lines of YAML The glamorous life of an API developer
9. The OpenAPI Journey: Episode 2 The good life of an API developer
10. The OpenAPI Journey: Episode 2 CRUD operations of my most important resource is just 249 lines of YAML The good life of an API developer
11. The OpenAPI Journey: Episode 3 The decent life of an API developer
12. The OpenAPI Journey: Episode 3 A sample real-world declarative de fi nition of an API in just 806 lines of YAML The decent life of an API developer
13. The OpenAPI Journey: Final Episode The “unless-there-is-tooling-I-will-switch-careers” API developer
14. The OpenAPI Journey: Final Episode A (great!) real-world production-ready declarative de fi nition of an API in JUST 143943 lines of YAML that GitHub will refuse to preview The “unless-there-is-tooling-I-will-switch-careers” API developer
15. The OpenAPI Journey: Final Episode A (great!) real-world production-ready declarative de fi nition of an API in JUST 143943 lines of YAML that GitHub will refuse to preview The “unless-there-is-tooling-I-will-switch-careers” API developer (At companies like Stripe, there indeed seems to be proprietary tooling for generating/managing this)
16. >80% of this is JSON Schema Either as re-usable components or inlined in path de fi nitions KEY OBSERVATION
17. >80% of this is JSON Schema Either as re-usable components or inlined in path de fi nitions KEY OBSERVATION
18. All of these speci fi cations are in a way wrappers around JSON Schema TAKING IT ONE STEP FURTHER…
19. All of these speci fi cations are in a way wrappers around JSON Schema TAKING IT ONE STEP FURTHER… JSON SCHEMA ALL THE WAY
20. If you want to improve your API speci fi cations…
21. If you want to improve your API speci fi cations… The best thing you can do is Improve your JSON Schemas
22. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience!
23. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation
24. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from
25. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all
26. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc
27. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc Schemas are overly complicated from what they are supposed to match (bad practices, etc)
28. The Most Common Pitfalls with JSON Schema At least based on my consultancy experience! De fi ning entire ontologies of schemas within the API speci fi cation Copy pasting schemas in various API speci fi cations because there is not a single place to reference them from Schemas are not being unit tested at all Schemas are plain invalid, using wrong keywords, etc Schemas are overly complicated from what they are supposed to match (bad practices, etc) Relying on non-fully-compliant JSON Schema implementations
29. The JSON Schema fi rst approach to API speci fi cations
30. The JSON Schema fi rst approach to API speci fi cations TLDR; Just treat you schemas as code. You already know all of this!
31. The JSON Schema fi rst approach to API speci fi cations TLDR; Just treat you schemas as code. You already know all of this!
32. Surely you don’t write all your projects as single huge code fi les
33. Surely you don’t write all your projects as single huge code fi les What about your OpenAPIs?
34. Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1
35. Extract your JSON Schemas as individual fi les on a GitHub repo • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! The JSON Schema fi rst approach: Step #1
36. Extract your JSON Schemas as individual fi les on a GitHub repo • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! • And you can share the same schemas with more than one API speci fi cation without copy-pasting (yay!) The JSON Schema fi rst approach: Step #1
37. Extract your JSON Schemas as individual fi les on a GitHub repo A “fundamental of getting your schema house in order” Kin Lane, the API Evangelist • Otherwise, its a lot harder to do everything within the constraints of i.e. an OpenAPI wrapper! • And you can share the same schemas with more than one API speci fi cation without copy-pasting (yay!) The JSON Schema fi rst approach: Step #1
38. Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1
39. Extract your JSON Schemas as individual fi les on a GitHub repo The JSON Schema fi rst approach: Step #1 https://wellshapedwords.com/posts/split- fi les-to-save-time/ There are various EXISTING tools to “unbundle” an OpenAPI and extract its schemas as separate fi les
40. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2
41. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 https://github.com/sourcemeta/jsonschema Shameless plug: You may enjoy my own CLI, as it was speci fi cally created with these use cases in mind
42. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Shameless plug: You may enjoy my own CLI, as it was speci fi cally created with these use cases in mind https://github.com/sourcemeta/jsonschema
43. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant!
44. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant!
45. Find a fully-compliant tool to work with JSON Schema The JSON Schema fi rst approach: Step #2 Avoid AJV-based tools! AJV is non-compliant! Because of it, many developers inadvertently create bad schemas
46. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3
47. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 This is analogous to checking if your code actually compiles
48. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 This is analogous to checking if your code actually compiles
49. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3
50. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems
51. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems
52. Check all schemas against their meta-schemas The JSON Schema fi rst approach: Step #3 The array variant of items in 2019-09 and before was replaced by prefixItems Sounds obvious, but you would be surprised at how many people upgrade their schemas by just bumping $schema without taking a look at anything else
53. Are you using any code formatters? Like prettier, rustfmt, gofmt, etc
54. Are you using any code formatters? Like prettier, rustfmt, gofmt, etc Same for your JSON Schemas, right?
55. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4
56. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4
57. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4
58. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Dialect fi rst, so we know how to read the rest
59. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Schema-wide metadata at the top
60. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Type information fi rst, if any
61. Format your schemas for readability and uni fi ed styling The JSON Schema fi rst approach: Step #4 Type-speci fi c constraints last
62. Are you using any code linters at work?
63. Are you using any code linters at work? You are linting your JSON Schemas too, right?
64. Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5
65. Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing
66. Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing
67. Lint your schemas to fi nd issues early and avoid bad practices The JSON Schema fi rst approach: Step #5 This constrain is doing nothing For most rules, you can do: jsonschema lint —fix schema.json
68. Do you write automated tests for your code?
69. Do you write automated tests for your code? You are testing your JSON Schemas too, right?
70. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
71. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
72. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
73. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
74. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
75. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
76. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
77. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
78. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6
79. Unit test your schemas to ensure they match what you intend The JSON Schema fi rst approach: Step #6 The syntax of my test runner is intentionally inspired by the of fi cial JSON Schema Test Suite
80. Just like you would do with your code, run all of this on CI/CD!
81. We provide an easy GitHub Actions integration
82. Make sure your schemas “compile”
83. Enforce a common readable style
84. Catch obvious issues and avoid bad practices
85. Make sure the schemas actually do what you intend
86. https://github.com/sourcemeta/jsonschema JSON Schema CLI Thanks a lot!