Automating Empathy: Test Your Docs with Swagger and Apivore

Ugh, documentation.

It's the afterthought of every system, scrambled together in the final days before launch, updated sparingly, generally out of date.

What if we could programmatically verify that our API documentation was accurate? What if this helped us build more intuitive APIs by putting our users first? What if documentation came first, and helped us write our code?

With Swagger and Apivore as our weapons of choice, we'll write documentation that will make your APIs better, your clients more satisfied, and you happier.

Transcript

1. AUTOMATING EMPATHY Test Your Documentation with Swagger and Apivore
2. EVERYONE HATES DOCUMENTATION
3. “Projects often start with high ideals for documentation, but they always fall by the wayside in the crush of business. I observed this and said to myself, "Perhaps people don't maintain detailed documentation because it isn't actually a good idea." If it hurts running your head into a brick wall over and over, perhaps you should figure out how to get along without running your head into the brick wall. - Kent Beck https://accu.org/index.php/journals/509
4. “Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan - The Agile Manifesto
5. 1. IMPLEMENTATION DOCUMENTATION
 VS.
 USAGE DOCUMENTATION
6. “I think that the original writers could have been more specific to remove some of the confusion and misinformation that has sprung up around this value. It might have been more appropriate if the original writers would have said, “Working software over comprehensive requirements and design documentation,” because I think this is more what they meant. -Larry “Agile Doctor” Apke http://www.agile-doctor.com/2016/08/16/agile-values-working-software-documentation/
7. 2. IT’S NOT A BURDEN WHEN IT’S A TOOL FOR ITERATION
8. 3. IF YOU CAME TO THIS TALK, YOU PROBABLY NEED DOCS
9. WHY IS IT SO DIFFICULT TO PRODUCE ACCURATE DOCUMENTATION?
10. “WE KEEP FORGETTING TO UPDATE THE DOCS. LET’S GET BETTER!”
11. “OUR API IS CHANGING ALL THE TIME. HOW CAN WE KEEP UP?
12. “I HATE UPDATING OUR DOCUMENTATION. IT’S SO ANNOYING!”
13. “CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!”
14. “CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!” COMPLEXITY
15. WE HAVE A PROBLEM. GUILT WON’T SOLVE IT. MAYBE WE CAN FIX IT IF WE MAKE IT EASIER.
16. THE MORE WE NEED TO HOLD IN OUR HEADS TO PRODUCE DOCUMENTATION, THE WORSE IT WILL BE.
17. WHAT IF WE DIDN’T NEED TO REMEMBER ANYTHING?
18. YOU DON’T NEED TO REMEMBER ANYTHING IF YOU WRITE THE DOCUMENTATION FIRST!
19. WE CAN WRITE PERFECT DOCUMENTATION IF A TEST FAILS UNTIL THE CODE MATCHES THE DOCS
20. DOCUMENTATION-DRIVEN DEVELOPMENT ➤ For a new endpoint: ➤ Document the endpoint ➤ The test suite complains that your documentation isn’t fully tested ➤ Write a test for the endpoint ➤ Write just enough code to make the test pass ➤ To change an endpoint: ➤ Update the documentation ➤ The test fails ➤ Write just enough code to make the test pass
21. USER-DRIVEN DOCUMENTATION UPDATES ➤ For a new endpoint: ➤ Make the endpoint ➤ Document it (unless you forget) ➤ Fix the documentation when a user complains about your API being broken ➤ To change an endpoint: ➤ Update the endpoint ➤ Probably forget to document it ➤ Fix the documentation when a user complains about your API being broken
22. @amcaplan amcaplan.ninja Hi! My name is Ariel Caplan
23. @amcaplan amcaplan.ninja
24. @amcaplan amcaplan.ninja Dev Empathy
 Book Club devempathybook.club
26. “ The only thing holding Ruby together was a hipster coder community of twenty-something year old nerds who are now thirty-something nerds. -Stefan Mischook, killerphp.com
27. I’M STILL A TWENTY-SOMETHING NERD!
28. HOW DO WE GET STARTED WITH DOCUMENTATION-DRIVEN DEVELOPMENT?
29. STEP 2: CREATE DOCUMENTATION THAT COMPUTERS CAN READ
32. THE ZEN OF SWAGGER
33. /packages/{id} POST /packages THE ZEN OF SWAGGER 201 422 200 200 404 200 422 200 410 GET GET DELETE PATCH
34. THE ZEN OF SWAGGER 201 422 /packages POST What information needs to be submitted so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails?
35. parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg { "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1,
 "weight": 3.3 }
36. parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg
37. THE ZEN OF SWAGGER What information needs to be submitted so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails? parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
38. /packages: post: summary: Create a Package operationId: createPackage tags: - Packages parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' THE ZEN OF SWAGGER 201 422 /packages POST parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
39. THE ZEN OF SWAGGER
40. THE ZEN OF SWAGGER
41. THE ZEN OF SWAGGER
42. STEP 3: TESTING YOUR DOCUMENTATION
44. SETTING UP APIVORE ➤ Include the apivore gem ➤ Tell Apivore::SwaggerChecker how to find your docs ➤ Write a test to assert all endpoints/statuses are tested ➤ For each endpoint/status: ➤ Set up context (like in any other RSpec test) ➤ Tell Apivore::SwaggerChecker: ➤ Which request to make ➤ With what params ➤ Which status code to expect
45. INITIAL BOILERPLATE require 'spec_helper' RSpec.describe 'the API', type: :apivore, order: :defined do subject { Apivore::SwaggerChecker.instance_for('/swagger.json') } context 'has valid paths' do # tests go here end context 'and' do it 'tests all documented routes' do expect(subject).to validate_all_paths end end end Where is the documentation? Assert all endpoints/statuses are tested
46. BACKFILLING AN API $ rspec F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 201 post /packages is untested for response code 422 # ./spec/api/api_spec.rb:12:in `block (3 levels) in <top (required)>' Finished in 0.0929 seconds (files took 4.76 seconds to load) 1 example, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:11 # the API and tests all documented routes
47. A SAMPLE HAPPY-PATH TEST let(:params) {{ "_data" => { "destination_id" => 114, "length" => 14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 201, params) }
48. A SAMPLE HAPPY-PATH TEST $ rspec .F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 422 # ./spec/api/api_spec.rb:22:in `block (3 levels) in <top (required)>' Finished in 0.09739 seconds (files took 2.02 seconds to load) 2 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:21 # the API and tests all documented routes
49. A SAMPLE FAILURE-PATH TEST let(:params) {{ "_data" => { "destination_id" => 114, "length" => -14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 422, params) }
50. A SAMPLE FAILURE-PATH TEST $ rspec ... Finished in 0.09168 seconds (files took 1.96 seconds to load) 3 examples, 0 failures
51. CREATING A NEW ENDPOINT /packages/{id}: patch: summary: Update a Package operationId: updatePackage tags: - Packages parameters: - name: body in: body required: true description: Updated Package information schema: $ref: '#/definitions/PackageUpdateModel' responses: '200': description: Package successfully updated schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
52. CREATING A NEW ENDPOINT /packages/{id}: patch: summary: Update a Package operationId: updatePackage tags: - Packages parameters: - name: body in: body required: true description: Updated Package information schema: $ref: '#/definitions/PackageUpdateModel' responses: '200': description: Package successfully updated schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' PackageUpdateModel: properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg
53. CREATING A NEW ENDPOINT $ rspec ..F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths patch /packages is untested for response code 200 patch /packages is untested for response code 422 # ./spec/api/api_spec.rb:36:in `block (3 levels) in <top (required)>' Finished in 0.11711 seconds (files took 1.94 seconds to load) 3 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:35 # the API and tests all documented routes
54. CREATING A NEW ENDPOINT let(:package) { Package.create!( destination_id: 114, length: 14.7, width: 12.2, height: 2.1, weight: 3.3 ) }
55. CREATING A NEW ENDPOINT context 'happy' do let(:params) {{ "id" => package.id, "_data" => { "length" => 8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 200, params) } end context 'sad' do let(:params) {{ "id" => package.id, "_data" => { "length" => -8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 422, params) } end
56. AFTER IMPLEMENTING…
57. CREATING A NEW ENDPOINT $ rspec ..... Finished in 0.1321 seconds (files took 2.06 seconds to load) 5 examples, 0 failures
58. UPDATING THE API PackageModel: required: [destination_id, length, width, height] properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg
59. PackageModel: required: [destination_id, length, width, height, volume] properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg UPDATING THE API volume: type: number format: float description: Volume of package in cm3
60. UPDATING THE API Failures: 1) the API has valid paths post /packages happy should validate that post / packages returns 201 Failure/Error: it { is_expected.to validate(:post, "/packages", 201, params) } '/packages#/' did not contain a required property of 'volume' Response body: { "id": 24, "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1, "weight": 3.3, "created_at": "2018-04-15T22:54:37.166Z", "updated_at": "2018-04-15T22:54:37.166Z" } # ./spec/api/api_spec.rb:17:in `block (5 levels) in <top (required)>'
61. CAVEAT TIME!
66. ❌
67. IT’S NOT PERFECT. IT’S STILL AN IMPROVEMENT.
68. STEP 1: CONVINCING YOUR MANAGER YOU NEED DOCUMENTATION TESTING
69. LET’S TRAVEL BACK IN TIME…
70. HOW WE STARTED ON SWAGGER AND APIVORE
71. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails
72. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team
73. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week
74. HOW WE STARTED ON SWAGGER AND APIVORE May 2016 I join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests
75. HOW WE STARTED ON SWAGGER AND APIVORE June 2016 Decided to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests August 18, 2016 Finished writing Apivore documentation tests
76. FAST FACTS ➤ 29 days ➤ 18 pull requests ➤ +1,835 LOC ➤ - 2,734 LOC
77. WHAT TOOK SO LONG?
78. DOCUMENTATION MISTAKES
79. UNNECESSARILY STUFFED API Why return timestamps
 on models that never change?
80. UNCONVENTIONAL STATUS CODES Create actions
 should return 201,
 not 200 DELETE to a
 nonexistent resource
 should be 200 or 410,
 not 404
81. OVERCOMPLICATED ROUTES We don’t need all of
 /doctors/:doctor_id/reviews/:id
 to know which Review is being requested
82. CONFUSING DOMAIN MODEL Are individual Answers an
 HTTP resource
 or an attribute of a Review?
83. MIXING DATABASE CONCERNS INTO DOMAIN MODEL Flags, Likes, and Dislikes
 are a single DB concept using STI.
 Why should the user know that?
84. INAPPROPRIATE BEHAVIOR ID of a reviewed doctor/facility
 isn’t a required field on a Review.
 Does that make sense?
85. CLIENT DATA EXPOSURE We seem to be returning Reviews
 belonging to all clients,
 not just the current one.
86. DATA CORRUPTION The error case where
 required questions aren’t answered
 returns a 201 and saves the Review.
87. INSUFFICIENT LIMITS ON PERMISSIONS We expose API endpoints
 to edit system-level data.
 That’s not right!
88. DOCUMENTATION TESTING HELPED US UNCOVER: ➤ Documentation Mistakes ➤ Unnecessarily Stuffed API ➤ Overcomplicated Routes ➤ Confusing Domain Model ➤ Mixing Database Concerns Into Domain Model ➤ Inappropriate Behavior ➤ Client Data Exposure ➤ Data Corruption ➤ Insufficient Limits on Permissions
89. DOCUMENTATION TESTING HELPED US UNCOVER: ➤ Documentation Mistakes ➤ Unnecessarily Stuffed API ➤ Overcomplicated Routes ➤ Confusing Domain Model ➤ Mixing Database Concerns Into Domain Model ➤ Inappropriate Behavior ➤ Client Data Exposure ➤ Data Corruption ➤ Insufficient Limits on Permissions Hey Manager, I saw a talk about how documentation testing can save you from a laundry list of problems. Maybe we can try it! That sounds great!
90. THIS WAS A TALENTED TEAM OF SEASONED DEVELOPERS WHO MADE EMBARRASSING MISTAKES
91. THE CODE WAS GREAT. THE DESIGN WASN’T (ALWAYS).
92. DOCUMENTATION TESTING FORCED US TO THINK ABOUT THE API FROM THE PERSPECTIVE OF A DOCUMENTATION USER
93. WHEN DESIGNING NEW THINGS, WE START BY DEFINING THE USER IMPACT
94. THE TAKEAWAY
96. “From the perspective of a user,
 if a feature is not documented,
 then it doesn't exist,
 and if a feature is documented incorrectly,
 then it's broken. - Zach Supalla, “Documentation-Driven Development” https://gist.github.com/zsup/9434452
97. ACCORDING TO USERS, DOCUMENTATION IS THE PRIMARY SOURCE OF TRUTH. LET’S EMBRACE THAT.
98. THANK YOU! @amcaplan amcaplan.ninja/talks devempathybook.club vitals.com