Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automating Empathy: Test Your Docs with Swagger and Apivore

Automating Empathy: Test Your Docs with Swagger and Apivore

Ugh, documentation.

It's the afterthought of every system, scrambled together in the final days before launch, updated sparingly, generally out of date.

What if we could programmatically verify that our API documentation was accurate? What if this helped us build more intuitive APIs by putting our users first? What if documentation came first, and helped us write our code?

With Swagger and Apivore as our weapons of choice, we'll write documentation that will make your APIs better, your clients more satisfied, and you happier.

7b5a451ee25044b9c869e3e98b79425d?s=128

Ariel Caplan

April 19, 2018
Tweet

More Decks by Ariel Caplan

Other Decks in Technology

Transcript

  1. AUTOMATING EMPATHY Test Your Documentation with Swagger and Apivore

  2. EVERYONE HATES DOCUMENTATION

  3. “Projects often start with high ideals for documentation, but they

    always fall by the wayside in the crush of business. I observed this and said to myself, "Perhaps people don't maintain detailed documentation because it isn't actually a good idea." If it hurts running your head into a brick wall over and over, perhaps you should figure out how to get along without running your head into the brick wall. - Kent Beck https://accu.org/index.php/journals/509
  4. “Individuals and interactions over processes and tools Working software over

    comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan - The Agile Manifesto
  5. 1. IMPLEMENTATION DOCUMENTATION
 VS.
 USAGE DOCUMENTATION

  6. “I think that the original writers could have been more

    specific to remove some of the confusion and misinformation that has sprung up around this value. It might have been more appropriate if the original writers would have said, “Working software over comprehensive requirements and design documentation,” because I think this is more what they meant. -Larry “Agile Doctor” Apke http://www.agile-doctor.com/2016/08/16/agile-values-working-software-documentation/
  7. 2. IT’S NOT A BURDEN WHEN IT’S A TOOL FOR

    ITERATION
  8. 3. IF YOU CAME TO THIS TALK, YOU PROBABLY NEED

    DOCS
  9. WHY IS IT SO DIFFICULT TO PRODUCE ACCURATE DOCUMENTATION?

  10. “WE KEEP FORGETTING TO UPDATE THE DOCS. LET’S GET BETTER!”

  11. “OUR API IS CHANGING ALL THE TIME. HOW CAN WE

    KEEP UP?
  12. “I HATE UPDATING OUR DOCUMENTATION. IT’S SO ANNOYING!”

  13. “CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!”

  14. “CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!” COMPLEXITY

  15. WE HAVE A PROBLEM. GUILT WON’T SOLVE IT. MAYBE WE

    CAN FIX IT IF WE MAKE IT EASIER.
  16. THE MORE WE NEED TO HOLD IN OUR HEADS TO

    PRODUCE DOCUMENTATION, THE WORSE IT WILL BE.
  17. WHAT IF WE DIDN’T NEED TO REMEMBER ANYTHING?

  18. YOU DON’T NEED TO REMEMBER ANYTHING IF YOU WRITE THE

    DOCUMENTATION FIRST!
  19. WE CAN WRITE PERFECT DOCUMENTATION IF A TEST FAILS UNTIL

    THE CODE MATCHES THE DOCS
  20. DOCUMENTATION-DRIVEN DEVELOPMENT ➤ For a new endpoint: ➤ Document the

    endpoint ➤ The test suite complains that your documentation isn’t fully tested ➤ Write a test for the endpoint ➤ Write just enough code to make the test pass ➤ To change an endpoint: ➤ Update the documentation ➤ The test fails ➤ Write just enough code to make the test pass
  21. USER-DRIVEN DOCUMENTATION UPDATES ➤ For a new endpoint: ➤ Make

    the endpoint ➤ Document it (unless you forget) ➤ Fix the documentation when a user complains about your API being broken ➤ To change an endpoint: ➤ Update the endpoint ➤ Probably forget to document it ➤ Fix the documentation when a user complains about your API being broken
  22. @amcaplan amcaplan.ninja Hi! My name is Ariel Caplan

  23. @amcaplan amcaplan.ninja

  24. @amcaplan amcaplan.ninja Dev Empathy
 Book Club devempathybook.club

  25. None
  26. “ The only thing holding Ruby together was a hipster

    coder community of twenty-something year old nerds who are now thirty-something nerds. -Stefan Mischook, killerphp.com
  27. I’M STILL A TWENTY-SOMETHING NERD!

  28. HOW DO WE GET STARTED WITH DOCUMENTATION-DRIVEN DEVELOPMENT?

  29. STEP 2: CREATE DOCUMENTATION THAT COMPUTERS CAN READ

  30. None
  31. None
  32. THE ZEN OF SWAGGER

  33. /packages/{id} POST /packages THE ZEN OF SWAGGER 201 422 200

    200 404 200 422 200 410 GET GET DELETE PATCH
  34. THE ZEN OF SWAGGER 201 422 /packages POST What information

    needs to be submitted so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails?
  35. parameters: - name: body in: body required: true description: Package

    to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg { "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1,
 "weight": 3.3 }
  36. parameters: - name: body in: body required: true description: Package

    to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg
  37. THE ZEN OF SWAGGER What information needs to be submitted

    so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails? parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
  38. /packages: post: summary: Create a Package operationId: createPackage tags: -

    Packages parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' THE ZEN OF SWAGGER 201 422 /packages POST parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
  39. THE ZEN OF SWAGGER

  40. THE ZEN OF SWAGGER

  41. THE ZEN OF SWAGGER

  42. STEP 3: TESTING YOUR DOCUMENTATION

  43. None
  44. SETTING UP APIVORE ➤ Include the apivore gem ➤ Tell

    Apivore::SwaggerChecker how to find your docs ➤ Write a test to assert all endpoints/statuses are tested ➤ For each endpoint/status: ➤ Set up context (like in any other RSpec test) ➤ Tell Apivore::SwaggerChecker: ➤ Which request to make ➤ With what params ➤ Which status code to expect
  45. INITIAL BOILERPLATE require 'spec_helper' RSpec.describe 'the API', type: :apivore, order:

    :defined do subject { Apivore::SwaggerChecker.instance_for('/swagger.json') } context 'has valid paths' do # tests go here end context 'and' do it 'tests all documented routes' do expect(subject).to validate_all_paths end end end Where is the documentation? Assert all endpoints/statuses are tested
  46. BACKFILLING AN API $ rspec F Failures: 1) the API

    and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 201 post /packages is untested for response code 422 # ./spec/api/api_spec.rb:12:in `block (3 levels) in <top (required)>' Finished in 0.0929 seconds (files took 4.76 seconds to load) 1 example, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:11 # the API and tests all documented routes
  47. A SAMPLE HAPPY-PATH TEST let(:params) {{ "_data" => { "destination_id"

    => 114, "length" => 14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 201, params) }
  48. A SAMPLE HAPPY-PATH TEST $ rspec .F Failures: 1) the

    API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 422 # ./spec/api/api_spec.rb:22:in `block (3 levels) in <top (required)>' Finished in 0.09739 seconds (files took 2.02 seconds to load) 2 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:21 # the API and tests all documented routes
  49. A SAMPLE FAILURE-PATH TEST let(:params) {{ "_data" => { "destination_id"

    => 114, "length" => -14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 422, params) }
  50. A SAMPLE FAILURE-PATH TEST $ rspec ... Finished in 0.09168

    seconds (files took 1.96 seconds to load) 3 examples, 0 failures
  51. CREATING A NEW ENDPOINT /packages/{id}: patch: summary: Update a Package

    operationId: updatePackage tags: - Packages parameters: - name: body in: body required: true description: Updated Package information schema: $ref: '#/definitions/PackageUpdateModel' responses: '200': description: Package successfully updated schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'
  52. CREATING A NEW ENDPOINT /packages/{id}: patch: summary: Update a Package

    operationId: updatePackage tags: - Packages parameters: - name: body in: body required: true description: Updated Package information schema: $ref: '#/definitions/PackageUpdateModel' responses: '200': description: Package successfully updated schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' PackageUpdateModel: properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg
  53. CREATING A NEW ENDPOINT $ rspec ..F Failures: 1) the

    API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths patch /packages is untested for response code 200 patch /packages is untested for response code 422 # ./spec/api/api_spec.rb:36:in `block (3 levels) in <top (required)>' Finished in 0.11711 seconds (files took 1.94 seconds to load) 3 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:35 # the API and tests all documented routes
  54. CREATING A NEW ENDPOINT let(:package) { Package.create!( destination_id: 114, length:

    14.7, width: 12.2, height: 2.1, weight: 3.3 ) }
  55. CREATING A NEW ENDPOINT context 'happy' do let(:params) {{ "id"

    => package.id, "_data" => { "length" => 8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 200, params) } end context 'sad' do let(:params) {{ "id" => package.id, "_data" => { "length" => -8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 422, params) } end
  56. AFTER IMPLEMENTING…

  57. CREATING A NEW ENDPOINT $ rspec ..... Finished in 0.1321

    seconds (files took 2.06 seconds to load) 5 examples, 0 failures
  58. UPDATING THE API PackageModel: required: [destination_id, length, width, height] properties:

    destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg
  59. PackageModel: required: [destination_id, length, width, height, volume] properties: destination_id: type:

    integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg UPDATING THE API volume: type: number format: float description: Volume of package in cm3
  60. UPDATING THE API Failures: 1) the API has valid paths

    post /packages happy should validate that post / packages returns 201 Failure/Error: it { is_expected.to validate(:post, "/packages", 201, params) } '/packages#/' did not contain a required property of 'volume' Response body: { "id": 24, "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1, "weight": 3.3, "created_at": "2018-04-15T22:54:37.166Z", "updated_at": "2018-04-15T22:54:37.166Z" } # ./spec/api/api_spec.rb:17:in `block (5 levels) in <top (required)>'
  61. CAVEAT TIME!

  62. None
  63. None
  64. None
  65. None
  66. IT’S NOT PERFECT. IT’S STILL AN IMPROVEMENT.

  67. STEP 1: CONVINCING YOUR MANAGER YOU NEED DOCUMENTATION TESTING

  68. LET’S TRAVEL BACK IN TIME…

  69. HOW WE STARTED ON SWAGGER AND APIVORE

  70. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS

    (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails
  71. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS

    (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team
  72. HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS

    (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week
  73. HOW WE STARTED ON SWAGGER AND APIVORE May 2016 I

    join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests
  74. HOW WE STARTED ON SWAGGER AND APIVORE June 2016 Decided

    to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests August 18, 2016 Finished writing Apivore documentation tests
  75. FAST FACTS ➤ 29 days ➤ 18 pull requests ➤

    +1,835 LOC ➤ - 2,734 LOC
  76. WHAT TOOK SO LONG?

  77. DOCUMENTATION MISTAKES

  78. UNNECESSARILY STUFFED API Why return timestamps
 on models that never

    change?
  79. UNCONVENTIONAL STATUS CODES Create actions
 should return 201,
 not 200

    DELETE to a
 nonexistent resource
 should be 200 or 410,
 not 404
  80. OVERCOMPLICATED ROUTES We don’t need all of
 /doctors/:doctor_id/reviews/:id
 to know

    which Review is being requested
  81. CONFUSING DOMAIN MODEL Are individual Answers an
 HTTP resource
 or

    an attribute of a Review?
  82. MIXING DATABASE CONCERNS INTO DOMAIN MODEL Flags, Likes, and Dislikes


    are a single DB concept using STI.
 Why should the user know that?
  83. INAPPROPRIATE BEHAVIOR ID of a reviewed doctor/facility
 isn’t a required

    field on a Review.
 Does that make sense?
  84. CLIENT DATA EXPOSURE We seem to be returning Reviews
 belonging

    to all clients,
 not just the current one.
  85. DATA CORRUPTION The error case where
 required questions aren’t answered


    returns a 201 and saves the Review.
  86. INSUFFICIENT LIMITS ON PERMISSIONS We expose API endpoints
 to edit

    system-level data.
 That’s not right!
  87. DOCUMENTATION TESTING HELPED US UNCOVER: ➤ Documentation Mistakes ➤ Unnecessarily

    Stuffed API ➤ Overcomplicated Routes ➤ Confusing Domain Model ➤ Mixing Database Concerns Into Domain Model ➤ Inappropriate Behavior ➤ Client Data Exposure ➤ Data Corruption ➤ Insufficient Limits on Permissions
  88. DOCUMENTATION TESTING HELPED US UNCOVER: ➤ Documentation Mistakes ➤ Unnecessarily

    Stuffed API ➤ Overcomplicated Routes ➤ Confusing Domain Model ➤ Mixing Database Concerns Into Domain Model ➤ Inappropriate Behavior ➤ Client Data Exposure ➤ Data Corruption ➤ Insufficient Limits on Permissions Hey Manager, I saw a talk about how documentation testing can save you from a laundry list of problems. Maybe we can try it! That sounds great!
  89. THIS WAS A TALENTED TEAM OF SEASONED DEVELOPERS WHO MADE

    EMBARRASSING MISTAKES
  90. THE CODE WAS GREAT. THE DESIGN WASN’T (ALWAYS).

  91. DOCUMENTATION TESTING FORCED US TO THINK ABOUT THE API FROM

    THE PERSPECTIVE OF A DOCUMENTATION USER
  92. WHEN DESIGNING NEW THINGS, WE START BY DEFINING THE USER

    IMPACT
  93. THE TAKEAWAY

  94. None
  95. “From the perspective of a user,
 if a feature is

    not documented,
 then it doesn't exist,
 and if a feature is documented incorrectly,
 then it's broken. - Zach Supalla, “Documentation-Driven Development” https://gist.github.com/zsup/9434452
  96. ACCORDING TO USERS, DOCUMENTATION IS THE PRIMARY SOURCE OF TRUTH.

    LET’S EMBRACE THAT.
  97. THANK YOU! @amcaplan amcaplan.ninja/talks devempathybook.club vitals.com