Automating Empathy: Test Your Docs with Swagger and Apivore

Slide 1

Slide 1 text

AUTOMATING EMPATHY Test Your Documentation with Swagger and Apivore

Slide 2

Slide 2 text

EVERYONE HATES DOCUMENTATION

Slide 3

Slide 3 text

“Projects often start with high ideals for documentation, but they always fall by the wayside in the crush of business. I observed this and said to myself, "Perhaps people don't maintain detailed documentation because it isn't actually a good idea." If it hurts running your head into a brick wall over and over, perhaps you should ﬁgure out how to get along without running your head into the brick wall. - Kent Beck https://accu.org/index.php/journals/509

Slide 4

Slide 4 text

“Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan - The Agile Manifesto

Slide 5

Slide 5 text

1. IMPLEMENTATION DOCUMENTATION  VS.  USAGE DOCUMENTATION

Slide 6

Slide 6 text

“I think that the original writers could have been more speciﬁc to remove some of the confusion and misinformation that has sprung up around this value. It might have been more appropriate if the original writers would have said, “Working software over comprehensive requirements and design documentation,” because I think this is more what they meant. -Larry “Agile Doctor” Apke http://www.agile-doctor.com/2016/08/16/agile-values-working-software-documentation/

Slide 7

Slide 7 text

2. IT’S NOT A BURDEN WHEN IT’S A TOOL FOR ITERATION

Slide 8

Slide 8 text

3. IF YOU CAME TO THIS TALK, YOU PROBABLY NEED DOCS

Slide 9

Slide 9 text

WHY IS IT SO DIFFICULT TO PRODUCE ACCURATE DOCUMENTATION?

Slide 10

Slide 10 text

“WE KEEP FORGETTING TO UPDATE THE DOCS. LET’S GET BETTER!”

Slide 11

Slide 11 text

“OUR API IS CHANGING ALL THE TIME. HOW CAN WE KEEP UP?

Slide 12

Slide 12 text

“I HATE UPDATING OUR DOCUMENTATION. IT’S SO ANNOYING!”

Slide 13

Slide 13 text

“CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!”

Slide 14

Slide 14 text

“CLEARLY, YOU JUST DON’T CARE ABOUT YOUR USERS!” COMPLEXITY

Slide 15

Slide 15 text

WE HAVE A PROBLEM. GUILT WON’T SOLVE IT. MAYBE WE CAN FIX IT IF WE MAKE IT EASIER.

Slide 16

Slide 16 text

THE MORE WE NEED TO HOLD IN OUR HEADS TO PRODUCE DOCUMENTATION, THE WORSE IT WILL BE.

Slide 17

Slide 17 text

WHAT IF WE DIDN’T NEED TO REMEMBER ANYTHING?

Slide 18

Slide 18 text

YOU DON’T NEED TO REMEMBER ANYTHING IF YOU WRITE THE DOCUMENTATION FIRST!

Slide 19

Slide 19 text

WE CAN WRITE PERFECT DOCUMENTATION IF A TEST FAILS UNTIL THE CODE MATCHES THE DOCS

Slide 20

Slide 20 text

DOCUMENTATION-DRIVEN DEVELOPMENT ➤ For a new endpoint: ➤ Document the endpoint ➤ The test suite complains that your documentation isn’t fully tested ➤ Write a test for the endpoint ➤ Write just enough code to make the test pass ➤ To change an endpoint: ➤ Update the documentation ➤ The test fails ➤ Write just enough code to make the test pass

Slide 21

Slide 21 text

USER-DRIVEN DOCUMENTATION UPDATES ➤ For a new endpoint: ➤ Make the endpoint ➤ Document it (unless you forget) ➤ Fix the documentation when a user complains about your API being broken ➤ To change an endpoint: ➤ Update the endpoint ➤ Probably forget to document it ➤ Fix the documentation when a user complains about your API being broken

Slide 22

Slide 22 text

@amcaplan amcaplan.ninja Hi! My name is Ariel Caplan

Slide 23

Slide 23 text

@amcaplan amcaplan.ninja

Slide 24

Slide 24 text

@amcaplan amcaplan.ninja Dev Empathy  Book Club devempathybook.club

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

“ The only thing holding Ruby together was a hipster coder community of twenty-something year old nerds who are now thirty-something nerds. -Stefan Mischook, killerphp.com

Slide 27

Slide 27 text

I’M STILL A TWENTY-SOMETHING NERD!

Slide 28

Slide 28 text

HOW DO WE GET STARTED WITH DOCUMENTATION-DRIVEN DEVELOPMENT?

Slide 29

Slide 29 text

STEP 2: CREATE DOCUMENTATION THAT COMPUTERS CAN READ

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

THE ZEN OF SWAGGER

Slide 33

Slide 33 text

/packages/{id} POST /packages THE ZEN OF SWAGGER 201 422 200 200 404 200 422 200 410 GET GET DELETE PATCH

Slide 34

Slide 34 text

THE ZEN OF SWAGGER 201 422 /packages POST What information needs to be submitted so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails?

Slide 35

Slide 35 text

parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg { "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1,  "weight": 3.3 }

Slide 36

Slide 36 text

parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' THE ZEN OF SWAGGER PackageModel: required: [destination_id, length, width, height] properties: - destination_id: type: integer format: int64 description: Canonical ID of the package destination - length: type: number format: float description: Length of package in cm - width: type: number format: float description: Width of package in cm - height: type: number format: float description: Height of package in cm - weight: type: number format: float description: Weight of package in kg

Slide 37

Slide 37 text

THE ZEN OF SWAGGER What information needs to be submitted so the API knows what to do? What will the API tell me when the request succeeds? What will the API tell me when the request fails? parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'

Slide 38

Slide 38 text

/packages: post: summary: Create a Package operationId: createPackage tags: - Packages parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' THE ZEN OF SWAGGER 201 422 /packages POST parameters: - name: body in: body required: true description: Package to insert into the system schema: $ref: '#/definitions/PackageModel' responses: '201': description: Package successfully created schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel'

Slide 39

Slide 39 text

THE ZEN OF SWAGGER

Slide 40

Slide 40 text

THE ZEN OF SWAGGER

Slide 41

Slide 41 text

THE ZEN OF SWAGGER

Slide 42

Slide 42 text

STEP 3: TESTING YOUR DOCUMENTATION

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

SETTING UP APIVORE ➤ Include the apivore gem ➤ Tell Apivore::SwaggerChecker how to ﬁnd your docs ➤ Write a test to assert all endpoints/statuses are tested ➤ For each endpoint/status: ➤ Set up context (like in any other RSpec test) ➤ Tell Apivore::SwaggerChecker: ➤ Which request to make ➤ With what params ➤ Which status code to expect

Slide 45

Slide 45 text

INITIAL BOILERPLATE require 'spec_helper' RSpec.describe 'the API', type: :apivore, order: :defined do subject { Apivore::SwaggerChecker.instance_for('/swagger.json') } context 'has valid paths' do # tests go here end context 'and' do it 'tests all documented routes' do expect(subject).to validate_all_paths end end end Where is the documentation? Assert all endpoints/statuses are tested

Slide 46

Slide 46 text

BACKFILLING AN API $ rspec F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 201 post /packages is untested for response code 422 # ./spec/api/api_spec.rb:12:in `block (3 levels) in ' Finished in 0.0929 seconds (files took 4.76 seconds to load) 1 example, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:11 # the API and tests all documented routes

Slide 47

Slide 47 text

A SAMPLE HAPPY-PATH TEST let(:params) {{ "_data" => { "destination_id" => 114, "length" => 14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 201, params) }

Slide 48

Slide 48 text

A SAMPLE HAPPY-PATH TEST $ rspec .F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths post /packages is untested for response code 422 # ./spec/api/api_spec.rb:22:in `block (3 levels) in ' Finished in 0.09739 seconds (files took 2.02 seconds to load) 2 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:21 # the API and tests all documented routes

Slide 49

Slide 49 text

A SAMPLE FAILURE-PATH TEST let(:params) {{ "_data" => { "destination_id" => 114, "length" => -14.7, "width" => 12.2, "height" => 2.1, "weight" => 3.3 }}} it { is_expected.to validate(:post, "/packages", 422, params) }

Slide 50

Slide 50 text

A SAMPLE FAILURE-PATH TEST $ rspec ... Finished in 0.09168 seconds (files took 1.96 seconds to load) 3 examples, 0 failures

Slide 51

Slide 51 text

Slide 52

Slide 52 text

CREATING A NEW ENDPOINT /packages/{id}: patch: summary: Update a Package operationId: updatePackage tags: - Packages parameters: - name: body in: body required: true description: Updated Package information schema: $ref: '#/definitions/PackageUpdateModel' responses: '200': description: Package successfully updated schema: $ref: '#/definitions/PackageModel' '422': description: Invalid package input schema: $ref: '#/definitions/ErrorModel' PackageUpdateModel: properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg

Slide 53

Slide 53 text

CREATING A NEW ENDPOINT $ rspec ..F Failures: 1) the API and tests all documented routes Failure/Error: expect(subject).to validate_all_paths patch /packages is untested for response code 200 patch /packages is untested for response code 422 # ./spec/api/api_spec.rb:36:in `block (3 levels) in ' Finished in 0.11711 seconds (files took 1.94 seconds to load) 3 examples, 1 failure Failed examples: rspec ./spec/api/api_spec.rb:35 # the API and tests all documented routes

Slide 54

Slide 54 text

CREATING A NEW ENDPOINT let(:package) { Package.create!( destination_id: 114, length: 14.7, width: 12.2, height: 2.1, weight: 3.3 ) }

Slide 55

Slide 55 text

CREATING A NEW ENDPOINT context 'happy' do let(:params) {{ "id" => package.id, "_data" => { "length" => 8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 200, params) } end context 'sad' do let(:params) {{ "id" => package.id, "_data" => { "length" => -8.3 } }} it { is_expected.to validate(:patch, '/packages/{id}', 422, params) } end

Slide 56

Slide 56 text

AFTER IMPLEMENTING…

Slide 57

Slide 57 text

CREATING A NEW ENDPOINT $ rspec ..... Finished in 0.1321 seconds (files took 2.06 seconds to load) 5 examples, 0 failures

Slide 58

Slide 58 text

UPDATING THE API PackageModel: required: [destination_id, length, width, height] properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg

Slide 59

Slide 59 text

PackageModel: required: [destination_id, length, width, height, volume] properties: destination_id: type: integer format: int64 description: Canonical ID of the package destination length: type: number format: float description: Length of package in cm width: type: number format: float description: Width of package in cm height: type: number format: float description: Height of package in cm weight: type: number format: float description: Weight of package in kg UPDATING THE API volume: type: number format: float description: Volume of package in cm3

Slide 60

Slide 60 text

UPDATING THE API Failures: 1) the API has valid paths post /packages happy should validate that post / packages returns 201 Failure/Error: it { is_expected.to validate(:post, "/packages", 201, params) } '/packages#/' did not contain a required property of 'volume' Response body: { "id": 24, "destination_id": 114, "length": 14.7, "width": 12.2, "height": 2.1, "weight": 3.3, "created_at": "2018-04-15T22:54:37.166Z", "updated_at": "2018-04-15T22:54:37.166Z" } # ./spec/api/api_spec.rb:17:in `block (5 levels) in '

Slide 61

Slide 61 text

CAVEAT TIME!

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

❌

Slide 67

Slide 67 text

IT’S NOT PERFECT. IT’S STILL AN IMPROVEMENT.

Slide 68

Slide 68 text

STEP 1: CONVINCING YOUR MANAGER YOU NEED DOCUMENTATION TESTING

Slide 69

Slide 69 text

LET’S TRAVEL BACK IN TIME…

Slide 70

Slide 70 text

HOW WE STARTED ON SWAGGER AND APIVORE

Slide 71

Slide 71 text

HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails

Slide 72

Slide 72 text

HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team

Slide 73

Slide 73 text

HOW WE STARTED ON SWAGGER AND APIVORE October 2015 PRS (Patient Rating System) is rapidly rewritten PHP/Laravel ➡ Ruby/Rails May 2016 I join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week

Slide 74

Slide 74 text

HOW WE STARTED ON SWAGGER AND APIVORE May 2016 I join the PRS team June 2016 Decided to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests

Slide 75

Slide 75 text

HOW WE STARTED ON SWAGGER AND APIVORE June 2016 Decided to use Swagger, Implemented in 1 week July 20, 2016 Started writing Apivore documentation tests August 18, 2016 Finished writing Apivore documentation tests

Slide 76

Slide 76 text

FAST FACTS ➤ 29 days ➤ 18 pull requests ➤ +1,835 LOC ➤ - 2,734 LOC

Slide 77

Slide 77 text

WHAT TOOK SO LONG?

Slide 78

Slide 78 text

DOCUMENTATION MISTAKES

Slide 79

Slide 79 text

UNNECESSARILY STUFFED API Why return timestamps  on models that never change?

Slide 80

Slide 80 text

UNCONVENTIONAL STATUS CODES Create actions  should return 201,  not 200 DELETE to a  nonexistent resource  should be 200 or 410,  not 404

Slide 81

Slide 81 text

OVERCOMPLICATED ROUTES We don’t need all of  /doctors/:doctor_id/reviews/:id  to know which Review is being requested

Slide 82

Slide 82 text

CONFUSING DOMAIN MODEL Are individual Answers an  HTTP resource  or an attribute of a Review?

Slide 83

Slide 83 text

MIXING DATABASE CONCERNS INTO DOMAIN MODEL Flags, Likes, and Dislikes  are a single DB concept using STI.  Why should the user know that?

Slide 84

Slide 84 text

INAPPROPRIATE BEHAVIOR ID of a reviewed doctor/facility  isn’t a required field on a Review.  Does that make sense?

Slide 85

Slide 85 text

CLIENT DATA EXPOSURE We seem to be returning Reviews  belonging to all clients,  not just the current one.

Slide 86

Slide 86 text

DATA CORRUPTION The error case where  required questions aren’t answered  returns a 201 and saves the Review.

Slide 87

Slide 87 text

INSUFFICIENT LIMITS ON PERMISSIONS We expose API endpoints  to edit system-level data.  That’s not right!

Slide 88

Slide 88 text

Slide 89

Slide 89 text

DOCUMENTATION TESTING HELPED US UNCOVER: ➤ Documentation Mistakes ➤ Unnecessarily Stuﬀed API ➤ Overcomplicated Routes ➤ Confusing Domain Model ➤ Mixing Database Concerns Into Domain Model ➤ Inappropriate Behavior ➤ Client Data Exposure ➤ Data Corruption ➤ Insuﬃcient Limits on Permissions Hey Manager, I saw a talk about how documentation testing can save you from a laundry list of problems. Maybe we can try it! That sounds great!

Slide 90

Slide 90 text

THIS WAS A TALENTED TEAM OF SEASONED DEVELOPERS WHO MADE EMBARRASSING MISTAKES

Slide 91

Slide 91 text

THE CODE WAS GREAT. THE DESIGN WASN’T (ALWAYS).

Slide 92

Slide 92 text

DOCUMENTATION TESTING FORCED US TO THINK ABOUT THE API FROM THE PERSPECTIVE OF A DOCUMENTATION USER

Slide 93

Slide 93 text

WHEN DESIGNING NEW THINGS, WE START BY DEFINING THE USER IMPACT

Slide 94

Slide 94 text

THE TAKEAWAY

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

“From the perspective of a user,  if a feature is not documented,  then it doesn't exist,  and if a feature is documented incorrectly,  then it's broken. - Zach Supalla, “Documentation-Driven Development” https://gist.github.com/zsup/9434452

Slide 97

Slide 97 text

ACCORDING TO USERS, DOCUMENTATION IS THE PRIMARY SOURCE OF TRUTH. LET’S EMBRACE THAT.

Slide 98

Slide 98 text

THANK YOU! @amcaplan amcaplan.ninja/talks devempathybook.club vitals.com