Web Development with OpenAPI Spec & Fast Testing

Web Development with OpenAPI Spec & Fast Testing

Modern web applications are mostly built on API and client side apps. Centralizing the business logic in API can prevent clients duplicating features on their sides.

However, in certain situations, we might face a tight schedule, work with clients who use waterfall workflow, or work with clients who are very sensitive about any unexpected behaviors.

This talk will cover how we utilize OpenAPI Spec for cross-team communication, keep the development speed by using slow but steady integration tests, and how we speed up those tests.

This talk was originally given at WebHack Meetup, a software engineering meetup in Tokyo. https://webhack.connpass.com/event/121030/

I gave this talk as a web engineer at Moneytree KK, which provides financial data platform for general customers, financial institutions and businesses. Check Moneytree's careers page for current open positions. https://www.moneytree.jp/careers

36b1f565fc83d9b67588123f2171b896?s=128

Yu-Cheng Chuang

February 27, 2019
Tweet

Transcript

  1. 2019 © Moneytree KK. Web Development with OpenAPI Spec &

    Fast Testing Yu-Cheng Chuang Feb 27, 2019 Slides will be available online
  2. @yorkxin @chitsaou Yu-Cheng Chuang
 Core Systems Engineer 
 at Moneytree

    From Taiwan BSD / Linux User since 2005 Web Developer since 2009 #typography #railway #cooking About me
  3. Moneytree is a financial data portability platform, comprised of a

    personal financial management app, and an API (Application Programming Interface) feed of customer data for financial institutions. Moneytree in Brief "
  4. Project Background

  5. • Backend server built on Ruby on Rails • Server

    is running on AWS ECS • User management on AWS Cognito • CircleCI / AWS Code Pipeline • Web App built on React • iOS, Android Apps The Stack Project Background
  6. # Customer™ $ Backend Developers % $ % Web Developers

    $ % $ % Mobile Developers $ % $ % & Project Managers '
  7. How do we develop individually and keep everything on track?

    ( ( (
  8. Actually I’ve started writing OpenAPI Spec! ' '

  9. Yes, our Project Managers can write Open API Spec!

  10. Use Machine-friendly API Spec as the Single Source of Truth

  11. ) pet-store-api-spec.git pet-store-server.git pet-store-web.git pet-store-mobile.git * + , % %

    % '%
  12. • Allows you to design / define HTTP API in

    YAML or JSON • Request & Responses can be defined as Schema • Using JSON Schema-like DSL Open API Spec (f.k.a. Swagger) Open API Spec
  13. paths: /pets: get: summary: List all pets operationId: listPets tags:

    - pets parameters: - name: limit in: query description: How many items to return at one time (max 100) required: false schema: type: integer format: int32 responses: '200': description: A paged array of pets headers: x-next: description: A link to the next page of responses schema: type: string content: application/json: schema: $ref: "#/components/schemas/Pets" default: description: unexpected error content: application/json: schema: $ref: "#/components/schemas/Error" components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true Pets: type: array items: $ref: "#/components/schemas/Pet" Error: required: - code - message properties: code: type: integer format: int32 message: type: string - - - ! ! !
  14. None
  15. • A language to define the data structure of JSON

    JSON Schema Open API Spec
  16. { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product

    from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } { "productId": 1, "productName": "A green door", "price": 12.50, "tags": [ "home", "green" ] } https:/ /json-schema.org/learn/getting-started-step-by-step.html
  17. { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product

    from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true ᶃ ᶃ ᶄ ᶄ OpenAPI Schema vs. JSON Schema
  18. https:/ /swagger.io/docs/specification/data-models/keywords/

  19. https:/ /philsturgeon.uk/api/2018/03/30/openapi-and-json-schema-divergence/ (c) 2018 Phil Sturgeon

  20. npm install openapi2schema Converting to JSON Schema

  21. ⋊> ~/D/pet-store openapi2schema -i pet-store.openapi.yml | jq { "/pets": {

    "get": { ...}, "post": {…} }, "/pets/{petId}": { "get": { "responses": { "200": { "type": "array", "items": { "required": [“id", "name"], "properties": { "id": { "type": "integer", "format": "int64", "minimum": -9223372036854776000, "maximum": 9223372036854776000 }, "name": { "type": "string" }, "tag": { "type": [ "string", "null" ] } } }, "$schema": "http://json-schema.org/draft-04/schema#" }, "default": {...} } } } } components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true
  22. • YAML / JSON = Machine-friendly • Schema is defined

    using JSON Schema (*partial compatible) Pros Open API Spec
  23. • Most libraries only support version 2 • See https:/

    /openapi.tools/ • To validate data structure, you need to
 convert Schema definitions to JSON Schema manually Cons Open API Spec
  24. Now I have schema definition for each endpoints…

  25. How do we ensure that API server implementation matches the

    spec? (
  26. Test API responses With Schema Definitions

  27. gem install json-schema Validating JSON Schema

  28. JSON::Validator.fully_validate(schema, hash, strict: true)

  29. def load_schema_file(path) # Converts to JSON Schema with a Node

    CLI tool JSON.parse(`npx openapi2schema -i #{path}`) end ※ Don’t do this in production (only in tests) Since we’re using Ruby…
  30. 3) GET /pets/{id} Should be successful matches response schema (success)

    Failure/Error: expect(response.parsed_body).to match_json_schema(schema) The property '#/pet/name' of type integer did not match the following type: string in schema eb4470d8-8767-56a6-83b7-789839be31e0 Shared Example Group: "Should conform OAS spec" called from ./spec/requests/ shared_examples/response_examples.rb:2
  31. • Seed data using test factories or database dumps •

    Test output with response schema from API Spec • Client developers can start working 
 with fake data and real response structure • Backend developers can start working on 
 JSON serializers Stub Server Extra Benefits
  32. Authoring Open API Spec

  33. • Swagger Viewer • openapi-lint Tools in VSCode Authoring API

    Specs
  34. None
  35. None
  36. None
  37. Test the API Spec!

  38. npm install swagger-cli Test the API spec

  39. ⋊> ~/D/pet-store find . -name *.yml -print0 | xargs -n

    1 -0 swagger-cli validate Swagger schema validation failed. Additional properties not allowed: responses at #/paths//pets JSON_OBJECT_VALIDATION_FAILED
  40. Now it’s time to implement endpoints…

  41. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake routes |

    wc -l 90
  42. Can I prevent future regressions? (

  43. Request Tests With Detailed Scenarios

  44. • Test scenarios are close to business cases • Think

    from Client Developer’s and User’s perspective • Useful for Regression Tests • Ensuring all responses to match API Spec • So that Client developers will be happy every day Pros Request Tests
  45. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake stats +----------------------+--------+--------+---------+---------+-----+-------+

    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+--------+--------+---------+---------+-----+-------+ | Controllers | 1489 | 1151 | 42 | 145 | 3 | 5 | | Helpers | 11 | 10 | 0 | 1 | 0 | 8 | | Jobs | 2 | 2 | 1 | 0 | 0 | 0 | | Models | 2822 | 1781 | 33 | 155 | 4 | 9 | | Mailers | 135 | 95 | 4 | 5 | 1 | 17 | | Channels | 8 | 8 | 2 | 0 | 0 | 0 | | JavaScripts | 1 | 0 | 0 | 0 | 0 | 0 | | Libraries | 155 | 119 | 4 | 8 | 2 | 12 | | Controller tests | 42 | 36 | 1 | 5 | 5 | 5 | | Model tests | 690 | 481 | 4 | 38 | 9 | 10 | | Mailer tests | 0 | 0 | 0 | 0 | 0 | 0 | | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 | | Model specs | 1040 | 834 | 0 | 0 | 0 | 0 | | Presenter specs | 637 | 528 | 4 | 7 | 1 | 73 | | Request specs | 11377 | 9330 | 0 | 78 | 0 | 117 | | Searcher specs | 361 | 298 | 0 | 0 | 0 | 0 | | Lib specs | 1197 | 994 | 0 | 0 | 0 | 0 | | Route specs | 43 | 38 | 0 | 0 | 0 | 0 | +----------------------+--------+--------+---------+---------+-----+-------+ | Total | 20010 | 15705 | 95 | 442 | 4 | 33 | +----------------------+--------+--------+---------+---------+-----+-------+ Code LOC: 3166 Test LOC: 12539 Code to Test Ratio: 1:4.0
  46. (Coverage Report from Simplecov)

  47. • Hard to debug due to deep stacks • Long

    Roundtrip → Slow • Very Slow ./0 Cons Request Tests
  48. Run tests In Parallel

  49. None
  50. 1 ⚙ ⚙ ⚙ ʊਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹ̴̧̰̲ɹ̸̸̘̳̾̽̓̾̽ɹʻ ʉ:?:?:?:?:?:?:?:?:?:?:?:?:?:?:ʉ

  51. 1 ⚙ 1 ⚙ 1 ⚙

  52. None
  53. • It assumes each job has same load • You

    need to write your own script to 
 distribute test files evenly by execution time or size Using GNU Parallel Run Test in Parallel
  54. Load Balancing Test Files

  55. Time 0 sec 500 sec 1,000 sec 1,500 sec 2,000

    sec 2,500 sec ~1600s (70%) ~650s (30%) Bottleneck ~250s
  56. • One endpoint per test file • It’s also easier

    for human to read • Avoid Race Condition — Reset Database for each case • Use as less requests for one scenario as possible • Data preparations (db insertions) are usually fast… • …Unless you’re inserting hundreds of rows Load Balancing Test Files Run Test in Parallel
  57. Took 441 seconds (7:21) real 7m21.858s user 26m6.616s sys 0m43.592s

    Took 278 seconds (4:38) real 4m39.159s user 27m46.976s sys 2m3.372s Took 439 seconds (7:19) real 7m19.914s user 28m23.356s sys 5m35.944s ⚙ ⚙ ⚙ × 4 × 8 × 16 ✅
  58. 8 processes might be faster than 16 • /proc/info may

    != # of CPU Cores • Be aware of startup time overhead (Rails is slow) • Find best number by measuring time • One CI machine might be enough Deciding # of Processes Speed up total test runtime
  59. Mistakes…

  60. 0 3000 6000 9000 12000 0 ms 120,000 ms 240,000

    ms 360,000 ms 480,000 ms 600,000 ms 720,000 ms Past ← Time → Now Total Runtime (ms) Number of Cases Time / Cases Ratio No junit.xml Reduced requests & Split into 2 machines Start using GNU Parallel 4 processes on 1 machine Found some files not tested Switched to Ruby parallel_tests Started 8 Processes Started testing JWT
  61. • Find an existing parallel testing solution 
 before you

    rollout your own one • Store test reports from day 1, if your CI supports it Lessons Parallel Testing
  62. AWS Cognito / JWT

  63. “Amazon Cognito User Pools provide a secure user directory that

    scales to hundreds of millions of users. As a fully managed service, User Pools are easy to set up without any worries about server infrastructure. User Pools provide user profiles and authentication tokens for users who sign up directly and for federated users who sign in with social and enterprise identity providers.” AWS Cognito User Pools AWS Cognito https:/ /aws.amazon.com/cognito/details/
  64. https:/ /docs.aws.amazon.com/cognito/latest/developerguide/cognito-scenarios.html OAuth 2.0 + Bearer Token (JWT)

  65. Use JWT to Remove Dependency on Auth Server

  66. • A Stateless, Cryptography-Based Token Standard JSON Web Token (RFC

    7519) JWT
  67. None
  68. Authorization: Bearer 11111111.22222222.33333333 Header Signature Payload

  69. • HMAC + SHA256 (Shared Secrets) • RSA / ECDSA

    (Asymmetric Signing) Algorithms JSON Web Token
  70. 4 4 JWT Issuer 5 6 API Server Signed by

    Private Key Verified by Public Key
  71. { "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", // Unique user 'sub' in Cognito User

    Pool "aud": "xxxxxxxxxxxxexample", // OAuth Client ID "email_verified": true, "token_use": "id", "auth_time": 1500009400, "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_example", // Token Issuer "cognito:username": "janedoe", "exp": 1500013000, // Token Valid Until "given_name": "Jane", "iat": 1500009400, "email": "janedoe@example.com" } A typical JWT Claim from AWS Cognito
  72. • The Customer™ trusts on AWS’s Security • Real Stateless

    Authentication • No (hashed) password stored in our database • Client App pinning (checking aud) • Easy to find user in database by inspecting the token Pros JSON Web Token
  73. • Token Revocation, obviously • Need to store JWKs (Public

    Keys) in every server daemon Cons JSON Web Token
  74. Just do what Cognito is doing • Generate an RSA

    Key Pair for Test Environment • Store the JWK of Public Key • Sign the JWT Token with Private Key • Verify the JWT with JWK from Public Key Mocking JWT Verification in Tests JSON Web Token
  75. npm install pem-jwk Converting RSA PEM to JWK

  76. ⋊> ~/ openssl genrsa 2048 > test.pem Generating RSA private

    key, 2048 bit long modulus ................................................................+++ ................................................+++ e is 65537 (0x10001) ⋊> ~/ npx pem-jwk test.pem {“kty”:”RSA”,”n”:”m3Knu8N5…”,”e”:”AQAB”,”d”:”X69q6lwo…”,"p":"zrFBKps h...","q":"wIfj...","dp":"f9WHPWe...","qi":"o_O3bEU..."}
  77. Black Box Integration Tests

  78. • Writing tests in the same framework may have blind

    spot • Incorrectly assuming Accept: application/json • Sending payload without serializing into JSON • Passing parameters in wrong data type • To make sure your service runs in Docker Why Black Box Integration Tests? Black Box Integration Tests
  79. • Docker Build • Docker Run • Database Migrations •

    PoC of an API Client using OpenAPI Spec • Typical scenarios (sign-up, create data, etc.) • Edge case scenarios that involve multiple actions • Job Workers What to test? Black Box Integration Tests
  80. Run your server in Docker and Write HTTP Client in

    Another Language
  81. SQL Database 1 Redis Cache 1 API Server 7 Job

    Worker ⚙ Fake Clients (Node.js) 89 Docker Compose
  82. 4 4 My Test Script 5 6 API Server Sign

    Token by my Private Key Verify Token by my Public Key
  83. npm install openapi-client-axios Blackbox Testing

  84. import OpenAPIClientAxios from 'openapi-client-axios'; import * as fs from 'fs';

    import * as jwt from 'jsonwebtoken'; import * as jsYaml from 'js-yaml'; export class MyAppClient extends OpenAPIClientAxios { constructor(host = 'localhost:3000') { const url = `http://${host}/`; const definition = jsYaml.safeLoad(fs.readFileSync('/path/to/api/spec.openapi.yml')); definition.servers = [{ url }]; super({ definition }); } async showPetById(id) { return await this.client.query('showPetById', id); } // Sign a JWT Token loginAs(sub, email = null) { const options = { algorithm: 'RS256', issuer: 'test-issuer', audience: 'test-audience' }; const cert = fs.readFileSync('/path/to/jwk.pem'); this.token = jwt.sign({ email, sub }, cert, options); } }
  85. async init() { await super.init(); // NOTE: workaround https://github.com/axios/axios/pull/1395 this.client.interceptors.request.use((request)

    => { request.headers.Authorization = `Token ${this.token}`; return request; }); } Workaround Axios per-instance header issue
  86. const client = new MyAppClient(); await client.init(); client.loginAs('john-appleseed', 'john@appleseed.com'); await

    client.showPetById(42);
  87. version: '3' services: app: build: . ports: - "3000:3000" env_file:

    - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] db: image: "postgres:9" ports: - "15432:5432" environment: POSTGRES_PASSWORD: "password" redis: image: "redis:5" ports: - "16379:6379" • Expose ports for dockerize heath checking
  88. build: machine: true steps: - checkout - run: name: Install

    Docker Compose command: | curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`una chmod +x ~/docker-compose sudo mv ~/docker-compose /usr/local/bin/docker-compose • Run CircleCI task outside a container so that we can mount local files • Install docker-compose if necessary
  89. - run: name: Build image and pull dependencies command: docker-compose

    up --no-start - run: name: Run Containers background: true command: docker-compose up - run: name: Wait for containers command: | dockerize -wait tcp://localhost:15432 \ -wait tcp://localhost:16379 \ -wait tcp://localhost:3000 • Run Docker build & pull in an individual task • Start containers in background after build & pull were done • Wait until containers ready before running anything else (some RDB takes few seconds to initialize)
  90. - run: name: Test - Database preparation command: | docker-compose

    exec app rake \ db:create \ db:migrate \ db:seed • Prepare database by migration instead of loading schema
  91. How do we test features that depend on time? (

  92. github.com/wolfcw/libfaketime • A dynamic linking library to modify date() system

    call • Download a binary from Debian APT Repository • Or build it on a Linux you use for server container • A macOS dylib won’t work libfaketime.so Mocking System Time
  93. SQL Database 1 Redis Cache 1 API Server 7 Job

    Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7
  94. # An API server that shifts machine time by +2d

    app-future: build: . ports: - “3001:3000” environment: LD_PRELOAD: /mnt/libfaketime/libfaketime.so.1 FAKETIME: +2d volumes: - ./path/to/libfaketime:/mnt/libfaketime env_file: - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] 1) Export to another host port 2) Load libfaketime.so for API server 3) Set Shifts 4) Mount faketime binaries into container ! ᶃ " ᶄ : ᶅ : ᶆ
  95. How about email content? (

  96. MailHog or MailDev

  97. SQL Database 1 Redis Cache 1 API Server 7 Job

    Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7 MailHog ; Scrape Emails through its API
  98. Conclusion

  99. • Write as detailed test cases as possible • Use

    them as regression tests • Run tests in parallel to reduce total execution time • Test Docker Images to prevent shipping broken image Your Time is < < < Summary
  100. 2019 © Moneytree KK. We’re hiring! visit moneytree.jp/careers/