Web Development with OpenAPI Spec & Fast Testing

Modern web applications are mostly built on API and client side apps. Centralizing the business logic in API can prevent clients duplicating features on their sides.

However, in certain situations, we might face a tight schedule, work with clients who use waterfall workflow, or work with clients who are very sensitive about any unexpected behaviors.

This talk will cover how we utilize OpenAPI Spec for cross-team communication, keep the development speed by using slow but steady integration tests, and how we speed up those tests.

This talk was originally given at WebHack Meetup, a software engineering meetup in Tokyo. https://webhack.connpass.com/event/121030/

I gave this talk as a web engineer at Moneytree KK, which provides financial data platform for general customers, financial institutions and businesses. Check Moneytree's careers page for current open positions. https://www.moneytree.jp/careers

Transcript

1. 2019 © Moneytree KK.
   Web Development with
   OpenAPI Spec & Fast Testing
   Yu-Cheng Chuang
   Feb 27, 2019
   Slides will be available online
   

   View Slide

2. @yorkxin
   @chitsaou
   Yu-Cheng Chuang

   Core Systems Engineer 

   at Moneytree
   From Taiwan
   BSD / Linux User since 2005
   Web Developer since 2009
   #typography #railway #cooking
   About me
   

   View Slide

3. Moneytree is a
   financial data
   portability platform,
   comprised of a
   personal financial
   management app,
   and an API
   (Application
   Programming
   Interface)
   feed of customer
   data for financial
   institutions.
   Moneytree
   in Brief
   "
   

   View Slide

4. Project Background
   

   View Slide

5. • Backend server built on Ruby on Rails
   • Server is running on AWS ECS
   • User management on AWS Cognito
   • CircleCI / AWS Code Pipeline
   • Web App built on React
   • iOS, Android Apps
   The Stack
   Project Background
   

   View Slide

6. #
   Customer™
   $
   Backend
   Developers
   %
   $
   %
   Web
   Developers
   $
   %
   $
   %
   Mobile
   Developers
   $
   %
   $
   %
   &
   Project
   Managers
   '
   

   View Slide

7. How do we develop individually and
   keep everything on track?
   ( ( (
   

   View Slide

8. Actually I’ve started writing OpenAPI Spec!
   '
   '
   

   View Slide

9. Yes, our Project Managers can write Open API Spec!
   

   View Slide

10. Use Machine-friendly API Spec as the
    Single Source of Truth
    

    View Slide

11. )
    pet-store-api-spec.git
    pet-store-server.git
    pet-store-web.git
    pet-store-mobile.git
    *
    +
    ,
    %
    %
    %
    '%
    

    View Slide

12. • Allows you to design / define HTTP API in YAML or
    JSON
    • Request & Responses can be defined as Schema
    • Using JSON Schema-like DSL
    Open API Spec (f.k.a. Swagger)
    Open API Spec
    

    View Slide

13. paths:
    /pets:
    get:
    summary: List all pets
    operationId: listPets
    tags:
    - pets
    parameters:
    - name: limit
    in: query
    description: How many items to return at one time (max 100)
    required: false
    schema:
    type: integer
    format: int32
    responses:
    '200':
    description: A paged array of pets
    headers:
    x-next:
    description: A link to the next page of responses
    schema:
    type: string
    content:
    application/json:
    schema:
    $ref: "#/components/schemas/Pets"
    default:
    description: unexpected error
    content:
    application/json:
    schema:
    $ref: "#/components/schemas/Error"
    components:
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true
    Pets:
    type: array
    items:
    $ref: "#/components/schemas/Pet"
    Error:
    required:
    - code
    - message
    properties:
    code:
    type: integer
    format: int32
    message:
    type: string
    -
    -
    -
    !
    !
    !
    

    View Slide

14. View Slide

15. • A language to define the data structure of JSON
    JSON Schema
    Open API Spec
    

    View Slide

16. {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "http://example.com/product.schema.json",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
    "productId": {
    "description": "The unique identifier for a product",
    "type": "integer"
    },
    "productName": {
    "description": "Name of the product",
    "type": "string"
    },
    "price": {
    "description": "The price of the product",
    "type": "number",
    "exclusiveMinimum": 0
    },
    "tags": {
    "description": "Tags for the product",
    "type": "array",
    "items": { "type": "string" },
    "minItems": 1,
    "uniqueItems": true
    }
    },
    "required": [ "productId", "productName", "price" ]
    }
    {
    "productId": 1,
    "productName": "A green door",
    "price": 12.50,
    "tags": [ "home", "green" ]
    }
    https:/
    /json-schema.org/learn/getting-started-step-by-step.html
    

    View Slide

17. {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "http://example.com/product.schema.json",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
    "productId": {
    "description": "The unique identifier for a product",
    "type": "integer"
    },
    "productName": {
    "description": "Name of the product",
    "type": "string"
    },
    "price": {
    "description": "The price of the product",
    "type": "number",
    "exclusiveMinimum": 0
    },
    "tags": {
    "description": "Tags for the product",
    "type": "array",
    "items": { "type": "string" },
    "minItems": 1,
    "uniqueItems": true
    }
    },
    "required": [ "productId", "productName", "price" ]
    }
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true
    ᶃ
    ᶃ
    ᶄ
    ᶄ
    OpenAPI Schema vs. JSON Schema
    

    View Slide

18. https:/
    /swagger.io/docs/specification/data-models/keywords/
    

    View Slide

19. https:/
    /philsturgeon.uk/api/2018/03/30/openapi-and-json-schema-divergence/
    (c) 2018 Phil Sturgeon
    

    View Slide

20. npm install openapi2schema
    Converting to JSON Schema
    

    View Slide

21. ⋊> ~/D/pet-store openapi2schema -i pet-store.openapi.yml | jq
    {
    "/pets": {
    "get": { ...},
    "post": {…}
    },
    "/pets/{petId}": {
    "get": {
    "responses": {
    "200": {
    "type": "array",
    "items": {
    "required": [“id", "name"],
    "properties": {
    "id": {
    "type": "integer",
    "format": "int64",
    "minimum": -9223372036854776000,
    "maximum": 9223372036854776000
    },
    "name": {
    "type": "string"
    },
    "tag": {
    "type": [
    "string",
    "null"
    ]
    }
    }
    },
    "$schema": "http://json-schema.org/draft-04/schema#"
    },
    "default": {...}
    }
    }
    }
    }
    components:
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true
    

    View Slide

22. • YAML / JSON = Machine-friendly
    • Schema is defined using JSON Schema (*partial compatible)
    Pros
    Open API Spec
    

    View Slide

23. • Most libraries only support version 2
    • See https:/
    /openapi.tools/
    • To validate data structure, you need to

    convert Schema definitions to JSON Schema manually
    Cons
    Open API Spec
    

    View Slide

24. Now I have schema definition for each endpoints…
    

    View Slide

25. How do we ensure that API server
    implementation matches the spec?
    (
    

    View Slide

26. Test API responses
    With Schema Definitions
    

    View Slide

27. gem install json-schema
    Validating JSON Schema
    

    View Slide

28. JSON::Validator.fully_validate(schema, hash, strict: true)
    

    View Slide

29. def load_schema_file(path)
    # Converts to JSON Schema with a Node CLI tool
    JSON.parse(`npx openapi2schema -i #{path}`)
    end
    ※ Don’t do this in production (only in tests)
    Since we’re using Ruby…
    

    View Slide

30. 3) GET /pets/{id} Should be successful matches response schema (success)
    Failure/Error: expect(response.parsed_body).to match_json_schema(schema)
    The property '#/pet/name' of type integer did not match the following
    type: string in schema eb4470d8-8767-56a6-83b7-789839be31e0
    Shared Example Group: "Should conform OAS spec" called from ./spec/requests/
    shared_examples/response_examples.rb:2
    

    View Slide

31. • Seed data using test factories or database dumps
    • Test output with response schema from API Spec
    • Client developers can start working 

    with fake data and real response structure
    • Backend developers can start working on 

    JSON serializers
    Stub Server
    Extra Benefits
    

    View Slide

32. Authoring Open API Spec
    

    View Slide

33. • Swagger Viewer
    • openapi-lint
    Tools in VSCode
    Authoring API Specs
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. Test the API Spec!
    

    View Slide

38. npm install swagger-cli
    Test the API spec
    

    View Slide

39. ⋊> ~/D/pet-store find . -name *.yml -print0 | xargs -n 1 -0 swagger-cli validate
    Swagger schema validation failed.
    Additional properties not allowed: responses at #/paths//pets
    JSON_OBJECT_VALIDATION_FAILED
    

    View Slide

40. Now it’s time to implement endpoints…
    

    View Slide

41. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake routes | wc -l
    90
    

    View Slide

42. Can I prevent future regressions?
    (
    

    View Slide

43. Request Tests
    With Detailed Scenarios
    

    View Slide

44. • Test scenarios are close to business cases
    • Think from Client Developer’s and User’s perspective
    • Useful for Regression Tests
    • Ensuring all responses to match API Spec
    • So that Client developers will be happy every day
    Pros
    Request Tests
    

    View Slide

45. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake stats
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Controllers | 1489 | 1151 | 42 | 145 | 3 | 5 |
    | Helpers | 11 | 10 | 0 | 1 | 0 | 8 |
    | Jobs | 2 | 2 | 1 | 0 | 0 | 0 |
    | Models | 2822 | 1781 | 33 | 155 | 4 | 9 |
    | Mailers | 135 | 95 | 4 | 5 | 1 | 17 |
    | Channels | 8 | 8 | 2 | 0 | 0 | 0 |
    | JavaScripts | 1 | 0 | 0 | 0 | 0 | 0 |
    | Libraries | 155 | 119 | 4 | 8 | 2 | 12 |
    | Controller tests | 42 | 36 | 1 | 5 | 5 | 5 |
    | Model tests | 690 | 481 | 4 | 38 | 9 | 10 |
    | Mailer tests | 0 | 0 | 0 | 0 | 0 | 0 |
    | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 |
    | Model specs | 1040 | 834 | 0 | 0 | 0 | 0 |
    | Presenter specs | 637 | 528 | 4 | 7 | 1 | 73 |
    | Request specs | 11377 | 9330 | 0 | 78 | 0 | 117 |
    | Searcher specs | 361 | 298 | 0 | 0 | 0 | 0 |
    | Lib specs | 1197 | 994 | 0 | 0 | 0 | 0 |
    | Route specs | 43 | 38 | 0 | 0 | 0 | 0 |
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Total | 20010 | 15705 | 95 | 442 | 4 | 33 |
    +----------------------+--------+--------+---------+---------+-----+-------+
    Code LOC: 3166 Test LOC: 12539 Code to Test Ratio: 1:4.0
    

    View Slide

46. (Coverage Report from Simplecov)
    

    View Slide

47. • Hard to debug due to deep stacks
    • Long Roundtrip → Slow
    • Very Slow ./0
    Cons
    Request Tests
    

    View Slide

48. Run tests
    In Parallel
    

    View Slide

49. View Slide

50. 1
    ⚙
    ⚙
    ⚙ ʊਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓʊ
    ʼɹ̴̧̰̲ɹ̸̸̘̳̾̽̓̾̽ɹʻ
    ʉ:?:?:?:?:?:?:?:?:?:?:?:?:?:?:ʉ
    

    View Slide

51. 1
    ⚙
    1
    ⚙
    1
    ⚙
    

    View Slide

52. View Slide

53. • It assumes each job has same load
    • You need to write your own script to 

    distribute test files evenly by execution time or size
    Using GNU Parallel
    Run Test in Parallel
    

    View Slide

54. Load Balancing
    Test Files
    

    View Slide

55. Time
    0 sec 500 sec 1,000 sec 1,500 sec 2,000 sec 2,500 sec
    ~1600s (70%) ~650s (30%)
    Bottleneck ~250s
    

    View Slide

56. • One endpoint per test file
    • It’s also easier for human to read
    • Avoid Race Condition — Reset Database for each case
    • Use as less requests for one scenario as possible
    • Data preparations (db insertions) are usually fast…
    • …Unless you’re inserting hundreds of rows
    Load Balancing Test Files
    Run Test in Parallel
    

    View Slide

57. Took 441 seconds (7:21)
    real 7m21.858s
    user 26m6.616s
    sys 0m43.592s
    Took 278 seconds (4:38)
    real 4m39.159s
    user 27m46.976s
    sys 2m3.372s
    Took 439 seconds (7:19)
    real 7m19.914s
    user 28m23.356s
    sys 5m35.944s
    ⚙
    ⚙
    ⚙
    × 4
    × 8
    × 16
    ✅
    

    View Slide

58. 8 processes might be faster than 16
    • /proc/info may != # of CPU Cores
    • Be aware of startup time overhead (Rails is slow)
    • Find best number by measuring time
    • One CI machine might be enough
    Deciding # of Processes
    Speed up total test runtime
    

    View Slide

59. Mistakes…
    

    View Slide

60. 0
    3000
    6000
    9000
    12000
    0 ms
    120,000 ms
    240,000 ms
    360,000 ms
    480,000 ms
    600,000 ms
    720,000 ms
    Past ← Time → Now
    Total Runtime (ms)
    Number of Cases
    Time / Cases Ratio
    No junit.xml
    Reduced requests &
    Split into 2 machines Start using
    GNU Parallel
    4 processes
    on 1 machine
    Found
    some files
    not tested
    Switched to
    Ruby
    parallel_tests
    Started 8
    Processes
    Started
    testing JWT
    

    View Slide

61. • Find an existing parallel testing solution 

    before you rollout your own one
    • Store test reports from day 1, if your CI supports it
    Lessons
    Parallel Testing
    

    View Slide

62. AWS Cognito / JWT
    

    View Slide

63. “Amazon Cognito User Pools provide a secure user
    directory that scales to hundreds of millions of users. As a
    fully managed service, User Pools are easy to set up without
    any worries about server infrastructure. User Pools provide
    user profiles and authentication tokens for users who sign
    up directly and for federated users who sign in with social
    and enterprise identity providers.”
    AWS Cognito User Pools
    AWS Cognito
    https:/
    /aws.amazon.com/cognito/details/
    

    View Slide

64. https:/
    /docs.aws.amazon.com/cognito/latest/developerguide/cognito-scenarios.html
    OAuth 2.0 + Bearer Token (JWT)
    

    View Slide

65. Use JWT to
    Remove Dependency on Auth Server
    

    View Slide

66. • A Stateless, Cryptography-Based Token Standard
    JSON Web Token (RFC 7519)
    JWT
    

    View Slide

67. View Slide

68. Authorization: Bearer 11111111.22222222.33333333
    Header Signature
    Payload
    

    View Slide

69. • HMAC + SHA256 (Shared Secrets)
    • RSA / ECDSA (Asymmetric Signing)
    Algorithms
    JSON Web Token
    

    View Slide

70. 4
    4
    JWT Issuer
    5 6
    API Server
    Signed by Private Key Verified by Public Key
    

    View Slide

71. {
    "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", // Unique user 'sub' in Cognito User Pool
    "aud": "xxxxxxxxxxxxexample", // OAuth Client ID
    "email_verified": true,
    "token_use": "id",
    "auth_time": 1500009400,
    "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_example", // Token Issuer
    "cognito:username": "janedoe",
    "exp": 1500013000, // Token Valid Until
    "given_name": "Jane",
    "iat": 1500009400,
    "email": "janedoe@example.com"
    }
    A typical JWT Claim from AWS Cognito
    

    View Slide

72. • The Customer™ trusts on AWS’s Security
    • Real Stateless Authentication
    • No (hashed) password stored in our database
    • Client App pinning (checking aud)
    • Easy to find user in database by inspecting the token
    Pros
    JSON Web Token
    

    View Slide

73. • Token Revocation, obviously
    • Need to store JWKs (Public Keys) in every server daemon
    Cons
    JSON Web Token
    

    View Slide

74. Just do what Cognito is doing
    • Generate an RSA Key Pair for Test Environment
    • Store the JWK of Public Key
    • Sign the JWT Token with Private Key
    • Verify the JWT with JWK from Public Key
    Mocking JWT Verification in Tests
    JSON Web Token
    

    View Slide

75. npm install pem-jwk
    Converting RSA PEM to JWK
    

    View Slide

76. ⋊> ~/ openssl genrsa 2048 > test.pem
    Generating RSA private key, 2048 bit long modulus
    ................................................................+++
    ................................................+++
    e is 65537 (0x10001)
    ⋊> ~/ npx pem-jwk test.pem
    {“kty”:”RSA”,”n”:”m3Knu8N5…”,”e”:”AQAB”,”d”:”X69q6lwo…”,"p":"zrFBKps
    h...","q":"wIfj...","dp":"f9WHPWe...","qi":"o_O3bEU..."}
    

    View Slide

77. Black Box Integration Tests
    

    View Slide

78. • Writing tests in the same framework may have blind spot
    • Incorrectly assuming Accept: application/json
    • Sending payload without serializing into JSON
    • Passing parameters in wrong data type
    • To make sure your service runs in Docker
    Why Black Box Integration Tests?
    Black Box Integration Tests
    

    View Slide

79. • Docker Build
    • Docker Run
    • Database Migrations
    • PoC of an API Client using OpenAPI Spec
    • Typical scenarios (sign-up, create data, etc.)
    • Edge case scenarios that involve multiple actions
    • Job Workers
    What to test?
    Black Box Integration Tests
    

    View Slide

80. Run your server in Docker and
    Write HTTP Client in Another Language
    

    View Slide

81. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker
    ⚙
    Fake Clients
    (Node.js)
    89
    Docker Compose
    

    View Slide

82. 4
    4
    My Test Script
    5 6
    API Server
    Sign Token by
    my Private Key
    Verify Token by
    my Public Key
    

    View Slide

83. npm install openapi-client-axios
    Blackbox Testing
    

    View Slide

84. import OpenAPIClientAxios from 'openapi-client-axios';
    import * as fs from 'fs';
    import * as jwt from 'jsonwebtoken';
    import * as jsYaml from 'js-yaml';
    export class MyAppClient extends OpenAPIClientAxios {
    constructor(host = 'localhost:3000') {
    const url = `http://${host}/`;
    const definition = jsYaml.safeLoad(fs.readFileSync('/path/to/api/spec.openapi.yml'));
    definition.servers = [{ url }];
    super({ definition });
    }
    async showPetById(id) {
    return await this.client.query('showPetById', id);
    }
    // Sign a JWT Token
    loginAs(sub, email = null) {
    const options = { algorithm: 'RS256', issuer: 'test-issuer', audience: 'test-audience' };
    const cert = fs.readFileSync('/path/to/jwk.pem');
    this.token = jwt.sign({ email, sub }, cert, options);
    }
    }
    

    View Slide

85. async init() {
    await super.init();
    // NOTE: workaround https://github.com/axios/axios/pull/1395
    this.client.interceptors.request.use((request) => {
    request.headers.Authorization = `Token ${this.token}`;
    return request;
    });
    }
    Workaround Axios per-instance header issue
    

    View Slide

86. const client = new MyAppClient();
    await client.init();
    client.loginAs('john-appleseed', 'john@appleseed.com');
    await client.showPetById(42);
    

    View Slide

87. version: '3'
    services:
    app:
    build: .
    ports:
    - "3000:3000"
    env_file:
    - __tests__/rails.env
    - __tests__/shared/jwt/jwks.env
    depends_on: ['db', 'redis']
    db:
    image: "postgres:9"
    ports:
    - "15432:5432"
    environment:
    POSTGRES_PASSWORD: "password"
    redis:
    image: "redis:5"
    ports:
    - "16379:6379"
    • Expose ports for dockerize heath
    checking
    

    View Slide

88. build:
    machine: true
    steps:
    - checkout
    - run:
    name: Install Docker Compose
    command: |
    curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`una
    chmod +x ~/docker-compose
    sudo mv ~/docker-compose /usr/local/bin/docker-compose
    • Run CircleCI task outside a
    container so that we can mount
    local files
    • Install docker-compose if necessary
    

    View Slide

89. - run:
    name: Build image and pull dependencies
    command: docker-compose up --no-start
    - run:
    name: Run Containers
    background: true
    command: docker-compose up
    - run:
    name: Wait for containers
    command: |
    dockerize -wait tcp://localhost:15432 \
    -wait tcp://localhost:16379 \
    -wait tcp://localhost:3000
    • Run Docker build & pull in an
    individual task
    • Start containers in background
    after build & pull were done
    • Wait until containers ready before
    running anything else (some RDB
    takes few seconds to initialize)
    

    View Slide

90. - run:
    name: Test - Database preparation
    command: |
    docker-compose exec app rake \
    db:create \
    db:migrate \
    db:seed
    • Prepare database by migration
    instead of loading schema
    

    View Slide

91. How do we test features that
    depend on time?
    (
    

    View Slide

92. github.com/wolfcw/libfaketime
    • A dynamic linking library to modify date() system call
    • Download a binary from Debian APT Repository
    • Or build it on a Linux you use for server container
    • A macOS dylib won’t work
    libfaketime.so
    Mocking System Time
    

    View Slide

93. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker
    ⚙
    Fake Clients
    (Node.js)
    89
    Docker Compose
    API Server + 2d
    7
    

    View Slide

94. # An API server that shifts machine time by +2d
    app-future:
    build: .
    ports:
    - “3001:3000”
    environment:
    LD_PRELOAD: /mnt/libfaketime/libfaketime.so.1
    FAKETIME: +2d
    volumes:
    - ./path/to/libfaketime:/mnt/libfaketime
    env_file:
    - __tests__/rails.env
    - __tests__/shared/jwt/jwks.env
    depends_on: ['db', 'redis']
    1) Export to another host port
    2) Load libfaketime.so for API server
    3) Set Shifts
    4) Mount faketime binaries into
    container
    ! ᶃ
    " ᶄ
    : ᶅ
    : ᶆ
    

    View Slide

95. How about email content?
    (
    

    View Slide

96. MailHog or MailDev
    

    View Slide

97. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker
    ⚙
    Fake Clients
    (Node.js)
    89
    Docker Compose
    API Server + 2d
    7
    MailHog
    ;
    Scrape
    Emails
    through its
    API
    

    View Slide

98. Conclusion
    

    View Slide

99. • Write as detailed test cases as possible
    • Use them as regression tests
    • Run tests in parallel to reduce total execution time
    • Test Docker Images to prevent shipping broken image
    Your Time is < < <
    Summary
    

    View Slide

100. 2019 © Moneytree KK.
     We’re hiring!
     visit moneytree.jp/careers/
     

     View Slide