$30 off During Our Annual Pro Sale. View Details »

Web Development with OpenAPI Spec & Fast Testing

Web Development with OpenAPI Spec & Fast Testing

Modern web applications are mostly built on API and client side apps. Centralizing the business logic in API can prevent clients duplicating features on their sides.

However, in certain situations, we might face a tight schedule, work with clients who use waterfall workflow, or work with clients who are very sensitive about any unexpected behaviors.

This talk will cover how we utilize OpenAPI Spec for cross-team communication, keep the development speed by using slow but steady integration tests, and how we speed up those tests.

This talk was originally given at WebHack Meetup, a software engineering meetup in Tokyo. https://webhack.connpass.com/event/121030/

I gave this talk as a web engineer at Moneytree KK, which provides financial data platform for general customers, financial institutions and businesses. Check Moneytree's careers page for current open positions. https://www.moneytree.jp/careers

Yu-Cheng Chuang

February 27, 2019
Tweet

More Decks by Yu-Cheng Chuang

Other Decks in Technology

Transcript

  1. 2019 © Moneytree KK.
    Web Development with
    OpenAPI Spec & Fast Testing
    Yu-Cheng Chuang
    Feb 27, 2019
    Slides will be available online

    View Slide

  2. @yorkxin
    @chitsaou
    Yu-Cheng Chuang

    Core Systems Engineer 

    at Moneytree
    From Taiwan
    BSD / Linux User since 2005
    Web Developer since 2009
    #typography #railway #cooking
    About me

    View Slide

  3. Moneytree is a
    financial data
    portability platform,
    comprised of a
    personal financial
    management app,
    and an API
    (Application
    Programming
    Interface)
    feed of customer
    data for financial
    institutions.
    Moneytree
    in Brief
    "

    View Slide

  4. Project Background

    View Slide

  5. • Backend server built on Ruby on Rails
    • Server is running on AWS ECS
    • User management on AWS Cognito
    • CircleCI / AWS Code Pipeline
    • Web App built on React
    • iOS, Android Apps
    The Stack
    Project Background

    View Slide

  6. #
    Customer™
    $
    Backend
    Developers
    %
    $
    %
    Web
    Developers
    $
    %
    $
    %
    Mobile
    Developers
    $
    %
    $
    %
    &
    Project
    Managers
    '

    View Slide

  7. How do we develop individually and
    keep everything on track?
    ( ( (

    View Slide

  8. Actually I’ve started writing OpenAPI Spec!
    '
    '

    View Slide

  9. Yes, our Project Managers can write Open API Spec!

    View Slide

  10. Use Machine-friendly API Spec as the
    Single Source of Truth

    View Slide

  11. )
    pet-store-api-spec.git
    pet-store-server.git
    pet-store-web.git
    pet-store-mobile.git
    *
    +
    ,
    %
    %
    %
    '%

    View Slide

  12. • Allows you to design / define HTTP API in YAML or
    JSON
    • Request & Responses can be defined as Schema
    • Using JSON Schema-like DSL
    Open API Spec (f.k.a. Swagger)
    Open API Spec

    View Slide

  13. paths:
    /pets:
    get:
    summary: List all pets
    operationId: listPets
    tags:
    - pets
    parameters:
    - name: limit
    in: query
    description: How many items to return at one time (max 100)
    required: false
    schema:
    type: integer
    format: int32
    responses:
    '200':
    description: A paged array of pets
    headers:
    x-next:
    description: A link to the next page of responses
    schema:
    type: string
    content:
    application/json:
    schema:
    $ref: "#/components/schemas/Pets"
    default:
    description: unexpected error
    content:
    application/json:
    schema:
    $ref: "#/components/schemas/Error"
    components:
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true
    Pets:
    type: array
    items:
    $ref: "#/components/schemas/Pet"
    Error:
    required:
    - code
    - message
    properties:
    code:
    type: integer
    format: int32
    message:
    type: string
    -
    -
    -
    !
    !
    !

    View Slide

  14. View Slide

  15. • A language to define the data structure of JSON
    JSON Schema
    Open API Spec

    View Slide

  16. {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "http://example.com/product.schema.json",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
    "productId": {
    "description": "The unique identifier for a product",
    "type": "integer"
    },
    "productName": {
    "description": "Name of the product",
    "type": "string"
    },
    "price": {
    "description": "The price of the product",
    "type": "number",
    "exclusiveMinimum": 0
    },
    "tags": {
    "description": "Tags for the product",
    "type": "array",
    "items": { "type": "string" },
    "minItems": 1,
    "uniqueItems": true
    }
    },
    "required": [ "productId", "productName", "price" ]
    }
    {
    "productId": 1,
    "productName": "A green door",
    "price": 12.50,
    "tags": [ "home", "green" ]
    }
    https:/
    /json-schema.org/learn/getting-started-step-by-step.html

    View Slide

  17. {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "http://example.com/product.schema.json",
    "title": "Product",
    "description": "A product from Acme's catalog",
    "type": "object",
    "properties": {
    "productId": {
    "description": "The unique identifier for a product",
    "type": "integer"
    },
    "productName": {
    "description": "Name of the product",
    "type": "string"
    },
    "price": {
    "description": "The price of the product",
    "type": "number",
    "exclusiveMinimum": 0
    },
    "tags": {
    "description": "Tags for the product",
    "type": "array",
    "items": { "type": "string" },
    "minItems": 1,
    "uniqueItems": true
    }
    },
    "required": [ "productId", "productName", "price" ]
    }
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true




    OpenAPI Schema vs. JSON Schema

    View Slide

  18. https:/
    /swagger.io/docs/specification/data-models/keywords/

    View Slide

  19. https:/
    /philsturgeon.uk/api/2018/03/30/openapi-and-json-schema-divergence/
    (c) 2018 Phil Sturgeon

    View Slide

  20. npm install openapi2schema
    Converting to JSON Schema

    View Slide

  21. ⋊> ~/D/pet-store openapi2schema -i pet-store.openapi.yml | jq
    {
    "/pets": {
    "get": { ...},
    "post": {…}
    },
    "/pets/{petId}": {
    "get": {
    "responses": {
    "200": {
    "type": "array",
    "items": {
    "required": [“id", "name"],
    "properties": {
    "id": {
    "type": "integer",
    "format": "int64",
    "minimum": -9223372036854776000,
    "maximum": 9223372036854776000
    },
    "name": {
    "type": "string"
    },
    "tag": {
    "type": [
    "string",
    "null"
    ]
    }
    }
    },
    "$schema": "http://json-schema.org/draft-04/schema#"
    },
    "default": {...}
    }
    }
    }
    }
    components:
    schemas:
    Pet:
    required:
    - id
    - name
    properties:
    id:
    type: integer
    format: int64
    name:
    type: string
    tag:
    type: string
    nullable: true

    View Slide

  22. • YAML / JSON = Machine-friendly
    • Schema is defined using JSON Schema (*partial compatible)
    Pros
    Open API Spec

    View Slide

  23. • Most libraries only support version 2
    • See https:/
    /openapi.tools/
    • To validate data structure, you need to

    convert Schema definitions to JSON Schema manually
    Cons
    Open API Spec

    View Slide

  24. Now I have schema definition for each endpoints…

    View Slide

  25. How do we ensure that API server
    implementation matches the spec?
    (

    View Slide

  26. Test API responses
    With Schema Definitions

    View Slide

  27. gem install json-schema
    Validating JSON Schema

    View Slide

  28. JSON::Validator.fully_validate(schema, hash, strict: true)

    View Slide

  29. def load_schema_file(path)
    # Converts to JSON Schema with a Node CLI tool
    JSON.parse(`npx openapi2schema -i #{path}`)
    end
    ※ Don’t do this in production (only in tests)
    Since we’re using Ruby…

    View Slide

  30. 3) GET /pets/{id} Should be successful matches response schema (success)
    Failure/Error: expect(response.parsed_body).to match_json_schema(schema)
    The property '#/pet/name' of type integer did not match the following
    type: string in schema eb4470d8-8767-56a6-83b7-789839be31e0
    Shared Example Group: "Should conform OAS spec" called from ./spec/requests/
    shared_examples/response_examples.rb:2

    View Slide

  31. • Seed data using test factories or database dumps
    • Test output with response schema from API Spec
    • Client developers can start working 

    with fake data and real response structure
    • Backend developers can start working on 

    JSON serializers
    Stub Server
    Extra Benefits

    View Slide

  32. Authoring Open API Spec

    View Slide

  33. • Swagger Viewer
    • openapi-lint
    Tools in VSCode
    Authoring API Specs

    View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. Test the API Spec!

    View Slide

  38. npm install swagger-cli
    Test the API spec

    View Slide

  39. ⋊> ~/D/pet-store find . -name *.yml -print0 | xargs -n 1 -0 swagger-cli validate
    Swagger schema validation failed.
    Additional properties not allowed: responses at #/paths//pets
    JSON_OBJECT_VALIDATION_FAILED

    View Slide

  40. Now it’s time to implement endpoints…

    View Slide

  41. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake routes | wc -l
    90

    View Slide

  42. Can I prevent future regressions?
    (

    View Slide

  43. Request Tests
    With Detailed Scenarios

    View Slide

  44. • Test scenarios are close to business cases
    • Think from Client Developer’s and User’s perspective
    • Useful for Regression Tests
    • Ensuring all responses to match API Spec
    • So that Client developers will be happy every day
    Pros
    Request Tests

    View Slide

  45. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake stats
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Controllers | 1489 | 1151 | 42 | 145 | 3 | 5 |
    | Helpers | 11 | 10 | 0 | 1 | 0 | 8 |
    | Jobs | 2 | 2 | 1 | 0 | 0 | 0 |
    | Models | 2822 | 1781 | 33 | 155 | 4 | 9 |
    | Mailers | 135 | 95 | 4 | 5 | 1 | 17 |
    | Channels | 8 | 8 | 2 | 0 | 0 | 0 |
    | JavaScripts | 1 | 0 | 0 | 0 | 0 | 0 |
    | Libraries | 155 | 119 | 4 | 8 | 2 | 12 |
    | Controller tests | 42 | 36 | 1 | 5 | 5 | 5 |
    | Model tests | 690 | 481 | 4 | 38 | 9 | 10 |
    | Mailer tests | 0 | 0 | 0 | 0 | 0 | 0 |
    | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 |
    | Model specs | 1040 | 834 | 0 | 0 | 0 | 0 |
    | Presenter specs | 637 | 528 | 4 | 7 | 1 | 73 |
    | Request specs | 11377 | 9330 | 0 | 78 | 0 | 117 |
    | Searcher specs | 361 | 298 | 0 | 0 | 0 | 0 |
    | Lib specs | 1197 | 994 | 0 | 0 | 0 | 0 |
    | Route specs | 43 | 38 | 0 | 0 | 0 | 0 |
    +----------------------+--------+--------+---------+---------+-----+-------+
    | Total | 20010 | 15705 | 95 | 442 | 4 | 33 |
    +----------------------+--------+--------+---------+---------+-----+-------+
    Code LOC: 3166 Test LOC: 12539 Code to Test Ratio: 1:4.0

    View Slide

  46. (Coverage Report from Simplecov)

    View Slide

  47. • Hard to debug due to deep stacks
    • Long Roundtrip → Slow
    • Very Slow ./0
    Cons
    Request Tests

    View Slide

  48. Run tests
    In Parallel

    View Slide

  49. View Slide

  50. 1


    ⚙ ʊਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓʊ
    ʼɹ̴̧̰̲ɹ̸̸̘̳̾̽̓̾̽ɹʻ
    ʉ:?:?:?:?:?:?:?:?:?:?:?:?:?:?:ʉ

    View Slide

  51. 1

    1

    1

    View Slide

  52. View Slide

  53. • It assumes each job has same load
    • You need to write your own script to 

    distribute test files evenly by execution time or size
    Using GNU Parallel
    Run Test in Parallel

    View Slide

  54. Load Balancing
    Test Files

    View Slide

  55. Time
    0 sec 500 sec 1,000 sec 1,500 sec 2,000 sec 2,500 sec
    ~1600s (70%) ~650s (30%)
    Bottleneck ~250s

    View Slide

  56. • One endpoint per test file
    • It’s also easier for human to read
    • Avoid Race Condition — Reset Database for each case
    • Use as less requests for one scenario as possible
    • Data preparations (db insertions) are usually fast…
    • …Unless you’re inserting hundreds of rows
    Load Balancing Test Files
    Run Test in Parallel

    View Slide

  57. Took 441 seconds (7:21)
    real 7m21.858s
    user 26m6.616s
    sys 0m43.592s
    Took 278 seconds (4:38)
    real 4m39.159s
    user 27m46.976s
    sys 2m3.372s
    Took 439 seconds (7:19)
    real 7m19.914s
    user 28m23.356s
    sys 5m35.944s



    × 4
    × 8
    × 16

    View Slide

  58. 8 processes might be faster than 16
    • /proc/info may != # of CPU Cores
    • Be aware of startup time overhead (Rails is slow)
    • Find best number by measuring time
    • One CI machine might be enough
    Deciding # of Processes
    Speed up total test runtime

    View Slide

  59. Mistakes…

    View Slide

  60. 0
    3000
    6000
    9000
    12000
    0 ms
    120,000 ms
    240,000 ms
    360,000 ms
    480,000 ms
    600,000 ms
    720,000 ms
    Past ← Time → Now
    Total Runtime (ms)
    Number of Cases
    Time / Cases Ratio
    No junit.xml
    Reduced requests &
    Split into 2 machines Start using
    GNU Parallel
    4 processes
    on 1 machine
    Found
    some files
    not tested
    Switched to
    Ruby
    parallel_tests
    Started 8
    Processes
    Started
    testing JWT

    View Slide

  61. • Find an existing parallel testing solution 

    before you rollout your own one
    • Store test reports from day 1, if your CI supports it
    Lessons
    Parallel Testing

    View Slide

  62. AWS Cognito / JWT

    View Slide

  63. “Amazon Cognito User Pools provide a secure user
    directory that scales to hundreds of millions of users. As a
    fully managed service, User Pools are easy to set up without
    any worries about server infrastructure. User Pools provide
    user profiles and authentication tokens for users who sign
    up directly and for federated users who sign in with social
    and enterprise identity providers.”
    AWS Cognito User Pools
    AWS Cognito
    https:/
    /aws.amazon.com/cognito/details/

    View Slide

  64. https:/
    /docs.aws.amazon.com/cognito/latest/developerguide/cognito-scenarios.html
    OAuth 2.0 + Bearer Token (JWT)

    View Slide

  65. Use JWT to
    Remove Dependency on Auth Server

    View Slide

  66. • A Stateless, Cryptography-Based Token Standard
    JSON Web Token (RFC 7519)
    JWT

    View Slide

  67. View Slide

  68. Authorization: Bearer 11111111.22222222.33333333
    Header Signature
    Payload

    View Slide

  69. • HMAC + SHA256 (Shared Secrets)
    • RSA / ECDSA (Asymmetric Signing)
    Algorithms
    JSON Web Token

    View Slide

  70. 4
    4
    JWT Issuer
    5 6
    API Server
    Signed by Private Key Verified by Public Key

    View Slide

  71. {
    "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", // Unique user 'sub' in Cognito User Pool
    "aud": "xxxxxxxxxxxxexample", // OAuth Client ID
    "email_verified": true,
    "token_use": "id",
    "auth_time": 1500009400,
    "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_example", // Token Issuer
    "cognito:username": "janedoe",
    "exp": 1500013000, // Token Valid Until
    "given_name": "Jane",
    "iat": 1500009400,
    "email": "[email protected]"
    }
    A typical JWT Claim from AWS Cognito

    View Slide

  72. • The Customer™ trusts on AWS’s Security
    • Real Stateless Authentication
    • No (hashed) password stored in our database
    • Client App pinning (checking aud)
    • Easy to find user in database by inspecting the token
    Pros
    JSON Web Token

    View Slide

  73. • Token Revocation, obviously
    • Need to store JWKs (Public Keys) in every server daemon
    Cons
    JSON Web Token

    View Slide

  74. Just do what Cognito is doing
    • Generate an RSA Key Pair for Test Environment
    • Store the JWK of Public Key
    • Sign the JWT Token with Private Key
    • Verify the JWT with JWK from Public Key
    Mocking JWT Verification in Tests
    JSON Web Token

    View Slide

  75. npm install pem-jwk
    Converting RSA PEM to JWK

    View Slide

  76. ⋊> ~/ openssl genrsa 2048 > test.pem
    Generating RSA private key, 2048 bit long modulus
    ................................................................+++
    ................................................+++
    e is 65537 (0x10001)
    ⋊> ~/ npx pem-jwk test.pem
    {“kty”:”RSA”,”n”:”m3Knu8N5…”,”e”:”AQAB”,”d”:”X69q6lwo…”,"p":"zrFBKps
    h...","q":"wIfj...","dp":"f9WHPWe...","qi":"o_O3bEU..."}

    View Slide

  77. Black Box Integration Tests

    View Slide

  78. • Writing tests in the same framework may have blind spot
    • Incorrectly assuming Accept: application/json
    • Sending payload without serializing into JSON
    • Passing parameters in wrong data type
    • To make sure your service runs in Docker
    Why Black Box Integration Tests?
    Black Box Integration Tests

    View Slide

  79. • Docker Build
    • Docker Run
    • Database Migrations
    • PoC of an API Client using OpenAPI Spec
    • Typical scenarios (sign-up, create data, etc.)
    • Edge case scenarios that involve multiple actions
    • Job Workers
    What to test?
    Black Box Integration Tests

    View Slide

  80. Run your server in Docker and
    Write HTTP Client in Another Language

    View Slide

  81. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker

    Fake Clients
    (Node.js)
    89
    Docker Compose

    View Slide

  82. 4
    4
    My Test Script
    5 6
    API Server
    Sign Token by
    my Private Key
    Verify Token by
    my Public Key

    View Slide

  83. npm install openapi-client-axios
    Blackbox Testing

    View Slide

  84. import OpenAPIClientAxios from 'openapi-client-axios';
    import * as fs from 'fs';
    import * as jwt from 'jsonwebtoken';
    import * as jsYaml from 'js-yaml';
    export class MyAppClient extends OpenAPIClientAxios {
    constructor(host = 'localhost:3000') {
    const url = `http://${host}/`;
    const definition = jsYaml.safeLoad(fs.readFileSync('/path/to/api/spec.openapi.yml'));
    definition.servers = [{ url }];
    super({ definition });
    }
    async showPetById(id) {
    return await this.client.query('showPetById', id);
    }
    // Sign a JWT Token
    loginAs(sub, email = null) {
    const options = { algorithm: 'RS256', issuer: 'test-issuer', audience: 'test-audience' };
    const cert = fs.readFileSync('/path/to/jwk.pem');
    this.token = jwt.sign({ email, sub }, cert, options);
    }
    }

    View Slide

  85. async init() {
    await super.init();
    // NOTE: workaround https://github.com/axios/axios/pull/1395
    this.client.interceptors.request.use((request) => {
    request.headers.Authorization = `Token ${this.token}`;
    return request;
    });
    }
    Workaround Axios per-instance header issue

    View Slide

  86. const client = new MyAppClient();
    await client.init();
    client.loginAs('john-appleseed', '[email protected]');
    await client.showPetById(42);

    View Slide

  87. version: '3'
    services:
    app:
    build: .
    ports:
    - "3000:3000"
    env_file:
    - __tests__/rails.env
    - __tests__/shared/jwt/jwks.env
    depends_on: ['db', 'redis']
    db:
    image: "postgres:9"
    ports:
    - "15432:5432"
    environment:
    POSTGRES_PASSWORD: "password"
    redis:
    image: "redis:5"
    ports:
    - "16379:6379"
    • Expose ports for dockerize heath
    checking

    View Slide

  88. build:
    machine: true
    steps:
    - checkout
    - run:
    name: Install Docker Compose
    command: |
    curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`una
    chmod +x ~/docker-compose
    sudo mv ~/docker-compose /usr/local/bin/docker-compose
    • Run CircleCI task outside a
    container so that we can mount
    local files
    • Install docker-compose if necessary

    View Slide

  89. - run:
    name: Build image and pull dependencies
    command: docker-compose up --no-start
    - run:
    name: Run Containers
    background: true
    command: docker-compose up
    - run:
    name: Wait for containers
    command: |
    dockerize -wait tcp://localhost:15432 \
    -wait tcp://localhost:16379 \
    -wait tcp://localhost:3000
    • Run Docker build & pull in an
    individual task
    • Start containers in background
    after build & pull were done
    • Wait until containers ready before
    running anything else (some RDB
    takes few seconds to initialize)

    View Slide

  90. - run:
    name: Test - Database preparation
    command: |
    docker-compose exec app rake \
    db:create \
    db:migrate \
    db:seed
    • Prepare database by migration
    instead of loading schema

    View Slide

  91. How do we test features that
    depend on time?
    (

    View Slide

  92. github.com/wolfcw/libfaketime
    • A dynamic linking library to modify date() system call
    • Download a binary from Debian APT Repository
    • Or build it on a Linux you use for server container
    • A macOS dylib won’t work
    libfaketime.so
    Mocking System Time

    View Slide

  93. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker

    Fake Clients
    (Node.js)
    89
    Docker Compose
    API Server + 2d
    7

    View Slide

  94. # An API server that shifts machine time by +2d
    app-future:
    build: .
    ports:
    - “3001:3000”
    environment:
    LD_PRELOAD: /mnt/libfaketime/libfaketime.so.1
    FAKETIME: +2d
    volumes:
    - ./path/to/libfaketime:/mnt/libfaketime
    env_file:
    - __tests__/rails.env
    - __tests__/shared/jwt/jwks.env
    depends_on: ['db', 'redis']
    1) Export to another host port
    2) Load libfaketime.so for API server
    3) Set Shifts
    4) Mount faketime binaries into
    container
    ! ᶃ
    " ᶄ
    : ᶅ
    : ᶆ

    View Slide

  95. How about email content?
    (

    View Slide

  96. MailHog or MailDev

    View Slide

  97. SQL Database
    1
    Redis Cache
    1
    API Server
    7
    Job Worker

    Fake Clients
    (Node.js)
    89
    Docker Compose
    API Server + 2d
    7
    MailHog
    ;
    Scrape
    Emails
    through its
    API

    View Slide

  98. Conclusion

    View Slide

  99. • Write as detailed test cases as possible
    • Use them as regression tests
    • Run tests in parallel to reduce total execution time
    • Test Docker Images to prevent shipping broken image
    Your Time is < < <
    Summary

    View Slide

  100. 2019 © Moneytree KK.
    We’re hiring!
    visit moneytree.jp/careers/

    View Slide