2019 © Moneytree KK. Web Development with OpenAPI Spec & Fast Testing Yu-Cheng Chuang Feb 27, 2019 Slides will be available online 

@yorkxin @chitsaou Yu-Cheng Chuang
 Core Systems Engineer 
 at Moneytree From Taiwan BSD / Linux User since 2005 Web Developer since 2009 #typography #railway #cooking About me 

Moneytree is a financial data portability platform, comprised of a personal financial management app, and an API (Application Programming Interface) feed of customer data for financial institutions. Moneytree in Brief " 

Project Background 

• Backend server built on Ruby on Rails • Server is running on AWS ECS • User management on AWS Cognito • CircleCI / AWS Code Pipeline • Web App built on React • iOS, Android Apps The Stack Project Background 

# Customer™ $ Backend Developers % $ % Web Developers $ % $ % Mobile Developers $ % $ % & Project Managers ' 

How do we develop individually and keep everything on track? ( ( ( 

Actually I’ve started writing OpenAPI Spec! ' ' 

Yes, our Project Managers can write Open API Spec! 

Use Machine-friendly API Spec as the Single Source of Truth 

) pet-store-api-spec.git pet-store-server.git pet-store-web.git pet-store-mobile.git * + , % % % '% 

• Allows you to design / define HTTP API in YAML or JSON • Request & Responses can be defined as Schema • Using JSON Schema-like DSL Open API Spec (f.k.a. Swagger) Open API Spec 

paths: /pets: get: summary: List all pets operationId: listPets tags: - pets parameters: - name: limit in: query description: How many items to return at one time (max 100) required: false schema: type: integer format: int32 responses: '200': description: A paged array of pets headers: x-next: description: A link to the next page of responses schema: type: string content: application/json: schema: $ref: "#/components/schemas/Pets" default: description: unexpected error content: application/json: schema: $ref: "#/components/schemas/Error" components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true Pets: type: array items: $ref: "#/components/schemas/Pet" Error: required: - code - message properties: code: type: integer format: int32 message: type: string - - - ! ! ! 

No content

• A language to define the data structure of JSON JSON Schema Open API Spec 

{ "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } { "productId": 1, "productName": "A green door", "price": 12.50, "tags": [ "home", "green" ] } https:/ /json-schema.org/learn/getting-started-step-by-step.html 

{ "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true ᶃ ᶃ ᶄ ᶄ OpenAPI Schema vs. JSON Schema 

https:/ /swagger.io/docs/specification/data-models/keywords/ 

https:/ /philsturgeon.uk/api/2018/03/30/openapi-and-json-schema-divergence/ (c) 2018 Phil Sturgeon 

npm install openapi2schema Converting to JSON Schema 

⋊> ~/D/pet-store openapi2schema -i pet-store.openapi.yml | jq { "/pets": { "get": { ...}, "post": {…} }, "/pets/{petId}": { "get": { "responses": { "200": { "type": "array", "items": { "required": [“id", "name"], "properties": { "id": { "type": "integer", "format": "int64", "minimum": -9223372036854776000, "maximum": 9223372036854776000 }, "name": { "type": "string" }, "tag": { "type": [ "string", "null" ] } } }, "$schema": "http://json-schema.org/draft-04/schema#" }, "default": {...} } } } } components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true 

• YAML / JSON = Machine-friendly • Schema is defined using JSON Schema (*partial compatible) Pros Open API Spec 

• Most libraries only support version 2 • See https:/ /openapi.tools/ • To validate data structure, you need to
 convert Schema definitions to JSON Schema manually Cons Open API Spec 

Now I have schema definition for each endpoints… 

How do we ensure that API server implementation matches the spec? ( 

Test API responses With Schema Definitions 

gem install json-schema Validating JSON Schema 

JSON::Validator.fully_validate(schema, hash, strict: true) 

def load_schema_file(path) # Converts to JSON Schema with a Node CLI tool JSON.parse(`npx openapi2schema -i #{path}`) end ※ Don’t do this in production (only in tests) Since we’re using Ruby… 

3) GET /pets/{id} Should be successful matches response schema (success) Failure/Error: expect(response.parsed_body).to match_json_schema(schema) The property '#/pet/name' of type integer did not match the following type: string in schema eb4470d8-8767-56a6-83b7-789839be31e0 Shared Example Group: "Should conform OAS spec" called from ./spec/requests/ shared_examples/response_examples.rb:2 

• Seed data using test factories or database dumps • Test output with response schema from API Spec • Client developers can start working 
 with fake data and real response structure • Backend developers can start working on 
 JSON serializers Stub Server Extra Benefits 

Authoring Open API Spec 

• Swagger Viewer • openapi-lint Tools in VSCode Authoring API Specs 

No content

No content

No content

Test the API Spec! 

npm install swagger-cli Test the API spec 

⋊> ~/D/pet-store find . -name *.yml -print0 | xargs -n 1 -0 swagger-cli validate Swagger schema validation failed. Additional properties not allowed: responses at #/paths//pets JSON_OBJECT_VALIDATION_FAILED 

Now it’s time to implement endpoints… 

⋊> ~/D/rocketscience on develop ◦ bundle exec rake routes | wc -l 90 

Can I prevent future regressions? ( 

Request Tests With Detailed Scenarios 

• Test scenarios are close to business cases • Think from Client Developer’s and User’s perspective • Useful for Regression Tests • Ensuring all responses to match API Spec • So that Client developers will be happy every day Pros Request Tests 

⋊> ~/D/rocketscience on develop ◦ bundle exec rake stats +----------------------+--------+--------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+--------+--------+---------+---------+-----+-------+ | Controllers | 1489 | 1151 | 42 | 145 | 3 | 5 | | Helpers | 11 | 10 | 0 | 1 | 0 | 8 | | Jobs | 2 | 2 | 1 | 0 | 0 | 0 | | Models | 2822 | 1781 | 33 | 155 | 4 | 9 | | Mailers | 135 | 95 | 4 | 5 | 1 | 17 | | Channels | 8 | 8 | 2 | 0 | 0 | 0 | | JavaScripts | 1 | 0 | 0 | 0 | 0 | 0 | | Libraries | 155 | 119 | 4 | 8 | 2 | 12 | | Controller tests | 42 | 36 | 1 | 5 | 5 | 5 | | Model tests | 690 | 481 | 4 | 38 | 9 | 10 | | Mailer tests | 0 | 0 | 0 | 0 | 0 | 0 | | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 | | Model specs | 1040 | 834 | 0 | 0 | 0 | 0 | | Presenter specs | 637 | 528 | 4 | 7 | 1 | 73 | | Request specs | 11377 | 9330 | 0 | 78 | 0 | 117 | | Searcher specs | 361 | 298 | 0 | 0 | 0 | 0 | | Lib specs | 1197 | 994 | 0 | 0 | 0 | 0 | | Route specs | 43 | 38 | 0 | 0 | 0 | 0 | +----------------------+--------+--------+---------+---------+-----+-------+ | Total | 20010 | 15705 | 95 | 442 | 4 | 33 | +----------------------+--------+--------+---------+---------+-----+-------+ Code LOC: 3166 Test LOC: 12539 Code to Test Ratio: 1:4.0 

(Coverage Report from Simplecov) 

• Hard to debug due to deep stacks • Long Roundtrip → Slow • Very Slow ./0 Cons Request Tests 

Run tests In Parallel 

No content

1 ⚙ ⚙ ⚙ ʊਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓʊ
ʼɹ̴̧̰̲ɹ̸̸̘̳̾̽̓̾̽ɹʻ
ʉ:?:?:?:?:?:?:?:?:?:?:?:?:?:?:ʉ 

1 ⚙ 1 ⚙ 1 ⚙ 

No content

• It assumes each job has same load • You need to write your own script to 
 distribute test files evenly by execution time or size Using GNU Parallel Run Test in Parallel 

Load Balancing Test Files 

Time 0 sec 500 sec 1,000 sec 1,500 sec 2,000 sec 2,500 sec ~1600s (70%) ~650s (30%) Bottleneck ~250s 

• One endpoint per test file • It’s also easier for human to read • Avoid Race Condition — Reset Database for each case • Use as less requests for one scenario as possible • Data preparations (db insertions) are usually fast… • …Unless you’re inserting hundreds of rows Load Balancing Test Files Run Test in Parallel 

Took 441 seconds (7:21) real 7m21.858s user 26m6.616s sys 0m43.592s Took 278 seconds (4:38) real 4m39.159s user 27m46.976s sys 2m3.372s Took 439 seconds (7:19) real 7m19.914s user 28m23.356s sys 5m35.944s ⚙ ⚙ ⚙ × 4 × 8 × 16 ✅ 

8 processes might be faster than 16 • /proc/info may != # of CPU Cores • Be aware of startup time overhead (Rails is slow) • Find best number by measuring time • One CI machine might be enough Deciding # of Processes Speed up total test runtime 

Mistakes… 

0 3000 6000 9000 12000 0 ms 120,000 ms 240,000 ms 360,000 ms 480,000 ms 600,000 ms 720,000 ms Past ← Time → Now Total Runtime (ms) Number of Cases Time / Cases Ratio No junit.xml Reduced requests & Split into 2 machines Start using GNU Parallel 4 processes on 1 machine Found some files not tested Switched to Ruby parallel_tests Started 8 Processes Started testing JWT 

• Find an existing parallel testing solution 
 before you rollout your own one • Store test reports from day 1, if your CI supports it Lessons Parallel Testing 

AWS Cognito / JWT 

“Amazon Cognito User Pools provide a secure user directory that scales to hundreds of millions of users. As a fully managed service, User Pools are easy to set up without any worries about server infrastructure. User Pools provide user profiles and authentication tokens for users who sign up directly and for federated users who sign in with social and enterprise identity providers.” AWS Cognito User Pools AWS Cognito https:/ /aws.amazon.com/cognito/details/ 

https:/ /docs.aws.amazon.com/cognito/latest/developerguide/cognito-scenarios.html OAuth 2.0 + Bearer Token (JWT) 

Use JWT to Remove Dependency on Auth Server 

• A Stateless, Cryptography-Based Token Standard JSON Web Token (RFC 7519) JWT 

No content

Authorization: Bearer 11111111.22222222.33333333 Header Signature Payload 

• HMAC + SHA256 (Shared Secrets) • RSA / ECDSA (Asymmetric Signing) Algorithms JSON Web Token 

4 4 JWT Issuer 5 6 API Server Signed by Private Key Verified by Public Key 

{ "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", // Unique user 'sub' in Cognito User Pool "aud": "xxxxxxxxxxxxexample", // OAuth Client ID "email_verified": true, "token_use": "id", "auth_time": 1500009400, "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_example", // Token Issuer "cognito:username": "janedoe", "exp": 1500013000, // Token Valid Until "given_name": "Jane", "iat": 1500009400, "email": "[email protected]" } A typical JWT Claim from AWS Cognito 

• The Customer™ trusts on AWS’s Security • Real Stateless Authentication • No (hashed) password stored in our database • Client App pinning (checking aud) • Easy to find user in database by inspecting the token Pros JSON Web Token 

• Token Revocation, obviously • Need to store JWKs (Public Keys) in every server daemon Cons JSON Web Token 

Just do what Cognito is doing • Generate an RSA Key Pair for Test Environment • Store the JWK of Public Key • Sign the JWT Token with Private Key • Verify the JWT with JWK from Public Key Mocking JWT Verification in Tests JSON Web Token 

npm install pem-jwk Converting RSA PEM to JWK 

⋊> ~/ openssl genrsa 2048 > test.pem Generating RSA private key, 2048 bit long modulus ................................................................+++ ................................................+++ e is 65537 (0x10001) ⋊> ~/ npx pem-jwk test.pem {“kty”:”RSA”,”n”:”m3Knu8N5…”,”e”:”AQAB”,”d”:”X69q6lwo…”,"p":"zrFBKps h...","q":"wIfj...","dp":"f9WHPWe...","qi":"o_O3bEU..."} 

Black Box Integration Tests 

• Writing tests in the same framework may have blind spot • Incorrectly assuming Accept: application/json • Sending payload without serializing into JSON • Passing parameters in wrong data type • To make sure your service runs in Docker Why Black Box Integration Tests? Black Box Integration Tests 

• Docker Build • Docker Run • Database Migrations • PoC of an API Client using OpenAPI Spec • Typical scenarios (sign-up, create data, etc.) • Edge case scenarios that involve multiple actions • Job Workers What to test? Black Box Integration Tests 

Run your server in Docker and Write HTTP Client in Another Language 

SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose 

4 4 My Test Script 5 6 API Server Sign Token by my Private Key Verify Token by my Public Key 

npm install openapi-client-axios Blackbox Testing 

import OpenAPIClientAxios from 'openapi-client-axios'; import * as fs from 'fs'; import * as jwt from 'jsonwebtoken'; import * as jsYaml from 'js-yaml'; export class MyAppClient extends OpenAPIClientAxios { constructor(host = 'localhost:3000') { const url = `http://${host}/`; const definition = jsYaml.safeLoad(fs.readFileSync('/path/to/api/spec.openapi.yml')); definition.servers = [{ url }]; super({ definition }); } async showPetById(id) { return await this.client.query('showPetById', id); } // Sign a JWT Token loginAs(sub, email = null) { const options = { algorithm: 'RS256', issuer: 'test-issuer', audience: 'test-audience' }; const cert = fs.readFileSync('/path/to/jwk.pem'); this.token = jwt.sign({ email, sub }, cert, options); } } 

async init() { await super.init(); // NOTE: workaround https://github.com/axios/axios/pull/1395 this.client.interceptors.request.use((request) => { request.headers.Authorization = `Token ${this.token}`; return request; }); } Workaround Axios per-instance header issue 

const client = new MyAppClient(); await client.init(); client.loginAs('john-appleseed', '[email protected]'); await client.showPetById(42); 

version: '3' services: app: build: . ports: - "3000:3000" env_file: - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] db: image: "postgres:9" ports: - "15432:5432" environment: POSTGRES_PASSWORD: "password" redis: image: "redis:5" ports: - "16379:6379" • Expose ports for dockerize heath checking 

build: machine: true steps: - checkout - run: name: Install Docker Compose command: | curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`una chmod +x ~/docker-compose sudo mv ~/docker-compose /usr/local/bin/docker-compose • Run CircleCI task outside a container so that we can mount local files • Install docker-compose if necessary 

- run: name: Build image and pull dependencies command: docker-compose up --no-start - run: name: Run Containers background: true command: docker-compose up - run: name: Wait for containers command: | dockerize -wait tcp://localhost:15432 \ -wait tcp://localhost:16379 \ -wait tcp://localhost:3000 • Run Docker build & pull in an individual task • Start containers in background after build & pull were done • Wait until containers ready before running anything else (some RDB takes few seconds to initialize) 

- run: name: Test - Database preparation command: | docker-compose exec app rake \ db:create \ db:migrate \ db:seed • Prepare database by migration instead of loading schema 

How do we test features that depend on time? ( 

github.com/wolfcw/libfaketime • A dynamic linking library to modify date() system call • Download a binary from Debian APT Repository • Or build it on a Linux you use for server container • A macOS dylib won’t work libfaketime.so Mocking System Time 

SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7 

# An API server that shifts machine time by +2d app-future: build: . ports: - “3001:3000” environment: LD_PRELOAD: /mnt/libfaketime/libfaketime.so.1 FAKETIME: +2d volumes: - ./path/to/libfaketime:/mnt/libfaketime env_file: - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] 1) Export to another host port 2) Load libfaketime.so for API server 3) Set Shifts 4) Mount faketime binaries into container ! ᶃ " ᶄ : ᶅ : ᶆ 

How about email content? ( 

MailHog or MailDev 

SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7 MailHog ; Scrape Emails through its API 

Conclusion 

• Write as detailed test cases as possible • Use them as regression tests • Run tests in parallel to reduce total execution time • Test Docker Images to prevent shipping broken image Your Time is < < < Summary 

2019 © Moneytree KK. We’re hiring! visit moneytree.jp/careers/