Scaling the Test Coverage of OpenAPI Generator for 30+ Programming Languages William Cheng Core team & founding member of OpenAPI Generator https:/ /openapi-generator.tech

Agenda - What's OpenAPI Generator - How do we test? - How do we scale? - Q&A 2

William Cheng ▰ #1 contributor to OpenAPI Generator ▰ Core team, founding member of OpenAPI Generator (https:/ /openapi-generator.tech) ▰ eBook Author: https:/ /gumroad.com/wing328 ▰ Founder of RESTUnited.com ▰ Morgan Stanley Alumni ▰ Github, Twitter: @wing328 3

What is OpenAPI Generator? 4 [OUTPUT] API clients, server stubs, documentation etc ● API clients (30 programming languages) ● 30+ server frameworks ● API Documentations ● GraphQL, Protobuf, Avro schemas ● Apache2 web server config ● MySQL DB schemas [INPUT] OpenAPI Spec (YAML/JSON) For REST APIs ● Swagger 1.2 ● Swagger/OpenAPI 2.x ● OpenAPI 3.x [PROCESS] OpenAPI Generator ● Java package (JAR) ● CLI ● Brew (MacOS) ● REST API ● Docker ● npm package wrapper (~120k weekly download) ● Plug-ins (Maven, Gradle, Bazel, etc) ● IDE integrations (Eclipse, IntelliJ IDEA, VS Code, etc)

5 Who are using OpenAPI Generator in production? Full list: https:/ /openapi-generator.tech/users

How do we test? How do we scale?

Day 1 7 OpenAPI 2.x, 3.x spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python

Test Input - OpenAPI spec files (YAML/JSON) OpenAPI (f.k.a Swagger) 2.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/2_0 - 59 JSON/YAML files (30296 lines) OpenAPI (f.k.a Swagger) 3.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/3_0 - 113 JSON/YAML files (24500 lines) OpenAPI 2.0 tests will be deprecated and we'll focus on OpenAPI 3.0 tests moving forward. 8

9 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python OpenAPI Generator module tests - Unit tests for different openapi-generator modules (CLI, online, maven plugin, core, etc) - Unit tests for generators (e.g. Ruby, PHP, Python, etc) and mustache templates

10 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python Integration tests - Tests the auto-generated clients (e.g Ruby, PHP) against a REST API test server - Example: make sure HTTP GET returns a proper list of objects

11 Continuous Integration - Run the tests in Travis CI (https:/ /travis-ci.org) - Tests the clients against a public REST API test server Public REST API test server Travis CI

Challenges - More test cases in the modules itself and the auto-generated clients - More client generators (e.g. Perl, TypeScript, Haskell, Erlang, Swift etc) - More server generators (e.g. Ruby on Rails, Rust, ASP.net core, etc) - More "other" generators (e.g. documentation, GraphQL schema converter, Protobuf schema converter, MySQL DB schema generator, etc) - More language versions to tests (e.g. JVM 7 vs JVM 8 vs JVM 11) - More platforms/OSs to tests (Swift clients in MacOS, C# clients in Windows) 12

And here come the problems…... - Travis CI timed out (> 50mins) - How to test Windows/Mac-based clients? - Single public REST API server results in race conditions (multiple PRs opened at the same time trigger multiple builds testing against the same test server) - Public module repositories (e.g. maven.org) throttled connections (e.g. connection reset, timeout) from the Travis build node due to too many connections from multiple jobs. - No budget for commercial offerings by Travis CI 13

14 Scale out the CIs - Use other CI providers (free-tier) in parallel: CircleCI, Shippable, Drone.io, GitHub Actions - AppVeyor for Windows tests and Bitrise.io for Swift (4.x, 5.x) tests - Run a local REST API server inside CIs - Install different versions of JDK in various CI providers AppVeyor CI JDK 8 JDK 11

More contributors joining …... - How to contribute a new generator (e.g. Rust)? - Too many PRs to review? - Oh some PRs accidentally remove a few tests or all test files for a particular client 15

Onboarding new contributors - Release checklist - Treat them like VIPs! - Use other PRs as references - Help them to get the new generators ASAP - Setup tests in CI for them 16

How the Rust client generator was created 17

Reviewing PRs …... - First and foremost, understand the change: What it fixes? What new feature to add? - Ask good questions, e.g. how to reproduce the issue? - Play with it! - Building a community of reviewers (technical committees) - Release often, e.g. bi-weekly patch releases 4.2.2 => 4.2.3 - Tests got removed!! ○ 1st attempt: restore test files from a particular folder ○ 2nd attempt: monitor the test files 18

Thank you our sponsors 19

Please support our voluntary work https:/ /opencollective.com/openapi_generator Or Search for "opencollective openapi generator" 20

Credits ▰ 1600+ contributors to the project ▰ Presentation template by SlidesCarnival ▰ Icons/Images ○ https:/ /dryicons.com/ ○ https:/ /www.flaticon.com/ ○ https:/ /www.wikipedia.org/ ○ Movie "300" 21

22 THANKS! Any questions? You can find me at ▰ @wing328 ▰ wing328hk@gmail.com