Scaling the Test Coverage of OpenAPI Generator for 30+ Programming Languages

OpenAPI Generator (https://openapi-generator.tech/) - a code generation tool covering 30+ programming languages and 30+ server-side frameworks for REST APIs, has gone mainstream with adoption by companies such as IBM, Yelp, and open-source project such as Kubernetes. Npm CLI wrapper download alone already reached 2.5 million. How can OpenAPI Generator scale from less than 10 generators to more than 130 generators with 1500+ contributors? What can we do to make it easier for open-source developers to contribute a new generator? How to engage the community to try it out and give us feedback? How to learn from mistakes in one generator and apply the lessons to the other generators? How do we test 30+ programming languages and 30+ server-side frameworks? What resources (services, tools, and more) we can leverage to help us improve the project? What to expect when testing so many programming languages and frameworks in a single project? William, the top contributor of the project, will walk you through the journey to scale the project in terms of the community, technical aspects, and test coverages so that you can apply similar techniques to your open-source and proprietary projects.

Presented on Open Source Summit Japan 2020 (Dec 4th, 2020)

Transcript

1. Scaling the Test Coverage of OpenAPI Generator for 30+ Programming Languages William Cheng Core team & founding member of OpenAPI Generator https:/ /openapi-generator.tech
2. Agenda - What's OpenAPI Generator - How do we test? - How do we scale? - Q&A 2
3. William Cheng ▰ #1 contributor to OpenAPI Generator ▰ Core team, founding member of OpenAPI Generator (https:/ /openapi-generator.tech) ▰ eBook Author: https:/ /gumroad.com/wing328 ▰ Founder of RESTUnited.com ▰ Morgan Stanley Alumni ▰ Github, Twitter: @wing328 3
4. What is OpenAPI Generator? 4 [OUTPUT] API clients, server stubs, documentation etc • API clients (30 programming languages) • 30+ server frameworks • API Documentations • GraphQL, Protobuf, Avro schemas • Apache2 web server config • MySQL DB schemas [INPUT] OpenAPI Spec (YAML/JSON) For REST APIs • Swagger 1.2 • Swagger/OpenAPI 2.x • OpenAPI 3.x [PROCESS] OpenAPI Generator • Java package (JAR) • CLI • Brew (MacOS) • REST API • Docker • npm package wrapper (~120k weekly download) • Plug-ins (Maven, Gradle, Bazel, etc) • IDE integrations (Eclipse, IntelliJ IDEA, VS Code, etc)
5. 5 Who are using OpenAPI Generator in production? Full list: https:/ /openapi-generator.tech/users
6. How do we test? How do we scale?
7. Day 1 7 OpenAPI 2.x, 3.x spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python
8. Test Input - OpenAPI spec files (YAML/JSON) OpenAPI (f.k.a Swagger) 2.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/2_0 - 59 JSON/YAML files (30296 lines) OpenAPI (f.k.a Swagger) 3.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/3_0 - 113 JSON/YAML files (24500 lines) OpenAPI 2.0 tests will be deprecated and we'll focus on OpenAPI 3.0 tests moving forward. 8
9. 9 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python OpenAPI Generator module tests - Unit tests for different openapi-generator modules (CLI, online, maven plugin, core, etc) - Unit tests for generators (e.g. Ruby, PHP, Python, etc) and mustache templates
10. 10 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java) API clients in Ruby, Java, PHP, Python Integration tests - Tests the auto-generated clients (e.g Ruby, PHP) against a REST API test server - Example: make sure HTTP GET returns a proper list of objects
11. 11 Continuous Integration - Run the tests in Travis CI (https:/ /travis-ci.org) - Tests the clients against a public REST API test server Public REST API test server Travis CI
12. Challenges - More test cases in the modules itself and the auto-generated clients - More client generators (e.g. Perl, TypeScript, Haskell, Erlang, Swift etc) - More server generators (e.g. Ruby on Rails, Rust, ASP.net core, etc) - More "other" generators (e.g. documentation, GraphQL schema converter, Protobuf schema converter, MySQL DB schema generator, etc) - More language versions to tests (e.g. JVM 7 vs JVM 8 vs JVM 11) - More platforms/OSs to tests (Swift clients in MacOS, C# clients in Windows) 12
13. And here come the problems…... - Travis CI timed out (> 50mins) - How to test Windows/Mac-based clients? - Single public REST API server results in race conditions (multiple PRs opened at the same time trigger multiple builds testing against the same test server) - Public module repositories (e.g. maven.org) throttled connections (e.g. connection reset, timeout) from the Travis build node due to too many connections from multiple jobs. - No budget for commercial offerings by Travis CI 13
14. 14 Scale out the CIs - Use other CI providers (free-tier) in parallel: CircleCI, Shippable, Drone.io, GitHub Actions - AppVeyor for Windows tests and Bitrise.io for Swift (4.x, 5.x) tests - Run a local REST API server inside CIs - Install different versions of JDK in various CI providers AppVeyor CI JDK 8 JDK 11
15. More contributors joining …... - How to contribute a new generator (e.g. Rust)? - Too many PRs to review? - Oh some PRs accidentally remove a few tests or all test files for a particular client 15
16. Onboarding new contributors - Release checklist - Treat them like VIPs! - Use other PRs as references - Help them to get the new generators ASAP - Setup tests in CI for them 16
17. How the Rust client generator was created 17
18. Reviewing PRs …... - First and foremost, understand the change: What it fixes? What new feature to add? - Ask good questions, e.g. how to reproduce the issue? - Play with it! - Building a community of reviewers (technical committees) - Release often, e.g. bi-weekly patch releases 4.2.2 => 4.2.3 - Tests got removed!! ◦ 1st attempt: restore test files from a particular folder ◦ 2nd attempt: monitor the test files 18
19. Thank you our sponsors 19
20. Please support our voluntary work https:/ /opencollective.com/openapi_generator Or Search for "opencollective openapi generator" 20
21. Credits ▰ 1600+ contributors to the project ▰ Presentation template by SlidesCarnival ▰ Icons/Images ◦ https:/ /dryicons.com/ ◦ https:/ /www.flaticon.com/ ◦ https:/ /www.wikipedia.org/ ◦ Movie "300" 21
22. 22 THANKS! Any questions? You can find me at ▰ @wing328 ▰ wing328hk@gmail.com