Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling the Test Coverage of OpenAPI Generator for 30+ Programming Languages

Scaling the Test Coverage of OpenAPI Generator for 30+ Programming Languages

OpenAPI Generator (https://openapi-generator.tech/) - a code generation tool covering 30+ programming languages and 30+ server-side frameworks for REST APIs, has gone mainstream with adoption by companies such as IBM, Yelp, and open-source project such as Kubernetes. Npm CLI wrapper download alone already reached 2.5 million. How can OpenAPI Generator scale from less than 10 generators to more than 130 generators with 1500+ contributors? What can we do to make it easier for open-source developers to contribute a new generator? How to engage the community to try it out and give us feedback? How to learn from mistakes in one generator and apply the lessons to the other generators? How do we test 30+ programming languages and 30+ server-side frameworks? What resources (services, tools, and more) we can leverage to help us improve the project? What to expect when testing so many programming languages and frameworks in a single project? William, the top contributor of the project, will walk you through the journey to scale the project in terms of the community, technical aspects, and test coverages so that you can apply similar techniques to your open-source and proprietary projects.

Presented on Open Source Summit Japan 2020 (Dec 4th, 2020)


William Cheng

February 02, 2021

More Decks by William Cheng

Other Decks in Programming


  1. Scaling the Test Coverage of OpenAPI Generator for 30+ Programming

    Languages William Cheng Core team & founding member of OpenAPI Generator https:/ /openapi-generator.tech
  2. Agenda - What's OpenAPI Generator - How do we test?

    - How do we scale? - Q&A 2
  3. William Cheng ▰ #1 contributor to OpenAPI Generator ▰ Core

    team, founding member of OpenAPI Generator (https:/ /openapi-generator.tech) ▰ eBook Author: https:/ /gumroad.com/wing328 ▰ Founder of RESTUnited.com ▰ Morgan Stanley Alumni ▰ Github, Twitter: @wing328 3
  4. What is OpenAPI Generator? 4 [OUTPUT] API clients, server stubs,

    documentation etc • API clients (30 programming languages) • 30+ server frameworks • API Documentations • GraphQL, Protobuf, Avro schemas • Apache2 web server config • MySQL DB schemas [INPUT] OpenAPI Spec (YAML/JSON) For REST APIs • Swagger 1.2 • Swagger/OpenAPI 2.x • OpenAPI 3.x [PROCESS] OpenAPI Generator • Java package (JAR) • CLI • Brew (MacOS) • REST API • Docker • npm package wrapper (~120k weekly download) • Plug-ins (Maven, Gradle, Bazel, etc) • IDE integrations (Eclipse, IntelliJ IDEA, VS Code, etc)
  5. 5 Who are using OpenAPI Generator in production? Full list:

    https:/ /openapi-generator.tech/users
  6. How do we test? How do we scale?

  7. Day 1 7 OpenAPI 2.x, 3.x spec files (YAML/JSON) OpenAPI

    Generator (written in Java) API clients in Ruby, Java, PHP, Python
  8. Test Input - OpenAPI spec files (YAML/JSON) OpenAPI (f.k.a Swagger)

    2.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/2_0 - 59 JSON/YAML files (30296 lines) OpenAPI (f.k.a Swagger) 3.0 test spec - https:/ /github.com/OpenAPITools/openapi-generator/tree/maste r/modules/openapi-generator/src/test/resources/3_0 - 113 JSON/YAML files (24500 lines) OpenAPI 2.0 tests will be deprecated and we'll focus on OpenAPI 3.0 tests moving forward. 8
  9. 9 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java)

    API clients in Ruby, Java, PHP, Python OpenAPI Generator module tests - Unit tests for different openapi-generator modules (CLI, online, maven plugin, core, etc) - Unit tests for generators (e.g. Ruby, PHP, Python, etc) and mustache templates
  10. 10 OpenAPI spec files (YAML/JSON) OpenAPI Generator (written in Java)

    API clients in Ruby, Java, PHP, Python Integration tests - Tests the auto-generated clients (e.g Ruby, PHP) against a REST API test server - Example: make sure HTTP GET returns a proper list of objects
  11. 11 Continuous Integration - Run the tests in Travis CI

    (https:/ /travis-ci.org) - Tests the clients against a public REST API test server Public REST API test server Travis CI
  12. Challenges - More test cases in the modules itself and

    the auto-generated clients - More client generators (e.g. Perl, TypeScript, Haskell, Erlang, Swift etc) - More server generators (e.g. Ruby on Rails, Rust, ASP.net core, etc) - More "other" generators (e.g. documentation, GraphQL schema converter, Protobuf schema converter, MySQL DB schema generator, etc) - More language versions to tests (e.g. JVM 7 vs JVM 8 vs JVM 11) - More platforms/OSs to tests (Swift clients in MacOS, C# clients in Windows) 12
  13. And here come the problems…... - Travis CI timed out

    (> 50mins) - How to test Windows/Mac-based clients? - Single public REST API server results in race conditions (multiple PRs opened at the same time trigger multiple builds testing against the same test server) - Public module repositories (e.g. maven.org) throttled connections (e.g. connection reset, timeout) from the Travis build node due to too many connections from multiple jobs. - No budget for commercial offerings by Travis CI 13
  14. 14 Scale out the CIs - Use other CI providers

    (free-tier) in parallel: CircleCI, Shippable, Drone.io, GitHub Actions - AppVeyor for Windows tests and Bitrise.io for Swift (4.x, 5.x) tests - Run a local REST API server inside CIs - Install different versions of JDK in various CI providers AppVeyor CI JDK 8 JDK 11
  15. More contributors joining …... - How to contribute a new

    generator (e.g. Rust)? - Too many PRs to review? - Oh some PRs accidentally remove a few tests or all test files for a particular client 15
  16. Onboarding new contributors - Release checklist - Treat them like

    VIPs! - Use other PRs as references - Help them to get the new generators ASAP - Setup tests in CI for them 16
  17. How the Rust client generator was created 17

  18. Reviewing PRs …... - First and foremost, understand the change:

    What it fixes? What new feature to add? - Ask good questions, e.g. how to reproduce the issue? - Play with it! - Building a community of reviewers (technical committees) - Release often, e.g. bi-weekly patch releases 4.2.2 => 4.2.3 - Tests got removed!! ◦ 1st attempt: restore test files from a particular folder ◦ 2nd attempt: monitor the test files 18
  19. Thank you our sponsors 19

  20. Please support our voluntary work https:/ /opencollective.com/openapi_generator Or Search for

    "opencollective openapi generator" 20
  21. Credits ▰ 1600+ contributors to the project ▰ Presentation template

    by SlidesCarnival ▰ Icons/Images ◦ https:/ /dryicons.com/ ◦ https:/ /www.flaticon.com/ ◦ https:/ /www.wikipedia.org/ ◦ Movie "300" 21
  22. 22 THANKS! Any questions? You can find me at ▰

    @wing328 ▰ wing328hk@gmail.com