Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Uniting Rust Servers and Clients through OpenAPI

Uniting Rust Servers and Clients through OpenAPI

Adam Leventhal

March 12, 2024

More Decks by Adam Leventhal

Other Decks in Technology


  1. OXIDE What is Oxide? • Rack scale computer • On-premises,

    private cloud in a box • All software included 2
  2. OXIDE Software? • The hardware is very cool–also there’s a

    ton of software that drives the platform • Data services like storage and VPC (networking) • A control plane for configuration and control • Divided up into services for various components 4
  3. OXIDE Why OpenAPI? • “Seems popular, let’s use it” •

    Used to describe internal services and the external API • A rare opportunity to be normal! • (foreshadowing) Vibrant, open ecosystem of tooling! 5
  4. OXIDE Why Rust? • In 2020 we knew there was

    lots of software at lots of layers we’d be building: embedded OS, host OS, hypervisor, storage, control plane, etc. • Rust was becoming the consensus choice for low level systems • Was not obvious (to me) that it was ideal for higher level software • Memory safety is important up and down the stack • So is memory consumption and general robustness • … and it’s kind of in the name of the company 6
  5. OXIDE Dropshot: our HTTP API framework • Evaluated Rust web

    frameworks and didn’t find what we needed • Dropshot has OpenAPI at its core–define the API in Rust code, and emit the OpenAPI description • Many OpenAPI advocates say (and said) that this is basically wrong • We wanted one source of truth, the code was changing fast, so the code was going to be the truth • Built dropshot into all our services… and started to consume those APIs (Dave Pacheco and I gave a talk on Dropshot in 2020; link at the end) 7
  6. OXIDE OpenAPI Ecosystem Fail • Our perception of the strength

    of the ecosystem was a huge attraction • Largely disappointed for what we needed • E.g. generated Rust client bindings 8
  7. OXIDE Progenitor • Generate Rust client bindings specifically for our

    Dropshot-based services • Fully automated–no manual intervention required • Immediately reactive to changes in the spec documents 11
  8. OXIDE Aside: Rust Macros • In Rust “!” is short

    for “pay attention, anything could be about to happen!” • Rust macros are spectacular: write Rust code to generate Rust tokens • The ecosystem around it is also spectacular: derive an AST from tokens, quote tokens, produce great error messages, etc. • Progenitor hides all the complexity of compiling OpenAPI into Rust within a macro invocation which makes it dead simple to use 13
  9. OXIDE Progenitor in action • Idiomatic SDKs • Automatic client

    generation eliminates a whole class of common errors • Like good tests: fearless refactors 14
  10. OXIDE Pagination • One goal of Dropshot was to make

    it easy to do things the right way and hard to do them the wrong way e.g. we wanted one pagination mechanism • Upshot: in Progenitor we can automatically iterate over pages of items: 15
  11. OXIDE Surprises • Started using Progenitor for our internal APIs–now

    using it for our public SDK • Progenitor: ~20K LOC; SDK: ~60K LOC (generated) • Using the same AST in Progenitor to emit our end-user CLI • And an HTTP mocking library (in part, to test the CLI) 16
  12. OXIDE Lesson: Rust is great for API clients • I

    get it: Rust people are a meme! • But seriously! • If you’re patient with Rust it will pay you back 18
  13. OXIDE Lesson: Write the tools you need • I was

    much too reluctant to start on Progenitor • If I had understood the complexity of OpenAPI and JSON Schema like I do now I might have been even more reluctant • You don’t need to solve the general problem! • For progenitor we would change the API to simplify generation! • Bryan Cantrill gave a great talk on “the primacy of toolmaking” (link at the end) 19
  14. OXIDE Lesson: Own your strategic weirdness • Our choice of

    OpenAPI was definitely right… even though the ecosystem wasn’t what we had hoped • We knew we wanted to do things a bit differently at the HTTP API layer • Dropshot and Progenitor were important components we needed to build • We don’t build everything from scratch (even though it seems like it) • Areas where you have hopes and ambitions deserve at least a prototype! 20
  15. OXIDE Thanks! • Oxide https://oxide.computer • Dropshot https://github.com/oxidecomputer/dropshot • 2020

    API Spec Conf Talk https://www.youtube.com/watch?v=EmSjZbSzA3A • Progenitor https://github.com/oxidecomputer/progenitor • Typify (JSON Schema → types) https://github.com/oxidecomputer/typify • Oxide Podcast: client generation https://share.transistor.fm/s/8460f5be • Bryan Cantrill: Sharpening the Axe https://www.youtube.com/watch?v=_GpBkplsGus 21