Uniting Rust Servers and Clients through OpenAPI

1. Uniting Rust Servers and Clients through OpenAPI Adam Leventhal
2. OXIDE What is Oxide? • Rack scale computer • On-premises, private cloud in a box • All software included 2
3. OXIDE 3
4. OXIDE Software? • The hardware is very cool–also there’s a ton of software that drives the platform • Data services like storage and VPC (networking) • A control plane for configuration and control • Divided up into services for various components 4
5. OXIDE Why OpenAPI? • “Seems popular, let’s use it” • Used to describe internal services and the external API • A rare opportunity to be normal! • (foreshadowing) Vibrant, open ecosystem of tooling! 5
6. OXIDE Why Rust? • In 2020 we knew there was lots of software at lots of layers we’d be building: embedded OS, host OS, hypervisor, storage, control plane, etc. • Rust was becoming the consensus choice for low level systems • Was not obvious (to me) that it was ideal for higher level software • Memory safety is important up and down the stack • So is memory consumption and general robustness • … and it’s kind of in the name of the company 6
7. OXIDE Dropshot: our HTTP API framework • Evaluated Rust web frameworks and didn’t find what we needed • Dropshot has OpenAPI at its core–define the API in Rust code, and emit the OpenAPI description • Many OpenAPI advocates say (and said) that this is basically wrong • We wanted one source of truth, the code was changing fast, so the code was going to be the truth • Built dropshot into all our services… and started to consume those APIs (Dave Pacheco and I gave a talk on Dropshot in 2020; link at the end) 7
8. OXIDE OpenAPI Ecosystem Fail • Our perception of the strength of the ecosystem was a huge attraction • Largely disappointed for what we needed • E.g. generated Rust client bindings 8
9. OXIDE openapi-generator 9
10. OXIDE openapi-generator 10
11. OXIDE Progenitor • Generate Rust client bindings specifically for our Dropshot-based services • Fully automated–no manual intervention required • Immediately reactive to changes in the spec documents 11
12. OXIDE Client Generation 12
13. OXIDE Aside: Rust Macros • In Rust “!” is short for “pay attention, anything could be about to happen!” • Rust macros are spectacular: write Rust code to generate Rust tokens • The ecosystem around it is also spectacular: derive an AST from tokens, quote tokens, produce great error messages, etc. • Progenitor hides all the complexity of compiling OpenAPI into Rust within a macro invocation which makes it dead simple to use 13
14. OXIDE Progenitor in action • Idiomatic SDKs • Automatic client generation eliminates a whole class of common errors • Like good tests: fearless refactors 14
15. OXIDE Pagination • One goal of Dropshot was to make it easy to do things the right way and hard to do them the wrong way e.g. we wanted one pagination mechanism • Upshot: in Progenitor we can automatically iterate over pages of items: 15
16. OXIDE Surprises • Started using Progenitor for our internal APIs–now using it for our public SDK • Progenitor: ~20K LOC; SDK: ~60K LOC (generated) • Using the same AST in Progenitor to emit our end-user CLI • And an HTTP mocking library (in part, to test the CLI) 16
17. OXIDE Lessons 17
18. OXIDE Lesson: Rust is great for API clients • I get it: Rust people are a meme! • But seriously! • If you’re patient with Rust it will pay you back 18
19. OXIDE Lesson: Write the tools you need • I was much too reluctant to start on Progenitor • If I had understood the complexity of OpenAPI and JSON Schema like I do now I might have been even more reluctant • You don’t need to solve the general problem! • For progenitor we would change the API to simplify generation! • Bryan Cantrill gave a great talk on “the primacy of toolmaking” (link at the end) 19
20. OXIDE Lesson: Own your strategic weirdness • Our choice of OpenAPI was definitely right… even though the ecosystem wasn’t what we had hoped • We knew we wanted to do things a bit differently at the HTTP API layer • Dropshot and Progenitor were important components we needed to build • We don’t build everything from scratch (even though it seems like it) • Areas where you have hopes and ambitions deserve at least a prototype! 20
21. OXIDE Thanks! • Oxide https://oxide.computer • Dropshot https://github.com/oxidecomputer/dropshot • 2020 API Spec Conf Talk https://www.youtube.com/watch?v=EmSjZbSzA3A • Progenitor https://github.com/oxidecomputer/progenitor • Typify (JSON Schema → types) https://github.com/oxidecomputer/typify • Oxide Podcast: client generation https://share.transistor.fm/s/8460f5be • Bryan Cantrill: Sharpening the Axe https://www.youtube.com/watch?v=_GpBkplsGus 21