Platform values, Rust, and the implications for system software

Talk given at Scale By The Bay 2018. Video is at https://www.youtube.com/watch?v=2wZ1pCpJUIM. If you are interested in this talk, you might also be interested in my talk on Platform as a Reflection of Values from Node Summit 2017: https://vimeo.com/230142234

Transcript

1. Platform values, Rust, and the implications for system software CTO bryan@joyent.com Bryan Cantrill @bcantrill
2. Software platform and values • Deciding on programming language, operating system, etc. — that is, deciding on a software platform — is a big, important decision with lasting consequences (even for small work) • One doesn’t merely select a language for where it is, but also where it’s going — and what it represents • In the post-open source world of software infrastructure, this is reflected and guided by a community’s values • Beyond right tool for the job, it is the right values for the job… • …and then the right software for the values
3. Some software platform values • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
4. Platform core values • All of these values are important — but they are in tension • Platforms aspire to balance all of them — but for every platform some small subset represents its core values • These core values attract a like-minded community — and become self-reinforcing…
5. Platform core values: C • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
6. Platform core values: K • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
7. Platform core values: OpenBSD • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
8. Platform core values: Awk • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
9. Platform core values: Scala? • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
10. Values: The challenge for system software • For system software — that is, the software that abstracts hardware and serves as a generic platform — values are defined by expectations • For some values, if your system software doesn’t have it, the system itself can’t ever have it!
11. Values we demand of system software: • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
12. Platform core values: C • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
13. Platform core values: C++ • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
14. The system software disconnect • There is a tremendous divide between the values held by system software programming languages and the values that must be held by the software itself • Over the years (decades), we in system software have been able to deliver relatively secure, relatively reliable systems… • …but it is despite the programming language, not because of it! • Can we do better?
15. Enter Rust? • Rust is a systems software programming language designed around safety, parallelism, and speed • Rust has a novel system of ownership, whereby it can statically determine when a memory object is no longer in use • This allows for the power of a garbage-collected language, but with the performance of manual memory management • This is important because — unlike C — Rust is highly composable, allowing for more sophisticated (and higher performing!) primitives
16. Platform core values: Rust • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
17. Rust performance (my experience) Source: http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performance-of-c-and-rust/
18. Rust: Beyond ownership • Rust has a number of other features that make it highly compelling for systems software implementation: • Algebraic types allow robust, concise error handling • Hygienic macros allow for safe syntax extensions • Foreign function interface allows for full-duplex integration with C without sacrificing performance • “unsafe” keyword allows for some safety guarantees to be surgically overruled (though with obvious peril) • Also: terrific community, thriving ecosystem, etc.
19. Systems software in Rust: The promise • Rust’s values are an excellent fit for systems software • The beauty of Rust is that it shifts cognitive load from the operation of software back to the developer • Further, Rust’s ownership restricts what the software can do • The process can be frustrating but the end result is often satisfying: Source: Programming Rust by Jim Blandy and Jason Orendorff, page 262
20. Systems software in Rust: The peril • While Rust’s advantages are themselves clear, it’s less clear what the advantage is when replacing otherwise working code • Rust may be especially challenging in large, monolithic systems like an OS kernel which often have multiply-owned structures • So an OS kernel — despite its historic appeal and superficial fit for Rust — may represent more challenge than its worth • But what of hybrid approaches?
21. Hybrid approach I: Rust subsystems • One appeal of Rust is its ability to interoperate with C • One hybrid approach to explore would be to retain a traditional (C-based) system while allowing for Rust-based components • This would allow for an incremental approach — and instead of rewriting, Rust can be used for new development • There is a prototype example of this in FreeBSD; others are presumably possible
22. Hybrid approach II: Rust system components • System software is much broader than the operating system! • System software consists of many user-level components: utilities, daemons, service-/device-/fault- management facilities, debuggers, etc. • And then the distributed system that represents a multi- computer control plane — that itself includes many components • These components are much more prone to run-time failure! • Many of these are an excellent candidate for Rust!
23. Hybrid approach III: Rust-based firmware • Below the operating system lurks hardware-facing special- purpose software: firmware • Firmware is a sewer of unobservable software with a long history of infamous quality problems • Firmware has some of the same challenges as kernel development (e.g., dealing with allocation failures), but may otherwise be more amenable to Rust • This is especially true when/where firmware is in user-space and is network-facing! (e.g., OpenBMC)
24. Looking forward: Systems software in Rust • Rust represents something that we haven’t seen in a long time: a modern language that represents an alternative throughout the stack of software abstraction: we can have nice things! • Rust’s values are an excellent fit for system software! • Rust’s interoperability allows hybrid approaches, allowing for productive kernel incrementalism rather than whole-system rewrites • Firmware and user-level operating system software are two very promising candidates for implementation in Rust!