Platform values, Rust, and the implications for system software

Talk given at Scale By The Bay 2018. Video is at https://www.youtube.com/watch?v=2wZ1pCpJUIM. If you are interested in this talk, you might also be interested in my talk on Platform as a Reflection of Values from Node Summit 2017: https://vimeo.com/230142234

Transcript

1. Platform values, Rust, and the
   implications for system software
   CTO
   bryan@joyent.com
   Bryan Cantrill
   @bcantrill
   

   View Slide

2. Software platform and values
   • Deciding on programming language, operating system, etc. —
   that is, deciding on a software platform — is a big, important
   decision with lasting consequences (even for small work)
   • One doesn’t merely select a language for where it is, but also
   where it’s going — and what it represents
   • In the post-open source world of software infrastructure, this is
   reflected and guided by a community’s values
   • Beyond right tool for the job, it is the right values for the job…
   • …and then the right software for the values
   

   View Slide

3. Some software platform values
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

4. Platform core values
   • All of these values are important — but they are in tension
   • Platforms aspire to balance all of them — but for every platform
   some small subset represents its core values
   • These core values attract a like-minded community — and
   become self-reinforcing…
   

   View Slide

5. Platform core values: C
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

6. Platform core values: K
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

7. Platform core values: OpenBSD
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

8. Platform core values: Awk
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

9. Platform core values: Scala?
   • Approachability
   • Availability
   • Compatibility
   • Composability
   • Debuggability
   • Expressiveness
   • Extensibility
   • Interoperability
   • Integrity
   • Maintainability
   • Measurability
   • Operability
   • Performance
   • Portability
   • Resiliency
   • Rigor
   • Robustness
   • Safety
   • Security
   • Simplicity
   • Stability
   • Thoroughness
   • Transparency
   • Velocity
   

   View Slide

10. Values: The challenge for system software
    • For system software — that is, the software that abstracts
    hardware and serves as a generic platform — values are
    defined by expectations
    • For some values, if your system software doesn’t have it, the
    system itself can’t ever have it!
    

    View Slide

11. Values we demand of system software:
    • Approachability
    • Availability
    • Compatibility
    • Composability
    • Debuggability
    • Expressiveness
    • Extensibility
    • Interoperability
    • Integrity
    • Maintainability
    • Measurability
    • Operability
    • Performance
    • Portability
    • Resiliency
    • Rigor
    • Robustness
    • Safety
    • Security
    • Simplicity
    • Stability
    • Thoroughness
    • Transparency
    • Velocity
    

    View Slide

12. Platform core values: C
    • Approachability
    • Availability
    • Compatibility
    • Composability
    • Debuggability
    • Expressiveness
    • Extensibility
    • Interoperability
    • Integrity
    • Maintainability
    • Measurability
    • Operability
    • Performance
    • Portability
    • Resiliency
    • Rigor
    • Robustness
    • Safety
    • Security
    • Simplicity
    • Stability
    • Thoroughness
    • Transparency
    • Velocity
    

    View Slide

13. Platform core values: C++
    • Approachability
    • Availability
    • Compatibility
    • Composability
    • Debuggability
    • Expressiveness
    • Extensibility
    • Interoperability
    • Integrity
    • Maintainability
    • Measurability
    • Operability
    • Performance
    • Portability
    • Resiliency
    • Rigor
    • Robustness
    • Safety
    • Security
    • Simplicity
    • Stability
    • Thoroughness
    • Transparency
    • Velocity
    

    View Slide

14. The system software disconnect
    • There is a tremendous divide between the values held by
    system software programming languages and the values that
    must be held by the software itself
    • Over the years (decades), we in system software have been
    able to deliver relatively secure, relatively reliable systems…
    • …but it is despite the programming language, not because of it!
    • Can we do better?
    

    View Slide

15. Enter Rust?
    • Rust is a systems software programming language designed
    around safety, parallelism, and speed
    • Rust has a novel system of ownership, whereby it can statically
    determine when a memory object is no longer in use
    • This allows for the power of a garbage-collected language, but
    with the performance of manual memory management
    • This is important because — unlike C — Rust is highly
    composable, allowing for more sophisticated (and higher
    performing!) primitives
    

    View Slide

16. Platform core values: Rust
    • Approachability
    • Availability
    • Compatibility
    • Composability
    • Debuggability
    • Expressiveness
    • Extensibility
    • Interoperability
    • Integrity
    • Maintainability
    • Measurability
    • Operability
    • Performance
    • Portability
    • Resiliency
    • Rigor
    • Robustness
    • Safety
    • Security
    • Simplicity
    • Stability
    • Thoroughness
    • Transparency
    • Velocity
    

    View Slide

17. Rust performance (my experience)
    Source: http://dtrace.org/blogs/bmc/2018/09/28/the-relative-performance-of-c-and-rust/
    

    View Slide

18. Rust: Beyond ownership
    • Rust has a number of other features that make it highly
    compelling for systems software implementation:
    • Algebraic types allow robust, concise error handling
    • Hygienic macros allow for safe syntax extensions
    • Foreign function interface allows for full-duplex integration
    with C without sacrificing performance
    • “unsafe” keyword allows for some safety guarantees to be
    surgically overruled (though with obvious peril)
    • Also: terrific community, thriving ecosystem, etc.
    

    View Slide

19. Systems software in Rust: The promise
    • Rust’s values are an excellent fit for systems software
    • The beauty of Rust is that it shifts cognitive load from the
    operation of software back to the developer
    • Further, Rust’s ownership restricts what the software can do
    • The process can be frustrating but the end result is often
    satisfying:
    Source: Programming Rust by Jim Blandy and Jason Orendorff, page 262
    

    View Slide

20. Systems software in Rust: The peril
    • While Rust’s advantages are themselves clear, it’s less clear
    what the advantage is when replacing otherwise working code
    • Rust may be especially challenging in large, monolithic systems
    like an OS kernel which often have multiply-owned structures
    • So an OS kernel — despite its historic appeal and superficial fit
    for Rust — may represent more challenge than its worth
    • But what of hybrid approaches?
    

    View Slide

21. Hybrid approach I: Rust subsystems
    • One appeal of Rust is its ability to interoperate with C
    • One hybrid approach to explore would be to retain a traditional
    (C-based) system while allowing for Rust-based components
    • This would allow for an incremental approach — and instead of
    rewriting, Rust can be used for new development
    • There is a prototype example of this in FreeBSD; others are
    presumably possible
    

    View Slide

22. Hybrid approach II: Rust system components
    • System software is much broader than the operating system!
    • System software consists of many user-level components:
    utilities, daemons, service-/device-/fault- management facilities,
    debuggers, etc.
    • And then the distributed system that represents a multi-
    computer control plane — that itself includes many components
    • These components are much more prone to run-time failure!
    • Many of these are an excellent candidate for Rust!
    

    View Slide

23. Hybrid approach III: Rust-based firmware
    • Below the operating system lurks hardware-facing special-
    purpose software: firmware
    • Firmware is a sewer of unobservable software with a long
    history of infamous quality problems
    • Firmware has some of the same challenges as kernel
    development (e.g., dealing with allocation failures), but may
    otherwise be more amenable to Rust
    • This is especially true when/where firmware is in user-space
    and is network-facing! (e.g., OpenBMC)
    

    View Slide

24. Looking forward: Systems software in Rust
    • Rust represents something that we haven’t seen in a long time:
    a modern language that represents an alternative throughout
    the stack of software abstraction: we can have nice things!
    • Rust’s values are an excellent fit for system software!
    • Rust’s interoperability allows hybrid approaches, allowing for
    productive kernel incrementalism rather than whole-system
    rewrites
    • Firmware and user-level operating system software are two very
    promising candidates for implementation in Rust!
    

    View Slide