Platform as reflection of values: Joyent, node.js, and beyond

My talk from Node Summit 2017. Video: https://www.youtube.com/watch?v=Xhx970_JKX4

1. Platform as a reflection of values
   Joyent, node.js, and beyond CTO bryan@joyent.com Bryan Cantrill @bcantrill
2. Platform and values • Software platform decisions are big, important decisions with lasting consequences • One doesn’t merely select component infrastructure for where it is, but also where it’s going — and what it represents • In the post-open source world of software infrastructure, this is reflected and guided by a community’s values • Beyond right tool for the job, it is the right values for the job… • …and then the right software for the values
3. Some platform values • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
4. Platform core values • All of these values are important — but they are in tension • Platforms aspire to balance all of them — but for every platform some small subset represents its core values • These core values attract a like-minded community — and become self-reinforcing…
5. Platform core values: Seventh Edition Unix • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
6. Platform core values: OpenBSD • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
7. Platform values: K • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
8. Platform values: Awk • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
9. Platform values: ZFS • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
10. Platform values: DTrace • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
11. Platform values: node.js circa 2010 • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
12. Joyent’s core values (historically): • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
13. Joyent’s core values (now): • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
14. Growing core values • While the core values can mutate over time, it usually isn’t with the same people; core values often track people • Ideally, a community is able to grow its core values by attracting and welcoming people with those new values • In 2010, there was reason for optimism for node.js, as it was enjoying an influx of technologists bearing the scars from failed experiments in Twisted Python, Event Machine, C, etc.
15. The node.js demographic circa 2010
16. Growing core values: Debuggability • As we at Joyent built our own production infrastructure on node.js, we began to need to assert some of our own values… • It started with debuggability as we added: • DTrace probes in node.js core (2010) • DTrace probes in arbitrary node.js (2011) • MDB support (postmortem!) for node.js with mdb_v8 (2011) • DTrace ustack() helper for node.js (2011)
17. Growing core values: Rigor • As we hit performance problems in production, we made heavy use of the DTrace ustack() helper for profiling • The need for rigor in understanding this output inspired us to invent flame graphs (2012) to visualize node.js stacks:
18. Growing core values: Maintainability • As we had more and more services in node.js, we found the need for a unified logging library… • We developed Bunyan in 2012, reflecting not only our values of maintainability but also debuggability (DTrace probes built in!) • Bunyan also reflects our value of stability: we needed abstractions that we could build upon!
19. Growing core values: Robustness • We found that the de facto modules in several critical areas didn’t entirely reflect our own values — especially around robustness and debuggability • For building REST-based services we developed Restify (2011), with an emphasis on RFC compliance and debuggability • For constructing robust asynchronous pipelines, we developed vasync (2012), with a focus on error handling and debuggability
20. Growing core values: Robustness • An early problem — and one with many dynamic languages — is increasing memory usage over time • It’s easy to errantly hold a reference to an entire object graph! • We added “::findjsobjects” to mdb_v8 (2012) and have used to find many thorny production problems • We later added the ability to iterate over all functions with “::jsfunctions” (!) — and then the ability to actually print the variables in a closure with “::jsclosure” (!!)
21. Growing core values: Stability • We maintained quite a few binary add-ons — and immediately saw the need for stability in the interface • Stability does not mean intransigence or stasis! • Stability means developing the right abstractions that can survive future change on both sides of the interface boundary • C++ can make this difficult task much more challenging… • We developed v8plus (2012), a C interface for add-ons that we continue to use in lieu of NAN (or N-API)
22. Growing core values: Resiliency • Our experience trying to incorporate resiliency in our node.js- based systems led us to taxonomize node.js errors • The resulting document quickly became authoritative; searching “node.js error handling” + “I’m feeling lucky” still yields it:
23. The challenge of changing values • Just because we believed in production-oriented values like debuggability, robustness, and stability doesn’t mean these values were widely shared! • Values aren’t necessarily contagious! • The node.js community broadly identified more closely with growth-oriented values like approachability, expressiveness, and velocity • By 2014, these values were increasingly coming into conflict, and a fracture was (in retrospect) inevitable…
24. Farewell Node.js
25. Farewell Node.js, cont.
26. Farewell to TJ • TJ Holowaychuk was the most prolific node module contributor; if nothing else, probably wrote more node.js than anyone else • TJ wasn’t the first; there had been foreshocks:
27. The challenge of large communities • The departures highlight a challenge with larger communities… • When communities are large, values can easily conflict or become diluted • Smaller communities identify more strongly with their values — but larger communities are more likely to fracture over them • Sometimes this is a fork, but more often, people just leave • Sometimes the values change because people leave, but more often people leave because the values change (or fail to!)
28. Divergent values • We erred in either not recognizing or not articulating that the io.js fork in late 2014 represented divergent values • Even to the degree that we recognized it and articulated it, not clear that it mattered: the problem was the divergence itself • The problem was even more basic at root: we held aspirations for the values of node.js that it didn’t hold for itself • The values of node.js are (and probably have long been) the values of JavaScript itself…
29. Platform values: node.js circa 2015 • Approachability • Availability • Compatibility • Composability • Debuggability • Expressiveness • Extensibility • Interoperability • Integrity • Maintainability • Measurability • Operability • Performance • Portability • Resiliency • Rigor • Robustness • Safety • Security • Simplicity • Stability • Thoroughness • Transparency • Velocity
30. node.js • The values for JavaScript/node.js and our values didn’t align — and it was clear that they weren’t going to • We transitioned node.js to a foundation in 2015 • But many in the community putatively believed in our values (or wanted to), so we remained engaged • This hasn’t been easy, and one issue in particular highlights the challenge in accommodating divergent values…
31. Promises • Promises are a different way of asynchronous programming, dating back to Argus, a CLU derivative from the mid-1980s:
 
 
 
 
 
 • node.js was originally implemented in terms of promises, but removed them entirely in February 2010 (0.2!)
32. Promises • Because of their use in broader JavaScript, demand grew to reintroduce promise support in node.js… • But JavaScript promises conflate operational errors with programmatic errors in that they both become exceptions • Unlike thrown exceptions absent promises, we cannot know if a thrown exception in a promise is going to be later caught or not • This makes it challenging to abort a process on, say, a type error with enough information to debug it post-mortem • Promises make writing robust software more difficult
33. Promises • Promises embody the divergence of values in modern node.js: approachability, compatibility and expressiveness in tension with robustness, debuggability, and simplicity • If node.js must do whatever JavaScript does in the name of compatibility, is node.js permitted its own values? • This conundrum was on vivid display in the util.promisify() PR:
 
 https://github.com/nodejs/node/pull/12442

34. https://github.com/nodejs/node/pull/12442
35. https://github.com/nodejs/node/pull/12442
36. Promises • This is not an issue of someone being right or wrong! • These are two different value systems — both admirable and reasonable — coming into conflict • This issue was put to a vote: 9 in favor, 1 against, 4 abstained • But elections do not resolve differences in values • Democracy does not necessarily imply leadership!
37. The challenge for Joyent • We at Joyent are resigned to the divergence in values • This doesn’t mean we’re abandoning node.js… • …but it does mean that we are actively considering other platforms when making future software platform decisions • In the coming year(s), you will certainly see us in the Go community — and perhaps Rust or others • These communities aren’t necessarily a perfect reflection of our values either — but we feel better prepared for any divergence
38. The challenge for node.js • node.js should define what its values are — and aren’t • Growth may be an objective, but growth is not a value! • Claptrap around values that node.js doesn’t in fact have only plants the seeds of disappointment and discord • node.js should be upfront that compatibility with JavaScript will trump robustness, debuggability, stability, etc. • We remain hopeful that node.js will be a big enough tent to not run so counter to our values that we abandon it entirely!
39. Platform as a reflection of values • When picking a platform, values matter • Get consensus on values before selecting a platform! • With values in hand, the challenge becomes: how does one determine the values of a community? • How a community deals with an open issue reflects much on its values around robustness, thoroughness, rigor, etc. • It’s a balance between your values and that of a community… • …that may require you to create your own platform and attract your own community bound by like values!