Leaping the chasm from proprietary to open: A survivor's guide

My talk from #oscon 2015. Video at https://www.youtube.com/watch?v=Zpnncakrelk

Transcript

1. Leaping the chasm from proprietary to open: A survivor’s guide CTO Bryan Cantrill @bcantrill
2. Open source: In the beginning • In the beginning, it was all open: software shipped with its source • IBM led the move to proprietary enterprise software with their move to an “object-only” model in 1983 • The rise of the personal computer in the 1980s led to the broader rise of proprietary software, with the tone set by Bill Gates’ infamous 1976 letter, “An Open Letter to Hobbyists” • By the mid-1990s, software was commercial, proprietary — and entirely ossified, with many domains (e.g., languages, operating systems, databases) considered “done”
3. Open source: The internet cometh • The rise of the internet (and especially, Perl and Apache) brought with it an ethos of sharing source to collaboratively develop it • Free-and-open dominated among de novo non-commercial systems software: languages (Perl, Python) databases (MySQL, Postgres), web servers (Apache), OSs (Linux, BSD) • Proprietary software remained, but with cost often driven to $0 • The early internet can be seen as a growing chasm between free- but-proprietary (e.g. Java) and free-and-open (e.g. Perl)
4. Open source in the 2000s • In the Dot Com Bust, economics dictated that open source (and commodity hardware!) be used wherever possible • Companies would occasionally contribute back the changes they made to the software they used, but only rarely did they open source software that they themselves invented • Counter-examples were in infrastructural software domains in which open source had become a hard requirement...
5. Leaping the chasm: OpenSolaris • Sun — alone among the proprietary Unix vendors — chose the arduous path of open sourcing its OS to assure its vitality • Solaris was open sourced in June 2005 — and was showcased by Sun engineers at OSCON in August (a decade ago!) • To balance community needs with IHV wants, Sun chose a weak copy-left (CDDL) that amounted to a cleaned-up MPL • After being open sourced, Solaris was developed entirely in the open, becoming perhaps the largest single software project to completely transition from proprietary to open
6. Oracle pulls OpenSolaris back into the abyss • After Oracle acquired Sun, it was clear that there was no interest in open source — and zero appreciation of its social contract • On Friday, August 13th an internal memo closed OpenSolaris: We will distribute updates to approved CDDL or other open source-licensed code following full releases of our enterprise Solaris operating system. In this manner, new technology innovations will show up in our releases before anywhere else. We will no longer distribute source code for the entirety of the Solaris operating system in real-time while it is developed, on a nightly basis. • Oracle’s depraved (and cowardly!) act foreshadowed even more diabolical behavior: the assertion that APIs can be copyrighted!
7. The rise of illumos • Oracle couldn’t actually close OpenSolaris; they could only fork it • A new community, illumos, formed from the ashes of OpenSolaris, aided by (but much larger than!) the Solaris diaspora • Five years later, illumos is thriving with an established track record for innovation (e.g., ZFS, DTrace, containers), new blood, and multiple distributions (e.g., OmniOS, Joyent’s SmartOS) • See http://illumos.org, http://omnios.omniti.com, http://smartos.org
8. Open sourcing shrink-wrapped software • The OpenSolaris experience can be viewed as an object lesson in open sourcing traditional, licensed, shrink-wrapped software • Leaping the chasm from proprietary to open is (ultimately) a business decision, and you will need to make the business case • You need to move past fear to opportunity: think of open source not as endangering revenue but rather as driving adoption • Reminder: those that will operate and self-support your software weren’t going to pay you anything anyway! • And no, your competitors aren’t going to steal it
9. Open sourcing shrink-wrapped software, cont. • Don’t be forkaphobic; the easier it is to fork your software, the harder it is to fork the community! • Don’t fetishize governance and/or process — for most open source projects, core team + consensus is just fine (there are exceptions to this; stay tuned!) • Prefer permissive (or weak copy-left) licenses over strong copy- left ones — license incompatibility impedes adoption • Elevate your people! Organizations don’t innovate, people do — and with open source, you can provide that transparency!
10. Open source and the cloud • With the rise of cloud computing in the late-2000s, open source became ever more entrenched, with more professionally-written software being born in the open • Trend was both accelerated and reflected by the rise of GitHub, which greatly reduced the cost of open sourcing software • Large companies began open sourcing important de novo software e.g. Yahoo’s Hadoop, Google’s V8, Facebook’s Hive • This has accelerated so much that it is now an engineering challenge to determine which open source project to use!
11. SaaS: Proprietary software strikes back! • But the cloud also represented a new kind of proprietary software: proprietary software made available as a service • Lock-in from services — especially platforms-as-a-service — can be even more acute than that from shrink-wrapped software! • Adding insult to injury, these proprietary services are often (quietly) built on open source components — without so much as patches being offered back! • The new vector for proprietary software reveals GPL’s copy-left to be an obviated notion — and that coercion is the wrong approach to encourage contribution
12. Leaping the chasm: SmartDataCenter • At Joyent, our core technologies — SmartOS and node.js — have always been open source, but the distributed systems and services that we built and operated based on them were not • First among these is SmartDataCenter, our SmartOS- and container-based cloud orchestration software that we run ourselves as the Joyent Public Cloud — and sell to others • Open sourcing software that runs your service is technically hard; it is easy for implicit operational dependencies to creep in • Risk is higher than with open sourcing software you merely use, as code (and histories!) must be carefully scrubbed of secrets
13. Leaping the chasm: Manta • But wait, it gets worse: proprietary systems and services can easily build dependencies on other proprietary services • We had built a significant dependency on a new, container-based storage service that we had built, Manta • We couldn’t meaningfully open source SmartDataCenter without also open sourcing Manta, complicating the effort significantly • If it needs to be said: it is much easier to be born open than to become open — all software should be built assuming that it is already open!
14. Leaping the chasm: SDC + Manta • For many years, we had wanted to open source SDC and Manta • It took much longer than one would like, in part because open sourcing proprietary software is so easy to veto or (worse!) defer • In making the case for open sourcing SDC + Manta, we made the familiar arguments for open sourcing shrink-wrapped software: open source as lead generation, as early evaluation program, etc. • But two other arguments ultimately carried the day…
15. Infrastructure software must be open source! • For infrastructure software (e.g., operating systems, databases, runtimes), open source is now a constraint on the problem • In this regard, open source has unequivocally won • This trend is continuing upstack — software used as foundation and/or component is overwhelmingly biased to be open in the limit: increased adoption feeds network effects • Docker is a particularly strong testament to the transformative power of open sourcing infrastructure software...
16. Open source is our best vector for hiring • Joyent — like most companies — struggles to find the right talent • University hiring has reasonable fidelity, but is increasingly expensive/competitive — and can result in a monoculture • Open source communities self-identify and self-select, attracting those who are drawn to the mission • Participation in our open source communities is a much better analogue for the work that we actually do! • Our open source communities are our highest fidelity, lowest cost vector for hiring — it’s our farm system!
17. Leaping the chasm: SDC + Manta! • On November 6, 2014, we open sourced SDC and Manta! • We moved to become an all-open company; the only proprietary software is that which contains true secrets (e.g., billing software) • e.g., Triton is our new service for running Docker containers directly on the metal — and was born open • We have already found that our technologies are being used in places and by people where it wouldn’t have been otherwise • And our first hire because of open source happened mere weeks after we open sourced it! (And others have followed…)
18. SDC + Manta: Licensing! • Based on our previous experiences, we: • Love permissive licensing, but hate license incompatibilities • Love contributors, but hate contributor license agreements • Based on this, we selected the Mozilla Public License 2.0 • For a commercial entity, the MPL has some nice CYA attributes: e.g., explicit warranting of original work, patent troll protection • The MPL is explicitly compatible with other open source licenses • The MPL is a great license that merits broader consideration!
19. SDC + Manta: Governance! • Most open source projects can be entirely consensus driven — the patch rate is low enough and the number of contributors small enough that everyone can effectively look at everything • If projects become more popular, the model can be extended with a core team that can review work and guide contributors • Most projects — including (for the moment) SDC and Manta — can stop there: GitHub is all the governance one needs • So while there is generally no need for elections or foundations or any other such madness, there are, of course, exceptions...
20. Open source: When to consider a foundation • Some (very few!) open source projects become so popular that multiple, new commercial entities are created around them • For these projects, no corporate steward is sufficiently neutral: commercial tensions will threaten to rip a project apart • In these rare cases, projects may be better served in a foundation • This can be very hard to internalize — especially to a corporate steward that believes that they are acting (and have acted) in the best interests of the community! • Yes, we learned this the hard way with node.js!
21. The limitations of foundations • Foundations should be used and created only sparingly! • Foundations pose more questions than they answer: • How are board seats determined? (Pay-to-play? Community elections? Code contributions?) • How are conflicts resolved? (By consensus? By voting?) • What are the principles that guide the work? (Does the mission of the foundation go beyond mere self-preservation?) • Foundations are only better than the alternatives
22. Leaping the chasm • While there is still a chasm between proprietary and open source software, open source is broadly winning — especially in infrastructure software • It is much easier to open source when a project is still young — and being open may take the project in interesting directions • For commercial entities, challenges have shifted from “when do I open source?” to “when does this need to be in a foundation?” • Participating in and leading open source efforts as a commercial entity is an ongoing education; check back at OSCON 2025!
23. Thank you! • SmartDataCenter: https://github.com/joyent/sdc • Manta: https://github.com/joyent/manta • Thanks to the many Joyent engineers who made open source SDC and Manta happen, especially @trentmick, @rmustacc, @dapsays, @jmclulow, Keith Wesolowski, @pfmooney, @joshwilsdon, Jerry Jelinek, @kusor, @notmatt and @orlandov • And did I mention that we’re hiring? ;) (Remote-friendly!)