The Container Revolution Reflections after the first decade CTO [email protected] Bryan Cantrill @bcantrill

The Container Revolution Reflections after the first third decade CTO [email protected] Bryan Cantrill @bcantrill

Containers are born: March 18, 1982 • Containers are not a new idea, having originated via filesystem containers with chroot in Seventh Edition Unix • chroot originated with Bill Joy, but specifics are blurry; according to Kirk McKusick, via Poul-Henning Kamp and Robert Watson:

Containers ca. 2000 • Seeking to provide a security mechanism, FreeBSD extended chroot into jails:

• To provide workload consolidation, Sun introduced complete operating system virtualization with zones (née Project Kevlar) Containers ca. 2002

Containers ca. 2006

Elsewhere ca. 2006...

Aside: Hardware-level virtualization • Since the 1960s, the preferred approach for operating legacy stacks unmodified has been to virtualize the hardware • A virtual machine is presented upon which each tenant runs an operating system that they choose (but must also manage) • Effective for running legacy stacks, but with a clear inefficiency: there are as many operating systems on a machine as tenants: • Operating systems are heavy and don’t play well with others with respect to resources like DRAM, CPU, I/O devices, etc.! • Still, hardware-level virtualization became de facto in the cloud

Containers ca. 2011

Containers ca. 2011, cont. • Via SmartOS zones, Joyent has run containers in multi-tenant production since ~2006... • Adding support for hardware-based virtualization circa 2011 strengthened our resolve with respect to OS-based virtualization! • We saw that the operational characteristics of OS containers — performance, elasticity, tenancy — opened up new possibilities...

Containers ca. 2012/2013

Containers ca. 2012/2013, cont. • Over 2012 and early 2013, we built Manta, a ZFS- and container- based internet-facing object store offering in situ compute • Containers allow the compute to be moved to the data instead of having to backhaul data to transient compute • And as a reminder, the OS — Unix! — was built around the notion of ad hoc computation

Aside: Unix • When Unix appeared in the early 1970s, it was not merely a new system, but a new way of thinking about systems • The Unix philosophy, as articulated by Doug McIlroy: • Write programs that do one thing and do it well • Write programs to work together • Write programs that handle text streams, because that is a universal interface • Four decades later, this philosophy remains as relevant as ever!

Aside: Doug McIlroy v. Don Knuth • In 1986, Jon Bentley posed the challenge that became the Epic Rap Battle of computer science history: Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. • Don Knuth’s solution: a purpose-built algorithm in WEB, a Pascal- like literate programming system of his own invention • Doug McIlroy’s solution shows the power of the Unix philosophy: tr -cs A-Za-z '\n' | tr A-Z a-z | \ sort | uniq -c | sort -rn | sed ${1}q

Containers ca. 2012/2013, cont. • Manta allows for an arbitrarily scalable variant of McIlroy’s solution to Bentley’s challenge: mfind -t o /bcantrill/public/v7/usr/man | \ mjob create -o -m "tr -cs A-Za-z '\n' | \ tr A-Z a-z | sort | uniq -c" -r \ "awk '{ x[\$2] += \$1 } END { for (w in x) { print x[w] \" \" w } }' | \ sort -rn | sed ${1}q" • As with McIlroy’s solution, the container allows for not just for terser expression but faster execution!

Containers ca. 2012/2013 • Manta served to strengthen our belief in the power of containers as not merely better infrastructure, but as transformative • Our biggest challenge with Manta was that containers were not well understood: it felt “easy” to us, but was confusing to many • Would the world ever really figure out containers?!

Containers ca. 2013

Containers ca. 2013/2014: Revolution at last! • Docker used the rapid provisioning + shared underlying filesystem of containers to allow developers to think operationally • Operational dependencies are encoded via an image • Images can be reliably and reproducibly deployed as a container • Images can be quickly deployed — and re-deployed • Coupled with the mainstream adoption of open source and the leading adoption of microservices... • Containers accelerate software development

Containers ca. 2014/2015

Containers ca. 2014/2015, cont. • Triton combines SmartOS and our cloud management software with a Docker Remote API endpoint • With Triton, the notion of a Docker host is virtualized: to the Docker client, the datacenter is a large Docker host • One never allocates VMs with Triton; one only allocates containers — and all Triton containers run directly on-the-metal • All of Triton is open source — you can download it and run it for yourself (or run it on our public cloud)

Containers ca. 2015

Containers ca. 2015

Containers ca. 2015

Containers ca. 2015

Containers ca. 2015

2016 and beyond: Frameworks v. Libraries • The crowded container ecosystem can be viewed broadly as two different approaches: • A framework approach in which the framework is in control, but alternatives are “pluggable” • A library approach in which the developer/operator is in control, composing functionality out of well-defined modules • The “all in” nature of frameworks may make initial adoption easier, but they ultimately sacrifice flexibility!

2016 and beyond: Frameworks v. Libraries • For example, a framework approach to orchestration may not accommodate applications that demand local persistence • We opted for orchestration that lives not in an all-knowing framework, but rather with the application, in the container • In-container logic allows for service registration/discovery, health checks, etc. — but does not handle restart • This is application-centric orchestration, and we have dubbed this the autopilot pattern: https://autopilotpattern.io

2016 and beyond: Containers on autopilot • We have developed ContainerPilot, an embodiment of the autopilot pattern for Docker containers • All open source (https://github.com/joyent/containerpilot) — and not Triton or Joyent specific; can be used anywhere

2016 and beyond: Container failure modes

2016 and beyond: Container failure modes

2016 and beyond: Container failure modes

2016 and beyond: Failure matters • When deploying containers + microservices, there is an unstated truth: you are developing a distributed system • While more resilient to certain classes of force majeure failure, distributed systems remain vulnerable to software defects • We must be able to debug such systems; hope is not a strategy! • Distributed systems are hard to debug — and are more likely to exhibit behavior non-reproducible in development • With container-based systems, we must think in terms of the system, not merely the program!

2016 and beyond: The Jevons paradox • The Jevons paradox seems very likely to hold for containers: greater efficiency will result in a net increase in consumption! • Efficiency gains from containers are in terms of developer time... • ...but requiring containers to be scheduled in VMs induces operational inefficiencies: every operator must now think like a cloud operator — maximizing density within fixed-cost VMs • Greater consumption + operational inefficiencies threaten to slow the container revolution — or make it explosive in terms of cost • To realize the full economic promise of the container revolution, we need container-native infrastructure!

2016 and beyond: Public and on-prem • Death of on-prem computing is greatly exaggerated! • There are three key determinants for public v. on-premises: • Economics: Rent vs. buy; OPEX vs. CAPEX • Risk Management: Security/compliance — and also risk factors associated with operator-as-threat • Latency: The speed of light is a constant! • Economics dominates: “private cloud” efforts that do not deliver public cloud economics are doomed to (continue to) fail!

2016 and beyond: #serverless • On the one hand, this term is clearly meaningless… • ...but on the other, it highlights that containers have liberated us to think in terms of higher-level functionality • Thinking in functions is great (viz. Manta), but confining functions to containers to single-tenant VMs undermines their economics • The virtual machine is a vestigial abstraction; we must reject container-based infrastructure that implicitly assumes it! • So don’t say “#serverless” when you mean...

2016 and beyond #vmless