$30 off During Our Annual Pro Sale. View Details »

The Container Revolution: Reflections after the first decade

Bryan Cantrill
September 09, 2016

The Container Revolution: Reflections after the first decade

Talk given at HashiConf 2016. Video: https://www.youtube.com/watch?v=xXWaECk9XqM

Bryan Cantrill

September 09, 2016


  1. The Container Revolution Reflections after the first decade CTO bryan@joyent.com

    Bryan Cantrill @bcantrill
  2. The Container Revolution Reflections after the first third decade CTO

    bryan@joyent.com Bryan Cantrill @bcantrill
  3. Containers are born: March 18, 1982 • Containers are not

    a new idea, having originated via filesystem containers with chroot in Seventh Edition Unix • chroot originated with Bill Joy, but specifics are blurry; according to Kirk McKusick, via Poul-Henning Kamp and Robert Watson:
  4. Containers ca. 2000 • Seeking to provide a security mechanism,

    FreeBSD extended chroot into jails:
  5. • To provide workload consolidation, Sun introduced complete operating system

    virtualization with zones (née Project Kevlar) Containers ca. 2002
  6. Containers ca. 2006

  7. Elsewhere ca. 2006...

  8. Aside: Hardware-level virtualization • Since the 1960s, the preferred approach

    for operating legacy stacks unmodified has been to virtualize the hardware • A virtual machine is presented upon which each tenant runs an operating system that they choose (but must also manage) • Effective for running legacy stacks, but with a clear inefficiency: there are as many operating systems on a machine as tenants: • Operating systems are heavy and don’t play well with others with respect to resources like DRAM, CPU, I/O devices, etc.! • Still, hardware-level virtualization became de facto in the cloud
  9. Containers ca. 2011

  10. Containers ca. 2011, cont. • Via SmartOS zones, Joyent has

    run containers in multi-tenant production since ~2006... • Adding support for hardware-based virtualization circa 2011 strengthened our resolve with respect to OS-based virtualization! • We saw that the operational characteristics of OS containers — performance, elasticity, tenancy — opened up new possibilities...
  11. Containers ca. 2012/2013

  12. Containers ca. 2012/2013, cont. • Over 2012 and early 2013,

    we built Manta, a ZFS- and container- based internet-facing object store offering in situ compute • Containers allow the compute to be moved to the data instead of having to backhaul data to transient compute • And as a reminder, the OS — Unix! — was built around the notion of ad hoc computation
  13. Aside: Unix • When Unix appeared in the early 1970s,

    it was not merely a new system, but a new way of thinking about systems • The Unix philosophy, as articulated by Doug McIlroy: • Write programs that do one thing and do it well • Write programs to work together • Write programs that handle text streams, because that is a universal interface • Four decades later, this philosophy remains as relevant as ever!
  14. Aside: Doug McIlroy v. Don Knuth • In 1986, Jon

    Bentley posed the challenge that became the Epic Rap Battle of computer science history: Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. • Don Knuth’s solution: a purpose-built algorithm in WEB, a Pascal- like literate programming system of his own invention • Doug McIlroy’s solution shows the power of the Unix philosophy: tr -cs A-Za-z '\n' | tr A-Z a-z | \ sort | uniq -c | sort -rn | sed ${1}q
  15. Containers ca. 2012/2013, cont. • Manta allows for an arbitrarily

    scalable variant of McIlroy’s solution to Bentley’s challenge: mfind -t o /bcantrill/public/v7/usr/man | \ mjob create -o -m "tr -cs A-Za-z '\n' | \ tr A-Z a-z | sort | uniq -c" -r \ "awk '{ x[\$2] += \$1 } END { for (w in x) { print x[w] \" \" w } }' | \ sort -rn | sed ${1}q" • As with McIlroy’s solution, the container allows for not just for terser expression but faster execution!
  16. Containers ca. 2012/2013 • Manta served to strengthen our belief

    in the power of containers as not merely better infrastructure, but as transformative • Our biggest challenge with Manta was that containers were not well understood: it felt “easy” to us, but was confusing to many • Would the world ever really figure out containers?!
  17. Containers ca. 2013

  18. Containers ca. 2013/2014: Revolution at last! • Docker used the

    rapid provisioning + shared underlying filesystem of containers to allow developers to think operationally • Operational dependencies are encoded via an image • Images can be reliably and reproducibly deployed as a container • Images can be quickly deployed — and re-deployed • Coupled with the mainstream adoption of open source and the leading adoption of microservices... • Containers accelerate software development
  19. Containers ca. 2014/2015

  20. Containers ca. 2014/2015, cont. • Triton combines SmartOS and our

    cloud management software with a Docker Remote API endpoint • With Triton, the notion of a Docker host is virtualized: to the Docker client, the datacenter is a large Docker host • One never allocates VMs with Triton; one only allocates containers — and all Triton containers run directly on-the-metal • All of Triton is open source — you can download it and run it for yourself (or run it on our public cloud)
  21. Containers ca. 2015

  22. Containers ca. 2015

  23. Containers ca. 2015

  24. Containers ca. 2015

  25. Containers ca. 2015

  26. 2016 and beyond: Frameworks v. Libraries • The crowded container

    ecosystem can be viewed broadly as two different approaches: • A framework approach in which the framework is in control, but alternatives are “pluggable” • A library approach in which the developer/operator is in control, composing functionality out of well-defined modules • The “all in” nature of frameworks may make initial adoption easier, but they ultimately sacrifice flexibility!
  27. 2016 and beyond: Frameworks v. Libraries • For example, a

    framework approach to orchestration may not accommodate applications that demand local persistence • We opted for orchestration that lives not in an all-knowing framework, but rather with the application, in the container • In-container logic allows for service registration/discovery, health checks, etc. — but does not handle restart • This is application-centric orchestration, and we have dubbed this the autopilot pattern: https://autopilotpattern.io
  28. 2016 and beyond: Containers on autopilot • We have developed

    ContainerPilot, an embodiment of the autopilot pattern for Docker containers • All open source (https://github.com/joyent/containerpilot) — and not Triton or Joyent specific; can be used anywhere
  29. 2016 and beyond: Container failure modes

  30. 2016 and beyond: Container failure modes

  31. 2016 and beyond: Container failure modes

  32. 2016 and beyond: Failure matters • When deploying containers +

    microservices, there is an unstated truth: you are developing a distributed system • While more resilient to certain classes of force majeure failure, distributed systems remain vulnerable to software defects • We must be able to debug such systems; hope is not a strategy! • Distributed systems are hard to debug — and are more likely to exhibit behavior non-reproducible in development • With container-based systems, we must think in terms of the system, not merely the program!
  33. 2016 and beyond: The Jevons paradox • The Jevons paradox

    seems very likely to hold for containers: greater efficiency will result in a net increase in consumption! • Efficiency gains from containers are in terms of developer time... • ...but requiring containers to be scheduled in VMs induces operational inefficiencies: every operator must now think like a cloud operator — maximizing density within fixed-cost VMs • Greater consumption + operational inefficiencies threaten to slow the container revolution — or make it explosive in terms of cost • To realize the full economic promise of the container revolution, we need container-native infrastructure!
  34. 2016 and beyond: Public and on-prem • Death of on-prem

    computing is greatly exaggerated! • There are three key determinants for public v. on-premises: • Economics: Rent vs. buy; OPEX vs. CAPEX • Risk Management: Security/compliance — and also risk factors associated with operator-as-threat • Latency: The speed of light is a constant! • Economics dominates: “private cloud” efforts that do not deliver public cloud economics are doomed to (continue to) fail!
  35. 2016 and beyond: #serverless • On the one hand, this

    term is clearly meaningless… • ...but on the other, it highlights that containers have liberated us to think in terms of higher-level functionality • Thinking in functions is great (viz. Manta), but confining functions to containers to single-tenant VMs undermines their economics • The virtual machine is a vestigial abstraction; we must reject container-based infrastructure that implicitly assumes it! • So don’t say “#serverless” when you mean...
  36. 2016 and beyond #vmless