The Container Operator’s Manual

The Container Operator’s Manual

Containers have been the future for five years now, featured on the stage of every major distributed systems conference in the world. But beyond the hype and the swag is a real technical solution, with real technical challenges, used for real problems at scale. And for the companies and engineers looking to adopt this solution, there’s little content on what awaits them.

In this talk, we’ll discuss some of the advantages and disadvantages of running containers, in production, at scale. We’ll address why to use containers, why not to, and the tradeoffs required at both the technical and human levels for implementing them. You will walk away with a better understanding of how containers could fit into your own architecture and what you’ll need to do to make that rollout a reality. Containers can be a great infrastructure solution, but no one should drive them without a manual.

C7b0422e97da85aabf114cc8591a10a2?s=128

Alice Goldfuss

June 27, 2018
Tweet

Transcript

  1. The Container Operator’s Manual @alicegoldfuss

  2. I’m Alice SRE @

  3. car commercial photo

  4. * Laboratory setting

  5. vintage car ad, car on pedestal

  6. vintage car ad, car on pedestal CONTAINER SAASY ‘18

  7. None
  8. None
  9. cars stuck in traffic

  10. cars stuck in traffic

  11. None
  12. My Credentials Running containers at production scale since early 2015

    Ran Dockerized Cassandra clusters Kubernetes platform team at GitHub
  13. What is a container? ?

  14. Host

  15. Host OS

  16. Host OS Daemon

  17. Host OS C C C Daemon

  18. Host OS C Daemon C C C

  19. $ top
 PID USER %CPU %MEM TIME+ COMMAND 34763 root

    49.4 0.2 22:42.70 python 32607 root 42.8 0.2 23:41.14 ruby 32631 root 40.1 0.2 23:10.20 dockerd 34900 root 38.5 0.2 22:47.33 ruby 32720 root 37.5 0.2 23:38.04 kubelet
  20. Containers are processes born from tarballs controlled by cgroups anchored

    to namespaces
  21. > Dockerfile FROM golang:1.9.2-alpine3.6 RUN apk add --no-cache git RUN

    go get github.com/golang/dep/cmd/dep COPY Gopkg.lock Gopkg.toml /go/src/project/ WORKDIR /go/src/project/ RUN dep ensure -vendor-only
 ...
  22. PID

  23. PID PID PID PID PID /data

  24. PID PID PID PID PID /data 2 GB mem 5

    CPU
  25. None
  26. None
  27. 4 Lessons

  28. 1. Containers have strengths

  29. Stateless Applications

  30. * data data

  31. * * *

  32. * * *

  33. * * *

  34. * * *

  35. pile of polaroids Ephemeral

  36. * *

  37. pile of polaroids Portable

  38. old new

  39. old new iterate

  40. Disaster Recovery

  41. * * *

  42. * * *

  43. Testing Environments

  44. 1.0 1.1 1.2

  45. 1.0 1.1 1.2

  46. None
  47. 2. Containers have weaknesses

  48. Apps Stateful

  49. Databases

  50. Are you Google?

  51. Are you routing billions of requests to thousands of databases

    across dozens of data centers per second?
  52. No

  53. Reasons to Try Faster provisioning Stability Recovery

  54. Reasons to Try Faster provisioning Stability Recovery

  55. Reasons to Try Faster provisioning Stability Recovery £££

  56. data

  57. 2 main solutions

  58. Storage

  59. Storage

  60. Storage

  61. Storage

  62. Network bound

  63. /data /data /data

  64. /data /data /data

  65. /data /data /data

  66. /data /data /data

  67. My Credentials Running containers at production scale since early 2015

    Ran Dockerized Cassandra clusters Kubernetes platform team at GitHub
  68. /data /data /data

  69. /data /data /data

  70. /data /data /data

  71. /data /data /data /data

  72. Network bound

  73. keep it small

  74. None
  75. Automatic failover Scalable dbs Read replicas Multi-regions Manual failover Scalable

    dbs Read replicas Multi-regions
  76. 3. Containers need friends

  77. It’s never “just” containers

  78. Dev Setups How will you build your container tarballs?

  79. Orchestration How will you schedule container resources?

  80. Management How will you manage clusters?

  81. Networking How will you handle routing, access control, service discovery?

  82. None
  83. None
  84. containers

  85. containers deployment

  86. containers deployment monitoring

  87. containers deployment monitoring provisioning

  88. containers deployment monitoring provisioning debugging

  89. gradual rollout

  90. 1 year

  91. * * * * * * + hybrid

  92. 4. Containers need headcount

  93. We need containers!

  94. Let’s give it to Ops!

  95. Ops Owns Inventory & provisioning Configuration management Networking Deployment tooling

    Monitoring Incident response
  96. Ops Owns Inventory & provisioning Configuration management Networking Deployment tooling

    Monitoring Incident response +containers!!
  97. new team!

  98. Operations

  99. Deployments

  100. Tooling

  101. Monitoring

  102. Kernel engineer

  103. Networking

  104. InfoSec

  105. Internal adoption

  106. Project manager

  107. None
  108. ideally 6-8 people

  109. at least 4 people

  110. empower them to succeed

  111. None
  112. Stateless Applications

  113. Databases

  114. It’s never “just” containers

  115. None
  116. “Should we use containers in prod?”

  117. Do You… Have stateless services? A large, heterogenous platform? Time,

    money, people, org support?
  118. Do You… Have stateless services? A large, heterogenous platform? Time,

    money, people, org support?
  119. Do You… Have a monolith and few services? A small

    team with no org support? Just want to spite me?
  120. Do You… Have a monolith and few services? A small

    team with no org support? Just want to spite me?
  121. Do you want containers or a blog post?

  122. Pssst
 it’s ok not to use containers!

  123. None
  124. Thanks! @alicegoldfuss

  125. Photo Credits by Slide Number 4 http://www.oldcaradvertising.com/Buick/1948/1948%20Buick%20Ad-01.html 6 https://vladimerbotsvadze.wordpress.com/2014/11/09/57-chevrolet-sweet-smooth-and-sassy-advertising-chevrolet/ 8

    http://www.tocmp.com/pix/Cadillac/pages/56Cadillac08-or_jpg.htm 10 Circa 1959, Photo by Ralph Crane http://grandmighty.com/2011/11/la-traffic/ 12 https://commons.wikimedia.org/wiki/File:Car_fire_I-90_Massachusetts.jpg 26 http://www.oldcaradvertising.com/Chevrolet%20Corvette/1956/1956%20Corvette%20Ad-03.html 27 http://www.oldcaradvertising.com/Plymouth/1955%20Plymouth/1955%20Plymouth%20Ad-03.html 30 https://www.magazine-advertisements.com/uploads/2/1/8/4/21844100/polaroid-sx-70-land-camera-1_orig.jpg 36 https://www.ebay.com/itm/19-1960s-70s-American-Muscle-Car-Photos-Polaroids-Ford-Mustang-Dodge-Charger/192242851917? _trkparms=aid%3D555017%26algo%3DPL.CASSINI%26ao%3D1%26asc%3D201607 06104836%26meid%3D0934e94015fa43119ec01b0813d56b72%26pid%3D100642%26rk%3D1%26rkt%3D1%26%26itm%3D192242851917&_trksid=p204 5573.c100642.m3226 38 http://tsiarde.tictail.com/product/polaroid-camera-instant-10-with-beautiful-rainbow-colored-handle 41 http://forums.popphoto.com/showthread.php?627010-Original-Polaroid-SX-70-Land-Camera 44 https://www.magazine-advertisements.com/polaroid-230-land-camera.html 47 https://www.ebay.com/itm/Lot-of-91-Vintage-Polaroid-photos-mixed-Lot-3/382437994619?hash=item590b14b07b:g:xiYAAOSwMGha1T9a 49 https://www.pinterest.com/pin/531284087276635567/ 57 https://i.pinimg.com/originals/e7/a6/9c/e7a69c81ffbd9373c722c6f2a52d78ca.jpg 63 http://vintage-magazine-ads.blogspot.com/ 74 http://www.voicesofeastanglia.com/wp-content/uploads/2012/01/Japanese-Make-up-Advert.jpg 78 http://www.goretro.com/2016/01/fond-of-fondue.html 84 http://myminkbetty.blogspot.com/2015/08/of-sewing-machines-and-tomato-soup.html 99 https://www.pinterest.com/pin/269441990180539638/ 100 https://www.archives.gov/files/research/african-americans/ww2-pictures/images/african-americans-wwii-065.jpg 101 http://www.shorpy.com/node/2974 102 https://imgur.com/gallery/2jEv6 103 https://www.cnn.com/style/article/radium-girls-radioactive-paint/index.html 104 https://en.wikipedia.org/wiki/United_States_home_front_during_World_War_II#/media/Fil e:World_War_II_woman_aircraft_worker,_Vega_Aircraft_Corporation,_Burbank,_California_1942.jpg 105 https://www.atomicheritage.org/history/african-americans-and-manhattan-project 106 http://www.zocalopublicsquare.org/2016/01/14/the-japanese-american-flower-growers-who-made-phoenix-bloom/chronicles/who-we-were/ 107 https://www.bet.com/news/fashion-and-beauty/photos/2012/03/black-female-activists/_jcr_content/mainCol/imagegallerycontainer/gal leryimage_4.custom1540fx865fx0xcrop.dimg/__1330980660498/030512-fashion-black-feminist-ruby-dee.jpg 124 http://blog.beforemario.com/2012/04/nintendo-twister-1966.html