The Container Operator’s Manual

Containers have been the future for five years now, featured on the stage of every major distributed systems conference in the world. But beyond the hype and the swag is a real technical solution, with real technical challenges, used for real problems at scale. And for the companies and engineers looking to adopt this solution, there’s little content on what awaits them.

In this talk, we’ll discuss some of the advantages and disadvantages of running containers, in production, at scale. We’ll address why to use containers, why not to, and the tradeoffs required at both the technical and human levels for implementing them. You will walk away with a better understanding of how containers could fit into your own architecture and what you’ll need to do to make that rollout a reality. Containers can be a great infrastructure solution, but no one should drive them without a manual.

Transcript

1. The Container Operator’s Manual @alicegoldfuss
2. I’m Alice SRE @
3. car commercial photo
4. * Laboratory setting
5. vintage car ad, car on pedestal
6. vintage car ad, car on pedestal CONTAINER SAASY ‘18
9. cars stuck in traffic
10. cars stuck in traffic
12. My Credentials Running containers at production scale since early 2015 Ran Dockerized Cassandra clusters Kubernetes platform team at GitHub
13. What is a container? ?
14. Host
15. Host OS
16. Host OS Daemon
17. Host OS C C C Daemon
18. Host OS C Daemon C C C
19. $ top
 PID USER %CPU %MEM TIME+ COMMAND 34763 root 49.4 0.2 22:42.70 python 32607 root 42.8 0.2 23:41.14 ruby 32631 root 40.1 0.2 23:10.20 dockerd 34900 root 38.5 0.2 22:47.33 ruby 32720 root 37.5 0.2 23:38.04 kubelet
20. Containers are processes born from tarballs controlled by cgroups anchored to namespaces
21. > Dockerfile FROM golang:1.9.2-alpine3.6 RUN apk add --no-cache git RUN go get github.com/golang/dep/cmd/dep COPY Gopkg.lock Gopkg.toml /go/src/project/ WORKDIR /go/src/project/ RUN dep ensure -vendor-only
 ...
22. PID
23. PID PID PID PID PID /data
24. PID PID PID PID PID /data 2 GB mem 5 CPU
27. 4 Lessons
28. 1. Containers have strengths
29. Stateless Applications
30. * data data
31. * * *
32. * * *
33. * * *
34. * * *
35. pile of polaroids Ephemeral
36. * *
37. pile of polaroids Portable
38. old new
39. old new iterate
40. Disaster Recovery
41. * * *
42. * * *
43. Testing Environments
44. 1.0 1.1 1.2
45. 1.0 1.1 1.2
47. 2. Containers have weaknesses
48. Apps Stateful
49. Databases
50. Are you Google?
51. Are you routing billions of requests to thousands of databases across dozens of data centers per second?
52. No
53. Reasons to Try Faster provisioning Stability Recovery
54. Reasons to Try Faster provisioning Stability Recovery
55. Reasons to Try Faster provisioning Stability Recovery £££
56. data
57. 2 main solutions
58. Storage
59. Storage
60. Storage
61. Storage
62. Network bound
63. /data /data /data
64. /data /data /data
65. /data /data /data
66. /data /data /data
67. My Credentials Running containers at production scale since early 2015 Ran Dockerized Cassandra clusters Kubernetes platform team at GitHub
68. /data /data /data
69. /data /data /data
70. /data /data /data
71. /data /data /data /data
72. Network bound
73. keep it small
75. Automatic failover Scalable dbs Read replicas Multi-regions Manual failover Scalable dbs Read replicas Multi-regions
76. 3. Containers need friends
77. It’s never “just” containers
78. Dev Setups How will you build your container tarballs?
79. Orchestration How will you schedule container resources?
80. Management How will you manage clusters?
81. Networking How will you handle routing, access control, service discovery?
84. containers
85. containers deployment
86. containers deployment monitoring
87. containers deployment monitoring provisioning
88. containers deployment monitoring provisioning debugging
89. gradual rollout
90. 1 year
91. * * * * * * + hybrid
92. 4. Containers need headcount
93. We need containers!
94. Let’s give it to Ops!
95. Ops Owns Inventory & provisioning Configuration management Networking Deployment tooling Monitoring Incident response
96. Ops Owns Inventory & provisioning Configuration management Networking Deployment tooling Monitoring Incident response +containers!!
97. new team!
98. Operations
99. Deployments
100. Tooling
101. Monitoring
102. Kernel engineer
103. Networking
104. InfoSec
105. Internal adoption
106. Project manager
108. ideally 6-8 people
109. at least 4 people
110. empower them to succeed
112. Stateless Applications
113. Databases
114. It’s never “just” containers
116. “Should we use containers in prod?”
117. Do You… Have stateless services? A large, heterogenous platform? Time, money, people, org support?
118. Do You… Have stateless services? A large, heterogenous platform? Time, money, people, org support?
119. Do You… Have a monolith and few services? A small team with no org support? Just want to spite me?
120. Do You… Have a monolith and few services? A small team with no org support? Just want to spite me?
121. Do you want containers or a blog post?
122. Pssst
 it’s ok not to use containers!
124. Thanks! @alicegoldfuss
125. Photo Credits by Slide Number 4 http://www.oldcaradvertising.com/Buick/1948/1948%20Buick%20Ad-01.html 6 https://vladimerbotsvadze.wordpress.com/2014/11/09/57-chevrolet-sweet-smooth-and-sassy-advertising-chevrolet/ 8 http://www.tocmp.com/pix/Cadillac/pages/56Cadillac08-or_jpg.htm 10 Circa 1959, Photo by Ralph Crane http://grandmighty.com/2011/11/la-traffic/ 12 https://commons.wikimedia.org/wiki/File:Car_fire_I-90_Massachusetts.jpg 26 http://www.oldcaradvertising.com/Chevrolet%20Corvette/1956/1956%20Corvette%20Ad-03.html 27 http://www.oldcaradvertising.com/Plymouth/1955%20Plymouth/1955%20Plymouth%20Ad-03.html 30 https://www.magazine-advertisements.com/uploads/2/1/8/4/21844100/polaroid-sx-70-land-camera-1_orig.jpg 36 https://www.ebay.com/itm/19-1960s-70s-American-Muscle-Car-Photos-Polaroids-Ford-Mustang-Dodge-Charger/192242851917? _trkparms=aid%3D555017%26algo%3DPL.CASSINI%26ao%3D1%26asc%3D201607 06104836%26meid%3D0934e94015fa43119ec01b0813d56b72%26pid%3D100642%26rk%3D1%26rkt%3D1%26%26itm%3D192242851917&_trksid=p204 5573.c100642.m3226 38 http://tsiarde.tictail.com/product/polaroid-camera-instant-10-with-beautiful-rainbow-colored-handle 41 http://forums.popphoto.com/showthread.php?627010-Original-Polaroid-SX-70-Land-Camera 44 https://www.magazine-advertisements.com/polaroid-230-land-camera.html 47 https://www.ebay.com/itm/Lot-of-91-Vintage-Polaroid-photos-mixed-Lot-3/382437994619?hash=item590b14b07b:g:xiYAAOSwMGha1T9a 49 https://www.pinterest.com/pin/531284087276635567/ 57 https://i.pinimg.com/originals/e7/a6/9c/e7a69c81ffbd9373c722c6f2a52d78ca.jpg 63 http://vintage-magazine-ads.blogspot.com/ 74 http://www.voicesofeastanglia.com/wp-content/uploads/2012/01/Japanese-Make-up-Advert.jpg 78 http://www.goretro.com/2016/01/fond-of-fondue.html 84 http://myminkbetty.blogspot.com/2015/08/of-sewing-machines-and-tomato-soup.html 99 https://www.pinterest.com/pin/269441990180539638/ 100 https://www.archives.gov/files/research/african-americans/ww2-pictures/images/african-americans-wwii-065.jpg 101 http://www.shorpy.com/node/2974 102 https://imgur.com/gallery/2jEv6 103 https://www.cnn.com/style/article/radium-girls-radioactive-paint/index.html 104 https://en.wikipedia.org/wiki/United_States_home_front_during_World_War_II#/media/Fil e:World_War_II_woman_aircraft_worker,_Vega_Aircraft_Corporation,_Burbank,_California_1942.jpg 105 https://www.atomicheritage.org/history/african-americans-and-manhattan-project 106 http://www.zocalopublicsquare.org/2016/01/14/the-japanese-american-flower-growers-who-made-phoenix-bloom/chronicles/who-we-were/ 107 https://www.bet.com/news/fashion-and-beauty/photos/2012/03/black-female-activists/_jcr_content/mainCol/imagegallerycontainer/gal leryimage_4.custom1540fx865fx0xcrop.dimg/__1330980660498/030512-fashion-black-feminist-ruby-dee.jpg 124 http://blog.beforemario.com/2012/04/nintendo-twister-1966.html