The Container Operator’s Manual

Containers have been the future for five years now, featured on the stage of every major distributed systems conference in the world. But beyond the hype and the swag is a real technical solution, with real technical challenges, used for real problems at scale. And for the companies and engineers looking to adopt this solution, there’s little content on what awaits them.

In this talk, we’ll discuss some of the advantages and disadvantages of running containers, in production, at scale. We’ll address why to use containers, why not to, and the tradeoffs required at both the technical and human levels for implementing them. You will walk away with a better understanding of how containers could fit into your own architecture and what you’ll need to do to make that rollout a reality. Containers can be a great infrastructure solution, but no one should drive them without a manual.

Transcript

1. The Container
   Operator’s Manual
   @alicegoldfuss
   

   View Slide

2. I’m Alice
   SRE @
   

   View Slide

3. car commercial photo
   

   View Slide

4. * Laboratory setting
   

   View Slide

5. vintage car ad, car on
   pedestal
   

   View Slide

6. vintage car ad, car on
   pedestal
   CONTAINER SAASY
   ‘18
   

   View Slide

7. View Slide

8. View Slide

9. cars stuck in traffic
   

   View Slide

10. cars stuck in traffic
    

    View Slide

11. View Slide

12. My Credentials
    Running containers at production scale since
    early 2015
    Ran Dockerized Cassandra clusters
    Kubernetes platform team at GitHub
    

    View Slide

13. What is
    a container?
    ?
    

    View Slide

14. Host
    

    View Slide

15. Host
    OS
    

    View Slide

16. Host
    OS
    Daemon
    

    View Slide

17. Host
    OS
    C C
    C
    Daemon
    

    View Slide

18. Host
    OS
    C
    Daemon
    C C
    C
    

    View Slide

19. $ top

    PID USER %CPU %MEM TIME+ COMMAND
    34763 root 49.4 0.2 22:42.70 python
    32607 root 42.8 0.2 23:41.14 ruby
    32631 root 40.1 0.2 23:10.20 dockerd
    34900 root 38.5 0.2 22:47.33 ruby
    32720 root 37.5 0.2 23:38.04 kubelet
    

    View Slide

20. Containers are processes
    born from tarballs
    controlled by cgroups
    anchored to namespaces
    

    View Slide

21. > Dockerfile
    FROM golang:1.9.2-alpine3.6
    RUN apk add --no-cache git
    RUN go get github.com/golang/dep/cmd/dep
    COPY Gopkg.lock Gopkg.toml /go/src/project/
    WORKDIR /go/src/project/
    RUN dep ensure -vendor-only

    ...
    

    View Slide

22. PID
    

    View Slide

23. PID
    PID
    PID
    PID
    PID
    /data
    

    View Slide

24. PID
    PID
    PID
    PID
    PID
    /data
    2 GB
    mem
    5
    CPU
    

    View Slide

25. View Slide

26. View Slide

27. 4 Lessons
    

    View Slide

28. 1. Containers have
    strengths
    

    View Slide

29. Stateless Applications
    

    View Slide

30. *
    data data
    

    View Slide

31. * *
    *
    

    View Slide

32. * *
    *
    

    View Slide

33. * *
    *
    

    View Slide

34. * *
    *
    

    View Slide

35. pile of polaroids
    Ephemeral
    

    View Slide

36. * *
    

    View Slide

37. pile of polaroids
    Portable
    

    View Slide

38. old new
    

    View Slide

39. old new
    iterate
    

    View Slide

40. Disaster Recovery
    

    View Slide

41. *
    * *
    

    View Slide

42. *
    * *
    

    View Slide

43. Testing
    Environments
    

    View Slide

44. 1.0 1.1 1.2
    

    View Slide

45. 1.0 1.1 1.2
    

    View Slide

46. View Slide

47. 2. Containers have
    weaknesses
    

    View Slide

48. Apps
    Stateful
    

    View Slide

49. Databases
    

    View Slide

50. Are you Google?
    

    View Slide

51. Are you routing billions of requests to
    thousands of databases across dozens
    of data centers per second?
    

    View Slide

52. No
    

    View Slide

53. Reasons to Try
    Faster provisioning
    Stability
    Recovery
    

    View Slide

54. Reasons to Try
    Faster provisioning
    Stability
    Recovery
    

    View Slide

55. Reasons to Try
    Faster provisioning
    Stability
    Recovery
    £££
    

    View Slide

56. data
    

    View Slide

57. 2 main solutions
    

    View Slide

58. Storage
    

    View Slide

59. Storage
    

    View Slide

60. Storage
    

    View Slide

61. Storage
    

    View Slide

62. Network bound
    

    View Slide

63. /data /data
    /data
    

    View Slide

64. /data /data
    /data
    

    View Slide

65. /data /data
    /data
    

    View Slide

66. /data /data
    /data
    

    View Slide

67. My Credentials
    Running containers at production scale since
    early 2015
    Ran Dockerized Cassandra clusters
    Kubernetes platform team at GitHub
    

    View Slide

68. /data /data
    /data
    

    View Slide

69. /data /data
    /data
    

    View Slide

70. /data /data
    /data
    

    View Slide

71. /data /data
    /data
    /data
    

    View Slide

72. Network bound
    

    View Slide

73. keep it small
    

    View Slide

74. View Slide

75. Automatic failover
    Scalable dbs
    Read replicas
    Multi-regions
    Manual failover
    Scalable dbs
    Read replicas
    Multi-regions
    

    View Slide

76. 3. Containers need
    friends
    

    View Slide

77. It’s
    never
    “just”
    containers
    

    View Slide

78. Dev Setups
    How will you build your container tarballs?
    

    View Slide

79. Orchestration
    How will you schedule container resources?
    

    View Slide

80. Management
    How will you manage clusters?
    

    View Slide

81. Networking
    How will you handle routing, access control,
    service discovery?
    

    View Slide

82. View Slide

83. View Slide

84. containers
    

    View Slide

85. containers
    deployment
    

    View Slide

86. containers
    deployment
    monitoring
    

    View Slide

87. containers
    deployment
    monitoring provisioning
    

    View Slide

88. containers
    deployment
    monitoring provisioning
    debugging
    

    View Slide

89. gradual rollout
    

    View Slide

90. 1 year
    

    View Slide

91. * *
    *
    *
    *
    *
    +
    hybrid
    

    View Slide

92. 4. Containers need
    headcount
    

    View Slide

93. We need containers!
    

    View Slide

94. Let’s give it to Ops!
    

    View Slide

95. Ops Owns
    Inventory & provisioning
    Configuration management
    Networking
    Deployment tooling
    Monitoring
    Incident response
    

    View Slide

96. Ops Owns
    Inventory & provisioning
    Configuration management
    Networking
    Deployment tooling
    Monitoring
    Incident response
    +containers!!
    

    View Slide

97. new team!
    

    View Slide

98. Operations
    

    View Slide

99. Deployments
    

    View Slide

100. Tooling
     

     View Slide

101. Monitoring
     

     View Slide

102. Kernel engineer
     

     View Slide

103. Networking
     

     View Slide

104. InfoSec
     

     View Slide

105. Internal adoption
     

     View Slide

106. Project manager
     

     View Slide

107. View Slide

108. ideally 6-8 people
     

     View Slide

109. at least 4 people
     

     View Slide

110. empower them to
     succeed
     

     View Slide

111. View Slide

112. Stateless Applications
     

     View Slide

113. Databases
     

     View Slide

114. It’s
     never
     “just”
     containers
     

     View Slide

115. View Slide

116. “Should we use
     containers in
     prod?”
     

     View Slide

117. Do You…
     Have stateless services?
     A large, heterogenous platform?
     Time, money, people, org support?
     

     View Slide

118. Do You…
     Have stateless services?
     A large, heterogenous platform?
     Time, money, people, org support?
     

     View Slide

119. Do You…
     Have a monolith and few services?
     A small team with no org support?
     Just want to spite me?
     

     View Slide

120. Do You…
     Have a monolith and few services?
     A small team with no org support?
     Just want to spite me?
     

     View Slide

121. Do you want
     containers or a
     blog post?
     

     View Slide

122. Pssst

     it’s ok not to use
     containers!
     

     View Slide

123. View Slide

124. Thanks!
     @alicegoldfuss
     

     View Slide

125. Photo Credits by Slide Number
     4 http://www.oldcaradvertising.com/Buick/1948/1948%20Buick%20Ad-01.html
     6 https://vladimerbotsvadze.wordpress.com/2014/11/09/57-chevrolet-sweet-smooth-and-sassy-advertising-chevrolet/
     8 http://www.tocmp.com/pix/Cadillac/pages/56Cadillac08-or_jpg.htm
     10 Circa 1959, Photo by Ralph Crane http://grandmighty.com/2011/11/la-traffic/
     12 https://commons.wikimedia.org/wiki/File:Car_fire_I-90_Massachusetts.jpg
     26 http://www.oldcaradvertising.com/Chevrolet%20Corvette/1956/1956%20Corvette%20Ad-03.html
     27 http://www.oldcaradvertising.com/Plymouth/1955%20Plymouth/1955%20Plymouth%20Ad-03.html
     30 https://www.magazine-advertisements.com/uploads/2/1/8/4/21844100/polaroid-sx-70-land-camera-1_orig.jpg
     36 https://www.ebay.com/itm/19-1960s-70s-American-Muscle-Car-Photos-Polaroids-Ford-Mustang-Dodge-Charger/192242851917?
     _trkparms=aid%3D555017%26algo%3DPL.CASSINI%26ao%3D1%26asc%3D201607
     06104836%26meid%3D0934e94015fa43119ec01b0813d56b72%26pid%3D100642%26rk%3D1%26rkt%3D1%26%26itm%3D192242851917&_trksid=p204
     5573.c100642.m3226
     38 http://tsiarde.tictail.com/product/polaroid-camera-instant-10-with-beautiful-rainbow-colored-handle
     41 http://forums.popphoto.com/showthread.php?627010-Original-Polaroid-SX-70-Land-Camera
     44 https://www.magazine-advertisements.com/polaroid-230-land-camera.html
     47 https://www.ebay.com/itm/Lot-of-91-Vintage-Polaroid-photos-mixed-Lot-3/382437994619?hash=item590b14b07b:g:xiYAAOSwMGha1T9a
     49 https://www.pinterest.com/pin/531284087276635567/
     57 https://i.pinimg.com/originals/e7/a6/9c/e7a69c81ffbd9373c722c6f2a52d78ca.jpg
     63 http://vintage-magazine-ads.blogspot.com/
     74 http://www.voicesofeastanglia.com/wp-content/uploads/2012/01/Japanese-Make-up-Advert.jpg
     78 http://www.goretro.com/2016/01/fond-of-fondue.html
     84 http://myminkbetty.blogspot.com/2015/08/of-sewing-machines-and-tomato-soup.html
     99 https://www.pinterest.com/pin/269441990180539638/
     100 https://www.archives.gov/files/research/african-americans/ww2-pictures/images/african-americans-wwii-065.jpg
     101 http://www.shorpy.com/node/2974
     102 https://imgur.com/gallery/2jEv6
     103 https://www.cnn.com/style/article/radium-girls-radioactive-paint/index.html
     104 https://en.wikipedia.org/wiki/United_States_home_front_during_World_War_II#/media/Fil
     e:World_War_II_woman_aircraft_worker,_Vega_Aircraft_Corporation,_Burbank,_California_1942.jpg
     105 https://www.atomicheritage.org/history/african-americans-and-manhattan-project
     106 http://www.zocalopublicsquare.org/2016/01/14/the-japanese-american-flower-growers-who-made-phoenix-bloom/chronicles/who-we-were/
     107 https://www.bet.com/news/fashion-and-beauty/photos/2012/03/black-female-activists/_jcr_content/mainCol/imagegallerycontainer/gal
     leryimage_4.custom1540fx865fx0xcrop.dimg/__1330980660498/030512-fashion-black-feminist-ruby-dee.jpg
     124 http://blog.beforemario.com/2012/04/nintendo-twister-1966.html
     

     View Slide