Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Container Operator’s Manual

The Container Operator’s Manual

Containers have been the future for five years now, featured on the stage of every major distributed systems conference in the world. But beyond the hype and the swag is a real technical solution, with real technical challenges, used for real problems at scale. And for the companies and engineers looking to adopt this solution, there’s little content on what awaits them.

In this talk, we’ll discuss some of the advantages and disadvantages of running containers, in production, at scale. We’ll address why to use containers, why not to, and the tradeoffs required at both the technical and human levels for implementing them. You will walk away with a better understanding of how containers could fit into your own architecture and what you’ll need to do to make that rollout a reality. Containers can be a great infrastructure solution, but no one should drive them without a manual.

Alice Goldfuss

June 27, 2018
Tweet

More Decks by Alice Goldfuss

Other Decks in Technology

Transcript

  1. The Container
    Operator’s Manual
    @alicegoldfuss

    View full-size slide

  2. I’m Alice
    SRE @

    View full-size slide

  3. car commercial photo

    View full-size slide

  4. * Laboratory setting

    View full-size slide

  5. vintage car ad, car on
    pedestal

    View full-size slide

  6. vintage car ad, car on
    pedestal
    CONTAINER SAASY
    ‘18

    View full-size slide

  7. cars stuck in traffic

    View full-size slide

  8. cars stuck in traffic

    View full-size slide

  9. My Credentials
    Running containers at production scale since
    early 2015
    Ran Dockerized Cassandra clusters
    Kubernetes platform team at GitHub

    View full-size slide

  10. What is
    a container?
    ?

    View full-size slide

  11. Host
    OS
    Daemon

    View full-size slide

  12. Host
    OS
    C C
    C
    Daemon

    View full-size slide

  13. Host
    OS
    C
    Daemon
    C C
    C

    View full-size slide

  14. $ top

    PID USER %CPU %MEM TIME+ COMMAND
    34763 root 49.4 0.2 22:42.70 python
    32607 root 42.8 0.2 23:41.14 ruby
    32631 root 40.1 0.2 23:10.20 dockerd
    34900 root 38.5 0.2 22:47.33 ruby
    32720 root 37.5 0.2 23:38.04 kubelet

    View full-size slide

  15. Containers are processes
    born from tarballs
    controlled by cgroups
    anchored to namespaces

    View full-size slide

  16. > Dockerfile
    FROM golang:1.9.2-alpine3.6
    RUN apk add --no-cache git
    RUN go get github.com/golang/dep/cmd/dep
    COPY Gopkg.lock Gopkg.toml /go/src/project/
    WORKDIR /go/src/project/
    RUN dep ensure -vendor-only

    ...

    View full-size slide

  17. PID
    PID
    PID
    PID
    PID
    /data

    View full-size slide

  18. PID
    PID
    PID
    PID
    PID
    /data
    2 GB
    mem
    5
    CPU

    View full-size slide

  19. 1. Containers have
    strengths

    View full-size slide

  20. Stateless Applications

    View full-size slide

  21. pile of polaroids
    Ephemeral

    View full-size slide

  22. pile of polaroids
    Portable

    View full-size slide

  23. old new
    iterate

    View full-size slide

  24. Disaster Recovery

    View full-size slide

  25. Testing
    Environments

    View full-size slide

  26. 2. Containers have
    weaknesses

    View full-size slide

  27. Apps
    Stateful

    View full-size slide

  28. Are you Google?

    View full-size slide

  29. Are you routing billions of requests to
    thousands of databases across dozens
    of data centers per second?

    View full-size slide

  30. Reasons to Try
    Faster provisioning
    Stability
    Recovery

    View full-size slide

  31. Reasons to Try
    Faster provisioning
    Stability
    Recovery

    View full-size slide

  32. Reasons to Try
    Faster provisioning
    Stability
    Recovery
    £££

    View full-size slide

  33. 2 main solutions

    View full-size slide

  34. Network bound

    View full-size slide

  35. /data /data
    /data

    View full-size slide

  36. /data /data
    /data

    View full-size slide

  37. /data /data
    /data

    View full-size slide

  38. /data /data
    /data

    View full-size slide

  39. My Credentials
    Running containers at production scale since
    early 2015
    Ran Dockerized Cassandra clusters
    Kubernetes platform team at GitHub

    View full-size slide

  40. /data /data
    /data

    View full-size slide

  41. /data /data
    /data

    View full-size slide

  42. /data /data
    /data

    View full-size slide

  43. /data /data
    /data
    /data

    View full-size slide

  44. Network bound

    View full-size slide

  45. keep it small

    View full-size slide

  46. Automatic failover
    Scalable dbs
    Read replicas
    Multi-regions
    Manual failover
    Scalable dbs
    Read replicas
    Multi-regions

    View full-size slide

  47. 3. Containers need
    friends

    View full-size slide

  48. It’s
    never
    “just”
    containers

    View full-size slide

  49. Dev Setups
    How will you build your container tarballs?

    View full-size slide

  50. Orchestration
    How will you schedule container resources?

    View full-size slide

  51. Management
    How will you manage clusters?

    View full-size slide

  52. Networking
    How will you handle routing, access control,
    service discovery?

    View full-size slide

  53. containers
    deployment

    View full-size slide

  54. containers
    deployment
    monitoring

    View full-size slide

  55. containers
    deployment
    monitoring provisioning

    View full-size slide

  56. containers
    deployment
    monitoring provisioning
    debugging

    View full-size slide

  57. gradual rollout

    View full-size slide

  58. * *
    *
    *
    *
    *
    +
    hybrid

    View full-size slide

  59. 4. Containers need
    headcount

    View full-size slide

  60. We need containers!

    View full-size slide

  61. Let’s give it to Ops!

    View full-size slide

  62. Ops Owns
    Inventory & provisioning
    Configuration management
    Networking
    Deployment tooling
    Monitoring
    Incident response

    View full-size slide

  63. Ops Owns
    Inventory & provisioning
    Configuration management
    Networking
    Deployment tooling
    Monitoring
    Incident response
    +containers!!

    View full-size slide

  64. Kernel engineer

    View full-size slide

  65. Internal adoption

    View full-size slide

  66. Project manager

    View full-size slide

  67. ideally 6-8 people

    View full-size slide

  68. at least 4 people

    View full-size slide

  69. empower them to
    succeed

    View full-size slide

  70. Stateless Applications

    View full-size slide

  71. It’s
    never
    “just”
    containers

    View full-size slide

  72. “Should we use
    containers in
    prod?”

    View full-size slide

  73. Do You…
    Have stateless services?
    A large, heterogenous platform?
    Time, money, people, org support?

    View full-size slide

  74. Do You…
    Have stateless services?
    A large, heterogenous platform?
    Time, money, people, org support?

    View full-size slide

  75. Do You…
    Have a monolith and few services?
    A small team with no org support?
    Just want to spite me?

    View full-size slide

  76. Do You…
    Have a monolith and few services?
    A small team with no org support?
    Just want to spite me?

    View full-size slide

  77. Do you want
    containers or a
    blog post?

    View full-size slide

  78. Pssst

    it’s ok not to use
    containers!

    View full-size slide

  79. Thanks!
    @alicegoldfuss

    View full-size slide

  80. Photo Credits by Slide Number
    4 http://www.oldcaradvertising.com/Buick/1948/1948%20Buick%20Ad-01.html
    6 https://vladimerbotsvadze.wordpress.com/2014/11/09/57-chevrolet-sweet-smooth-and-sassy-advertising-chevrolet/
    8 http://www.tocmp.com/pix/Cadillac/pages/56Cadillac08-or_jpg.htm
    10 Circa 1959, Photo by Ralph Crane http://grandmighty.com/2011/11/la-traffic/
    12 https://commons.wikimedia.org/wiki/File:Car_fire_I-90_Massachusetts.jpg
    26 http://www.oldcaradvertising.com/Chevrolet%20Corvette/1956/1956%20Corvette%20Ad-03.html
    27 http://www.oldcaradvertising.com/Plymouth/1955%20Plymouth/1955%20Plymouth%20Ad-03.html
    30 https://www.magazine-advertisements.com/uploads/2/1/8/4/21844100/polaroid-sx-70-land-camera-1_orig.jpg
    36 https://www.ebay.com/itm/19-1960s-70s-American-Muscle-Car-Photos-Polaroids-Ford-Mustang-Dodge-Charger/192242851917?
    _trkparms=aid%3D555017%26algo%3DPL.CASSINI%26ao%3D1%26asc%3D201607
    06104836%26meid%3D0934e94015fa43119ec01b0813d56b72%26pid%3D100642%26rk%3D1%26rkt%3D1%26%26itm%3D192242851917&_trksid=p204
    5573.c100642.m3226
    38 http://tsiarde.tictail.com/product/polaroid-camera-instant-10-with-beautiful-rainbow-colored-handle
    41 http://forums.popphoto.com/showthread.php?627010-Original-Polaroid-SX-70-Land-Camera
    44 https://www.magazine-advertisements.com/polaroid-230-land-camera.html
    47 https://www.ebay.com/itm/Lot-of-91-Vintage-Polaroid-photos-mixed-Lot-3/382437994619?hash=item590b14b07b:g:xiYAAOSwMGha1T9a
    49 https://www.pinterest.com/pin/531284087276635567/
    57 https://i.pinimg.com/originals/e7/a6/9c/e7a69c81ffbd9373c722c6f2a52d78ca.jpg
    63 http://vintage-magazine-ads.blogspot.com/
    74 http://www.voicesofeastanglia.com/wp-content/uploads/2012/01/Japanese-Make-up-Advert.jpg
    78 http://www.goretro.com/2016/01/fond-of-fondue.html
    84 http://myminkbetty.blogspot.com/2015/08/of-sewing-machines-and-tomato-soup.html
    99 https://www.pinterest.com/pin/269441990180539638/
    100 https://www.archives.gov/files/research/african-americans/ww2-pictures/images/african-americans-wwii-065.jpg
    101 http://www.shorpy.com/node/2974
    102 https://imgur.com/gallery/2jEv6
    103 https://www.cnn.com/style/article/radium-girls-radioactive-paint/index.html
    104 https://en.wikipedia.org/wiki/United_States_home_front_during_World_War_II#/media/Fil
    e:World_War_II_woman_aircraft_worker,_Vega_Aircraft_Corporation,_Burbank,_California_1942.jpg
    105 https://www.atomicheritage.org/history/african-americans-and-manhattan-project
    106 http://www.zocalopublicsquare.org/2016/01/14/the-japanese-american-flower-growers-who-made-phoenix-bloom/chronicles/who-we-were/
    107 https://www.bet.com/news/fashion-and-beauty/photos/2012/03/black-female-activists/_jcr_content/mainCol/imagegallerycontainer/gal
    leryimage_4.custom1540fx865fx0xcrop.dimg/__1330980660498/030512-fashion-black-feminist-ruby-dee.jpg
    124 http://blog.beforemario.com/2012/04/nintendo-twister-1966.html

    View full-size slide