The Challenges of Container Configuration

Slide 1

Slide 1 text

(without introducing more risk) The Challenges of Container Conﬁguration Puppet Gareth Rushgrove New capabilities and associated problems

Slide 2

Slide 2 text

(without introducing more risk) Gareth Rushgrove @garethr

Slide 3

Slide 3 text

(without introducing more risk) Gareth Rushgrove

Slide 4

Slide 4 text

(without introducing more risk) Conﬁguration What is it and why should I care

Slide 5

Slide 5 text

(without introducing more risk) Gareth Rushgrove

Slide 6

Slide 6 text

(without introducing more risk) synonyms: design, grouping, marshalling Gareth Rushgrove

Slide 7

Slide 7 text

(without introducing more risk) Designing your infrastructure Gareth Rushgrove

Slide 8

Slide 8 text

(without introducing more risk) Marshalling your containers Gareth Rushgrove

Slide 9

Slide 9 text

(without introducing more risk) - Immutability and containers - Runtime vs build time - Who conﬁgures the orchestrator? Gareth Rushgrove

Slide 10

Slide 10 text

(without introducing more risk) Mainly Docker and Kubernetes examples, but should be generally applicable Gareth Rushgrove

Slide 11

Slide 11 text

(without introducing more risk) Everything is immutable now? Assumptions vs reality

Slide 12

Slide 12 text

(without introducing more risk) Gareth Rushgrove

Slide 13

Slide 13 text

(without introducing more risk) Gareth Rushgrove $ docker run -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"

Slide 14

Slide 14 text

(without introducing more risk) Gareth Rushgrove $ docker exec a7a01beb14de touch /tmp/surprise

Slide 15

Slide 15 text

(without introducing more risk) Gareth Rushgrove $ docker diff a7a01beb14de C /tmp A /tmp/surprise

Slide 16

Slide 16 text

(without introducing more risk) Containers are not immutable by default Gareth Rushgrove

Slide 17

Slide 17 text

(without introducing more risk) Containers are not immutable by default Gareth Rushgrove

Slide 18

Slide 18 text

(without introducing more risk) Gareth Rushgrove $ docker run --read-only -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"

Slide 19

Slide 19 text

(without introducing more risk) Gareth Rushgrove $ docker exec 379150b2cf05 touch /tmp/surprise touch: cannot touch '/tmp/surprise': Read-only file syste

Slide 20

Slide 20 text

(without introducing more risk) Suggestion Enable read-only where possible Gareth Rushgrove

Slide 21

Slide 21 text

(without introducing more risk) Many applications won’t start with a read-only ﬁlesystem Gareth Rushgrove

Slide 22

Slide 22 text

(without introducing more risk) Gareth Rushgrove 1 import logging 2 from logging.handlers import RotatingFileHandler 3 4 from flask import Flask 5 6 app = Flask(__name__) 7 8 @app.route('/') 9 def home(): 10 app.logger.info('log request') 11 return "We'll never get to here" 12 13 if __name__ == '__main__': 14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount 15 handler.setLevel(logging.INFO) 16 app.logger.addHandler(handler) 17 app.run(debug=True, host='0.0.0.0')

Slide 23

Slide 23 text

Slide 24

Slide 24 text

(without introducing more risk) Gareth Rushgrove $ docker run --read-only -p 5000:5000 garethr/flaskapp Traceback (most recent call last): File "app.py", line 14, in handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1) File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__ BaseRotatingHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 30] Read-only file system: '/app/app.log'

Slide 25

Slide 25 text

(without introducing more risk) tmpfs support added in Docker 1.10 Gareth Rushgrove

Slide 26

Slide 26 text

(without introducing more risk) Gareth Rushgrove $ docker run --read-only --tmpfs /tmp \ -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"

Slide 27

Slide 27 text

(without introducing more risk) Gareth Rushgrove $ docker exec 222331443d28 touch /tmp/surprise

Slide 28

Slide 28 text

(without introducing more risk) Gareth Rushgrove $ docker diff 222331443d28

Slide 29

Slide 29 text

(without introducing more risk) Gareth Rushgrove $ docker exec 222331443d28 ls /tmp surprise

Slide 30

Slide 30 text

(without introducing more risk) Suggestion Use tmpfs only where needed Gareth Rushgrove

Slide 31

Slide 31 text

(without introducing more risk) Remember Without technical controls you only have social guarantees of immutability Gareth Rushgrove

Slide 32

Slide 32 text

(without introducing more risk) Build vs Run And the relationship between them

Slide 33

Slide 33 text

(without introducing more risk) Given an image - What machine built this image? - Are all the licenses compatible? - Who supports this image? - Does this image contain malware? Gareth Rushgrove

Slide 34

Slide 34 text

(without introducing more risk) Given a running container - Who built it? - How was it built? - What software does it contain? - Is the software up-to-date? Gareth Rushgrove

Slide 35

Slide 35 text

(without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && \ apt-get install -y python-pip python-dev build-essent apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"]

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

(without introducing more risk) The importance of time Gareth Rushgrove

Slide 41

Slide 41 text

(without introducing more risk) How often are images rebuilt? Gareth Rushgrove

Slide 42

Slide 42 text

(without introducing more risk) Rebuilding only on code change ignores environmental factors Gareth Rushgrove

Slide 43

Slide 43 text

(without introducing more risk) Versioning and metadata Gareth Rushgrove

Slide 44

Slide 44 text

(without introducing more risk) Gareth Rushgrove LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"

Slide 45

Slide 45 text

(without introducing more risk) Gareth Rushgrove LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12" What time? What timezone?

Slide 46

Slide 46 text

(without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" \ 4fa6e0f0c678 | jq { "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" }

Slide 47

Slide 47 text

(without introducing more risk) Suggestion Decide upon and enforce metadata standards Gareth Rushgrove

Slide 48

Slide 48 text

(without introducing more risk) Gareth Rushgrove LABEL com.example.git.repository="https://github.com/pupp com.example.git.sha="dc123cfb5ed4dca43a84be34a99d7c com.example.build.time="2016-04-24T15:43:05+00:00" com.example.build.builder=“jenkins1.example.com" \ com.example.docs="https://github.com/puppetlabs/doc ...

Slide 49

Slide 49 text

(without introducing more risk) Suggestion Embed Dockerﬁles in images Gareth Rushgrove

Slide 50

Slide 50 text

(without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq { "net.morethanseven.dockerfile": "/Dockerfile", }

Slide 51

Slide 51 text

(without introducing more risk) Gareth Rushgrove $ docker run -i -t garethr/alpine cat /Dockerfile FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /

Slide 52

Slide 52 text

(without introducing more risk) Suggestion Provide an API for your containers Gareth Rushgrove

Slide 53

Slide 53 text

(without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq { "com.example.api.packages": "apk info -vv" }

Slide 54

Slide 54 text

(without introducing more risk) Gareth Rushgrove $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2 - the musl c library (libc) implementation busybox-1.23.2-r0 - Size optimized toolbox of many common UNIX alpine-baselayout-2.3.2-r0 - Alpine base dir structure and init openrc-0.15.1-r3 - OpenRC manages the services, startup and shu alpine-conf-3.2.1-r6 - Alpine configuration management scripts

Slide 55

Slide 55 text

(without introducing more risk) DEMO

Slide 56

Slide 56 text

(without introducing more risk) Who conﬁgures the scheduler? Higher level conﬁguration

Slide 57

Slide 57 text

(without introducing more risk) Schedulers/orchestrators abstract you from - Where individual containers run - Balancing due to new resources - Balancing due to failed resources Gareth Rushgrove

Slide 58

Slide 58 text

(without introducing more risk) This results in a constraints based system Gareth Rushgrove

Slide 59

Slide 59 text

(without introducing more risk) Which means those constraints need to be explicit and correct Gareth Rushgrove

Slide 60

Slide 60 text

(without introducing more risk) Gareth Rushgrove $ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"

Slide 61

Slide 61 text

(without introducing more risk) Gareth Rushgrove $ docker run -d -P \ -e constraint:storage==ssd --name db mysql

Slide 62

Slide 62 text

(without introducing more risk) Gareth Rushgrove 1 template: 2 metadata: 3 labels: 4 app: guestbook 5 tier: frontend 6 spec: 7 containers: 8 - name: php-redis 9 image: gcr.io/google_samples/gb-frontend:v4 10 resources: 11 requests: 12 cpu: 100m 13 memory: 100Mi 14 env: 15 - name: GET_HOSTS_FROM 16 value: dns 17 # If your cluster config does not include a dns service, th 18 # instead access environment variables to find service host 19 # info, comment out the 'value: dns' line above, and uncomm

Slide 63

Slide 63 text

(without introducing more risk) How do you manage properties for all of your hosts? Gareth Rushgrove

Slide 64

Slide 64 text

(without introducing more risk) Gareth Rushgrove $ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd" Does this machine really have an SSD? What if someone swaps the drive?

Slide 65

Slide 65 text

(without introducing more risk) Suggestion Use properties of hosts as labels Gareth Rushgrove

Slide 66

Slide 66 text

(without introducing more risk) Gareth Rushgrove $ facter | head -n 20 architecture => x86_64 domain => local facterversion => 2.4.6 fqdn => Pro.local gid => staff hardwareisa => i386 hardwaremodel => x86_64 hostname => Pro id => garethr interfaces => lo0,gif0,stf0,en0,p2p0,awdl0,en1,en2,bridge0,vboxnet0,vboxnet1,v ipaddress => 192.168.0.5 ipaddress_en0 => 192.168.0.5

Slide 67

Slide 67 text

(without introducing more risk) Gareth Rushgrove $ facter -j os | jq { "os": { "name": "Darwin", "family": "Darwin", "release": { "major": "14", "minor": "5", "full": "14.5.0" } } }

Slide 68

Slide 68 text

(without introducing more risk) Gareth Rushgrove $ docker daemon \ --label net.example.os=`facter operatingsystem` \ --label net.example.virtual=`facter is_virtual` \ --label net.example.kernel=`facter kernelversion` \ ...

Slide 69

Slide 69 text

(without introducing more risk) Orchestrators also tend to introduce new higher-level primitives Gareth Rushgrove

Slide 70

Slide 70 text

(without introducing more risk) Docker Networks, Kubernetes Services, ReplicationControllers, Chronos Jobs Gareth Rushgrove

Slide 71

Slide 71 text

(without introducing more risk) Many with imperative interfaces Gareth Rushgrove

Slide 72

Slide 72 text

(without introducing more risk) Gareth Rushgrove $ kubectl get pod mypod -o yaml \ | sed 's/$image: myimage$:.*$/\1:v4/' \ | kubectl replace -f -

Slide 73

Slide 73 text

(without introducing more risk) Gareth Rushgrove $ docker network create bob c0a0f4538d259515813b771264688d37aaedb41098379a0d73ec0ca08 $ docker network create bob Error response from daemon: network with name bob already

Slide 74

Slide 74 text

And everything conﬁgured in YAML Gareth Rushgrove

Slide 75

Slide 75 text

Code plus data has advantages over data alone Gareth Rushgrove

Slide 76

Slide 76 text

The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modiﬁcation of this data should be done in a real programming language Gareth Rushgrove Borg, Omega, and Kubernetes, ACM Queue, Volume 14, issue 1 http://queue.acm.org/detail.cfm?id=2898444 “

Slide 77

Slide 77 text

Avoid repetition Combine external inputs Correctness Abstractions Gareth Rushgrove - - - -

Slide 78

Slide 78 text

Suggestion Use a higher level programming tool for generating conﬁg data Gareth Rushgrove

Slide 79

Slide 79 text

(without introducing more risk) Gareth Rushgrove jsonnet.org

Slide 80

Slide 80 text

(without introducing more risk) Gareth Rushgrove $ cat test.jsonnet // Example jsonnet file { person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: "Bob" }, }

Slide 81

Slide 81 text

(without introducing more risk) Gareth Rushgrove $ jsonnet test.jsonnet | jq { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" } }

Slide 82

Slide 82 text

(without introducing more risk) Gareth Rushgrove $ jsonnet test.jsonnet | json2yaml --- person1: name: "Alice" welcome: "Hello Alice!" person2: name: "Bob" welcome: "Hello Bob!"

Slide 83

Slide 83 text

(without introducing more risk) DEMO

Slide 84

Slide 84 text

(without introducing more risk) Gareth Rushgrove garethr/kubernetes

Slide 85

Slide 85 text

(without introducing more risk) Gareth Rushgrove kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }

Slide 86

Slide 86 text

(without introducing more risk) Gareth Rushgrove apiVersion: v1 kind: Pod metadata: namespace: default name: sample-pod spec: container: - image: nginx name: container-name

Slide 87

Slide 87 text

(without introducing more risk) Gareth Rushgrove controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379, }

Slide 88

Slide 88 text

(without introducing more risk) Gareth Rushgrove apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis tier: backend role: master spec: ports: # the port that this service should serve on - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master --- apiVersion: v1 kind: ReplicationController metadata: name: redis-master # these labels can be applied automatically # from the labels in the pod template if not set labels: app: redis role: master tier: backend spec: # this replicas value is default # modify it according to your case replicas: 1 # selector can be applied automatically # from the labels in the pod template if not set # selector: # app: guestbook # role: master # tier: backend

Slide 89

Slide 89 text

(without introducing more risk) DEMO

Slide 90

Slide 90 text

(without introducing more risk) Conclusions New technology means old problems

Slide 91

Slide 91 text

(without introducing more risk) The difference between how you think something works and how it actually works risks hard-to-debug production issues Gareth Rushgrove

Slide 92

Slide 92 text

(without introducing more risk) Containers introduce new and old conﬁguration problems Gareth Rushgrove

Slide 93

Slide 93 text

(without introducing more risk) Conﬁguration management is the discipline aimed at minimising those risks Gareth Rushgrove

Slide 94

Slide 94 text

(without introducing more risk) Start with principles Gareth Rushgrove

Slide 95

Slide 95 text

(without introducing more risk) - Identification - Control - Status accounting - Verification Gareth Rushgrove Military Handbook Configuration Management Guidance MIL-HDBK-61B

Slide 96

Slide 96 text

(without introducing more risk) Apply them to your container based infrastructure today Gareth Rushgrove

Slide 97

Slide 97 text

(without introducing more risk) Questions? And thanks for listening