Writing container-friendly applications

Slide 1

Slide 1 text

WRITING CONTAINER-FRIENDLY APPLICATIONS

Slide 2

Slide 2 text

WRITING CONTAINER-FRIENDLY APPLICATIONS

Slide 3

Slide 3 text

CONTAINER-UNFRIENDLY?

Slide 4

Slide 4 text

@AhmetAlpBalkan

Slide 5

Slide 5 text

@AhmetAlpBalkan I work for

Slide 6

Slide 6 text

@AhmetAlpBalkan and all my code is open source. I work for

Slide 7

Slide 7 text

CONTAINERS

Slide 8

Slide 8 text

DOCKER CONTAINERS

Slide 9

Slide 9 text

DOCKER CONTAINERS $ docker run mysql image

Slide 10

Slide 10 text

$ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ mysql env DOCKER CONTAINERS

Slide 11

Slide 11 text

$ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql volume DOCKER CONTAINERS

Slide 12

Slide 12 text

$ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql --verbose args DOCKER CONTAINERS

Slide 13

Slide 13 text

$ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql --verbose image DOCKER CONTAINERS

Slide 14

Slide 14 text

Container Images

Slide 15

Slide 15 text

docker run debian

Slide 16

Slide 16 text

docker run python

Slide 17

Slide 17 text

docker run memcached

Slide 18

Slide 18 text

docker run mysql

Slide 19

Slide 19 text

docker run nginx

Slide 20

Slide 20 text

DOCKER OFFICIAL REPOSITORIES …you are using

Slide 21

Slide 21 text

We all use them

Slide 22

Slide 22 text

One less thing to worry about

Slide 23

Slide 23 text

Always up-to-date

Slide 24

Slide 24 text

150,000,000 downloads * even more containers created

Slide 25

Slide 25 text

Maintained by community

Slide 26

Slide 26 text

Creating Docker images in a nutshell:

Slide 27

Slide 27 text

Creating Docker images FROM debian RUN apt-get install memcached ENTRYPOINT memcached in a nutshell:

Slide 28

Slide 28 text

Creating Docker images FROM debian RUN apt-get install memcached ENTRYPOINT memcached “Dockerfile” in a nutshell:

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

THIS HAVE TO DO WITH ME? WHAT DOES

Slide 31

Slide 31 text

Code Container

Slide 32

Slide 32 text

Code Container

Slide 33

Slide 33 text

Code Container

Slide 34

Slide 34 text

Code Container

Slide 35

Slide 35 text

RECIPES for SUCCESS

Slide 36

Slide 36 text

CONFIGURATION

Slide 37

Slide 37 text

ﬁles > ./program conﬁg.cfg command-line args > ./program --debug --db 127.0.0.1:3036 environment > DEBUG=1 DB=127.0.01:3036 ./program

Slide 38

Slide 38 text

conﬁguration ﬁles

Slide 39

Slide 39 text

my.cnf (MySQL) [client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql explicit_defaults_for_timestamp # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. #bind-address= 127.0.0.1

Slide 40

Slide 40 text

zoo.cfg (Apache ZooKeeper) tickTime=2000 initLimit=10 syncLimit=5 dataDir=/tmp/zookeeper clientPort=2181

Slide 41

Slide 41 text

cassandra.yaml cluster_name: 'Test Cluster' num_tokens: 256 hinted_handoff_enabled: true hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 batchlog_replay_throttle_in_kb: 1024 authenticator: AllowAllAuthenticator authorizer: AllowAllAuthorizer role_manager: CassandraRoleManager roles_validity_in_ms: 2000 permissions_validity_in_ms: 2000 data_file_directories: - /var/lib/cassandra/data commitlog_directory: /var/lib/cassandra/commitlog seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "127.0.0.1" concurrent_reads: 32

Slide 42

Slide 42 text

nginx.conf user nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on;

Slide 43

Slide 43 text

Slide 44

Slide 44 text

conﬁguration ﬁles they kinda suck

Slide 45

Slide 45 text

conﬁguration ﬁles they kinda suck don’t invent another

Slide 46

Slide 46 text

conﬁguration ﬁles they kinda suck don’t invent another try to avoid them!

Slide 47

Slide 47 text

command line arguments

Slide 48

Slide 48 text

tip 0: do not write your own parser command line arguments the language already has one

Slide 49

Slide 49 text

command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening.

Slide 50

Slide 50 text

command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening. typo!?

Slide 51

Slide 51 text

tip 1: have strict validation command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening. typo!?

Slide 52

Slide 52 text

tip 1: have strict validation command line arguments $ cassandra --help getopt: unrecognized option '--help' $ $ $ INFO 16:56:01 Loading settings from file:/etc/cassandra/cassandra.yaml INFO 16:56:01 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog;

Slide 53

Slide 53 text

tip 2: print your conﬁguration command line arguments $ server --host localhost --port 5000 [DEBUG] bind=localhost [DEBUG] port=5000 [DEBUG] workers=10 [DEBUG] logfile=/dev/stdout [INFO] Server started listening.

Slide 54

Slide 54 text

tip 3: have good defaults command line arguments memcached: 32 options, 0 required redis: 50 options, 0 required docker: 41 options, 0 required

Slide 55

Slide 55 text

environment variables

Slide 56

Slide 56 text

long command line arguments environment variables $ program \ --user root \ --password sEzMaB4UmnUMwLgHvikxmmUtco Ed2EkcYQUkdXHsC(CRt8ZHXKhMFxDyZtzNWeRF;92 72ymGGOh$hitUsFseBmbR

Slide 57

Slide 57 text

long command line arguments environment variables $ program \ --user root \ --password sEzMaB4UmnUMwLgHvikxmmUtco Ed2EkcYQUkdXHsC(CRt8ZHXKhMFxDyZtzNWeRF;92 72ymGGOh$hitUsFseBmbR

Slide 58

Slide 58 text

override arguments environment variables $ program --help Usage: server [OPTIONS] Options: --user Name of the victim for the container gods --password Authentication key [$PASSWORD]

Slide 59

Slide 59 text

override arguments environment variables $ program --help Usage: server [OPTIONS] Options: --user Name of the victim for the container gods --password Authentication key [$PASSWORD]

Slide 60

Slide 60 text

Doing IT WRONG

Slide 61

Slide 61 text

Doing IT WRONG # comment out a few problematic configuration values # don't reverse lookup hostnames, they are usually another container RUN sed -Ei 's/^(bind-address|log)/#&/' /etc/mysql/my.cnf \ && echo 'skip-host-cache\nskip-name-resolve' | \ awk '{ print } $1 == "[mysqld]" && c == 0 { \ c = 1; system("cat") }' /etc/mysql/my.cnf > /tmp/my.cnf \ && mv /tmp/my.cnf /etc/mysql/my.cnf MySQL Dockerﬁle uses sed/awk to disable a couple of conﬁguration keys https://github.com/docker-library/cassandra

Slide 62

Slide 62 text

Doing IT WRONG sed -ri 's/(- seeds:) "127.0.0.1"/\1 "'"$CASSANDRA_SEEDS"'"/' "$CASSANDRA_CONFIG/cassandra.yaml" for yaml in broadcast_address broadcast_rpc_address \ cluster_name endpoint_snitch listen_address num_tokens; do var="CASSANDRA_${yaml^^}" val=“${!var}”; if [ "$val" ]; then sed -ri 's/^(# )?('"$yaml"':).*/\2 '"$val"'/' \ "$CASSANDRA_CONFIG/cassandra.yaml" fi done fi exec "$@" Apache Cassandra entrypoint script override conﬁgs in YAML ﬁle from env https://github.com/docker-library/cassandra

Slide 63

Slide 63 text

command line args environment variables let the user override with conﬁguration ﬁles options 95% of users will not ever change:

Slide 64

Slide 64 text

LOGGING

Slide 65

Slide 65 text

DON’T YOU EVER LOG TO FILES

Slide 66

Slide 66 text

where are your logs? DON’T YOU EVER LOG TO FILES

Slide 67

Slide 67 text

you can’t run inside readonly containers DON’T YOU EVER LOG TO FILES

Slide 68

Slide 68 text

just log to stdout/stderr

Slide 69

Slide 69 text

just log to stdout/stderr let your init system do it sysvinit runit upstart systemd supervisor monit

Slide 70

Slide 70 text

just log to stdout/stderr let your init system do it sysvinit runit upstart systemd supervisor monit docker ✨ ✨ ✨ ✨

Slide 71

Slide 71 text

DOCKER LOGGING 101 docker daemon container STDOUT STDERR

Slide 72

Slide 72 text

DOCKER LOGGING 101 docker daemon container STDOUT STDERR json-ﬁle

Slide 73

Slide 73 text

DOCKER LOGGING 101 docker daemon container STDOUT STDERR json-ﬁle ﬂuentd syslog journald gelf

Slide 74

Slide 74 text

DOING IT WRONG

Slide 75

Slide 75 text

DOING IT WRONG Apache HTTP Server

Slide 76

Slide 76 text

Apache HTTP Server FROM debian:jessie RUN apt-get install [dependencies] RUN [get source code] RUN [compile it] && make install DOING IT WRONG

Slide 77

Slide 77 text

Apache HTTP Server FROM debian:jessie RUN apt-get install [dependencies] RUN [get source code] RUN [compile it] && make install RUN sed -ri ' \ s!^(\s*CustomLog)\s+\S+!\1 /proc/self/fd/1!g; \ s!^(\s*ErrorLog)\s+\S+!\1 /proc/self/fd/2!g; \ ' /usr/local/apache2/conf/httpd.conf DOING IT WRONG

Slide 78

Slide 78 text

Slide 79

Slide 79 text

nginx HTTP Server * and 80 other things DOING IT WRONG

Slide 80

Slide 80 text

nginx HTTP Server FROM debian:jessie […] ENV NGINX_VERSION 1.9.3-1~jessie RUN apt-get install nginx=${NGINX_VERSION} DOING IT WRONG

Slide 81

Slide 81 text

nginx HTTP Server FROM debian:jessie […] ENV NGINX_VERSION 1.9.3-1~jessie RUN apt-get install nginx=${NGINX_VERSION} # forward logs to docker log collector RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf /dev/stderr /var/log/nginx/error.log * and 80 other things DOING IT WRONG

Slide 82

Slide 82 text

nginx HTTP Server $ docker run --read-only nginx [emerg] open() "/var/run/nginx.pid" failed (30: Read-only file system) DOING IT WRONG

Slide 83

Slide 83 text

Log to STDOUT/STDERR You will be ﬁne.

Slide 84

Slide 84 text

DAEMONIZING Photo: jchapiewsky (Flickr) CC 2.0-BY-SA

Slide 85

Slide 85 text

DOING IT WRONG

Slide 86

Slide 86 text

$ nginx DOING IT WRONG

Slide 87

Slide 87 text

$ nginx ⏎ $ DOING IT WRONG

Slide 88

Slide 88 text

$ nginx ⏎ $ what? DOING IT WRONG

Slide 89

Slide 89 text

DOING IT WRONG $ nginx ⏎ $ $ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:00 bash 9 ? Ss 0:00 nginx: master process nginx 10 ? S 0:00 \_ nginx: worker process 13 ? R+ 0:00 ps afx what?

Slide 90

Slide 90 text

DOING IT WRONG $ nginx ⏎ $ $ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:00 bash 9 ? Ss 0:00 nginx: master process nginx 10 ? S 0:00 \_ nginx: worker process 13 ? R+ 0:00 ps afx what?

Slide 91

Slide 91 text

So don’t do that

Slide 92

Slide 92 text

Do not daemonize yourself

Slide 93

Slide 93 text

When the ENTRYPOINT exits, Docker stops the container

Slide 94

Slide 94 text

docker run ––restart=always …

Slide 95

Slide 95 text

https://github.com/nginxinc/docker-nginx/ nginx Dockerﬁle DOING IT WRONG

Slide 96

Slide 96 text

FROM debian:jessie RUN apt-get install nginx=${NGINX_VERSION} … EXPOSE 80/tcp EXPOSE 443/tcp ENTRYPOINT ["nginx", "-g", "daemon off;"] https://github.com/nginxinc/docker-nginx/ nginx Dockerﬁle DOING IT WRONG

Slide 97

Slide 97 text

Apache httpd Dockerﬁle ENTRYPOINT ["httpd", "-DFOREGROUND"] https://github.com/docker-library/cassandra Cassandra Dockerﬁle ENTRYPOINT ["cassandra", "-f"] https://github.com/docker-library/httpd DOING IT WRONG

Slide 98

Slide 98 text

ASP.NET 5 Docker Image DOING IT WRONG Problem: “my container suddenly exits after running for 5 minutes”

Slide 99

Slide 99 text

DOING IT WRONG var ignored = Task.Run(() => { Console.WriteLine("Started"); Console.ReadLine(); appShutdownService.RequestShutdown(); }); Microsoft.AspNet.Hosting/Program.cs: ASP.NET 5 Docker Image https://github.com/aspnet/Hosting/

Slide 100

Slide 100 text

Slide 101

Slide 101 text

So don’t do that either

Slide 102

Slide 102 text

Write programs like it’s 1960

Slide 103

Slide 103 text

Command line arguments Use OS built-ins Environment variables Non-demonized processes Log to stdout/stderr

Slide 104

Slide 104 text

HELPYOURSELF & PEOPLE PACKAGING YOUR SOFTWARE

Slide 105

Slide 105 text

HALL OF FAME

Slide 106

Slide 106 text

Docker Registry $ registry --help usage: /bin/registry

Slide 107

Slide 107 text

version: 0.1 log: fields: service: registry storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: :5000 tls: certificate: /path/to/x509/public key: /path/to/x509/private conﬁg.yml Docker Registry

Slide 108

Slide 108 text

Slide 109

Slide 109 text

etcd $ etcd --my-flag foo $ export ETCD_MY_FLAG=foo $ etcd

Slide 110

Slide 110 text

Docker Swarm $ swarm join --help docker run -it swarm join --help Usage: swarm join [OPTIONS] join a docker cluster Arguments: discovery service to use [$SWARM_DISCOVERY] * token:// * consul:/// * etcd://,/ * file://path/to/file * zk://,/ * , Options: --advertise, address of this Docker Engine [$SWARM_ADVERTISE] --heartbeat “20s" period between each heartbeat --ttl "60s" sets the expiration of an ephemeral node