Writing container-friendly applications

Presented at LinuxCon/ContainerCon North America 2015. (http://sched.co/44LN)

This talk is about how the OS primitives and sticking to the practices that has been out there since Unix philosophy came out helps us applying the best practices for configuration, logging, secrets, command line interfaces, output streams in writing application that are going to live in containers.

The talk goes through examples from the real-world popular open source projects and how they are containerized as official Docker images.

Transcript

1. WRITING CONTAINER-FRIENDLY APPLICATIONS
2. WRITING CONTAINER-FRIENDLY APPLICATIONS
3. CONTAINER-UNFRIENDLY?
4. @AhmetAlpBalkan
5. @AhmetAlpBalkan I work for
6. @AhmetAlpBalkan and all my code is open source. I work for
7. CONTAINERS
8. DOCKER CONTAINERS
9. DOCKER CONTAINERS $ docker run mysql image
10. $ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ mysql env DOCKER CONTAINERS
11. $ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql volume DOCKER CONTAINERS
12. $ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql --verbose args DOCKER CONTAINERS
13. $ docker run \ -e MYSQL_ROOT_PASSWORD=12345 \ -v /data:/var/lib/mysql \ mysql --verbose image DOCKER CONTAINERS
14. Container Images
15. docker run debian
16. docker run python
17. docker run memcached
18. docker run mysql
19. docker run nginx
20. DOCKER OFFICIAL REPOSITORIES …you are using
21. We all use them
22. One less thing to worry about
23. Always up-to-date
24. 150,000,000 downloads * even more containers created
25. Maintained by community
26. Creating Docker images in a nutshell:
27. Creating Docker images FROM debian RUN apt-get install memcached ENTRYPOINT memcached in a nutshell:
28. Creating Docker images FROM debian RUN apt-get install memcached ENTRYPOINT memcached “Dockerfile” in a nutshell:
30. THIS HAVE TO DO WITH ME? WHAT DOES
31. Code Container
32. Code Container
33. Code Container
34. Code Container
35. RECIPES for SUCCESS
36. CONFIGURATION
37. files > ./program config.cfg command-line args > ./program --debug --db 127.0.0.1:3036 environment > DEBUG=1 DB=127.0.01:3036 ./program
38. configuration files
39. my.cnf (MySQL) [client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql explicit_defaults_for_timestamp # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. #bind-address= 127.0.0.1
40. zoo.cfg (Apache ZooKeeper) tickTime=2000 initLimit=10 syncLimit=5 dataDir=/tmp/zookeeper clientPort=2181
41. cassandra.yaml cluster_name: 'Test Cluster' num_tokens: 256 hinted_handoff_enabled: true hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 batchlog_replay_throttle_in_kb: 1024 authenticator: AllowAllAuthenticator authorizer: AllowAllAuthorizer role_manager: CassandraRoleManager roles_validity_in_ms: 2000 permissions_validity_in_ms: 2000 data_file_directories: - /var/lib/cassandra/data commitlog_directory: /var/lib/cassandra/commitlog seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "127.0.0.1" concurrent_reads: 32
42. nginx.conf user nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on;
43. nginx.conf user nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; W HAT LANGUAGE IS THIS?
44. configuration files they kinda suck
45. configuration files they kinda suck don’t invent another
46. configuration files they kinda suck don’t invent another try to avoid them!
47. command line arguments
48. tip 0: do not write your own parser command line arguments the language already has one
49. command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening.
50. command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening. typo!?
51. tip 1: have strict validation command line arguments $ server --host localhost --portt 5000 [INFO] Server started listening. typo!?
52. tip 1: have strict validation command line arguments $ cassandra --help getopt: unrecognized option '--help' $ $ $ INFO 16:56:01 Loading settings from file:/etc/cassandra/cassandra.yaml INFO 16:56:01 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog;
53. tip 2: print your configuration command line arguments $ server --host localhost --port 5000 [DEBUG] bind=localhost [DEBUG] port=5000 [DEBUG] workers=10 [DEBUG] logfile=/dev/stdout [INFO] Server started listening.
54. tip 3: have good defaults command line arguments memcached: 32 options, 0 required redis: 50 options, 0 required docker: 41 options, 0 required
55. environment variables
56. long command line arguments environment variables $ program \ --user root \ --password sEzMaB4UmnUMwLgHvikxmmUtco Ed2EkcYQUkdXHsC(CRt8ZHXKhMFxDyZtzNWeRF;92 72ymGGOh$hitUsFseBmbR
57. long command line arguments environment variables $ program \ --user root \ --password sEzMaB4UmnUMwLgHvikxmmUtco Ed2EkcYQUkdXHsC(CRt8ZHXKhMFxDyZtzNWeRF;92 72ymGGOh$hitUsFseBmbR
58. override arguments environment variables $ program --help Usage: server [OPTIONS] Options: --user Name of the victim for the container gods --password Authentication key [$PASSWORD]
59. override arguments environment variables $ program --help Usage: server [OPTIONS] Options: --user Name of the victim for the container gods --password Authentication key [$PASSWORD]
60. Doing IT WRONG
61. Doing IT WRONG # comment out a few problematic configuration values # don't reverse lookup hostnames, they are usually another container RUN sed -Ei 's/^(bind-address|log)/#&/' /etc/mysql/my.cnf \ && echo 'skip-host-cache\nskip-name-resolve' | \ awk '{ print } $1 == "[mysqld]" && c == 0 { \ c = 1; system("cat") }' /etc/mysql/my.cnf > /tmp/my.cnf \ && mv /tmp/my.cnf /etc/mysql/my.cnf MySQL Dockerfile uses sed/awk to disable a couple of configuration keys https://github.com/docker-library/cassandra
62. Doing IT WRONG sed -ri 's/(- seeds:) "127.0.0.1"/\1 "'"$CASSANDRA_SEEDS"'"/' "$CASSANDRA_CONFIG/cassandra.yaml" for yaml in broadcast_address broadcast_rpc_address \ cluster_name endpoint_snitch listen_address num_tokens; do var="CASSANDRA_${yaml^^}" val=“${!var}”; if [ "$val" ]; then sed -ri 's/^(# )?('"$yaml"':).*/\2 '"$val"'/' \ "$CASSANDRA_CONFIG/cassandra.yaml" fi done fi exec "$@" Apache Cassandra entrypoint script override configs in YAML file from env https://github.com/docker-library/cassandra
63. command line args environment variables let the user override with configuration files options 95% of users will not ever change:
64. LOGGING
65. DON’T YOU EVER LOG TO FILES
66. where are your logs? DON’T YOU EVER LOG TO FILES
67. you can’t run inside readonly containers DON’T YOU EVER LOG TO FILES
68. just log to stdout/stderr
69. just log to stdout/stderr let your init system do it sysvinit runit upstart systemd supervisor monit
70. just log to stdout/stderr let your init system do it sysvinit runit upstart systemd supervisor monit docker ✨ ✨ ✨ ✨
71. DOCKER LOGGING 101 docker daemon container STDOUT STDERR
72. DOCKER LOGGING 101 docker daemon container STDOUT STDERR json-file
73. DOCKER LOGGING 101 docker daemon container STDOUT STDERR json-file fluentd syslog journald gelf
74. DOING IT WRONG
75. DOING IT WRONG Apache HTTP Server
76. Apache HTTP Server FROM debian:jessie RUN apt-get install [dependencies] RUN [get source code] RUN [compile it] && make install DOING IT WRONG
77. Apache HTTP Server FROM debian:jessie RUN apt-get install [dependencies] RUN [get source code] RUN [compile it] && make install RUN sed -ri ' \ s!^(\s*CustomLog)\s+\S+!\1 /proc/self/fd/1!g; \ s!^(\s*ErrorLog)\s+\S+!\1 /proc/self/fd/2!g; \ ' /usr/local/apache2/conf/httpd.conf DOING IT WRONG
78. Apache HTTP Server FROM debian:jessie RUN apt-get install [dependencies] RUN [get source code] RUN [compile it] && make install RUN sed -ri ' \ s!^(\s*CustomLog)\s+\S+!\1 /proc/self/fd/1!g; \ s!^(\s*ErrorLog)\s+\S+!\1 /proc/self/fd/2!g; \ ' /usr/local/apache2/conf/httpd.conf DOING IT WRONG
79. nginx HTTP Server * and 80 other things DOING IT WRONG
80. nginx HTTP Server FROM debian:jessie […] ENV NGINX_VERSION 1.9.3-1~jessie RUN apt-get install nginx=${NGINX_VERSION} DOING IT WRONG
81. nginx HTTP Server FROM debian:jessie […] ENV NGINX_VERSION 1.9.3-1~jessie RUN apt-get install nginx=${NGINX_VERSION} # forward logs to docker log collector RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf /dev/stderr /var/log/nginx/error.log * and 80 other things DOING IT WRONG
82. nginx HTTP Server $ docker run --read-only nginx [emerg] open() "/var/run/nginx.pid" failed (30: Read-only file system) DOING IT WRONG
83. Log to STDOUT/STDERR You will be fine.
84. DAEMONIZING Photo: jchapiewsky (Flickr) CC 2.0-BY-SA
85. DOING IT WRONG
86. $ nginx DOING IT WRONG
87. $ nginx ⏎ $ DOING IT WRONG
88. $ nginx ⏎ $ what? DOING IT WRONG
89. DOING IT WRONG $ nginx ⏎ $ $ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:00 bash 9 ? Ss 0:00 nginx: master process nginx 10 ? S 0:00 \_ nginx: worker process 13 ? R+ 0:00 ps afx what?
90. DOING IT WRONG $ nginx ⏎ $ $ ps afx PID TTY STAT TIME COMMAND 1 ? Ss 0:00 bash 9 ? Ss 0:00 nginx: master process nginx 10 ? S 0:00 \_ nginx: worker process 13 ? R+ 0:00 ps afx what?
91. So don’t do that
92. Do not daemonize yourself
93. When the ENTRYPOINT exits, Docker stops the container
94. docker run ––restart=always …
95. https://github.com/nginxinc/docker-nginx/ nginx Dockerfile DOING IT WRONG
96. FROM debian:jessie RUN apt-get install nginx=${NGINX_VERSION} … EXPOSE 80/tcp EXPOSE 443/tcp ENTRYPOINT ["nginx", "-g", "daemon off;"] https://github.com/nginxinc/docker-nginx/ nginx Dockerfile DOING IT WRONG
97. Apache httpd Dockerfile ENTRYPOINT ["httpd", "-DFOREGROUND"] https://github.com/docker-library/cassandra Cassandra Dockerfile ENTRYPOINT ["cassandra", "-f"] https://github.com/docker-library/httpd DOING IT WRONG
98. ASP.NET 5 Docker Image DOING IT WRONG Problem: “my container suddenly exits after running for 5 minutes”
99. DOING IT WRONG var ignored = Task.Run(() => { Console.WriteLine("Started"); Console.ReadLine(); appShutdownService.RequestShutdown(); }); Microsoft.AspNet.Hosting/Program.cs: ASP.NET 5 Docker Image https://github.com/aspnet/Hosting/
100. DOING IT WRONG var ignored = Task.Run(() => { Console.WriteLine("Started"); Console.ReadLine(); appShutdownService.RequestShutdown(); }); Microsoft.AspNet.Hosting/Program.cs: ASP.NET 5 Docker Image https://github.com/aspnet/Hosting/
101. So don’t do that either
102. Write programs like it’s 1960
103. Command line arguments Use OS built-ins Environment variables Non-demonized processes Log to stdout/stderr
104. HELPYOURSELF & PEOPLE PACKAGING YOUR SOFTWARE
105. HALL OF FAME
106. Docker Registry $ registry --help usage: /bin/registry <config>
107. version: 0.1 log: fields: service: registry storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: :5000 tls: certificate: /path/to/x509/public key: /path/to/x509/private config.yml Docker Registry
108. version: 0.1 log: fields: service: registry storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: :5000 tls: certificate: /path/to/x509/public key: /path/to/x509/private config.yml Docker Registry -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=… O verride
109. etcd $ etcd --my-flag foo $ export ETCD_MY_FLAG=foo $ etcd
110. Docker Swarm $ swarm join --help docker run -it swarm join --help Usage: swarm join [OPTIONS] <discovery> join a docker cluster Arguments: <discovery> discovery service to use [$SWARM_DISCOVERY] * token://<token> * consul://<ip>/<path> * etcd://<ip1>,<ip2>/<path> * file://path/to/file * zk://<ip1>,<ip2>/<path> * <ip1>,<ip2> Options: --advertise, address of this Docker Engine [$SWARM_ADVERTISE] --heartbeat “20s" period between each heartbeat --ttl "60s" sets the expiration of an ephemeral node
111. Docker Swarm $ swarm join --help docker run -it swarm join --help Usage: swarm join [OPTIONS] <discovery> join a docker cluster Arguments: <discovery> discovery service to use [$SWARM_DISCOVERY] * token://<token> * consul://<ip>/<path> * etcd://<ip1>,<ip2>/<path> * file://path/to/file * zk://<ip1>,<ip2>/<path> * <ip1>,<ip2> Options: --advertise, address of this Docker Engine [$SWARM_ADVERTISE] --heartbeat “20s" period between each heartbeat --ttl "60s" sets the expiration of an ephemeral node
113. Thank you @AhmetAlpBalkan We are hiring!