Docker for Ruby Developers

Originally presented at NYC.rb.

Transcript

1. Docker for Ruby Developers NYC.rb, August 2015 Frank Macreery CTO, Aptible @fancyremarker https://speakerdeck.com/fancyremarker
3. Docker What’s it all about?
6. Docker Containers: The new virtual machines?
7. https://www.docker.com/whatisdocker Then: Virtual Machines
8. https://www.docker.com/whatisdocker
9. https://www.docker.com/whatisdocker Now: Containers
10. https://www.docker.com/whatisdocker
11. Getting Started with Docker
12. boot2docker
13. Kitematic
14. Getting Started with Docker Install boot2docker via http://boot2docker.io/#installation echo 'eval "$(boot2docker shellinit)"' >> ~/.bashrc
15. Getting Started with Docker Alternatively (if you have Homebrew and VirtualBox installed)…
 
 brew install boot2docker brew install docker
16. Getting Started with Docker Images and containers
17. docker pull quay.io/aptible/nginx:latest
18. docker pull quay.io/aptible/nginx:latest Reference to a Docker image
19. docker pull quay.io/aptible/nginx:latest Image "repositories" can have many "tags"
20. docker pull quay.io/aptible/nginx:latest Pulls an image from a Docker "registry"
21. A single Docker image consists of many "layers"
22. docker images
23. docker run -d -p 80:80 -p 443:443 \ quay.io/aptible/nginx:latest
24. docker run -d -p 80:80 -p 443:443 \ quay.io/aptible/nginx:latest Launches a new container from an existing image
25. docker run -d -p 80:80 -p 443:443 \ quay.io/aptible/nginx:latest Forward container ports to the host (host port:container port)
26. docker run -d -p 80:80 -p 443:443 \ quay.io/aptible/nginx:latest Run container in background
27. docker ps
30. Simplifying SOA Service-oriented architectures without the complexity
31. Dev/prod parity?
33. What makes dev/prod parity so hard?
34. What makes dev/prod parity so hard? 1 production deployment, many development/ staging environments
35. What makes dev/prod parity so hard? SOA simplifies each service’s responsibilities, but often at the cost of additional deployment complexity
36. What makes dev/prod parity so hard? The more services you have, the harder it is to achieve dev/prod parity
37. What makes dev/prod parity so hard? The more engineers you have, the harder it is to standardize dev parity
41. README != Automation
42. Dev/prod parity via Docker
43. Dev/prod parity via Docker Define services in terms of Docker images
44. # docker-compose.yml for Aptible Auth/API auth: build: auth.aptible.com/ ports: "4000:4000" links: - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_auth_development api: build: api.aptible.com/ ports: "4001:4001" links: - redis - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_api_development REDIS_URL: redis://redis APTIBLE_AUTH_ROOT_URL: http://docker:4000 redis: image: quay.io/aptible/redis:latest ports: "6379:6379" postgresql: image: quay.io/aptible/postgresql:aptible-seeds ports: "5432:5432"
45. # docker-compose.yml for Aptible Auth/API auth: build: auth.aptible.com/ ports: "4000:4000" links: - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_auth_development api: build: api.aptible.com/ ports: "4001:4001" links: - redis - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_api_development REDIS_URL: redis://redis APTIBLE_AUTH_ROOT_URL: http://docker:4000 redis: image: quay.io/aptible/redis:latest ports: "6379:6379" postgresql: image: quay.io/aptible/postgresql:aptible-seeds ports: "5432:5432"
46. # docker-compose.yml for Aptible Auth/API auth: build: auth.aptible.com/ ports: "4000:4000" links: - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_auth_development api: build: api.aptible.com/ ports: "4001:4001" links: - redis - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_api_development REDIS_URL: redis://redis APTIBLE_AUTH_ROOT_URL: http://docker:4000 redis: image: quay.io/aptible/redis:latest ports: "6379:6379" postgresql: image: quay.io/aptible/postgresql:aptible-seeds ports: "5432:5432"
47. # docker-compose.yml for Aptible Auth/API auth: build: auth.aptible.com/ ports: "4000:4000" links: - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_auth_development api: build: api.aptible.com/ ports: "4001:4001" links: - redis - postgresql environment: RAILS_ENV: development DATABASE_URL: postgresql://postgresql/aptible_api_development REDIS_URL: redis://redis APTIBLE_AUTH_ROOT_URL: http://docker:4000 redis: image: quay.io/aptible/redis:latest ports: "6379:6379" postgresql: image: quay.io/aptible/postgresql:aptible-seeds ports: "5432:5432"
48. Dev/prod parity via Docker Use the same service/image configuration in production as in development (Docker Compose, Swarm, Kubernetes…)
49. Containerized SSL Infrastructure management made easy
50. Elastic Load Balancer (ELB) EC2 Instance EC2 Instance NGiNX NGiNX
51. TCP/HTTPS TCP/HTTPS HTTP HTTP
52. How to configure NGiNX with multiple dynamic upstreams? Chef? Salt? Ansible?
53. ENV configuration $UPSTREAM_SERVERS
54. docker run -d -p 80:80 -p 443:443 \ -e UPSTREAM_SERVERS=docker:4000,docker:4001 \ quay.io/aptible/nginx:latest
55. docker run -d -p 80:80 -p 443:443 \ -e UPSTREAM_SERVERS=docker:4000,docker:4001 \ quay.io/aptible/nginx:latest
56. ENV configuration Makes testing easier
57. https://github.com/sstephenson/bats
58. # Dockerfile # Install and configure NGiNX... # ... ADD test /tmp/test RUN bats /tmp/test https://github.com/aptible/docker-nginx Image: quay.io/aptible/nginx
59. #!/usr/bin/env bats # /tmp/test/nginx.bats @test "It should accept a list of UPSTREAM_SERVERS" { simulate_upstream UPSTREAM_SERVERS=localhost:4000 wait_for_nginx run curl localhost 2>/dev/null [[ "$output" =~ "Hello World!" ]] }
60. #!/usr/bin/env bats # /tmp/test/nginx.bats @test "It should accept a list of UPSTREAM_SERVERS" { simulate_upstream UPSTREAM_SERVERS=localhost:4000 wait_for_nginx run curl localhost 2>/dev/null [[ "$output" =~ "Hello World!" ]] }
61. @test "It should accept a list of UPSTREAM_SERVERS" { simulate_upstream UPSTREAM_SERVERS=localhost:4000 wait_for_nginx run curl localhost 2>/dev/null [[ "$output" =~ "Hello World!" ]] } simulate_upstream() { nc -l -p 4000 127.0.0.1 < upstream-response.txt }
62. ENV configuration Abstracts implementation details: could be NGiNX, HAProxy, …
63. ENV configuration Simplifies configuration management: central store doesn’t need to know parameters in advance
64. ENV configuration $UPSTREAM_SERVERS $FORCE_SSL $DISABLE_WEAK_CIPHER_SUITES (…)
65. Vulnerability Response Fix, test, docker push, restart
66. Heartbleed
67. Heartbleed POODLEbleed
68. Heartbleed POODLEbleed xBleed???
69. Integration Tests Document and test every vulnerability response
70. #!/usr/bin/env bats # /tmp/test/nginx.bats @test "It should pass an external Heartbleed test" { install_heartbleed wait_for_nginx Heartbleed localhost:443 uninstall_heartbleed }
71. @test "It should pass an external Heartbleed test" { install_heartbleed wait_for_nginx Heartbleed localhost:443 uninstall_heartbleed } install_heartbleed() { export GOPATH=/tmp/gocode export PATH=${PATH}:/usr/local/go/bin:${GOPATH}/bin go get github.com/FiloSottile/Heartbleed go install github.com/FiloSottile/Heartbleed }
72. Integration tests happen during each image build
73. Integration tests happen during each image build Images are built automatically via Quay Build Triggers
75. Integration tests happen during each image build Build status is easy to verify at a glance
77. Integration tests happen during each image build Quay Time Machine lets us roll back an image to any previous state
79. Database Deployment Standardizing an "API" across databases
81. New databases mean new dependencies
82. New databases mean new dependencies How to document setup steps for engineers?
83. New databases mean new dependencies How to deploy in production?
84. New databases mean new dependencies How to perform common admin tasks? Backups? Replication? CLI access? Read-only mode?
85. Wrap Databases in a Uniform API Standardizing an "API" across databases
86. #!/bin/bash # run-database.sh command="/usr/lib/postgresql/$PG_VERSION/bin/postgres -D "$DATA_DIRECTORY" -c config_file=/etc/postgresql/ $PG_VERSION/main/postgresql.conf" if [[ "$1" == "--initialize" ]]; then chown -R postgres:postgres "$DATA_DIRECTORY" su postgres <<COMMANDS /usr/lib/postgresql/$PG_VERSION/bin/initdb -D "$DATA_DIRECTORY" /etc/init.d/postgresql start psql --command "CREATE USER ${USERNAME:-aptible} WITH SUPERUSER PASSWORD '$PASSPHRASE'" psql --command "CREATE DATABASE ${DATABASE:-db}" /etc/init.d/postgresql stop COMMANDS elif [[ "$1" == "--client" ]]; then [ -z "$2" ] && echo "docker run -it aptible/postgresql --client postgresql://..." && exit psql "$2" elif [[ "$1" == "--dump" ]]; then [ -z "$2" ] && echo "docker run aptible/postgresql --dump postgresql://... > dump.psql" && exit pg_dump "$2" elif [[ "$1" == "--restore" ]]; then [ -z "$2" ] && echo "docker run -i aptible/postgresql --restore postgresql://... < dump.psql" && exit psql "$2"
87. #!/bin/bash # run-database.sh command="/usr/lib/postgresql/$PG_VERSION/bin/postgres -D "$DATA_DIRECTORY" -c config_file=/etc/postgresql/ $PG_VERSION/main/postgresql.conf" if [[ "$1" == "--initialize" ]]; then chown -R postgres:postgres "$DATA_DIRECTORY" su postgres <<COMMANDS /usr/lib/postgresql/$PG_VERSION/bin/initdb -D "$DATA_DIRECTORY" /etc/init.d/postgresql start psql --command "CREATE USER ${USERNAME:-aptible} WITH SUPERUSER PASSWORD '$PASSPHRASE'" psql --command "CREATE DATABASE ${DATABASE:-db}" /etc/init.d/postgresql stop COMMANDS elif [[ "$1" == "--client" ]]; then [ -z "$2" ] && echo "docker run -it aptible/postgresql --client postgresql://..." && exit psql "$2" elif [[ "$1" == "--dump" ]]; then [ -z "$2" ] && echo "docker run aptible/postgresql --dump postgresql://... > dump.psql" && exit pg_dump "$2" elif [[ "$1" == "--restore" ]]; then [ -z "$2" ] && echo "docker run -i aptible/postgresql --restore postgresql://... < dump.psql" && exit psql "$2"
88. --initialize: Initialize data directory --client: Start a CLI client --dump: Dump database to STDOUT --restore: Restore from dump --readonly: Start database in RO mode
89. db-launch () { container=$(head -c 32 /dev/urandom | md5); passphrase=${PASSPHRASE:-foobar}; image="${@: -1}"; docker create --name $container $image docker run --volumes-from $container \ -e USERNAME=aptible -e PASSPHRASE=$passphrase \ -e DB=db $image --initialize docker run --volumes-from $container $@ } http://bit.ly/aptible-dblaunch
90. docker create --name $container $image http://bit.ly/aptible-dblaunch 1. Create "volume container"
91. docker run --volumes-from $container \ -e USERNAME=aptible -e PASSPHRASE=$passphrase \ -e DB=db $image --initialize http://bit.ly/aptible-dblaunch 2. Initialize database data volume
92. docker run --volumes-from $container $@ http://bit.ly/aptible-dblaunch 3. Run database
94. Thank you
95. @fancyremarker frank@aptible.com
 https://speakerdeck.com/fancyremarker