Docker in Production (Docker Edinburgh, January 2016)

Docker - the awesome utility that allows you to create a single, runnable, distributable application … but is it really that awesome?

In this talk I’ll discuss some of the common pitfalls and hurdles that one must navigate to have an enjoyable and fruitful relationship with Docker; giving you the confidence to take that relationship to the next stage: production.

Transcript

1. Docker in Production
2. david@rawkode.com @rawkode github.com/rawkode Me! ★ Director of Development for TeamRock ★ Docker Glasgow Organiser Taking on consultancy work from Feb: • PHP ~ Go ~ Java ~ Obj-C • CQRS & Event Sourcing • DevOps / CI / CD / Docker • Domain-Driven Design • Framework agnostic code • TDD / BDD
3. Who are TeamRock?
4. TeamRock’s Portfolio
5. The Numbers
6. Magazines Tens of thousands of copies sold each month … and dropping
7. TeamRock: The Digital Transition The world’s leading rock and metal web siteTM
8. Our Corner of The Internet
9. Website Somewhere between 1 and 10 million unique visitors per month … give or take ten million
10. Website It’s news … we can’t predict the news
11. Our biggest challenge is scale
14. This is my journey. Your milage may vary
15. Challenge #1 AutoScaling wasn’t quick enough New instances need to bootstrap FAST
16. Hello, Docker!
17. Challenge #1 - Bootstrapping a Machine (Old Way) • Instance boots • USER DATA script installs SaltStack (or whatever provisioning tool you use) • SaltStack Executes ◦ apt-get update ◦ apt-get install apache php5 php5-memcached ◦ Modify a bunch of configuration files for apache and php ◦ git clone codebase ◦ Get project dependencies (composer install | bundle install | goget, etc) ◦ Warm up caches ◦ Execute database migrations ◦ Finally … start apache
18. Challenge #1 - Bootstrapping a Machine (New Way) • Instance boots • USER DATA script installs SaltStack (or whatever provisioning tool you use) • SaltStack Executes ◦ wget -qO- https://get.docker.com/ | sh ◦ docker pull ◦ docker run
19. Simpler. More Maintainable. Faster.
20. Faster Still • If you’re using AWS ◦ Use S3 for your registry ◦ Build your own AMI with Docker installed ▪ Just remember to keep it updated / security patches
21. Great Success.
22. Challenge #2 What resources do my containers really need? Monitoring
23. Options • Sensu • Prometheus • Nagios • DataDog • statsd
24. DataDog • No infrastructure to manage • It’s cheap • It’s awesome
25. Installing the Agent DD_API_KEY=nope bash -c "$(curl -L https://raw. githubusercontent.com/DataDog/dd- agent/master/packaging/datadog- agent/source/install_agent.sh)"
26. Container the Agent? You can - you lose out on: • Disk IO metrics • System process metrics • Network metrics
27. Docker Metrics
28. Infrastructure Map
29. A Single Zone
30. Agent Info
31. Docker Metrics
32. Integrations • MySQL • PostgreSQL • Mongo • Cassandra • All AWS services • PagerDuty • OpsGenie
33. Future • Still looking at getting Prometheus or Sensu running • Smarter AutoScaling, based on containers - rather than instances
34. Great Success.
35. Challenge #3 Logging
36. Logging Rolled our own LogStash cluster
37. Disclaimer
38. Our centralised logging, at this moment in time, is FUBAR
39. How It Was Supposed to Work • Data Container Awesomeness • Container for forwarding • Sing the sweet praises of Docker
40. Docker Log Driver Since Docker >= 1.6 --log-driver=<driver> • json-file • syslog • fluentd • awslogs • gelf (LogStash and Graylog)
41. Work in Progress
42. Challenge #4 Orchestration
43. Options • Universal Control Plane • Mesos • ECS • Kubernetes • Swarm • SaltStack
44. SaltStack Keeping it simple teamrock/blackbird:latest dockerng.image_present
45. SaltStack custom-docker-registry: https://registry:5000: email: deploy@teamrock.com username: deployment password: indeed Keeping it simple
46. SaltStack Keeping it simple my-container: dockerng.running: - image: teamrock/blackbird:latest - require: - dockerng: my-container-pre my-container-pre: dockerng.running: - image: teamrock/blackbird:latest - entrypoint: /opt/deploy/pre.sh
47. SaltStack Keeping it simple salt --async \ -C "G@ec2_tags:Environment:Production" \ state.sls deploy \ "pillar={ 'project': 'blackbird',\ 'buildNumber': ‘BB-184’ }"
48. Great Success. … For now ...
49. Challenge #5 Container Variants
50. Options • Build Time Variables • Consul • SaltStack
51. Build Time Variables Works really well! … until you change a variable
52. Consul Looks amazing! But you need to manage infrastructure
53. SaltStack Pillars can live in: MySQL, PostgreSQL, redis, files, … my-container: dockerng.running: - image: teamrock/blackbird:latest - environment: - POSTGRES_URL: {{ salt[‘pillar’].get (‘postgres_url’) }} - require: - dockerng: my-container-pre
54. Great Success.
55. Other Challenges? • Frontend People • OSX ◦ File Syncing ◦ eval eval eval • Frontend People
56. Docker Tips
57. Docker Tips Containers die: --restart
58. Docker Tips docker save / load
59. Docker Tips overlayfs
60. Docker Tips ONBUILD
61. Docker Tips Don’t use community images
62. Docker Tips Port based service discovery
63. Docker Tips Random Ports for Workers
64. Docker Tips Just say no to --privileged
65. Questions? @rawkode https://goo.gl/mI1ozM My dog as a Christmas pudding!