Docker Ecosystem

Slides for my talk at the Google Developer Group meetup on August 27th in Zurich, Switzerland about tools of the Docker ecosystem and how to use them to build a Docker based platform.

http://www.meetup.com/de/GDG-Zurich/events/224231328/

Transcript

1. Docker Ecosystem Georg Kunz, CloudGear 27. August 2015
2. About Me Ruby and infrastructure freelancer and founder of CloudGear, previous web architect at local.ch Interests: Rails/Ruby/Go/Cloud/Docker/PaaS/Infrastructure Web: http://georgkunz.com https://www.cloudgear.net Twitter: @geku Email: georg@cloudgear.net
3. Building a Platform CC BY-SA 2.0 http://bit.ly/1GgCb8H
4. Reality Check What we would like to ship What we have to ship Images by Derell Licht https://flic.kr/p/pnSzL licensed under CC BY-ND 2.0 and Chris https://flic.kr/p/5TtPFZ licensed under CC BY 2.0
5. standardize simplifies operations
6. speed up time from dev to production CC BY 2.0 http://bit.ly/1L2dEdY
7. • microservices • polyglot landscape • continuous delivery Buzzwords CC BY 2.0 http://bit.ly/1IQNpVK
8. How? CC BY 2.0 http://bit.ly/1Gu9SqD by using the Docker ecosystem
9. First Things First We still need servers CC BY 2.0 http://bit.ly/1Guan42
10. Container Runtime
11. Isolation, isolation, isolation Immutability Container Runtime CC BY 2.0 http://bit.ly/1epJ7uu
12. Routing • Expose services on port 80/443 • Domain based routing • Load balancing (HTTP & TCP) CC BY 2.0 http://bit.ly/1MNLEcC
13. Port 80 Node a.com container c.com container Routing Port 80 Node A a.com container b.com container Internet Node B
14. Port 80 Node a.com 34205 c.com 33789 Routing Port 80 Node A a.com 32781 b.com 36491 Node B nginx reverse proxy config: upstream a-com { server nodeA:32781; server nodeB:34205; } server { listen 80; server_name a.com www.a.com; location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For ... proxy_set_header Host $http_host; proxy_pass http://a-com; } }
15. Discovery Inventory and Source of truth Provides lookup table CC BY 2.0 http://bit.ly/1CeVe2C
16. Container Discovery a.com: nodeA:32506 nodeB:33555 nodeB:34912 nodeC:38144 b.com: nodeA:36707 .... Node Node A a.com container b.com container 0.com container Node Node B a.com container a.com container 0.com container Node Node C e.com container a.com container f.com container register/deregister container
17. Consul $ http PUT 192.168.59.103:8500/v1/agent/service/register \ ID=redis.1 Name=redis Address="192.168.59.103" Port:=9500 $ http 192.168.59.103:8500/v1/catalog/services { "consul": [], "redis": [] } $ http 192.168.59.103:8500/v1/catalog/service/redis [ { "Address": "192.168.59.103", "Node": "n1", "ServiceID": "redis.1", "ServiceName": "redis", "ServicePort": 9500, }, { "Address": "192.168.59.103", "Node": "n1", "ServiceID": "redis.2", "ServiceName": "redis", "ServicePort": 8500, } ] 1. register 2. list services 3. list instances
18. Routing & Discovery Node Node A a.com container b.com container b.com container 2. register new container 1. add container for b.com watch services backend 1 3. send event “new instance” 4. backend 2 gets added
19. Clustering • Node discovery • Leader election • Health checks CC BY 2.0 http://bit.ly/1Im3WhE
20. Node Discovery nodeA:192.168.1.10 nodeB:192.168.1.20 nodeC:192.168.1.30 ... register/deregister node Node C Consul Agent $ http 192.168.1.10:8500/v1/catalog/nodes [ { "Node": "nodeA", "Address": "192.168.1.20" }, { "Node": "nodeB", "Address": "192.168.1.20" } ] Node C Consul Agent Node C Consul Agent
21. Distributed Storage • Consistent key/value storage • Locks and master election • Change notifications CC0 1.0 http://bit.ly/1KQ86j9
22. Tools • Key/value stores/discovery: • Consul (https://www.consul.io/) • etcd (https://coreos.com/etcd/) • Discovery: • Docker links • Consul and Registrator (https://github.com/gliderlabs/registrator) • SkyDNS (https://github.com/skynetservices/skydns) • Smartstack (http://nerds.airbnb.com/smartstack-service-discovery-cloud/) • Routing • nginx-proxy (https://github.com/jwilder/nginx-proxy) • confd for etcd (https://github.com/kelseyhightower/confd) • Vulcand (https://github.com/mailgun/vulcand) • Consul Template (https://github.com/hashicorp/consul-template)
23. Deploy Pipeline • git push or callback • Docker build with Dockerfile • Image storage • Instance updates CC BY 2.0 http://bit.ly/1QFnF5E
24. Deploy Pipeline GIT client repo 1. git push Builder docker build ... docker push image bare GIT repo Docker Registry Repository 2. build Image 3. push image to registry Node A a.com v1 0.com Node B a.com v1 0.com a.com v2 Image 4. replace running instances
25. Tools • gitreceive to call simple build script (https://github.com/progrium/gitreceive) • Docker Registry Commercial or open source • Heroku Buildpack support with buildstep (https://github.com/progrium/buildstep)
26. Recap Port 80 Node Registrator c.com container Port 80 Node A Registrator b.com container Internet Node B GIT repo Builder Node C Docker Registry Developer
27. Storage • Manage stateful containers (volumes) • Redundant storage • Backup/snapshots • Container migration CC BY 2.0 http://bit.ly/1JoeGPS
28. Tools • flocker (https://github.com/ClusterHQ/flocker) EBS, OpenStack Cinder, EMC products • convoy (https://github.com/rancher/convoy) devicemapper and NFS • or distributed database
29. Network • Ideal case: each Container has routable IP • direct container to container connections • each port directly accessible CC BY-SA 2.0 http://bit.ly/1JA3pgx
30. Network: Solutions • Overlay network • flannel (https://github.com/coreos/flannel) • Weave (http://weave.works/) • “native” • IPv6 (difficult in cloud)
31. • Images • use few base images (security updates) • Understand what’s inside! • Start today step by step, e.g. • Consul • Dev or Staging with Docker & compose CC BY 2.0 http://bit.ly/1eqbk4j
32. Final Words You made it! CC BY 2.0 http://bit.ly/1eqdwsI
33. Final Words • Ask me questions anytime • Please follow @cloudgear_net or sign up on the mailing list • Support us with open source development CC BY 2.0 http://bit.ly/1eqdwsI
34. ? CC BY 2.0 http://bit.ly/1JWCVae