Upgrade to Pro — share decks privately, control downloads, hide ads and more …

docker-swarming-groove

 docker-swarming-groove

**Docker Swarming with groove talk**

Here you can find my docker slide container decks from my docker orchestration with swarming mode talks:

- [DevOpsCon Munich 2016 - Orchestration with Docker Swarming][1].
- open slidefire presentation container with `docker run -d -p 8000:80 rossbachp/docker-swarming-groove`
- pdf [Orchestration with Docker Swarming][2] is now avaiable!

Open slidefire presentation container with your browser `http :8000/docker-swarming-groove`

Source and examples can be find at

* https://github.com/bee42/traefik-with-docker

```
$ cd examples/docker-swarming-dind
$ ./start-dind.sh
$ ./start-traefik.sh
$ ./start-whoami
$ open 127.0.0.1:5080
```

Feedback welcome

[Peter Rossbach][3]

[1]: https://devopsconference.de/session/docker-orchestration-leicht-gemacht/
[2]: https://speakerdeck.com/rossbachp/docker-swarming-groove
[3]: http://twitter.com/PRossbach

Peter Rossbach

December 06, 2016
Tweet

More Decks by Peter Rossbach

Other Decks in Programming

Transcript

  1. 1 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Find a new groove for your services with swarming @PRossbach bee42 solutions gmbh [email protected]
  2. 2 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Build, Ship and Run Any App, Anywhere
  3. 5 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Desired State Reconciliation Types Replicated Services Global Services Configurable Updates Parallelism Delay Restart Policies Failure Detection Auto Loadbalancing Scheduling Default Security
  4. 7 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Resource Awareness Constraints IDs names labels (e.g. node.labels.foo!=bar1) Strategies Spread strategy Least loaded nodes Respect the constraints Respect the resource requirements
  5. 8 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon State Store Replicated (Raft based) Extremely fast (in-memory reads) Save snapshots Topology Management Managers Workers Node Management Pause/Unpause Drain/Activate Auto failure detection
  6. 9 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Mutual TLS Create CA or use a user CA Manage certs to all nodes Acceptance Policy Auto accept Manually accept Require a secret to join the cluster Certificate Rotation Rotated and reloaded transparently on every node Default is 3 months, the minimum is 30 minutes
  7. 10 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Create new machines with digitalocean|scaleway|xyhve Setup network (overlay) Manage a swarm on it :) Use Docker > 1.12.x
  8. 11 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon https://github.com/zchee/docker-machine-driver-xhyve https://github.com/scaleway/docker-machine-driver-scaleway https://github.com/bee42/docker-machine-scaleway-creator
  9. 12 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Create your docker-machine Login to this machine Create a cluster manager and worker
  10. 13 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ REGION=asm2 $ SIZE=1gb $ DOT=<token> $ docker-machine create \ --driver digitalocean \ --digitalocean-region $REGION \ --digitalocean-size $SIZE\ --digitalocean-access-token $DOT \ --digitalocean-private-networking \ --engine-label "cluster=moby2016" \ --engine-label "role=master" \ --engine-label "region=$REGION" \ swarm-master Create a DigitalOcean account with this promo link Test Release: --engine-install-url "https://test.docker.com" https://m.do.co/c/1b93d52f958f
  11. 14 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker $(docker-machine config swarm-master) swarm init \ --advertise-addr $(docker-machine ip swarm-master):2377 $ SWARM_TOKEN=`docker $(docker-machine config swarm-master) \ swarm join-token worker -q`
  12. 15 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ REGION=asm2 $ SIZE=1gb $ MACHINE=swarm-01 $ DOT=xxx $ docker-machine create \ --driver digitalocean \ --digitalocean-region $REGION \ --digitalocean-size $SIZE\ --digitalocean-access-token $DOT \ --digitalocean-private-networking \ --engine-label "cluster=moby2016" \ --engine-label "role=swarm" \ --engine-label "region=$REGION" \ $MACHINE $ docker $(docker-machine config $MACHINE) swarm join \ --token $SWARM_TOKEN \ $(docker-machine ip swarm-master):2377
  13. 16 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Use a whoami service Service Options Docker on ARM Uuid rolling update Visualizer of docker swarming Monitor docker and services with Prometheus ELK Stack Simple DIND Swarming Loadbalancing with Traefik Docker Meetup Bochum 1.12:)
  14. 17 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon https://github.com/emilevauge/whoami $ docker run -d -P --name whoami emilevauge/whoami $ docker inspect --format '{{ .NetworkSettings.Ports }}' whoami map[80/tcp:[{0.0.0.0 32769}]] $ curl "http://0.0.0.0:32769" Hostname : 6e0030e67d6a IP : 127.0.0.1 IP : ::1 IP : 172.17.0.27 IP : fe80::42:acff:fe11:1b GET / HTTP/1.1 Host: 0.0.0.0:32769 User-Agent: curl/7.35.0 Accept: */*
  15. 18 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create --name whoami emilevauge/whoami $ docker service scale whoami=2 $ docker service inspect whoami [ { "ID": "ch1cizq1k61qigbswzdd0sryj", "Version": { "Index": 717 }, "CreatedAt": "2016-06-19T11:01:39.630354919Z", "UpdatedAt": "2016-06-19T11:02:20.137102719Z", "Spec": { "Name": "whoami", ... ] $ docker service inspect whoami |jq "{ name: .[].Spec.Name , replicas: .[].Spec.Mode.Replica { "name": "whoami", "replicas": 2 }
  16. 19 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ eval $(docker-machine env swarm_master) $ docker service create --name cadvisor --mode global \ --mount type=bind,source=/,target=/rootfs/,writable=false \ --mount type=bind,source=/var/run/,target=/var/run/ \ --mount type=bind,source=/sys/,target=/sys/,writable=false \ --mount type=bind,source=/var/lib/docker/,target=/var/lib/docker/,writable= --publish 8080:8080 \ google/cadvisor:latest Add a new node, and service is started...
  17. 20 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker node update --label-add role=frontend $ docker node update --label-add role=backend --label-add storage=ssd Usage: docker node update [OPTIONS] NODE Update a node Options: --availability string Availability of the node (active/pause/drain) --help Print usage --label-add value Add or update a node label (key=value) (defaul --label-rm value Remove a node label if exists (default []) --role string Role of the node (worker/manager)
  18. 21 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Node attribute Matches Example node.id node ID node.id == 2ivku8v2gvtg4 node.hostname node hostname node.hostname != node-2 node.role node role: manager node.role == manager node.labels user defined node labels node.labels.security == high engine.labels Docker Engine’s labels engine.labels.operatingsystem == ubuntu 14.04
  19. 22 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create \ --name redis_2 \ --constraint 'node.labels.type == queue' \ redis:3.0.6
  20. 23 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service update \ --constraint-add 'node.labels.storage == ssd' \ redis_2 Option Description --constraint-add value Add or update placement constraints (default []) --constraint-rm value Remove a constraint (default []) --container- label-add value Add or update container labels (default []) --container- label-rm value Remove a container label by its key (default [])
  21. 24 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create \ --name redis_2 \ --label com.example.foo="bar" --label bar=baz \ redis:3.0.6
  22. 25 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon id label name $ docker service ls -f "id=0bcjw" $ docker service ls --filter label=project $ docker service ls --filter label=project=project-a $ docker service ls --filter name=redis
  23. 26 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Mount Named Volume Mount Anonymous Volume Bind Mount Volume
  24. 27 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create \ --name my-service \ --replicas 3 \ --mount type=volume,source=my-volume,destination=/path/in/container,volume nginx:alpine
  25. 28 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create \ --name my-service \ --replicas 3 \ --mount type=volume,destination=/path/in/container \ nginx:alpine
  26. 29 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create \ --name my-service \ --mount type=bind,source=/path/on/host,destination=/path/in/container \ nginx:alpine
  27. 30 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon UUID swarming # login to swarm manager $ docker build --build-arg UUID_VERSION=0.0.1 \ -t bee42/uuid:0.0.1 -f Dockerfile.uuid . $ docker tag bee42/uuid:0.0.1 queenshive:5000/bee42/uuid:0.0.1 $ docker push queenshive:5000/bee42/uuid:0.0.1 $ docker service create --name uuid queenshive:5000/bee42/uuid:0.0.1 $ docker service update --publish-add 9080:8080 uuid $ docker service update --replicas 2 uuid UUID next version $ docker build --build-arg UUID_VERSION=0.0.2 \ -t bee42/uuid:0.0.2 -f Dockerfile.uuid . $ docker tag bee42/uuid:0.0.2 queenshive:5000/bee42/uuid:0.0.2 $ docker push queenshive:5000/bee42/uuid:0.0.2 $ docker service update \ --replicas 4 \ --update-delay 10s \ --image queenshive:5000/bee42/uuid:0.0.2 uuid
  28. 31 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ curl 192.168.5.1:9080/uuid/index.jsp { "Container": "9e1daabe3fb2", "UUID": "37a971b7-1467-47de-90e2-20cd2d9de4df", "Date": "2016/10/25 14:30:25", "Timestamp": "1477405825", "Version": "0.0.2" }
  29. 32 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker run -it -d -p 5000:5000 \ -v /var/run/docker.sock:/var/run/docker.sock \ queenshive:5000/bee42/swarm-visualizer https://github.com/ManoMarks/docker-swarm-visualizer
  30. 35 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon https://github.com/JulienBreux/docker-swarm-gui
  31. 36 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create --name node --mode global --network prom \ --mount type=bind,source=/proc,target=/host/proc \ --mount type=bind,source=/sys,target=/host/sys \ --mount type=bind,source=/,target=/rootfs \ prom/node-exporter \ -collector.procfs /host/proc \ -collector.sysfs /host/proc \ -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/) $ docker service create --name cadvisor --network prom --mode global \ --mount type=bind,source=/,target=/rootfs \ --mount type=bind,source=/var/run,target=/var/run \ --mount type=bind,source=/sys,target=/sys \ --mount type=bind,source=/var/lib/docker,target=/var/lib/docker \ google/cadvisor:latest
  32. 37 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ cat >prometheus.conf <<EOF global: scrape_interval: 1s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'node' dns_sd_configs: - names: ['tasks.node'] type: 'A' port: 9100 - job_name: 'cadvisor' dns_sd_configs: - names: ['tasks.cadvisor'] type: 'A' port: 8080 EOF $ cat >Dockerfile <<EOF FROM prom/prometheus COPY prometheus.yml /etc/prometheus/prometheus.yml EOF $ docker build -t queenshive:5000/prometheus . $ docker push queenshive:5000/prometheus $ docker service create --network prom --name prom \ --publish 9090:9090 queenshive:5000/prometheus
  33. 38 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker network create --driver overlay logging $ docker service create --network logging \ --name elasticsearch elasticsearch:2.4 $ docker service create --network logging \ --name kibana --publish 5601:5601 \ -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana:4.6
  34. 39 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ cat >logstash.conf <<EOF input { # Listens on 514/udp and 514/tcp by default; change that to non-privileged port syslog { port => 51415 } # Default port is 12201/udp gelf { } heartbeat { } } filter { ruby { code => " event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' } " } } output { elasticsearch { hosts => ["elasticsearch:9200"] } stdout { codec => rubydebug } } EOF
  35. 40 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker service create --network logging \ --name logstash -p 12201:12201/udp \ logstash:2.4 -e "$(cat ./logstash.conf)" $ docker service ps logstash $ CID=$(docker ps -q --filter label=com.docker.swarm.service.name=logstash) $ docker logs --follow $CID $ docker run --log-driver gelf \ --log-opt gelf-address=udp://127.0.0.1:12201 \ --rm alpine echo hello $ open http://$(docker-machine ip worker1):5601/ $ for SERVICE in app1 app2 ; do docker service update $SERVICE \ --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201 done
  36. 41 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Auto update to next software release A/B Testing Canary Deployment Multi-Network Handling Control the liveness and health with Healthcheck Swarming with prometheus - no scraper at this moment! Create a Visualizer for complex setups Volume Management and Docker Plugin's Check the API DAB Format and docker-compose
  37. 43 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon https://github.com/bee42/traefik-with-docker
  38. 44 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker swarm init $ SWARM_TOKEN=$(docker swarm join-token -q worker) $ SWARM_MASTER=$(docker info | grep -w 'Node Address' | awk '{print $3}') $ NUM_WORKERS=3 $ for WORKER_NUMBER in $(seq ${NUM_WORKERS}); do docker run -d --privileged --name worker-${WORKER_NUMBER} \ --hostname=worker-${WORKER_NUMBER} \ -p ${WORKER_NUMBER}2375:2375 docker:1.13.0-rc2-dind docker --host=127.0.0.1:${WORKER_NUMBER}2375 swarm join \ --token ${SWARM_TOKEN} \ ${SWARM_MASTER}:2377 done $ docker run -it -d -p 5080:8080 --name visualizer \ -v /var/run/docker.sock:/var/run/docker.sock \ manomarks/visualizer $ docker service create --name registry \ --constraint 'node.role == manager' \ --publish 5000:5000 registry:2 $ curl 127.0.0.1:5000/v2/_catalog https://medium.com/@alexeiled/docker-swarm-cluster-with-docker-in-docker-on-macos- bdbb97d6bb07#.yry5s0jiv http://blog.terranillius.com/post/swarm_dind/ https://github.com/ManoMarks/docker-swarm-visualizer
  39. 45 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon ɪ https://traefik.io/ https://github.com/bee42/traefik-with-docker
  40. 47 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon ɪ $ docker-machine ssh swarm-master \ "docker network create --driver=overlay traefik-net" $ docker-machine ssh swarm-master "docker service create \ --name traefik \ --constraint=node.role==manager \ --publish 80:80 \ --publish 8080:8080 \ --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \ --network traefik-net \ traefik:v1.1.0 \ --docker \ --docker.swarmmode \ --docker.domain=traefik \ --docker.watch \ --web" https://github.com/containous/traefik/blob/master/docs/user- guide/swarm-mode.md
  41. 48 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon $ docker-machine ssh swarm-master "docker service create \ --name whoami0 \ --label traefik.port=80 \ --network traefik-net \ 127.0.0.1:5000/emilevauge/whoami" $ docker-machine ssh swarm-master "docker service create \ --name whoami1 \ --label traefik.port=80 \ --network traefik-net \ 127.0.0.1:5000/emilevauge/whoami" $ curl -H Host:whoami0.traefik http://$(docker-machine ip manager)
  42. 50 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Use old docker swarm Use consul, etcd with registrator New docker swarm 1.12 is available >1.1.0! https://hub.docker.com/r/library/traefik/
  43. 52 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Declarative management toolkit for orchestrating infrastructure Describe infrastructure encapsulation and composition Active control Monitor infrastructure status Detect state divergence Take action Continuous monitoring and reconciliation Rolling update https://github.com/docker/infrakit http://de.slideshare.net/chanezon/whats-new-in-docker- infrakit-docker-meetup-berlin-2016
  44. 54 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Nice concept and easy to use Simple and powerful orchestration solution Play around with fun Mesh Network Rolling Update ToDo: Better support of storage handling Auto Loadbalancing Add Serverless and Job scheduling support More options to service create (security and resource handling) Integration of Infrakit Better support for Multi-Architecture setups Check your networking skills!
  45. 65 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Reduce you coupling Add metrics Design for failure Setup machines and network automatically Handling your persistence states on classic infrastructure Your developer need a prod like environment Safe your persistence data
  46. 68 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Start this presentation with docker run -d -ti -p 4221:80 rossbachp/docker- swarming-groove open http://<dockerhost>:4221/docker-swarming- groove Peter Rossbach follow the blog www.infrabricks.de @PRossbach More Training: http://www.bee42.com/training/
  47. 69 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon 28.11 Microservice Summit Berlin 1.12 Docker Meetup Bochum - Traefik 5.12 DevOpsCon Docker Basic Workshop (Munich) 8.12 DevOpsCOn Docker Expert OpenSpace Workshop (Munich) 12-14.12 Docker Camp at Berlin 21.12 Docker Meetup Dortmund 17.1 Docker Meetup Münster - Orchestration http://devops-training.de/
  48. 70 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon We offer TOP-IT jobs... http://bee42.github.io http://www.bee42.com
  49. 71 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Living an Agile culture Influence us to DevOps-Journey Enable people to use modern technologies Build applications that living at the clouds Design services with clever API's Think infrastructure first Send your offering to [email protected]
  50. 72 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon Create beautiful human responsive services Find new ways to create creative web service Find good solutions with HTML/CSS/Javascript to create simple services Think mobile first Send your offering to [email protected]
  51. 73 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon We offer jobs... Administrator Experiences with Maintain a cluster of linux machines Setup Docker infrastructure Config management with ansible Release java applications Linux and mysql skills Developer Experiences with Create java based microservices Spring Boot and Netflix Stack Docker knowledge send your offering to [email protected]
  52. 76 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon This slides shows shell commands executed in different places. The line prefix shows where to run them: $ ... – local machine > ... – remote machine # ... – docker container
  53. 77 / 77 © 2016 <[email protected]>, @PRossbach, Docker Workshop 12/2016

    DevOpsCon http://www.infrabricks.de http://www.bee42.com http://bee42.github.io https://github.com/bee42/docker-on-rpi-lab https://www.docker.com https://github.com/docker/docker https://github.com/docker/machine https://github.com/docker/swarm https://github.com/docker/compose https://github.com/docker/swarmkit https://github.com/docker/infrakit