Building Your Docker Tech Stack

Building Your Docker Tech Stack

This session will focus on the practicals of building a fully-functional stack of container cluster tools, with different options for stacking those tools from the OS-up.

We’ve all seen examples of common technologies stacks, like the good ol’ LAMP and MEAN stacks for apps, but what about lower-level infrastructure? And can we get it without cloud vendor lock-in please? Oh and pure containers and infrastructure-as-code too?

With Docker, sure thing! This session will cover:

Which OS/Distro and Kernel to use
VM’s or Bare Metal
Recommended Swarm architectures
Tool stacks for “pure open source”, “cloud-service based”, and “Docker EE” scenarios
Demos of these tools working together including InfraKit, Docker, Swarm, Flow-Proxy, ELK, Prometheus, REX-Ray, and more.

86b88885327486213bf122579d697793?s=128

Bret Fisher

June 13, 2018
Tweet

Transcript

  1. BRET FISHER Docker Captain, DevOps Dude, Author of Docker Mastery

    Building Your Docker Tech Stack bretfisher.com/dockercon18 @bretfisher
  2. People ask "Where's my LAMP for container clusters?" Your solutions

    will be a "stack" of infrastructure tools Cloud-native container tools are new We need patterns and examples of full cluster stacks Problem: No Server Tool Lives In Isolation
  3. Build examples of full-er/ish stacks on different tools Options for

    solo to medium-sized DevOps/Ops teams Use Docker Swarm latest stable as orchestrator Avoid cloud vendor lock-in* 4 Goals for Today * Lock-in: A service I can't swap out in my server stack
  4. Limit "going production Docker" project scope. Go Lean! Focus first

    on quality Dockerfiles Stay on your familiar host OS with 4.x Kernel Use base images of familiar OS (keep same pkg mgr) Swarm CE can be 1 or more nodes, use it everywhere Swarm EE is your "easy button" for security and ops Last Time On Bret's DockerCon Talk
  5. Two Stacks, Same Core Docker CE Swarm Used for Dev/Test

    Heavy use of OSS/free Gluttony of choice for 3rd party Newest engine features Docker EE Swarm Used for Staging/Prod Heavy use of paid support Use Docker Solution Briefs Mature engine with hotfixes + +
  6. Clusters are Complex Start small and simple, grow them as

    you grow
  7. EE Platform Architecture Physical Virtualization Public Cloud Platform Security Developer

    Services Registry Services Access Policies App Lifecycle Management Automation & Extensibility Networking Orchestration Storage Container Engine ENTERPRISE EDITION PLATFORM
  8. CE Platform Architecture Public Cloud Overlay Swarm Container Engine COMMUNITY

    EDITION PLATFORM Physical Virtualization Platform Security
  9. v Sci-Fi, Am I Right!

  10. v Epic Battle Royale, on Swarm dogvs.cat VS

  11. v Epic Battle Royale, on Swarm dogvs.cat Sci-Fi Sounds Edition

  12. dogvs.cat App Services www.dogvs.cat vote.dogvs.cat blog.dogvs.cat result.dogvs.cat (Stack Files)

  13. dogvs.cat Swarm CE, (Han) Solo Sysadmin or

  14. Cloud agnostic, minimal infra Apps auto-recover on node fail Incoming

    TLS Centralized logging Centralized monitoring Healthcheck all containers Infra Requirements of dogvs.cat Performance auto-scaling Self-healing nodes Support serverless functions Services highly available Han Solo Requirements Optional Requirements for Later
  15. Simple Infrastructure, Easy Deployment "How can I deploy a multi-tier

    app on a few servers, with all the bells and whistles of orchestration with load balancing and auto recovery?" Han Solo, The Sysadmin
  16. 3+ Droplets (Ubuntu 16.04) Block Storage (Volumes) Load Balancer (incoming

    HTTP) Digital Ocean dogvs.cat Services Needed for High Availability
  17. App Services www.dogvs.cat vote.dogvs.cat blog.dogvs.cat result.dogvs.cat

  18. App Services + L7 Proxy www.dogvs.cat vote.dogvs.cat blog.dogvs.cat result.dogvs.cat

  19. App Services + L7 Proxy + Overlay www.dogvs.cat vote.dogvs.cat blog.dogvs.cat

    result.dogvs.cat
  20. App Services + L7 Proxy + Ops www.dogvs.cat vote.dogvs.cat blog.dogvs.cat

    result.dogvs.cat
  21. Cluster + External Load Balancer www.dogvs.cat vote.dogvs.cat blog.dogvs.cat result.dogvs.cat

  22. Open Source Stack Swarm GUI Portainer Central Monitoring Prometheus +

    Grafana Central Logging Elastic ELK Layer 7 Proxy Traefik + Let's Encrypt Storage REX-Ray + Digital Ocean Volumes Networking Docker Swarm Overlay Orchestration Docker Swarm Runtime Docker CE HW / OS Docker Machine + Digital Ocean
  23. Deploy Nodes: Docker Machine ./create-servers.sh ./enable-monitoring.sh ./create-swarm.sh

  24. Deploy Storage: RexRay Plug-in docker stack deploy -c stack-rexray.yml rexray

  25. Deploy Proxy Stack: Traefik with Let's Encrypt docker network create

    --driver overlay proxy docker stack deploy -c stack-proxy.yml proxy http://www.dogvs.cat:8080/dashboard/
  26. Deploy Ops Stacks: Prometheus + ELK + Portainer docker stack

    deploy -c stack-swarmprom.yml prom docker stack deploy -c stack-elk.yml elk docker stack deploy -c stack-portainer.yml portainer
  27. Our Apps: Voting + Ghost + Static Site docker stack

    deploy -c stack-ghost.yml ghost docker stack deploy -c stack-voting.yml vote docker stack deploy -c stack-menu.yml menu
  28. Deploy Stacks: Maintenance Tasks docker stack deploy -c stack-prune.yml prune

    backups (in stack file with app)
  29. Day Two Operations: Updates stack deploy ∞ micromanage update_config and

    healthcheck tune your limits and reservations
  30. Security? host setup scanning: Docker Bench image scanning: Aqua Microscanner

    behavior monitoring: Sysdig Falco user namespaces
  31. Next Steps more nodes? make 'em workers CI/CD stacks: gitlab,

    jenkins make redis, mysql, psql HA add app metrics to Prometheus swap Overlay for Weave Net swap ELK for Papertrail, etc. swap Prometheus for Sysdig, Datadog, Librato, etc. add socat proxy to Traefik
  32. Swarm EE, Amazonian DevOps dogvscat.biz or

  33. Complex Infrastructure, Harder Deployment "How can I deploy many multi-tier

    app on a many servers, with all the b e l l s a n d w h i s t l e s o f H A orchestration, have load balancing at all levels, with failover and auto recovery?" Amazonian Team
  34. CE Platform Architecture Physical Virtualization Public Cloud Platform Security Networking

    Orchestration Container Engine COMMUNITY EDITION PLATFORM
  35. EE Platform Architecture Physical Virtualization Public Cloud Platform Security Developer

    Services Registry Services Access Policies App Lifecycle Management Automation & Extensibility Networking Orchestration Storage Container Engine ENTERPRISE EDITION PLATFORM
  36. No More One Size Fits All Docker for AWS Docker

    for Azure
  37. Docker Certified Infrastructure

  38. v Reference Architecture Automation Tools Ecosystem Integration Docker Certified Infrastructure

  39. DCI AWS

  40. Docker EE on AWS Stack Swarm GUI Docker EE UCP

    Central Monitoring AWS Cloudwatch + Telegraph Central Logging AWS Cloudwatch Logs Registry Docker EE DTR Layer 7 Proxy HTTP Routing Mesh (Interlock+Nginx) Storage Docker Cloudstor EBS/EFS Networking Docker Swarm Overlay Orchestration Docker Swarm Runtime Docker EE HW / OS Terraform + Ansible + AWS
  41. Deploy Nodes: Terraform + Ansible terraform apply ansible-playbook -i inventory

    install.yml
  42. Advantages more flexible deployment tools more SecOps tools ops tools

    are fully HA ops tools are team-ready deploy to K8s just as easy
  43. Deploy Apps docker stack deploy -c stack-ghost.yml ghost docker stack

    deploy -c stack-voting.yml vote docker stack deploy -c stack-menu.yml menu
  44. Next Steps All the things in Swarm CE monitoring via

    CloudWatch and Telegraph logging via CloudWatch Logs
  45. Summary Infrastructure as code, make everything repeatable No "special" nodes,

    use remote management Grow as you go, assume you'll resize Look for compose files of popular tools to make stacks Don't throw out the good in search of the perfect
  46. I'd like to thank the internet

  47. Support the open source you use

  48. PLEASE VOTE SO I WIN ALL THE DOCKERS FRIDAY! Thanks!

    bretfisher.com/dockercon18 "Building Your Docker Tech Stack"