Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DNS Service Discovery for Docker Swarm Clusters

DNS Service Discovery for Docker Swarm Clusters

I gave this talk at DockerCon EU 2015 in Barcelona, Spain on Nov 16, 2015. The talk is about state of the service discovery problem in today's container clusters and a solution I developed for Docker Swarm called “wagl”.

More info about wagl and this talk:
- https://github.com/ahmetalpbalkan/wagl
- https://ahmetalpbalkan.com/blog/wagl/

Ahmet Alp Balkan

November 16, 2015
Tweet

More Decks by Ahmet Alp Balkan

Other Decks in Technology

Transcript

  1. DNS Service Discovery for Docker Swarm Ahmet Alp Balkan @AhmetAlpBalkan

    Software Engineer, Microsoft
  2. About The Speaker 2 I am Ahmet Alp Balkan, a

    software engineer at Microsoft. I contribute to Open Source. Follow me at @AhmetAlpBalkan.
  3. About This Talk 3 Service Discovery Service Discovery Methods A

    peek into various solutions Thought exercises Service Discovery for Docker Swarm Can a drop-in tool just do it™ for Swarm? Where are we headed?
  4. Before we begin 4 how many of you… … use

    Docker Swarm? … used a Service Discovery method? … wrote or configured a DNS server?
  5. Service Discovery 5 Cluster

  6. Service Discovery 6 Cluster machine1 machine3 machine4 machine2 machine5 machine6

    machine7 machine8
  7. Service Discovery 7 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    db db db machine6 machine1
  8. Service Discovery 8 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    api api api db db db machine6 machine1
  9. Service Discovery 9 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db machine6 machine1
  10. Service Discovery 10 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
  11. Service Discovery 11 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
  12. Service Discovery 12 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
  13. Service Discovery 13 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
  14. Service Discovery 14 Cluster machine3 machine4 machine2 machine5 machine7 machine8

    web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
  15. Service Discovery 15 ServiceA ServiceB help services find and talk

    to each other
  16. Service Discovery 16 ServiceA ServiceB addr help services find and

    talk to each other
  17. Service Discovery 17 ServiceA ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB

    ServiceB
  18. Service Discovery 18 ServiceA ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB

    ServiceB
  19. Service Discovery Methods 19 before we begin…

  20. Service Discovery Methods 20 before we begin… SPOILER: No method

    is actually really good.

  21. Service Discovery Methods 21 before we begin… SPOILER: No method

    is actually really good.
 This is still an unsolved problem.
 Food for thought, not a comparison.
  22. Service Discovery in my dreams… 22 it comes with the

    orchestrator
 …or it is a “setup and forget about it”
 does not require code modification in microservice
 uses a reliable networking stack,
 does not have too many moving parts.

  23. Common Approaches 23 to Service Discovery Overlay Networks
 Mixing Tools

    (docker bridge + template + rev proxy)
 Port Scanning
 Domain Name Service
  24. 24 Overlay networks

  25. Overlay Networks 25 node0 ServiceA node1 ServiceB *magic* serviceB:80

  26. Overlay Networks 26 Good at container-to-container networking
 Static port allocation

    not a problem IP address per container
 Seamless, does not change the application code
 Introduces network latency overhead*
  27. Docker 1.9 Multi-Host Networking 27 Container-to-container overlay network
 Discovery* through

    /etc/hosts entries (DNS)
 
 
 
 
 
 
 
 
 Lacking a load balancer (how to http://serviceA?)
  28. 28 Docker 1.9 Multi-Host Networking

  29. 29 Mixin’ Tools
 (because we can)

  30. 30 docker bridge template reverse proxy service registry HAProxy NGINX

    Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
  31. 31 Reverse Proxies

  32. Reverse Proxies (TCP/IP Load Balancers) 32 HAProxy, NGINX, Træfik, kube-proxy…

    Route load balance traffic to multiple backends can route traffic from/to any port list of backends can be dynamically updated
 ServiceA ServiceB Proxy node1:32012 ServiceB node2:33406 ServiceB node7:32104
  33. 33 HAProxy, NGINX, Træfik, kube-proxy… Load balancing
 Health checks do

    not route traffic to unhealthy backends ServiceA ServiceB Proxy node1:32012 ServiceB node2:33406 ServiceB node7:32104 health probes Reverse Proxies (TCP/IP Load Balancers)
  34. 34 Sticky sessions route a client’s traffic to the same


    backend between requests/connections
 
 Origin-based access control (ACLs)
 HAProxy, NGINX, Træfik, kube-proxy… ServiceA ServiceB Proxy ServiceC ServiceB Proxy Reverse Proxies (TCP/IP Load Balancers) 172.0.1.10 172.0.1.22
  35. 35 Connection draining* wait all connections to close before removing

    the backend from the routing list
 Allows blue-green deployments flip the switch → the new version of 
 your service starts getting traffic
 HAProxy, NGINX, Træfik, kube-proxy… Reverse Proxies (TCP/IP Load Balancers)
  36. 36 Downside: another moving part that can fail what if

    the proxy server crashes?
 Downside: discovery of the proxy server itself where do you place the proxy server(s)? what happens when they get rescheduled
 to another host? how do you discover proxy servers?
 Downside: introduces latency overhead HAProxy, NGINX, Træfik, kube-proxy… Reverse Proxies (TCP/IP Load Balancers)
  37. 37 Bridging Reverse Proxies with Docker interlock registrator

  38. 38 Discover new containers through Docker Events API on container

    events, update NGINX/HAProxy Plugin model write your own event handler Interlock by @ehazlett docker engine interlock event: start nginx update nginx.conf Get Container Details SIGHUP event: stop event: die
  39. 39 Discover new containers through Docker Events API
 Writes service

    definitions to consul/etcd Registrator by @progrium docker engine registrator event: start consul Save Get Containers event: stop event: die
  40. 40 You can then use consul-template/confd update haproxy/nginx backends list

    Registrator by @progrium docker engine registrator event: start consul Save Get Containers event: stop event: die consul-template Watch nginx update nginx.conf
  41. 41 docker bridge template reverse proxy service registry HAProxy NGINX

    Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
  42. 42 docker bridge template reverse proxy service registry HAProxy NGINX

    Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
  43. Mixin’ Tools 43 Far too many moving parts
 How do

    you deploy these components HA?
 You still have N point of failures & additional latency
 Connection draining feature is a lie: …unless orchestrator coordinates with the
 reverse proxy Stopping the container will just
 drop the connections.
  44. Connection draining done right 44 kube-proxy handles load balancing in

    Kubernetes.
 When you stop a pod, it is not stopped right away.
 Remaining open connections stay alive for T.
 (T=grace period, configurable)
 Also pre-start/post-start hooks for containers in pods
 “Zero downtime rolling upgrades in 1M requests/sec”
 
 http://blog.kubernetes.io/2015/11/one-million-requests-per-second- dependable-and-dynamic-distributed-systems-at-scale.html

  45. 45 even more crazy stuff:
 
 Port Scanning
 (nmap)

  46. Port Scanning in Overlay Networks 46 by Jeff Nickoloff (github.com/allingeek/nmap-sd)

    Add connected containers to a network
 (such as Docker 1.9 overlay driver)
 Scan open ports in the network’s subnet periodically
 (as long as your subnet is small, it’s very reasonable)
 Reports accessible ports to a file (bind volume)
 Refresh reverse proxy config, route the traffic!
  47. 47 DNS
 (domain name system)

  48. Motives for DNS 48 Started in 1984, roughly at the

    same time as TCP/IP
 Humans suck at remembering IP addresses
 google.com → 2a00:1450:4003:806::200e and IP addresses do not stick around forever
 Can this 30-year old tech save us?
  49. DNS Service Discovery 49 ServiceA ServiceB addr DNS IPs

  50. DNS Service Discovery 50 ServiceA ServiceB IPs DNS

  51. Intro to DNS Resource Records 51 Type A/AAAA records <hostname>

    → <IP>
 
 $ dig A +short docker.com.
 52.7.79.61
 52.22.96.108
 54.84.192.71 ugly truth: has no port information can’t support dynamic port-assigned containers :(
  52. Intro to DNS Resource Records 52 Type SRV records <hostname>

    → <IP, port, weight>
 
 $ dig SRV +short _database._tcp.local.
 1 1 32770 192.168.0.4
 1 1 32769 192.168.0.7
 1 1 32801 192.168.0.6
 ugly truth: SRV is neither used anywhere,
 nor getting adopted.
 
 new MySQLDriver(“_database._tcp.local”) ain’t happenin'
  53. Bad News 53 SRV is cool but not getting any

    adoption at all.
 We are left with A/AAAA records = IP addresses Works if all your instances are on static ports (such as docker run -p 80:80) When you do dynamic ports (docker run -P),
 you need to resolve the port from SRV rec.
 
 host, port = resolveSRV(“_database._tcp.local”)
 … = new MySQLDriver(host, port) you don’t want to do this all the time
  54. DNS 54 Advantage: very simple, far less moving parts
 Disadvantage:

    goodbye dynamic port allocation
 Advantage: reduces load on middleware (DNS TTL)
 Disadvantage: some languages* do not obey TTLs
 Advantage: uses existing network stack
 Disadvantage: no resilient way to do health checks
 Advantage: load balancing by shuffling IPs :)

  55. 55 Mesos-DNS

  56. Mesos-DNS 56 github.com/mesosphere/mesos-dns Deploy once, forget about it Designed for

    Apache Mesos Queries /state.json periodically to
 discover new tasks mesos-master task sync mesos-dns mesos-master mesos-master DNS query DNS records
  57. Mesos-DNS 57 github.com/mesosphere/mesos-dns Stateless Easy to replicate and make it

    HA
 Provides HTTP REST API For example, write your own SRV router
 without doing any SRV calls
 Many features IPSec, SOA/SRV/A records, DNS over TCP…
  58. 58 SkyDNS

  59. SkyDNS 59 github.com/skynetservices/skydns2 Very similar to Mesos-DNS.
 Closely coupled to

    etcd.
 Really complicated, probably does everything. Kinda hard to set up, too.
 Embraces plugin model, but only plugin is etcd.
 Used by Kubernetes as default DNS add-on.
  60. 60 wagl
 Minimalistic DNS Service Discovery for Docker Swarm

  61. 61 wagl
 Minimalistic DNS Service Discovery for Docker Swarm

  62. Service Discovery in my dreams… 62 it comes with the

    orchestrator
 …or it is a “setup and forget about it”
 does not require code modification in microservice
 uses a reliable networking stack,
 does not have too many moving parts.

  63. wagl 63 github.com/ahmetalpbalkan/wagl Inspired by Mesos-DNS, built for Swarm ♥


    Install and forget about it
 Stateless, easy to replicate
 Speaks Docker language, runs inside container
 Minimalistic feature set, because I’m lazy
  64. wagl architecture 64 swarm wagl events service Get Containers refresh

    periodically DNS query DNS records
  65. wagl Placement 65 swarm manager wagl master0 swarm manager wagl

    master2 swarm manager wagl master1 stuff node0 stuff stuff stuff … … docker run --dns master0 --dns master1 --dns master2 … node1 nodeN
  66. Service Naming 66 Using “Docker Labels” docker run -p 80:80

    \
 -l dns.service=api \
 nginx
 http://api.swarm.
  67. More labels… 67 docker run -p 80:80 \
 -l dns.service=api

    \
 -l dns.service=billing \
 nginx
 http://api.billing.swarm.
  68. Features 68 Only DNS A/SRV records
 Natural Load Balancing by

    shuffling DNS records
 External DNS recursion
 Works well with Docker TLS authentication
  69. Deploying wagl 69 Just run:
 
 docker run -d --restart=always

    --name=dns \
 -p 53:53/udp \
 --link=swarm-master:swarm \
 ahmet/wagl \
 wagl --swarm tcp://swarm:2375
 
 If it can get any easier, it means I have failed.
  70. 70 wagl in Action
 (demo time)

  71. Embrace & Contribute 71 Source code:
 
 https://github.com/AhmetAlpBalkan/wagl

  72. 72 Where are we headed?

  73. Where are we headed? 73 These are just baby steps

    (expect innovation here)
 We need a complete and seamless solution
 The solution will not change the application code
 A combination of DNS + Reverse Proxy can be it
 Watch for what orchestrators are going to adapt
  74. Find more at github.com/AhmetAlpBalkan/wagl

  75. Thank you! Ahmet Alp Balkan @AhmetAlpBalkan