DNS Service Discovery for Docker Swarm Clusters

I gave this talk at DockerCon EU 2015 in Barcelona, Spain on Nov 16, 2015. The talk is about state of the service discovery problem in today's container clusters and a solution I developed for Docker Swarm called “wagl”.

More info about wagl and this talk:
- https://github.com/ahmetalpbalkan/wagl
- https://ahmetalpbalkan.com/blog/wagl/

Transcript

1. DNS Service Discovery for Docker Swarm Ahmet Alp Balkan @AhmetAlpBalkan Software Engineer, Microsoft
2. About The Speaker 2 I am Ahmet Alp Balkan, a software engineer at Microsoft. I contribute to Open Source. Follow me at @AhmetAlpBalkan.
3. About This Talk 3 Service Discovery Service Discovery Methods A peek into various solutions Thought exercises Service Discovery for Docker Swarm Can a drop-in tool just do it™ for Swarm? Where are we headed?
4. Before we begin 4 how many of you… … use Docker Swarm? … used a Service Discovery method? … wrote or configured a DNS server?
5. Service Discovery 5 Cluster
6. Service Discovery 6 Cluster machine1 machine3 machine4 machine2 machine5 machine6 machine7 machine8
7. Service Discovery 7 Cluster machine3 machine4 machine2 machine5 machine7 machine8 db db db machine6 machine1
8. Service Discovery 8 Cluster machine3 machine4 machine2 machine5 machine7 machine8 api api api db db db machine6 machine1
9. Service Discovery 9 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db machine6 machine1
10. Service Discovery 10 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
11. Service Discovery 11 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
12. Service Discovery 12 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
13. Service Discovery 13 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
14. Service Discovery 14 Cluster machine3 machine4 machine2 machine5 machine7 machine8 web web api api api db db db stuff stuff stuff stuff stuff machine6 machine1
15. Service Discovery 15 ServiceA ServiceB help services find and talk to each other
16. Service Discovery 16 ServiceA ServiceB addr help services find and talk to each other
17. Service Discovery 17 ServiceA ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB
18. Service Discovery 18 ServiceA ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB ServiceB
19. Service Discovery Methods 19 before we begin…
20. Service Discovery Methods 20 before we begin… SPOILER: No method is actually really good.

21. Service Discovery Methods 21 before we begin… SPOILER: No method is actually really good.
 This is still an unsolved problem.
 Food for thought, not a comparison.
22. Service Discovery in my dreams… 22 it comes with the orchestrator
 …or it is a “setup and forget about it”
 does not require code modification in microservice
 uses a reliable networking stack,
 does not have too many moving parts.

23. Common Approaches 23 to Service Discovery Overlay Networks
 Mixing Tools (docker bridge + template + rev proxy)
 Port Scanning
 Domain Name Service
24. 24 Overlay networks
25. Overlay Networks 25 node0 ServiceA node1 ServiceB *magic* serviceB:80
26. Overlay Networks 26 Good at container-to-container networking
 Static port allocation not a problem IP address per container
 Seamless, does not change the application code
 Introduces network latency overhead*
27. Docker 1.9 Multi-Host Networking 27 Container-to-container overlay network
 Discovery* through /etc/hosts entries (DNS)
 
 
 
 
 
 
 
 
 Lacking a load balancer (how to http://serviceA?)
28. 28 Docker 1.9 Multi-Host Networking
29. 29 Mixin’ Tools
 (because we can)
30. 30 docker bridge template reverse proxy service registry HAProxy NGINX Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
31. 31 Reverse Proxies
32. Reverse Proxies (TCP/IP Load Balancers) 32 HAProxy, NGINX, Træfik, kube-proxy… Route load balance traffic to multiple backends can route traffic from/to any port list of backends can be dynamically updated
 ServiceA ServiceB Proxy node1:32012 ServiceB node2:33406 ServiceB node7:32104
33. 33 HAProxy, NGINX, Træfik, kube-proxy… Load balancing
 Health checks do not route traffic to unhealthy backends ServiceA ServiceB Proxy node1:32012 ServiceB node2:33406 ServiceB node7:32104 health probes Reverse Proxies (TCP/IP Load Balancers)
34. 34 Sticky sessions route a client’s traffic to the same
 backend between requests/connections
 
 Origin-based access control (ACLs)
 HAProxy, NGINX, Træfik, kube-proxy… ServiceA ServiceB Proxy ServiceC ServiceB Proxy Reverse Proxies (TCP/IP Load Balancers) 172.0.1.10 172.0.1.22
35. 35 Connection draining* wait all connections to close before removing the backend from the routing list
 Allows blue-green deployments flip the switch → the new version of 
 your service starts getting traffic
 HAProxy, NGINX, Træfik, kube-proxy… Reverse Proxies (TCP/IP Load Balancers)
36. 36 Downside: another moving part that can fail what if the proxy server crashes?
 Downside: discovery of the proxy server itself where do you place the proxy server(s)? what happens when they get rescheduled
 to another host? how do you discover proxy servers?
 Downside: introduces latency overhead HAProxy, NGINX, Træfik, kube-proxy… Reverse Proxies (TCP/IP Load Balancers)
37. 37 Bridging Reverse Proxies with Docker interlock registrator
38. 38 Discover new containers through Docker Events API on container events, update NGINX/HAProxy Plugin model write your own event handler Interlock by @ehazlett docker engine interlock event: start nginx update nginx.conf Get Container Details SIGHUP event: stop event: die
39. 39 Discover new containers through Docker Events API
 Writes service definitions to consul/etcd Registrator by @progrium docker engine registrator event: start consul Save Get Containers event: stop event: die
40. 40 You can then use consul-template/confd update haproxy/nginx backends list Registrator by @progrium docker engine registrator event: start consul Save Get Containers event: stop event: die consul-template Watch nginx update nginx.conf
41. 41 docker bridge template reverse proxy service registry HAProxy NGINX Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
42. 42 docker bridge template reverse proxy service registry HAProxy NGINX Træfik interlock confd etcd consul zookeeper interlock registrator … Mixin’ Tools
 (because we can) consul-template
43. Mixin’ Tools 43 Far too many moving parts
 How do you deploy these components HA?
 You still have N point of failures & additional latency
 Connection draining feature is a lie: …unless orchestrator coordinates with the
 reverse proxy Stopping the container will just
 drop the connections.
44. Connection draining done right 44 kube-proxy handles load balancing in Kubernetes.
 When you stop a pod, it is not stopped right away.
 Remaining open connections stay alive for T.
 (T=grace period, configurable)
 Also pre-start/post-start hooks for containers in pods
 “Zero downtime rolling upgrades in 1M requests/sec”
 
 http://blog.kubernetes.io/2015/11/one-million-requests-per-second- dependable-and-dynamic-distributed-systems-at-scale.html

45. 45 even more crazy stuff:
 
 Port Scanning
 (nmap)
46. Port Scanning in Overlay Networks 46 by Jeff Nickoloff (github.com/allingeek/nmap-sd) Add connected containers to a network
 (such as Docker 1.9 overlay driver)
 Scan open ports in the network’s subnet periodically
 (as long as your subnet is small, it’s very reasonable)
 Reports accessible ports to a file (bind volume)
 Refresh reverse proxy config, route the traffic!
47. 47 DNS
 (domain name system)
48. Motives for DNS 48 Started in 1984, roughly at the same time as TCP/IP
 Humans suck at remembering IP addresses
 google.com → 2a00:1450:4003:806::200e and IP addresses do not stick around forever
 Can this 30-year old tech save us?
49. DNS Service Discovery 49 ServiceA ServiceB addr DNS IPs
50. DNS Service Discovery 50 ServiceA ServiceB IPs DNS
51. Intro to DNS Resource Records 51 Type A/AAAA records <hostname> → <IP>
 
 $ dig A +short docker.com.
 52.7.79.61
 52.22.96.108
 54.84.192.71 ugly truth: has no port information can’t support dynamic port-assigned containers :(
52. Intro to DNS Resource Records 52 Type SRV records <hostname> → <IP, port, weight>
 
 $ dig SRV +short _database._tcp.local.
 1 1 32770 192.168.0.4
 1 1 32769 192.168.0.7
 1 1 32801 192.168.0.6
 ugly truth: SRV is neither used anywhere,
 nor getting adopted.
 
 new MySQLDriver(“_database._tcp.local”) ain’t happenin'
53. Bad News 53 SRV is cool but not getting any adoption at all.
 We are left with A/AAAA records = IP addresses Works if all your instances are on static ports (such as docker run -p 80:80) When you do dynamic ports (docker run -P),
 you need to resolve the port from SRV rec.
 
 host, port = resolveSRV(“_database._tcp.local”)
 … = new MySQLDriver(host, port) you don’t want to do this all the time
54. DNS 54 Advantage: very simple, far less moving parts
 Disadvantage: goodbye dynamic port allocation
 Advantage: reduces load on middleware (DNS TTL)
 Disadvantage: some languages* do not obey TTLs
 Advantage: uses existing network stack
 Disadvantage: no resilient way to do health checks
 Advantage: load balancing by shuffling IPs :)

55. 55 Mesos-DNS
56. Mesos-DNS 56 github.com/mesosphere/mesos-dns Deploy once, forget about it Designed for Apache Mesos Queries /state.json periodically to
 discover new tasks mesos-master task sync mesos-dns mesos-master mesos-master DNS query DNS records
57. Mesos-DNS 57 github.com/mesosphere/mesos-dns Stateless Easy to replicate and make it HA
 Provides HTTP REST API For example, write your own SRV router
 without doing any SRV calls
 Many features IPSec, SOA/SRV/A records, DNS over TCP…
58. 58 SkyDNS
59. SkyDNS 59 github.com/skynetservices/skydns2 Very similar to Mesos-DNS.
 Closely coupled to etcd.
 Really complicated, probably does everything. Kinda hard to set up, too.
 Embraces plugin model, but only plugin is etcd.
 Used by Kubernetes as default DNS add-on.
60. 60 wagl
 Minimalistic DNS Service Discovery for Docker Swarm
61. 61 wagl
 Minimalistic DNS Service Discovery for Docker Swarm
62. Service Discovery in my dreams… 62 it comes with the orchestrator
 …or it is a “setup and forget about it”
 does not require code modification in microservice
 uses a reliable networking stack,
 does not have too many moving parts.

63. wagl 63 github.com/ahmetalpbalkan/wagl Inspired by Mesos-DNS, built for Swarm ♥
 Install and forget about it
 Stateless, easy to replicate
 Speaks Docker language, runs inside container
 Minimalistic feature set, because I’m lazy
64. wagl architecture 64 swarm wagl events service Get Containers refresh periodically DNS query DNS records
65. wagl Placement 65 swarm manager wagl master0 swarm manager wagl master2 swarm manager wagl master1 stuff node0 stuff stuff stuff … … docker run --dns master0 --dns master1 --dns master2 … node1 nodeN
66. Service Naming 66 Using “Docker Labels” docker run -p 80:80 \
 -l dns.service=api \
 nginx
 http://api.swarm.
67. More labels… 67 docker run -p 80:80 \
 -l dns.service=api \
 -l dns.service=billing \
 nginx
 http://api.billing.swarm.
68. Features 68 Only DNS A/SRV records
 Natural Load Balancing by shuffling DNS records
 External DNS recursion
 Works well with Docker TLS authentication
69. Deploying wagl 69 Just run:
 
 docker run -d --restart=always --name=dns \
 -p 53:53/udp \
 --link=swarm-master:swarm \
 ahmet/wagl \
 wagl --swarm tcp://swarm:2375
 
 If it can get any easier, it means I have failed.
70. 70 wagl in Action
 (demo time)
71. Embrace & Contribute 71 Source code:
 
 https://github.com/AhmetAlpBalkan/wagl
72. 72 Where are we headed?
73. Where are we headed? 73 These are just baby steps (expect innovation here)
 We need a complete and seamless solution
 The solution will not change the application code
 A combination of DNS + Reverse Proxy can be it
 Watch for what orchestrators are going to adapt
74. Find more at github.com/AhmetAlpBalkan/wagl
75. Thank you! Ahmet Alp Balkan @AhmetAlpBalkan