Yocalhost - Yelp's One Weird Trick for Avoiding Links, Sidekicks, and Overlay Networks with Docker

Yelp uses Docker + Mesos for services, but we don't want to run any fancy overlay network or mess with docker links, or run "sidekicks". This is how we getaway with none of that.

Transcript

1. Yocalhost Yelp's one weird trick for avoiding links, sidekicks, and overlay networks
2. What is the problem we are trying to solve? • We want a solid, flexible service-discovery mechanism without lock-in (smartstack) • We don’t want to add complexity to services running in docker (sidekicks) • We want to co-exist with service that do *not* run in docker (or --net=host)
3. What have we got? • Smartstack • Things on the host already (postfix, sensu, etc) • A function that returns a hash of service=>ports for all services on a box (puppet ones, classic ones, docker ones)
4. Q: What is Smartstack? A: Totally underrated service-discovery tool Legend Server1 Nerve Server2 Synapse HTTP Zookeeper HAProxy Service1:8080 HTTP Request to Localhost:28080 Service2:8081 Nerve HAProxy HTTP Request to Localhost:28081 Synapse File
5. Q: What is Smartstack Consul? Server1 Server2 HAProxy Service1:8080 HTTP Request to Localhost:28080 Service2:8081 HAProxy HTTP Request to Localhost:28081 Consul Consul Consul-template + HAProxy Consul Consul-template + HAProxy
6. Q: What is Smartstack Etcd? Server1 Server2 HAProxy Service1:8080 HTTP Request to Localhost:28080 Service2:8081 HAProxy HTTP Request to Localhost:28081 etcd docker-register / fleet Vulcand docker-register / fleet Vulcand
7. But wait, localhost? • How is that going to work in this new docker world? • Are we going to run a nerve and synapse container for each service? (2 sidekicks?) • Are we going to link everything to a nerve and synapse containers? What if they restart?
8. What is Yocalhost? • Yocalhost is 169.254.255.254 • This is a link-local ip, and it allows services *running on the host* to be reached from *inside* a nat’d docker container • We encourage everyone to use yocalhost all the time, to future-proof yourself • With yocalhost, you only need to run 1 service- discovery daemon per server (important for scaling)
9. Why do we need yocalhost? Server With Docker Localhost IP 127.0.0.1 Yocalhost IP 169.254.255.254 Main IP: 10.0.X.Y Docker Bridge IP: 169.254.1.1 Docker Container 1 Localhost IP 127.0.0.1 Docker IP 169.254.14.17 Docker Container 2 Localhost IP 127.0.0.1 Docker IP 169.254.1.18 Docker Container 3 Localhost IP 127.0.0.1 Docker IP 169.254.1.19 Yocalhost works here, but *also* works on servers *without* docker. Yocalhost is the only IP you can depend on being the same everywhere! If we bind haproxy here, everyone can use it and we only have to run one copy!
10. Server 2 With Docker Localhost IP 127.0.0.1 Yocalhost IP 169.254.255.254 Main IP: 10.0.X.Y Docker Bridge IP: 169.254.1.1 But Kyle, how do the containers talk to each other without an overlay network? Server 1 With Docker Localhost IP 127.0.0.1 Yocalhost IP 169.254.255.254 Main IP: 10.0.X.Y Docker Bridge IP: 169.254.1.1 Docker Container 1 Localhost IP 127.0.0.1 Docker IP 169.254.14.17 Docker Container 2 Localhost IP 127.0.0.1 Docker IP 169.254.1.18 Docker Container 3 Localhost IP 127.0.0.1 Docker IP 169.254.1.19 HA Proxy (via Synapse) Dock Loca Dock
11. Conclusion • HAProxy *is* our overlay “network” • 169.254.255.254 is the only ip you can trust • docker/puppet/mesos/rocket/who-knows, we’ ll be able to announce it and discover it • Nerve is our “sidekick”, one per physical host • Who is going to let your application know when /etc/hosts changed? (docker links)