Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Yocalhost - Yelp's One Weird Trick for Avoiding Links, Sidekicks, and Overlay Networks with Docker

Yocalhost - Yelp's One Weird Trick for Avoiding Links, Sidekicks, and Overlay Networks with Docker

Yelp uses Docker + Mesos for services, but we don't want to run any fancy overlay network or mess with docker links, or run "sidekicks". This is how we getaway with none of that.

Kyle Anderson

June 25, 2015
Tweet

More Decks by Kyle Anderson

Other Decks in Technology

Transcript

  1. Yocalhost
    Yelp's one weird trick for
    avoiding links, sidekicks, and overlay networks

    View full-size slide

  2. What is the problem we are trying to solve?
    ● We want a solid, flexible service-discovery
    mechanism without lock-in (smartstack)
    ● We don’t want to add complexity to services
    running in docker (sidekicks)
    ● We want to co-exist with service that do
    *not* run in docker (or --net=host)

    View full-size slide

  3. What have we got?
    ● Smartstack
    ● Things on the host already (postfix, sensu,
    etc)
    ● A function that returns a hash of
    service=>ports for all services on a box
    (puppet ones, classic ones, docker ones)

    View full-size slide

  4. Q: What is Smartstack?
    A: Totally underrated service-discovery tool
    Legend
    Server1
    Nerve
    Server2
    Synapse
    HTTP
    Zookeeper
    HAProxy
    Service1:8080
    HTTP Request
    to Localhost:28080
    Service2:8081
    Nerve
    HAProxy
    HTTP Request
    to Localhost:28081
    Synapse
    File

    View full-size slide

  5. Q: What is Smartstack Consul?
    Server1 Server2
    HAProxy
    Service1:8080
    HTTP Request
    to Localhost:28080
    Service2:8081
    HAProxy
    HTTP Request
    to Localhost:28081
    Consul
    Consul
    Consul-template
    +
    HAProxy
    Consul
    Consul-template
    +
    HAProxy

    View full-size slide

  6. Q: What is Smartstack Etcd?
    Server1 Server2
    HAProxy
    Service1:8080
    HTTP Request
    to Localhost:28080
    Service2:8081
    HAProxy
    HTTP Request
    to Localhost:28081
    etcd
    docker-register /
    fleet
    Vulcand
    docker-register /
    fleet
    Vulcand

    View full-size slide

  7. But wait, localhost?
    ● How is that going to work in this new docker
    world?
    ● Are we going to run a nerve and synapse
    container for each service? (2 sidekicks?)
    ● Are we going to link everything to a nerve
    and synapse containers? What if they
    restart?

    View full-size slide

  8. What is Yocalhost?
    ● Yocalhost is 169.254.255.254
    ● This is a link-local ip, and it allows services
    *running on the host* to be reached from
    *inside* a nat’d docker container
    ● We encourage everyone to use yocalhost all the
    time, to future-proof yourself
    ● With yocalhost, you only need to run 1 service-
    discovery daemon per server (important for
    scaling)

    View full-size slide

  9. Why do we need yocalhost?
    Server With Docker
    Localhost IP 127.0.0.1
    Yocalhost IP 169.254.255.254
    Main IP: 10.0.X.Y
    Docker Bridge IP: 169.254.1.1
    Docker Container 1
    Localhost IP 127.0.0.1
    Docker IP 169.254.14.17
    Docker Container 2
    Localhost IP 127.0.0.1
    Docker IP 169.254.1.18
    Docker Container 3
    Localhost IP 127.0.0.1
    Docker IP 169.254.1.19
    Yocalhost works here, but *also* works on servers *without* docker.
    Yocalhost is the only IP you can depend on being the same everywhere!
    If we bind haproxy here,
    everyone can use it and
    we only have to run one
    copy!

    View full-size slide

  10. Server 2 With Docker
    Localhost IP 127.0.0.1
    Yocalhost IP 169.254.255.254
    Main IP: 10.0.X.Y
    Docker Bridge IP: 169.254.1.1
    But Kyle, how do the containers talk
    to each other without an overlay
    network?
    Server 1 With Docker
    Localhost IP 127.0.0.1
    Yocalhost IP 169.254.255.254
    Main IP: 10.0.X.Y
    Docker Bridge IP: 169.254.1.1
    Docker Container 1
    Localhost IP 127.0.0.1
    Docker IP 169.254.14.17
    Docker Container 2
    Localhost IP 127.0.0.1
    Docker IP 169.254.1.18
    Docker Container 3
    Localhost IP 127.0.0.1
    Docker IP 169.254.1.19
    HA Proxy
    (via Synapse)
    Dock
    Loca
    Dock

    View full-size slide

  11. Conclusion
    ● HAProxy *is* our overlay “network”
    ● 169.254.255.254 is the only ip you can trust
    ● docker/puppet/mesos/rocket/who-knows, we’
    ll be able to announce it and discover it
    ● Nerve is our “sidekick”, one per physical host
    ● Who is going to let your application know
    when /etc/hosts changed? (docker links)

    View full-size slide