Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Use ZooKeeper for discovery at converge time

Use ZooKeeper for discovery at converge time

DevOpsDays Austin


Matthew Hooker

April 30, 2013


  1. ZooKeeper and you or don’t solve your discovery problems with

    search Tuesday, April 30, 13
  2. ZooKeeper “ZooKeeper is a centralized service for maintaining configuration information,

    naming, providing distributed synchronization, and providing group services.” [1] Tuesday, April 30, 13
  3. Chef “The Chef Server acts as a hub for configuration

    data. The Chef Server stores cookbooks, the policies that are applied to cookbooks, and metadata that describes each registered node in the infrastructure.”[2] Tuesday, April 30, 13
  4. Why ZooKeeper? ZooKeeper is highly available* ZK is distributed ZK

    uses familiar semantics (k-v) or (znode-data) Solves discovery elegantly. If you use SOA, you have discovery problems *unless there is a network partition Tuesday, April 30, 13
  5. Writing to ZK Each znode is versioned ZooKeeper supports MVCC

    Multiversion concurrency control a way of enforcing consistency ensures multiple writers don’t clobber each other Tuesday, April 30, 13
  6. Writes to ZK are committed to a majority (aka quorum)

    of nodes before success is communicated some nodes may have old data Reads happen from any node Writes are forwarded through master As ensemble size grows, read performance increases while write performance decreases. ZooKeeper can only work if a majority of servers are correct (i.e., with 2f + 1 server we can tolerate f failures). [3] Tuesday, April 30, 13
  7. vs. chef 10 data store is backed by solr impedance

    mismatch Not HA recipes that depend on search cannot (easily) be run by chef-solo see edelight[4] Tuesday, April 30, 13
  8. Replacing Chef Server Hidden agenda chef server <=10 can fail

    at any time. Depending on chef-server at converge-time is a SPOF Tuesday, April 30, 13
  9. Back to reality We can’t replace chef server in one

    fell swoop One step at a time replace search with ZooKeeper Tuesday, April 30, 13
  10. cluster service discovery[5] def  all_providers_for_service  service    search(:node,  "provides_service:#{service}").  

         find_all{|server|  server[:provides_service][service]  &&  server[:provides_service][service        sort_by{|server|  server[:provides_service][service]['timestamp']  }  rescue  [] end def  provide_service  service,  info={}    node.set[:provides_service][service]  =  info    node.save end Tuesday, April 30, 13
  11. Sequence Nodes When creating a sequence znode, ZK appends a

    monotonically increasing counter to the end of path. e.g. 2 calls to create sequence znodes at /lock- will result in /lock-0 /lock-1 sequences are unique to the parent Tuesday, April 30, 13
  12. ZooKeeper discovery at converge time zk.set( ‘/services/myservice/’, node[:cloud][:public_hostname]) zk.children(‘/services/myservice/’) Useful

    for statically configured clusters. Databases, caches. Tuesday, April 30, 13
  13. Ephemeral Nodes Only exist as long as the creator maintains

    a connection to ZK Disappear after a timeout Querying them will effectively tell you what other clients are alive. Tuesday, April 30, 13
  14. ZooKeeper discovery at run-time hard-coding your cluster membership can be

    problematic solution is to have services register themselves with zookeeper when they come up membership always correct Tuesday, April 30, 13
  15. ZooKeeper Cookbook[6] Uses Netflix’s Exhibitor[7] to provide ops support. http

    endpoint for ensemble discovery only one name needed to discover every service abstracts services from their location Tuesday, April 30, 13
  16. ZK Cookbook One method to get zk connection string Helpers

    for interacting with ZK. Useful for priming chroots Tuesday, April 30, 13
  17. Cookbook example servers = zk_connect_str( discover_zookeepers( node[:exhibitor][:hostname])) Tuesday, April 30,

  18. Facebook is adopting private chef http://www.youtube.com/watch? v=SYZ2GzYAw_Q key point: stateless

    chef not the only ones... Tuesday, April 30, 13
  19. References 1.http://zookeeper.apache.org/ 2.http://docs.opscode.com/chef_overview_server.html 3.https://github.com/infochimps/cluster_chef/tree/master/site- cookbooks/cluster_service_discovery 4.https://github.com/edelight/chef-solo-search 5.http://static.usenix.org/event/usenix10/tech/full_papers/ Hunt.pdf 6.https://github.com/SimpleFinance/chef-zookeeper 7.https://github.com/Netflix/exhibitor

    Tuesday, April 30, 13
  20. Thank you! Matthew Hooker mwhooker@gmail.com twitter @mwhooker github https://github.com/mwhooker https://speakerdeck.com/mwhooker/chef-zookeeper

    https://github.com/SimpleFinance/chef-zookeeper Tuesday, April 30, 13