Use ZooKeeper for discovery at converge time

DevOpsDays Austin

1. ZooKeeper and you or don’t solve your discovery problems with search Tuesday, April 30, 13
2. ZooKeeper “ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.” [1] Tuesday, April 30, 13
3. Chef “The Chef Server acts as a hub for configuration data. The Chef Server stores cookbooks, the policies that are applied to cookbooks, and metadata that describes each registered node in the infrastructure.”[2] Tuesday, April 30, 13
4. Why ZooKeeper? ZooKeeper is highly available* ZK is distributed ZK uses familiar semantics (k-v) or (znode-data) Solves discovery elegantly. If you use SOA, you have discovery problems *unless there is a network partition Tuesday, April 30, 13
5. Writing to ZK Each znode is versioned ZooKeeper supports MVCC Multiversion concurrency control a way of enforcing consistency ensures multiple writers don’t clobber each other Tuesday, April 30, 13
6. Writes to ZK are committed to a majority (aka quorum) of nodes before success is communicated some nodes may have old data Reads happen from any node Writes are forwarded through master As ensemble size grows, read performance increases while write performance decreases. ZooKeeper can only work if a majority of servers are correct (i.e., with 2f + 1 server we can tolerate f failures). [3] Tuesday, April 30, 13
7. vs. chef 10 data store is backed by solr impedance mismatch Not HA recipes that depend on search cannot (easily) be run by chef-solo see edelight[4] Tuesday, April 30, 13
8. Replacing Chef Server Hidden agenda chef server <=10 can fail at any time. Depending on chef-server at converge-time is a SPOF Tuesday, April 30, 13
9. Back to reality We can’t replace chef server in one fell swoop One step at a time replace search with ZooKeeper Tuesday, April 30, 13
10. cluster service discovery[5] def  all_providers_for_service  service    search(:node,  "provides_service:#{service}").        find_all{|server|  server[:provides_service][service]  &&  server[:provides_service][service        sort_by{|server|  server[:provides_service][service]['timestamp']  }  rescue  [] end def  provide_service  service,  info={}    node.set[:provides_service][service]  =  info    node.save end Tuesday, April 30, 13
11. Sequence Nodes When creating a sequence znode, ZK appends a monotonically increasing counter to the end of path. e.g. 2 calls to create sequence znodes at /lock- will result in /lock-0 /lock-1 sequences are unique to the parent Tuesday, April 30, 13
12. ZooKeeper discovery at converge time zk.set( ‘/services/myservice/’, node[:cloud][:public_hostname]) zk.children(‘/services/myservice/’) Useful for statically configured clusters. Databases, caches. Tuesday, April 30, 13
13. Ephemeral Nodes Only exist as long as the creator maintains a connection to ZK Disappear after a timeout Querying them will effectively tell you what other clients are alive. Tuesday, April 30, 13
14. ZooKeeper discovery at run-time hard-coding your cluster membership can be problematic solution is to have services register themselves with zookeeper when they come up membership always correct Tuesday, April 30, 13
15. ZooKeeper Cookbook[6] Uses Netflix’s Exhibitor[7] to provide ops support. http endpoint for ensemble discovery only one name needed to discover every service abstracts services from their location Tuesday, April 30, 13
16. ZK Cookbook One method to get zk connection string Helpers for interacting with ZK. Useful for priming chroots Tuesday, April 30, 13
17. Cookbook example servers = zk_connect_str( discover_zookeepers( node[:exhibitor][:hostname])) Tuesday, April 30, 13
18. Facebook is adopting private chef http://www.youtube.com/watch? v=SYZ2GzYAw_Q key point: stateless chef not the only ones... Tuesday, April 30, 13
19. References 1.http://zookeeper.apache.org/ 2.http://docs.opscode.com/chef_overview_server.html 3.https://github.com/infochimps/cluster_chef/tree/master/site- cookbooks/cluster_service_discovery 4.https://github.com/edelight/chef-solo-search 5.http://static.usenix.org/event/usenix10/tech/full_papers/ Hunt.pdf 6.https://github.com/SimpleFinance/chef-zookeeper 7.https://github.com/Netflix/exhibitor Tuesday, April 30, 13
20. Thank you! Matthew Hooker mwhooker@gmail.com twitter @mwhooker github https://github.com/mwhooker https://speakerdeck.com/mwhooker/chef-zookeeper https://github.com/SimpleFinance/chef-zookeeper Tuesday, April 30, 13