PyCon Ireland 2013: Taming Pythons with ZooKeeper

(via http://python.ie/pycon/2013/talks/taming_pythons_with_zookeeper/)
Speaker: Jyrki Pulliainen (http://twitter.com/nailor)

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient.

Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to ues and how?

This talk will walk you through ZooKeeper and how to use it Python. We'll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo.

You'll see how to do things in ZooKeeper and how to implement them using Kazoo. We'll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we've used Kazoo and ZooKeeper to build at Spotify.

Transcript

1. October 7, 2013 Taming Pythons with ZooKeeper
2. $ whoami
3. Jyrki Pulliainen @nailor jyrki@spotify.com
4. *
5. ZooKeeper?
6. ZooKeeper!
7. * The Tao of ZooKeeper
8. *
9. October 7, 2013 One library for the enterprise Curator, from Netflix
10. October 7, 2013 Seven for the pythonistas gevent-zookeeper zkpython zc.zk pykeeper twitter’s zookeeper library zoop txzookeeper
11. One Library to rule them all* * unless you are running twisted
12. Kazoo
13. * CRUD ZooKeeper CRUD Let’s talk about KaZoo
14. First we need to connect! Text Text zk = KazooClient(hosts='127.0.0.1:2181/namespace,127.0.0.2:2181') Text zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop()
15. Create zk.create('/pyconie', b'2013') zk.create('/pyconie/jyrki', ephemeral=True) zk.create('/pyconie/sequential', sequence=True)
16. Read Text zk.get('/pyconie') zk.exists('/pyconie/jyrki') zk.get_children('/pyconie')
17. Update & Delete zk.set_data('/pyconie/jyrki', b'nervous') zk.delete('/pyconie/jyrki')
18. *
19. Distributed locks? Barrier? Semaphores? Counters? Elections?
20. Text from kazoo.recipe import <your-favorite-ting>
21. Want to know when things change?
22. watchers
23. Text zk.exists('/pyconie/guinness', watch=callback) zk.get('/pyconie/redbreast', watch=callback) zk.get_children('/pyconie/dinners', watch=callback)
24. from kazoo.recipe.watchers import DataWatch, ChildWatch @DataWatch('/path/to/node') def data_callback(data, stat): # ... do_something @ChildWatch('/path/to/node') def child_callback(children): # ... do_something
25. TESTS
26. What if something goes WRONG?
27. * sys.exit()
28. Stand back! It’s time for real life example
30. * SEARCHED FOR THE NEWEST JUSTIN BIEBER Encryption keys were not available
31. SUMMARY
32. Guess what, we’re hiring
33. Thank You! jyrki@spotify.com @nailor