Taming Pythons with ZooKeeper (EP2013)

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient.

Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to use and how?

This talk will walk you through ZooKeeper and how to use it with Python. We’ll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo.

You’ll see how to do things in ZooKeeper and how to implement them using Kazoo. We’ll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we’ve used Kazoo and ZooKeeper to build at Spotify.

Transcript

1. June 29, 2013 Taming Pythons with ZooKeeper Wednesday, 3 July 13
2. $ whoami Wednesday, 3 July 13
3. Jyrki Pulliainen @nailor jyrki@spotify.com Wednesday, 3 July 13
4. Wednesday, 3 July 13
5. ZooKeeper? Wednesday, 3 July 13 - This talk is about ZooKeeper. - Preaching Java software in Python conference. I can think of more healthier things to do too. - On side note, our Product Owner is really damn good at this game
6. ZooKeeper! Wednesday, 3 July 13 - Apache project, Yahoo 2007 - Consistency & Partition tolerance - Filesystem like, actually can be viewed as a trie, store data in directories too - In memory, limits the dataset you can store. Maximum zookeeper node, znode, data size 1M
7. The Tao of ZooKeeper 7 Wednesday, 3 July 13 Orderly, Reliable, Efficient, Timely, Contention free, ambition free
8. Wednesday, 3 July 13
9. June 29, 2013 One library for the enterprise Curator, from Netflix Wednesday, 3 July 13
10. June 29, 2013 Seven for the pythonistas gevent-zookeeper zkpython zc.zk pykeeper twitter’s zookeeper library zoop txzookeeper Wednesday, 3 July 13 - gevent-zookeeper -> Spotify - zkpython segfaults - Others have somewhat OK implementations, but lack core features
11. One Library to rule them all* * unless you are running twisted Wednesday, 3 July 13 - txzookeeper still valid for twisted
12. Kazoo Wednesday, 3 July 13 - Origins from the Nimbus project - Ben Bangert as the Sauron of ZooKeeper - Not Frodo, that bastard wanted to destroy the perfectly good ring. - All Python, including the protocol. No more segfaults!
13. CRUD ZooKeeper CRUD Let’s talk about KaZoo 13 Wednesday, 3 July 13 Create, Read, Update, Delete All basic operations available as async too, but we’ll focus on the synchronous use
14. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() Text Text Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
15. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() Text Text zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
16. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() Text zk = KazooClient(hosts='127.0.0.1:2181/namespace,127.0.0.2:2181') Text zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
17. Create zk.create('/europython', b'2013') Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
18. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
19. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) zk.create('/europython/sequential', sequence=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
20. Read Text zk.get('/europython') Wednesday, 3 July 13
21. Read Text zk.get('/europython') zk.exists('/europython/jyrki') Wednesday, 3 July 13
22. Read Text zk.get('/europython') zk.exists('/europython/jyrki') zk.get_children('/europython') Wednesday, 3 July 13
23. Update & Delete zk.set_data('/europython/jyrki', b'nervous') Wednesday, 3 July 13
24. Update & Delete zk.set_data('/europython/jyrki', b'nervous') zk.delete('/europython/jyrki') Wednesday, 3 July 13
25. Wednesday, 3 July 13
26. Distributed locks? Barrier? Semaphores? Counters? Elections? Wednesday, 3 July 13
27. Text from kazoo.recipe import <your-favourite-thing> Wednesday, 3 July 13 Of course it does not have everything from curator
28. Want to know when things change? Wednesday, 3 July 13
29. watchers Wednesday, 3 July 13
30. Text zk.exists('/europython/wine', watch=callback) zk.get('/europython/wine', watch=callback) zk.get_children('/europython/dinners', watch=callback) Wednesday, 3 July 13
31. from kazoo.recipe.watchers import DataWatch, ChildWatch @DataWatch('/path/to/node') def data_callback(data, stat): # ... do_something @ChildWatch('/path/to/node') def child_callback(children): # ... do_something Wednesday, 3 July 13
32. TESTS Wednesday, 3 July 13
33. What if something goes WRONG? Wednesday, 3 July 13
34. sys.exit() Wednesday, 3 July 13
35. Stand back! It’s time for real life example Wednesday, 3 July 13
36. Wednesday, 3 July 13
37. SEARCHED FOR THE NEWEST JUSTIN BIEBER Encryption keys were not available to play it. Wednesday, 3 July 13
38. SUMMARY Wednesday, 3 July 13
39. Guess what, we’re hiring Wednesday, 3 July 13
40. Thank You! jyrki@spotify.com @nailor Wednesday, 3 July 13