Taming Pythons with ZooKeeper (BCNDEVCON edition)

Taming Pythons with ZooKeeper

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient.

In the past we used to deploy new indices using sequential scripts, which would move files around in network and then tell services to upgrade their indices via complex command line calls over SSH. As number of services and thus indices and machines grew, this approach came more and more fragile.

To address the fragility, we decided to start providing index distribution as a service. No longer we would need to deal with individual services but aim only to offer robust and trustworthy way of orchestrating the publishing of new indices. Due to the nature of indices, where things need to happen in a certain order, we needed to be able to have consistency over the services inside a datacenter. For this problem, we decided to take the highly consistent ZooKeeper to help us.

With the help of ZooKeeper we built a self-orchestrating fault resilient, distributed system that we now can offer as a service for other teams needing to distribute indices.

The content of the talk will be in three parts: - Introducing our system and problem domain of index distribution - How we used to deal with this in the past - Why did we pick ZooKeeper - How do we built our system on top of primitives provided by ZooKeeper using Python.

This talk is more ZooKeeper and system focused than the equivalent in PyCon Finland.

Transcript

1. Taming Pythons with ZooKeeper Barcelona Developer Conference 2012
3. 20 million users
4. Billion playlists
5. 18 million songs
6. @nailor
7. Content Engineer
8. Day to day work
9. <xml />
11. Photo by hyper7pro (cc-by)
14. Shipping
16. Pre-defined order
17. The naïve way
20. Guess what? It breaks
21. ...and fixing isn't simple
22. CAP Theorem
23. Pick two. Any two will do.
24. We went for CP
25. How?
28. Apache ZooKeeper
29. Tree-like structure / /dir/ / /subdir/ /dir2/
30. Sequentially consistent
31. Simple primitives
32. Dos and Don'ts
33. Don't ship large chunks
34. Monitor the ZooKeeper
35. Don't write there all the time
36. Stay in one DC
37. Spotify & ZooKeeper
38. Photo by deltaMike
40. For me? Deploy! Meh. No :( Downloaded? No Download Yes! Queue Yes No First? Publish! Yes Dequeue
42. Self-organizing updates
43. Good API
44. Agnostic
45. ...but most importantly...
46. It was fun to build \o/
47. Questions?
48. Thank you @nailor jyrki@spotify.com