ZooKeeper. Taming your server farm.

The presentation was given on the local gathering of Ulyanovsk software engineers — ULCAMP::DEV (http://ulcamp.ru/dev) on April 5th, 2013.

Transcript

1. ZOOKEEPER
   Taming your server farm
   Alexander Zhuravlev, aboutecho.com
   ULCAMP::Dev — 2013-04-05
   

   View Slide

2. @zaa
   

   View Slide

3. aboutecho.com
   

   View Slide

4. ECHO PLATFORM
   IN NUMBERS
   

   View Slide

5. ~25 programmers
   ~450 servers
   ~50 database servers
   ~45000 requests per second
   

   View Slide

6. DISTRIBUTED
   SYSTEMS
   

   View Slide

7. “A distributed system is one
   in which the failure of a
   computer you didn't even
   know existed can render your
   own computer unusable.”
   — Laslie Lamport
   

   View Slide

8. FALLACIES OF
   DISTRIBUTED
   COMPUTING
   

   View Slide

9. 1. The network is reliable.
   2. Latency is zero.
   3. Bandwidth is infinite.
   4. The network is secure.
   5. Topology doesn't change.
   6. There is one administrator.
   7. Transport cost is zero.
   8. The network is homogeneous.
   

   View Slide

10. DR. ERIC BREWER’S
    CAP THEOREM
    

    View Slide

11. Consistency
    Availability
    Partition tolerance
    

    View Slide

12. “You can’t sacrifice partition
    tolerance.
    In the event of failures, which
    will this system sacrifice?
    Consistency or availability?”
    — Coda Hale
    

    View Slide

13. BASE
    Basically
    Available
    Soft-state services with
    Eventual-consistency
    

    View Slide

14. ACID
    Atomicity
    Consistency
    Isolation
    Durability
    

    View Slide

15. APACHE
    ZOOKEEPER
    

    View Slide

16. “ZooKeeper is a centralized
    service for maintaining
    configuration information,
    naming, providing distributed
    synchronization, and
    providing group services”
    

    View Slide

17. TIMELINE
    June 2007: First version, Yahoo
    Oct 2007: Sourceforge
    June 2008: Move to Apache
    Nov 2010: Top level project
    

    View Slide

18. WIDELY USED
    Yahoo Web Crawler
    LinkedIn Apache Kafka
    Twitter Storm
    Facebook Haystack
    Cloudera Apache Hadoop
    

    View Slide

19. THROUGHPUT
    

    View Slide

20. LATENCY
    

    View Slide

21. TAO OF ZOOKEEPER
    ZooKeepers keep order
    ZooKeepers are reliable
    ZooKeepers are efficient
    ZooKeepers are timely
    ZooKeepers avoid contention
    ZooKeepers are ambition free
    

    View Slide

22. ZOOKEEPER
    ARCHITECTURE
    

    View Slide

23. METADATA
    FILESYSTEM
    

    View Slide

24. /
    /a
    /a/2
    /a/1
    /b
    

    View Slide

25. ZOOKEEPER
    ATOMIC
    BROADCAST
    PROTOCOL
    

    View Slide

26. 2F+1 NODES?
    

    View Slide

27. SESSIONS
    

    View Slide

28. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    SESSIONS
    

    View Slide

29. SIMPLE API
    

    View Slide

30. API
    create
    delete
    exists
    getData
    setData
    getChildren
    

    View Slide

31. WATCHES
    

    View Slide

32. WATCHES
    getData
    getChildren
    exists
    

    View Slide

33. READ PATH
    

    View Slide

34. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    X=9
    X=9
    X=9
    getData “/X”
    return 9
    READ
    

    View Slide

35. WRITE PATH
    

    View Slide

36. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    X=10
    X=10
    X=10
    setData “/X”, 10
    ACK
    WRITE
    

    View Slide

37. WATCH PATH
    

    View Slide

38. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    X=9
    X=9
    X=9
    getData “/X”, true
    SET WATCH
    return 9
    

    View Slide

39. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    X=10
    X=10
    X=10
    setData “/X”, 10
    ACK
    WRITE HAPPENS
    

    View Slide

40. FOLLOWER
    FOLLOWER
    LEADER
    ENSEMBLE
    CLIENT
    CLIENT
    CLIENT
    X=10
    X=10
    X=10
    WATCH NOTIFICATION
    NOTIFICATION
    

    View Slide

41. DEMO TIME
    

    View Slide

42. USE CASES
    

    View Slide

43. USE CASE
    LOCK
    

    View Slide

44. USE CASE
    LEADER
    ELECTION
    

    View Slide

45. USE CASE
    GROUP
    MEMBERSHIP
    

    View Slide

46. USE CASE
    CONFIGURATION
    MANAGEMENT
    

    View Slide

47. HOW DOES ECHO
    USE ZOOKEEPER?
    

    View Slide

48. /
    /postgresql
    /postgresql/client1 /postgresql/client2 /postgresql/clientN
    ...
    

    View Slide

49. SHAMELESS PLUG
    If you are interested in learning
    about topics like this, get in touch
    with us, we are looking for interns
    jobs@aboutecho.com
    

    View Slide

50. THANKS
    

    View Slide

51. QUESTIONS?
    Alexander Zhuravlev
    http://twitter.com/zaa
    zaa@aboutecho.com
    

    View Slide

52. ATTRIBUTION
    Photos from "The Field Museum Library"
    http://www.flickr.com/photos/field_museum_library/3405476048/
    http://www.flickr.com/photos/field_museum_library/3405475952/
    http://www.flickr.com/photos/field_museum_library/4586895529/
    http://www.flickr.com/photos/field_museum_library/3404663989/
    http://www.flickr.com/photos/field_museum_library/4986435461/
    http://www.flickr.com/photos/field_museum_library/3795473195/
    http://www.flickr.com/photos/field_museum_library/4986457001/
    Photos from "Powerhouse Museum Collection"
    http://www.flickr.com/photos/powerhouse_museum/2759437054/
    Throughput and latency graphs from Flavio Junqueira's presentation
    "Distributed Coordination via ZooKeeper":
    https://cwiki.apache.org/confluence/download/attachments/24193445/keynote-hic-2011-web.pdf
    

    View Slide