Pro Yearly is on sale from $80 to $50! »

Raft: Consensus for Rubyists

Raft: Consensus for Rubyists

B6a8f005f39d23ffc930508ac9da68b9?s=128

Patrick Van Stee

November 09, 2013
Tweet

Transcript

  1. Raft Consensus for Rubyists

  2. @vanstee Big Nerd Ranch

  3. Coffee Snob Scenario

  4. Matt Amy Kim Dan

  5. Matt Amy Kim Dan

  6. Matt Amy Kim Dan Want to meet for coffee?

  7. Matt Amy Kim Dan Totally! I’m in. zzzzz…

  8. Matt Amy Kim Dan Ok, let’s meet at Octane.

  9. Matt Amy Kim Dan Sounds good. I’ll be there. …

  10. Matt Amy Kim Dan Who wants coffee? *wakes up

  11. Matt Amy Kim Dan Who wants coffee?

  12. Matt Amy Kim Dan Oh, we’re going to Octane.

  13. Delicious Coffee Consensus

  14. What did we just do?

  15. • Leader election • State replication • Partition tolerance

  16. Now write an algorithm.

  17. Humans > Computers > Congress

  18. Why Ruby

  19. Multiple db servers? Multiple app servers? Multiple clients?

  20. Multiple db servers? Distributed Multiple app servers? Multiple clients?

  21. Multiple db servers? Distributed Multiple app servers? Distributed Multiple clients?

  22. Multiple db servers? Distributed Multiple app servers? Distributed Multiple clients?

    Distributed
  23. If you build webapps, you build distributed systems.

  24. Raft

  25. @ongardie Diego Ongaro John Ousterhout and

  26. con·sen·sus /kənˈsensəs/ ! Agreeing upon state across distributed processes even

    in the presence of failures.
  27. When should I use it?

  28. • Distributed locks • Configuration • Background jobs

  29. Why not Paxos?

  30. Laying the Groundwork

  31. {…} Log State Machine Consensus Module

  32. Leader Follower Follower Follower

  33. Leader Follower Follower Follower

  34. Leader Follower Follower Follower

  35. Follower

  36. Leader Election Log Replication Safety

  37. Follower Candidate Leader

  38. Follower Candidate Leader

  39. Follower Candidate Leader Times out, Starts election

  40. Follower Candidate Leader Times out, Starts election Wins election

  41. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election
  42. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election Discovers current or new leader, Steps down
  43. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election Discovers new leader, Steps down Discovers current or new leader, Steps down
  44. RequestVote Used by candidates to ask for votes during an

    election. Log information included for comparison. ! AppendEntries Used by leaders to tell followers which entries to replicate and commit. Also used as a heartbeat to remain leader. Follower Candidate Vote for me please. Follower Here are some entries for your log. Oh, and btw I’m still the leader. Leader
  45. Term Higher numbers are used to determine leaders and check

    log entries. The term is incremented each time an election is started. Any command with an old term is ignored. Term 1 Term 2 Term 3 Election Normal Operation Split vote
  46. Follower Follower Follower Follower

  47. Candidate Follower Follower Follower Votes Times out, starts election

  48. Candidate Follower Follower Follower Votes Votes for self

  49. Candidate Follower Follower Follower Votes Sends out RequestVotes

  50. Candidate Follower Follower Votes Responds with success

  51. Leader Follower Votes Yay! I won! Follower

  52. Leader Election Log Replication Safety

  53. Log Entries Explained

  54. SET X = 1 SET Y = 2 SET Z

    = 3 SET X = 4 SET X = 5 1 1 1 2 2 SET X = 1 SET Y = 2 SET Z = 3 SET X = 4 1 1 1 2 Leader Follower 1 2 3 4 5 6
  55. { :entries => [{ 4 => 'SET X = 4'

    }], :term => 1, :prev_log_term => 1, :prev_log_index => 3, :leader_commit => 3, :leader_id => '192.168.1.101/7238' } APPEND ENTRIES EXAMPLE
  56. SET X = 1 SET Y = 2 SET Z

    = 3 1 1 1 Before 1 1 1 1 After SET X = 1 SET Y = 2 SET Z = 3 SET X = 4 1 2 3 4 5 6 1 2 3 4 5 6
  57. Log entries are always committed in the same order and

    are never uncommitted.
  58. Happy Log Entry

  59. Leader Follower Follower

  60. Leader Follower Follower

  61. Leader Follower Follower

  62. Leader Follower Follower

  63. Leader Follower Follower

  64. Sad Log Entry

  65. Leader Follower Follower

  66. Leader Follower Follower

  67. Leader Follower Follower

  68. Leader Follower Follower

  69. Leader Follower Leader

  70. Leader Follower Leader

  71. Leader Follower Leader

  72. Leader Follower Leader

  73. Leader Follower Leader

  74. Follower Follower Follower Leader

  75. Facepalm Log Entry

  76. Leader Follower Follower

  77. Leader Follower Follower

  78. Leader Follower Follower

  79. Leader Follower Follower

  80. Leader Follower Follower

  81. Leader Follower Follower

  82. Leader Follower Leader

  83. How do we guard against losing log entries?

  84. Leader Election Log Replication Safety

  85. Only cast votes for nodes with logs that contain at

    least as many entries as your own. 1
  86. New leaders must commit a log entry form their new

    term before committing old entries. 2
  87. Bonus Round

  88. • Cluster changes • Log compaction • Client specifics

  89. Why Ruby

  90. None
  91. Ruby is great for expressing complex problems.

  92. Then why doesn’t the academic community love it?

  93. • Community • Understandability • Learning Material

  94. • Read Papers • Go To Conferences • Talk To

    People
  95. Raft Paper Raft Implementations Raft Website ThinkDistributed Ricon Google Scholar

  96. Question Time Make sure I repeat your questions.