Raft: The Understandable Distributed Consensus Protocol

Transcript

1. Raft !" U#$"r%&'#$'b(" D)%&r)b*&"$ C+#%"#%*% Pr+&+,+( @benbjohnson

2. W-'& )% D)%&r)b*&"$ C+#%"#%*%?

3. Distributed = Many nodes Consensus = Agreement

4. Distributed = Many nodes Consensus = Agreement

5. Leader Election Data Replication Distributed Locks

6. A Really Short History Of Distributed Consensus Protocols

7. A Really Short History Of Distributed Consensus Protocols Paxos (1989)

8. Paxos In A Nutshell

9. Paxos In A Nutshell Client

10. Paxos In A Nutshell Client Proposer Client requests change to

    system

11. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Proposer

    tells Acceptors to get ready for a change Ready?

12. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Acceptors

    confirm to Proposer that they’re ready Hell yeah!

13. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Proposer

    sends change to Acceptors Here you go

14. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Learner

    Learner Learner Learner Acceptors propagate change to Learners

15. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Learner

    Learner Learner Learner Proposer is now recognized as leader Leader

16. Paxos In A Nutshell Client Proposer Acceptor Acceptor Acceptor Learner

    Learner Learner Learner Repeat for every new change to the system Leader

17. Fun Raft Facts

18. Created By:

19. Diego Ongaro Ph.D. Student Stanford University

20. John Ousterhout Professor of Computer Science Stanford University Diego Ongaro

    Ph.D. Student Stanford University

21. 28 Implementations across various languages

22. In Commercial Use CoreOS (etcd) go-raft

23. Raft Basics

24. Three Roles:

25. The Leader

26. The Follower

27. The Candidate

28. High-Level Example:

29. F F F

30. C F F

31. C F F Vote for me! Vote for me!

32. C F F Ok! Ok!

33. L F F

34. L F F Log entries Log entries

35. L F F Heartbeats Heartbeats

36. X F F

37. X C F

38. X C F Vote for me!

39. X C F Ok!

40. X L F Log Entries & Heartbeats

41. Leader Election

42. F1 F1 F1 Leader Election t(ms) 0 200 400 600

    800 1000 0ms

43. C2 F1 F1 Leader Election t(ms) 0 200 400 600

    800 1000 Request Vote 150ms Request Vote (Fails) One follower becomes a candidate after an election timeout and requests votes

44. C2 F2 F1 Leader Election t(ms) 0 200 400 600

    800 1000 Grant Vote 155ms Candidate receives one vote from a peer and one vote from self

45. L2 F2 F2 Leader Election t(ms) 0 200 400 600

    800 1000 156ms Two votes is a majority so candidate becomes leader

46. Leader Election (Split Vote)

47. F1 F1 F1 Leader Election t(ms) 0 200 400 600

    800 1000 0ms F1

48. C2 F1 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F1 Request Vote Request Vote 150ms Two followers become candidates simultaneously and begin requesting votes

49. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 Vote Granted Vote Granted 155ms Each candidate receives a vote from themselves and from one peer

50. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 Request Vote 156ms Each candidate requests a vote from a peer who has already voted

51. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 Vote Denied 160ms Vote requests are denied because the follower has already voted

52. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 Request Vote 161ms Candidates try to request votes from each other

53. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 Vote Denied 165ms Vote requests are denied because candidates voted for themselves

54. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 200ms Candidates wait for a randomized election timeout to occur (150ms - 300ms)

55. C2 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 250ms Still waiting...

56. C3 F2 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F2 300ms Request Vote Request Vote One candidate begins election term #3

57. L3 F3 C2 Leader Election t(ms) 0 200 400 600

    800 1000 F3 305ms Vote Granted Vote Granted Candidate receives vote from itself and two peer votes so it becomes leader for election term #3

58. L3 F3 C3 Leader Election t(ms) 0 200 400 600

    800 1000 F3 306ms Request Vote Request Vote Second candidate doesn’t know first candidate won the term and begins requesting votes

59. L3 F3 C3 Leader Election t(ms) 0 200 400 600

    800 1000 F3 306ms Vote Denied Vote Denied Peers already voted so votes are denied

60. L3 F3 F3 Leader Election t(ms) 0 200 400 600

    800 1000 F3 310ms Leader notifies peers of election and other candidate steps down

61. Log Replication

62. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 0ms <“”> <“”> <“”>

63. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 10ms 1 “sally” <“”> <“”> <“”> A new uncommitted log entry is added to the leader

64. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 20ms <“”> <“”> <“”> 1 “sally” 1 “sally” 1 “sally” Append Entries Append Entries At the next heartbeat, the log entry is replicated to followers

65. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 OK <“sally”> <“”> <“”> 1 “sally” 1 “sally” 1 “sally” 22ms A majority of nodes have written the log entry written to disk so it becomes committed

66. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“”> <“”> 1 “sally” 1 “sally” 1 “sally” OK 25ms

67. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“sally”> <“sally”> 1 “sally” 1 “sally” 1 “sally” Append Entries Append Entries 40ms At the next heartbeat, the leader notifies followers of updated committed entries

68. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“sally”> <“sally”> 1 “sally” 1 “sally” 1 “sally” 50ms

69. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“sally”> <“sally”> 1 “sally” 1 “sally” 1 “sally” Append Entries Append Entries 60ms At the next heartbeat, no new log information is sent

70. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“sally”> <“sally”> 1 “sally” 1 “sally” 1 “sally” 70ms

71. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 1 “sally” 2 “bob” 1 “sally” 1 “sally” <“sally”> <“sally”> <“sally”> 75ms A new uncommitted log entry is added to the leader

72. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“sally”> <“sally”> <“sally”> 1 “sally” 2 “bob” 1 “sally” 2 “bob” 1 “sally” 2 “bob” Append Entries Append Entries 80ms At the next heartbeat, the entry is replicated to the followers

73. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 OK <“bob”> <“sally”> <“sally”> 1 “sally” 2 “bob” 1 “sally” 1 “sally” OK 82ms The entry is committed once the followers acknowledge the request

74. L1 F1 F1 Log Replication t(ms) 0 200 400 600

    800 1000 <“bob”> <“bob”> <“bob”> 1 “sally” 2 “bob” 1 “sally” 2 “bob” 1 “sally” 2 “bob” Append Entries Append Entries 100ms At the next heartbeat, the leader notifies the followers of the new committed entry

75. Log Replication (with Network Partitions)

76. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 0ms <“”> <“”> F1 <“”> F1 <“”> F1 <“”>

77. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 10ms 1 “sally” <“”> <“”> F1 <“”> F1 <“”> F1 <“”> A new uncommitted log entry is added to the leader

78. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 20ms 1 “sally” 1 “sally” <“”> <“”> F1 1 “sally” <“”> F1 1 “sally” <“”> F1 1 “sally” <“”> Append Entries On the next heartbeat, the entry is replicated to the followers

79. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 25ms 1 “sally” 1 “sally” <“sally”> <“”> F1 1 “sally” <“”> F1 1 “sally” <“”> F1 1 “sally” <“”> OK The followers acknowledge the entry and the entry is committed

80. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 40ms 1 “sally” 1 “sally” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> Append Entries On the next heartbeat, the committed entry is replicated to the followers

81. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 1 “sally” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> 50ms

82. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 1 “sally” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> A network partition makes a majority of nodes inaccessible from the leader 60ms

83. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> A new log entry is added to the leader 70ms

84. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> The leader replicates the entry to the only accessible follower Append Entries 80ms

85. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> The follower acknowledges the entry but there is not a quorum OK 85ms

86. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> 90ms

87. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> C2 1 “sally” <“sally”> F1 1 “sally” <“sally”> F1 1 “sally” <“sally”> After an election timeout, one disconnected follower becomes a candidate Request Vote 190ms

88. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” <“sally”> F2 1 “sally” <“sally”> F2 1 “sally” <“sally”> The candidate receives a majority of votes and becomes a leader Vote Granted 195ms

89. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” <“sally”> F2 1 “sally” <“sally”> F2 1 “sally” <“sally”> 200ms

90. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“sally”> F2 1 “sally” <“sally”> F2 1 “sally” <“sally”> A log entry is added to the new leader 210ms

91. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“sally”> F2 1 “sally” 2 “tom” <“sally”> F2 1 “sally” 2 “tom” <“sally”> The log entry is replicated to the accessible followers Append Entries 220ms

92. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“sally”> F2 1 “sally” 2 “tom” <“sally”> A majority of nodes acknowledge the entry so it becomes committed OK 225ms

93. Append Entries L1 F1 Log Replication t(ms) 0 200 400

    600 800 1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> On the next heartbeat, the followers are notified the entry is committed 240ms

94. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> 250ms

95. L1 F1 Log Replication t(ms) 0 200 400 600 800

    1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> The network recovers and there is no longer a partition 255ms

96. Append Entries L1 F1 Log Replication t(ms) 0 200 400

    600 800 1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> The new leader sends a heartbeat on the next heartbeat timeout 260ms

97. Append Entries F2 F2 Log Replication t(ms) 0 200 400

    600 800 1000 1 “sally” 2 “bob” 1 “sally” 2 “bob” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> The leader of term #1 steps down after seeing a new leader in term #2 260ms

98. Append Entries F2 F2 Log Replication t(ms) 0 200 400

    600 800 1000 1 “sally” 1 “sally” <“sally”> <“sally”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> Uncommitted entries from disconnected nodes are discarded 260ms

99. Append Entries F2 F2 Log Replication t(ms) 0 200 400

    600 800 1000 1 “sally” 2 “tom” 1 “sally” 2 “tom” <“tom”> <“tom”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> New log entries are appended to the previously disconnected nodes 260ms

100. F2 F2 Log Replication t(ms) 0 200 400 600 800

     1000 1 “sally” 2 “tom” 1 “sally” 2 “tom” <“tom”> <“tom”> L2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> F2 1 “sally” 2 “tom” <“tom”> 260ms

101. Log Compaction

102. Unbounded log can grow until there’s no more disk

103. Recovery time increases as log length increases

104. Three Log Compaction Strategies

105. #1: Leader-Initiated, Stored in Log [start] chunk entry chunk entry

     [end] Raft Library

106. #2: Leader-Initiated, Stored Externally entry entry entry Snapshot Raft Library

107. #3: Independently-Initiated, Stored Externally Application entry entry entry Snapshot

108. Questions? Twitter: @benbjohnson GitHub: benbjohnson ben@skylandlabs.com

109. Image Attribution Database designed by Sergey Shmidt from The Noun

     Project Question designed by Greg Pabst from The Noun Project Lock from The Noun Project Floppy Disk designed by Mike Wirth from The Noun Project Movie designed by Anna Weiss from The Noun Project