etcd over gossip

Strongly consistent semantics in a weakly consistent world.

Transcript

1. weaveworks-
   etcd over gossip
   Strongly-consistent semantics

   in a weakly-consistent world
   

   View Slide

2. weaveworks-
   ~or~

   Embedding etcd in your application
   

   View Slide

3. weaveworks-
   ~or~

   Porting etcd to a different transport
   

   View Slide

4. weaveworks-
   weavemesh
   

   View Slide

5. weaveworks-
   Gossip communication protocol
   CRDT data model
   

   View Slide

6. weaveworks-
   Gossip communication protocol
   CRDT data model
   Your application
   

   View Slide

7. weaveworks-
   Gossip communication protocol
   CRDT data model
   weavenet
   Virtual network interface + IP
   

   View Slide

8. weaveworks-
   Gossip communication protocol
   CRDT data model
   Your application
   

   View Slide

9. weaveworks-
   Your application
   Router
   :6379
   Gossip Receiver
   Gossip Sender
   Gossip Data
   

   View Slide

10. weaveworks-
    Your application
    Router
    :6379
    Gossip Receiver
    Gossip Sender
    Gossip Data
    

    View Slide

11. weaveworks-
    Router
    Router
    Router
    Router
    Router
    

    View Slide

12. weaveworks-
    Router
    Router
    Router
    Router
    Router
    

    View Slide

13. weaveworks-
    Your application
    Router
    :6379
    Gossip Receiver
    Gossip Sender
    Gossip Data
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    OnGossip(src Peer, m Msg)
    Peers() []Peer
    

    View Slide

14. weaveworks-
    Gossip communication protocol
    Your application
    

    View Slide

15. weaveworks-
    Gossip communication protocol
    etcd
    

    View Slide

16. weaveworks-
    Raft
    

    View Slide

17. weaveworks-
    

    View Slide

18. weaveworks-
    Raft node Other nodes
    Write(X, 123)
    Proposal accepted
    X=123
    Sounds good
    Read(X)
    123
    
    

    View Slide

19. weaveworks-
    Send(dst Node, m Msg)
    

    View Slide

20. weaveworks-
    Send(dst Node, m Msg)
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    

    View Slide

21. weaveworks-
    etcd
    

    View Slide

22. weaveworks-
    Raft core
    (raft)
    Storage

    (store)
    Persistence

    (WAL)
    Transport

    (HTTP)
    API
    (etcdserverpb)
    

    View Slide

23. weaveworks-
    Raft core
    (raft)
    Storage

    (store)
    Persistence

    (WAL)
    Transport

    (Mesh)
    API
    (etcdserverpb)
    ✕
    

    View Slide

24. weaveworks-
    Raft core
    Storage Persistence
    Transport
    API
    mechanical bits
    propose/commit
    

    View Slide

25. weaveworks-
    Raft
    Storage Persistence
    Transport
    API
    mechanical bits
    propose/commit
    
    lol nope
    

    View Slide

26. weaveworks-
    Deep dive
    

    View Slide

27. weaveworks-
    

    View Slide

28. weaveworks-
    

    View Slide

29. weaveworks-
    

    View Slide

30. weaveworks-
    

    View Slide

31. weaveworks-
    

    View Slide

32. weaveworks-
    

    View Slide

33. weaveworks-
    

    View Slide

34. weaveworks-
    

    View Slide

35. weaveworks-
    

    View Slide

36. weaveworks-
    

    View Slide

37. weaveworks-
    

    View Slide

38. weaveworks-
    raft.Node driver
    

    View Slide

39. weaveworks-
    raft.Node
    

    View Slide

40. weaveworks-
    ProposeEntry
    ProposeConf
    raft.Node
    

    View Slide

41. weaveworks-
    ProposeEntry
    ProposeConf
    Commit
    Error
    raft.Node
    

    View Slide

42. weaveworks-
    ProposeEntry
    ProposeConf
    Storage
    Persistence
    Commit
    Error
    Transport
    raft.Node
    

    View Slide

43. weaveworks-
    type Proposer interface {
    ProposeEntry(Entry) error
    ProposeConfChange(ConfChange) error
    }
    type CommitReceiver interface {
    ReceiveCommit(Entry)
    }
    type ErrorReceiver interface {
    ReceiveError(error)
    }
    

    View Slide

44. weaveworks-
    type Proposer interface {
    ProposeEntry(Entry) error
    ProposeConfChange(ConfChange) error
    }
    type CommitReceiver interface {
    ReceiveCommit(Entry)
    }
    type ErrorReceiver interface {
    ReceiveError(error)
    }
    func main() {
    n := newRaftNode(...)
    s := newStore(...)
    a := newAPI(...)
    n.commitReceiver = s
    n.errorReceiver = s
    a.store = s
    a.proposer = n
    // ...
    }
    

    View Slide

45. weaveworks-
    type Proposer interface {
    ProposeEntry(Entry) error
    ProposeConfChange(ConfChange) error
    }
    type CommitReceiver interface {
    ReceiveCommit(Entry)
    }
    type ErrorReceiver interface {
    ReceiveError(error)
    }
    func main() {
    n := newRaftNode(...)
    s := newStore(...)
    a := newAPI(...)
    n.commitReceiver = s
    n.errorReceiver = s
    a.store = s
    a.proposer = n
    // ...
    }
    ✕
    

    View Slide

46. weaveworks-
    ProposeEntry
    ProposeConf
    Commit
    Error
    raft.Node
    chan entry
    chan confchange
    chan entry
    chan error
    

    View Slide

47. weaveworks-
    func main() {
    var (
    entryc = make(chan entry)
    confchangec = make(chan confchange)
    commitc = make(chan entry)
    errorc = make(chan error)
    )
    n := newRaftNode(entryc, confchangec, commitc, errorc, ...)
    s := newStore(commitc, ...)
    a := newAPI(entryc, ...)
    // ...
    }
    

    View Slide

48. weaveworks-
    type raftNode struct {
    node raft.Node
    entryc confchangec commitc chanerrorc chan// storage?
    // transport?
    // ...
    quitc chan struct{}
    }
    

    View Slide

49. weaveworks-
    

    View Slide

50. weaveworks-
    

    View Slide

51. weaveworks-
    

    View Slide

52. weaveworks-
    Transport
    

    View Slide

53. weaveworks-
    Router
    Router
    Router
    Router
    Router
    

    View Slide

54. weaveworks-
    Router
    Router
    Router
    Router
    Router
    

    View Slide

55. weaveworks-
    Router
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    OnGossip(src Peer, m Msg)
    

    View Slide

56. weaveworks-
    type PacketConn interface {
    ReadFrom(b []byte) (n int, addr Addr, err error)
    WriteTo(b []byte, addr Addr) (n int, err error)
    Close() error
    LocalAddr() Addr
    SetDeadline(t time.Time) error
    SetReadDeadline(t time.Time) error
    SetWriteDeadline(t time.Time) error
    }
    

    View Slide

57. weaveworks-
    type PacketConn interface {
    ReadFrom(b []byte) (n int, addr Addr, err error)
    WriteTo(b []byte, addr Addr) (n int, err error)
    Close() error
    LocalAddr() Addr
    SetDeadline(t time.Time) error
    SetReadDeadline(t time.Time) error
    SetWriteDeadline(t time.Time) error
    }
    

    View Slide

58. weaveworks-
    Router
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    OnGossip(src Peer, m Msg)
    Adapter?
    ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
    

    View Slide

59. weaveworks-
    Router
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    OnGossip(src Peer, m Msg)
    meshconn.Peer
    ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
    

    View Slide

60. weaveworks-
    Router
    GossipUnicast(dst Peer, m Msg)
    GossipBroadcast(m Msg)
    OnGossip(src Peer, m Msg)
    meshconn.Peer
    ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
    incomingc chan Msg outgoingc chan Msg
    

    View Slide

61. weaveworks-
    type raftNode struct {
    node raft.Node
    entryc confchangec commitc chanerrorc chan// storage?
    incoming outgoing chan// ...
    quitc chan struct{}
    }
    

    View Slide

62. weaveworks-
    The controller
    

    View Slide

63. weaveworks-
    

    View Slide

64. weaveworks-
    for {
    select {
    case c.node.Tick()
    case r := // save entries
    // send messages
    // publish entries
    // advance
    case return
    }
    }
    

    View Slide

65. weaveworks-
    for {
    select {
    case c.node.Tick()
    case r := // save entries
    // send messages
    // publish entries
    // advance
    case return
    }
    }
    

    View Slide

66. weaveworks-
    

    View Slide

67. weaveworks-
    case r := // save entries
    // send messages
    // publish entries
    // advance
    

    View Slide

68. weaveworks-
    case r := // save entries
    // send messages
    // publish entries
    // advance
    

    View Slide

69. weaveworks-
    case r := // save entries
    // send messages
    // publish entries
    // advance
    

    View Slide

70. weaveworks-
    case r := c.readySave(r)
    c.readySend(r)
    c.readyApply(r)
    c.readyAdvance()
    

    View Slide

71. weaveworks-
    case r := if !c.readySave(r) { return }
    c.readySend(r)
    if !c.readyApply(r) { return }
    c.readyAdvance()
    

    View Slide

72. weaveworks-
    

    View Slide

73. weaveworks-
    

    View Slide

74. weaveworks-
    

    View Slide

75. weaveworks-
    

    View Slide

76. weaveworks-
    

    View Slide

77. weaveworks-
    

    View Slide

78. weaveworks-
    

    View Slide

79. weaveworks-
    

    View Slide

80. weaveworks-
    type raftNode struct {
    node raft.Node
    incomingc unreachablec confchangec proposalc outgoingc chanentryc chansnapshotc chanremovedc chanstopc chan struct{} // from the user
    terminatedc chan struct{} // to the user
    // storage?
    }
    

    View Slide

81. weaveworks-
    type raftNode struct {
    node raft.Node
    incomingc unreachablec confchangec proposalc outgoingc chanentryc chansnapshotc chanremovedc chanstopc chan struct{} // from the user
    terminatedc chan struct{} // to the user
    // storage?
    }
    
    

    View Slide

82. weaveworks-
    type raftNode struct {
    node raft.Node
    incomingc unreachablec confchangec proposalc outgoingc chanentryc chansnapshotc chanremovedc chanstopc chan struct{} // from the user
    terminatedc chan struct{} // to the user
    // storage?
    }
    
    

    View Slide

83. weaveworks-
    type raftNode struct {
    node raft.Node
    incomingc unreachablec confchangec proposalc outgoingc chanentryc chansnapshotc chanremovedc chanstopc chan struct{} // from the user
    terminatedc chan struct{} // to the user
    // storage?
    }
    

    View Slide

84. weaveworks-
    Drawing
    

    View Slide

85. weaveworks-
    Controller
    raft.Node
    

    View Slide

86. weaveworks-
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    

    View Slide

87. weaveworks-
    Transport
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    

    View Slide

88. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    

    View Slide

89. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    unreachablec ReportUnreachable
    

    View Slide

90. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

91. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    entryc
    unreachablec ReportUnreachable
    confchangec
    

    View Slide

92. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec ReportUnreachable
    confchangec
    

    View Slide

93. weaveworks-
    Transport
    Router
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec ReportUnreachable
    confchangec
    

    View Slide

94. weaveworks-
    Transport
    Router
    Membership
    meshconn
    Controller
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

95. weaveworks-
    Transport
    Router
    Membership
    meshconn
    Controller
    State
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

96. weaveworks-
    Transport
    Router
    Membership
    meshconn
    Controller
    State
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    snapshotc
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

97. weaveworks-
    Transport
    Router
    Membership
    meshconn
    Controller
    State
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    ProposeEntry
    proposalc
    snapshotc
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

98. weaveworks-
    

    View Slide

99. weaveworks-
    Transport
    Router
    Membership
    meshconn
    Controller
    State
    incomingc
    outgoingc
    raft.Node
    Step
    ProposeConfChange
    ProposeEntry
    proposalc
    snapshotc
    entryc
    Demuxer Normal entries
    ConfChange entries
    unreachablec
    confchangec
    ReportUnreachable
    

    View Slide

100. weaveworks-
     type raftNode struct {
     node raft.Node
     incomingc unreachablec confchangec proposalc outgoingc chanentryc chansnapshotc chanremovedc chanstopc chan struct{} // from the user
     terminatedc chan struct{} // to the user
     // storage?
     }
     

     View Slide

101. weaveworks-
     Transport
     Router
     Membership
     meshconn
     Controller
     State
     incomingc
     outgoingc
     raft.Node
     Step
     ProposeConfChange
     ProposeEntry
     proposalc
     snapshotc
     entryc
     Demuxer Normal entries
     ConfChange entries
     unreachablec
     confchangec
     ReportUnreachable
     

     View Slide

102. weaveworks-
     State & the API
     

     View Slide

103. weaveworks-
     State
     

     View Slide

104. weaveworks-
     State API?
     

     View Slide

105. weaveworks-
     State etcd V3
     

     View Slide

106. weaveworks-
     

     View Slide

107. weaveworks-
     State
     etcd V3

     KV
     

     View Slide

108. weaveworks-
     

     View Slide

109. weaveworks-
     

     View Slide

110. weaveworks-
     

     View Slide

111. weaveworks-
     

     View Slide

112. weaveworks-
     Transport
     Router
     Membership
     meshconn
     Controller
     State
     incomingc
     outgoingc
     raft.Node
     Step
     ProposeConfChange
     ProposeEntry
     proposalc
     snapshotc
     entryc
     Demuxer Normal entries
     ConfChange entries
     unreachablec
     confchangec
     ReportUnreachable
     

     View Slide

113. weaveworks-
     

     View Slide

114. weaveworks-
     

     View Slide

115. weaveworks-
     

     View Slide

116. weaveworks-
     

     View Slide

117. weaveworks-
     

     View Slide

118. weaveworks-
     

     View Slide

119. weaveworks-
     ¯\_(ϑ)_/¯
     

     View Slide

120. weaveworks-
     Demo
     

     View Slide

121. weaveworks-
     Conclusion & next steps
     

     View Slide

122. weaveworks-
     I've made a
     huge mistake.
     

     View Slide

123. weaveworks-
     Transport
     Router
     Membership
     meshconn
     Controller
     State
     incomingc
     outgoingc
     raft.Node
     Step
     ProposeConfChange
     ProposeEntry
     proposalc
     snapshotc
     entryc
     Demuxer Normal entries
     ConfChange entries
     unreachablec
     confchangec
     ReportUnreachable
     

     View Slide

124. weaveworks-
     Transport
     Router
     Membership
     meshconn
     Controller
     State
     incomingc
     outgoingc
     raft.Node
     Step
     ProposeConfChange
     ProposeEntry
     proposalc
     snapshotc
     entryc
     Demuxer Normal entries
     ConfChange entries
     unreachablec
     confchangec
     ReportUnreachable
     

     View Slide

125. weaveworks-
     The purpose of this project may be
     to serve as a warning to others.
     

     View Slide

126. weaveworks-
     github.com/weaveworks/mesh/tree/master/metcd
     

     View Slide

127. weaveworks-
     Thanks for listening!
     (I'll be at the bar.)
     (I assume there's a bar.)
     

     View Slide