Upgrade to Pro — share decks privately, control downloads, hide ads and more …

etcd over gossip

etcd over gossip

Strongly consistent semantics in a weakly consistent world.

Eb68eb9603a0e8822eda69e7ca721866?s=128

Peter Bourgon

May 09, 2016
Tweet

Transcript

  1. weaveworks- etcd over gossip Strongly-consistent semantics
 in a weakly-consistent world

  2. weaveworks- ~or~
 Embedding etcd in your application

  3. weaveworks- ~or~
 Porting etcd to a different transport

  4. weaveworks- weavemesh

  5. weaveworks- Gossip communication protocol CRDT data model

  6. weaveworks- Gossip communication protocol CRDT data model Your application

  7. weaveworks- Gossip communication protocol CRDT data model weavenet Virtual network

    interface + IP
  8. weaveworks- Gossip communication protocol CRDT data model Your application

  9. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip

    Data
  10. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip

    Data
  11. weaveworks- Router Router Router Router Router

  12. weaveworks- Router Router Router Router Router

  13. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip

    Data GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) Peers() []Peer
  14. weaveworks- Gossip communication protocol Your application

  15. weaveworks- Gossip communication protocol etcd

  16. weaveworks- Raft

  17. weaveworks-

  18. weaveworks- Raft node Other nodes Write(X, 123) Proposal accepted X=123

    Sounds good Read(X) 123
  19. weaveworks- Send(dst Node, m Msg)

  20. weaveworks- Send(dst Node, m Msg) GossipUnicast(dst Peer, m Msg) GossipBroadcast(m

    Msg)
  21. weaveworks- etcd

  22. weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (HTTP)

    API (etcdserverpb)
  23. weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (Mesh)

    API (etcdserverpb) ✕
  24. weaveworks- Raft core Storage Persistence Transport API mechanical bits propose/commit

  25. weaveworks- Raft Storage Persistence Transport API mechanical bits propose/commit lol

    nope
  26. weaveworks- Deep dive

  27. weaveworks-

  28. weaveworks-

  29. weaveworks-

  30. weaveworks-

  31. weaveworks-

  32. weaveworks-

  33. weaveworks-

  34. weaveworks-

  35. weaveworks-

  36. weaveworks-

  37. weaveworks-

  38. weaveworks- raft.Node driver

  39. weaveworks- raft.Node

  40. weaveworks- ProposeEntry ProposeConf raft.Node

  41. weaveworks- ProposeEntry ProposeConf Commit Error raft.Node

  42. weaveworks- ProposeEntry ProposeConf Storage Persistence Commit Error Transport raft.Node

  43. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error }

    type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) }
  44. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error }

    type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... }
  45. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error }

    type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... } ✕
  46. weaveworks- ProposeEntry ProposeConf Commit Error raft.Node chan entry chan confchange

    chan entry chan error
  47. weaveworks- func main() { var ( entryc = make(chan entry)

    confchangec = make(chan confchange) commitc = make(chan entry) errorc = make(chan error) ) n := newRaftNode(entryc, confchangec, commitc, errorc, ...) s := newStore(commitc, ...) a := newAPI(entryc, ...) // ... }
  48. weaveworks- type raftNode struct { node raft.Node entryc <-chan entry

    confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? // transport? // ... quitc chan struct{} }
  49. weaveworks-

  50. weaveworks-

  51. weaveworks-

  52. weaveworks- Transport

  53. weaveworks- Router Router Router Router Router

  54. weaveworks- Router Router Router Router Router

  55. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer,

    m Msg)
  56. weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr

    Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }
  57. weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr

    Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }
  58. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer,

    m Msg) Adapter? ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
  59. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer,

    m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
  60. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer,

    m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ... incomingc chan Msg outgoingc chan Msg
  61. weaveworks- type raftNode struct { node raft.Node entryc <-chan entry

    confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? incoming <-chan msg // transport outgoing chan<- msg // transport // ... quitc chan struct{} }
  62. weaveworks- The controller

  63. weaveworks-

  64. weaveworks- for { select { case <-ticker: c.node.Tick() case r

    := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }
  65. weaveworks- for { select { case <-ticker: c.node.Tick() case r

    := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }
  66. weaveworks-

  67. weaveworks- case r := <-c.node.Ready(): // save entries // send

    messages // publish entries // advance
  68. weaveworks- case r := <-c.node.Ready(): // save entries // send

    messages // publish entries // advance
  69. weaveworks- case r := <-c.node.Ready(): // save entries // send

    messages // publish entries // advance
  70. weaveworks- case r := <-c.node.Ready(): c.readySave(r) c.readySend(r) c.readyApply(r) c.readyAdvance()

  71. weaveworks- case r := <-c.node.Ready(): if !c.readySave(r) { return }

    c.readySend(r) if !c.readyApply(r) { return } c.readyAdvance()
  72. weaveworks-

  73. weaveworks-

  74. weaveworks-

  75. weaveworks-

  76. weaveworks-

  77. weaveworks-

  78. weaveworks-

  79. weaveworks-

  80. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message

    // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
  81. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message

    // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
  82. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message

    // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
  83. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message

    // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
  84. weaveworks- Drawing

  85. weaveworks- Controller raft.Node

  86. weaveworks- meshconn Controller incomingc outgoingc raft.Node Step

  87. weaveworks- Transport meshconn Controller incomingc outgoingc raft.Node Step

  88. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step

  89. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step unreachablec

    ReportUnreachable
  90. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange

    unreachablec confchangec ReportUnreachable
  91. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange

    entryc unreachablec ReportUnreachable confchangec
  92. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange

    entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec
  93. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange

    entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec
  94. weaveworks- Transport Router Membership meshconn Controller incomingc outgoingc raft.Node Step

    ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  95. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  96. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  97. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  98. weaveworks-

  99. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  100. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message

    // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
  101. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  102. weaveworks- State & the API

  103. weaveworks- State

  104. weaveworks- State API?

  105. weaveworks- State etcd V3

  106. weaveworks-

  107. weaveworks- State etcd V3
 KV

  108. weaveworks-

  109. weaveworks-

  110. weaveworks-

  111. weaveworks-

  112. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  113. weaveworks-

  114. weaveworks-

  115. weaveworks-

  116. weaveworks-

  117. weaveworks-

  118. weaveworks-

  119. weaveworks- ¯\_(ϑ)_/¯

  120. weaveworks- Demo

  121. weaveworks- Conclusion & next steps

  122. weaveworks- I've made a huge mistake.

  123. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  124. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node

    Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
  125. weaveworks- The purpose of this project may be to serve

    as a warning to others.
  126. weaveworks- github.com/weaveworks/mesh/tree/master/metcd

  127. weaveworks- Thanks for listening! (I'll be at the bar.) (I

    assume there's a bar.)