etcd over gossip

Strongly consistent semantics in a weakly consistent world.

Transcript

1. weaveworks- etcd over gossip Strongly-consistent semantics
 in a weakly-consistent world
2. weaveworks- ~or~
 Embedding etcd in your application
3. weaveworks- ~or~
 Porting etcd to a different transport
4. weaveworks- weavemesh
5. weaveworks- Gossip communication protocol CRDT data model
6. weaveworks- Gossip communication protocol CRDT data model Your application
7. weaveworks- Gossip communication protocol CRDT data model weavenet Virtual network interface + IP
8. weaveworks- Gossip communication protocol CRDT data model Your application
9. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data
10. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data
11. weaveworks- Router Router Router Router Router
12. weaveworks- Router Router Router Router Router
13. weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) Peers() []Peer
14. weaveworks- Gossip communication protocol Your application
15. weaveworks- Gossip communication protocol etcd
16. weaveworks- Raft
17. weaveworks-
18. weaveworks- Raft node Other nodes Write(X, 123) Proposal accepted X=123 Sounds good Read(X) 123
19. weaveworks- Send(dst Node, m Msg)
20. weaveworks- Send(dst Node, m Msg) GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg)
21. weaveworks- etcd
22. weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (HTTP) API (etcdserverpb)
23. weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (Mesh) API (etcdserverpb) ✕
24. weaveworks- Raft core Storage Persistence Transport API mechanical bits propose/commit
25. weaveworks- Raft Storage Persistence Transport API mechanical bits propose/commit lol nope
26. weaveworks- Deep dive
27. weaveworks-
28. weaveworks-
29. weaveworks-
30. weaveworks-
31. weaveworks-
32. weaveworks-
33. weaveworks-
34. weaveworks-
35. weaveworks-
36. weaveworks-
37. weaveworks-
38. weaveworks- raft.Node driver
39. weaveworks- raft.Node
40. weaveworks- ProposeEntry ProposeConf raft.Node
41. weaveworks- ProposeEntry ProposeConf Commit Error raft.Node
42. weaveworks- ProposeEntry ProposeConf Storage Persistence Commit Error Transport raft.Node
43. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) }
44. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... }
45. weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... } ✕
46. weaveworks- ProposeEntry ProposeConf Commit Error raft.Node chan entry chan confchange chan entry chan error
47. weaveworks- func main() { var ( entryc = make(chan entry) confchangec = make(chan confchange) commitc = make(chan entry) errorc = make(chan error) ) n := newRaftNode(entryc, confchangec, commitc, errorc, ...) s := newStore(commitc, ...) a := newAPI(entryc, ...) // ... }
48. weaveworks- type raftNode struct { node raft.Node entryc <-chan entry confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? // transport? // ... quitc chan struct{} }
49. weaveworks-
50. weaveworks-
51. weaveworks-
52. weaveworks- Transport
53. weaveworks- Router Router Router Router Router
54. weaveworks- Router Router Router Router Router
55. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg)
56. weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }
57. weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }
58. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) Adapter? ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
59. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...
60. weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ... incomingc chan Msg outgoingc chan Msg
61. weaveworks- type raftNode struct { node raft.Node entryc <-chan entry confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? incoming <-chan msg // transport outgoing chan<- msg // transport // ... quitc chan struct{} }
62. weaveworks- The controller
63. weaveworks-
64. weaveworks- for { select { case <-ticker: c.node.Tick() case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }
65. weaveworks- for { select { case <-ticker: c.node.Tick() case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }
66. weaveworks-
67. weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance
68. weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance
69. weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance
70. weaveworks- case r := <-c.node.Ready(): c.readySave(r) c.readySend(r) c.readyApply(r) c.readyAdvance()
71. weaveworks- case r := <-c.node.Ready(): if !c.readySave(r) { return } c.readySend(r) if !c.readyApply(r) { return } c.readyAdvance()
72. weaveworks-
73. weaveworks-
74. weaveworks-
75. weaveworks-
76. weaveworks-
77. weaveworks-
78. weaveworks-
79. weaveworks-
80. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
81. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
82. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
83. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
84. weaveworks- Drawing
85. weaveworks- Controller raft.Node
86. weaveworks- meshconn Controller incomingc outgoingc raft.Node Step
87. weaveworks- Transport meshconn Controller incomingc outgoingc raft.Node Step
88. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step
89. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step unreachablec ReportUnreachable
90. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange unreachablec confchangec ReportUnreachable
91. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc unreachablec ReportUnreachable confchangec
92. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec
93. weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec
94. weaveworks- Transport Router Membership meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
95. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
96. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
97. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
98. weaveworks-
99. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
100. weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }
101. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
102. weaveworks- State & the API
103. weaveworks- State
104. weaveworks- State API?
105. weaveworks- State etcd V3
106. weaveworks-
107. weaveworks- State etcd V3
 KV
108. weaveworks-
109. weaveworks-
110. weaveworks-
111. weaveworks-
112. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
113. weaveworks-
114. weaveworks-
115. weaveworks-
116. weaveworks-
117. weaveworks-
118. weaveworks-
119. weaveworks- ¯\_(ϑ)_/¯
120. weaveworks- Demo
121. weaveworks- Conclusion & next steps
122. weaveworks- I've made a huge mistake.
123. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
124. weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable
125. weaveworks- The purpose of this project may be to serve as a warning to others.
126. weaveworks- github.com/weaveworks/mesh/tree/master/metcd
127. weaveworks- Thanks for listening! (I'll be at the bar.) (I assume there's a bar.)