Tweet
Tweet

Slide 1

Slide 1 text

weaveworks- etcd over gossip Strongly-consistent semantics
 in a weakly-consistent world

Slide 2

Slide 2 text

weaveworks- ~or~
 Embedding etcd in your application

Slide 3

Slide 3 text

weaveworks- ~or~
 Porting etcd to a different transport

Slide 4

Slide 4 text

weaveworks- weavemesh

Slide 5

Slide 5 text

weaveworks- Gossip communication protocol CRDT data model

Slide 6

Slide 6 text

weaveworks- Gossip communication protocol CRDT data model Your application

Slide 7

Slide 7 text

weaveworks- Gossip communication protocol CRDT data model weavenet Virtual network interface + IP

Slide 8

Slide 8 text

weaveworks- Gossip communication protocol CRDT data model Your application

Slide 9

Slide 9 text

weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data

Slide 10

Slide 10 text

weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data

Slide 11

Slide 11 text

weaveworks- Router Router Router Router Router

Slide 12

Slide 12 text

weaveworks- Router Router Router Router Router

Slide 13

Slide 13 text

weaveworks- Your application Router :6379 Gossip Receiver Gossip Sender Gossip Data GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) Peers() []Peer

Slide 14

Slide 14 text

weaveworks- Gossip communication protocol Your application

Slide 15

Slide 15 text

weaveworks- Gossip communication protocol etcd

Slide 16

Slide 16 text

weaveworks- Raft

Slide 17

Slide 17 text

weaveworks-

Slide 18

Slide 18 text

weaveworks- Raft node Other nodes Write(X, 123) Proposal accepted X=123 Sounds good Read(X) 123

Slide 19

Slide 19 text

weaveworks- Send(dst Node, m Msg)

Slide 20

Slide 20 text

weaveworks- Send(dst Node, m Msg) GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg)

Slide 21

Slide 21 text

weaveworks- etcd

Slide 22

Slide 22 text

weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (HTTP) API (etcdserverpb)

Slide 23

Slide 23 text

weaveworks- Raft core (raft) Storage
 (store) Persistence
 (WAL) Transport
 (Mesh) API (etcdserverpb) ✕

Slide 24

Slide 24 text

weaveworks- Raft core Storage Persistence Transport API mechanical bits propose/commit

Slide 25

Slide 25 text

weaveworks- Raft Storage Persistence Transport API mechanical bits propose/commit lol nope

Slide 26

Slide 26 text

weaveworks- Deep dive

Slide 27

Slide 27 text

weaveworks-

Slide 28

Slide 28 text

weaveworks-

Slide 29

Slide 29 text

weaveworks-

Slide 30

Slide 30 text

weaveworks-

Slide 31

Slide 31 text

weaveworks-

Slide 32

Slide 32 text

weaveworks-

Slide 33

Slide 33 text

weaveworks-

Slide 34

Slide 34 text

weaveworks-

Slide 35

Slide 35 text

weaveworks-

Slide 36

Slide 36 text

weaveworks-

Slide 37

Slide 37 text

weaveworks-

Slide 38

Slide 38 text

weaveworks- raft.Node driver

Slide 39

Slide 39 text

weaveworks- raft.Node

Slide 40

Slide 40 text

weaveworks- ProposeEntry ProposeConf raft.Node

Slide 41

Slide 41 text

weaveworks- ProposeEntry ProposeConf Commit Error raft.Node

Slide 42

Slide 42 text

weaveworks- ProposeEntry ProposeConf Storage Persistence Commit Error Transport raft.Node

Slide 43

Slide 43 text

weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) }

Slide 44

Slide 44 text

weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... }

Slide 45

Slide 45 text

weaveworks- type Proposer interface { ProposeEntry(Entry) error ProposeConfChange(ConfChange) error } type CommitReceiver interface { ReceiveCommit(Entry) } type ErrorReceiver interface { ReceiveError(error) } func main() { n := newRaftNode(...) s := newStore(...) a := newAPI(...) n.commitReceiver = s n.errorReceiver = s a.store = s a.proposer = n // ... } ✕

Slide 46

Slide 46 text

weaveworks- ProposeEntry ProposeConf Commit Error raft.Node chan entry chan confchange chan entry chan error

Slide 47

Slide 47 text

weaveworks- func main() { var ( entryc = make(chan entry) confchangec = make(chan confchange) commitc = make(chan entry) errorc = make(chan error) ) n := newRaftNode(entryc, confchangec, commitc, errorc, ...) s := newStore(commitc, ...) a := newAPI(entryc, ...) // ... }

Slide 48

Slide 48 text

weaveworks- type raftNode struct { node raft.Node entryc <-chan entry confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? // transport? // ... quitc chan struct{} }

Slide 49

Slide 49 text

weaveworks-

Slide 50

Slide 50 text

weaveworks-

Slide 51

Slide 51 text

weaveworks-

Slide 52

Slide 52 text

weaveworks- Transport

Slide 53

Slide 53 text

weaveworks- Router Router Router Router Router

Slide 54

Slide 54 text

weaveworks- Router Router Router Router Router

Slide 55

Slide 55 text

weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg)

Slide 56

Slide 56 text

weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }

Slide 57

Slide 57 text

weaveworks- type PacketConn interface { ReadFrom(b []byte) (n int, addr Addr, err error) WriteTo(b []byte, addr Addr) (n int, err error) Close() error LocalAddr() Addr SetDeadline(t time.Time) error SetReadDeadline(t time.Time) error SetWriteDeadline(t time.Time) error }

Slide 58

Slide 58 text

weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) Adapter? ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...

Slide 59

Slide 59 text

weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ...

Slide 60

Slide 60 text

weaveworks- Router GossipUnicast(dst Peer, m Msg) GossipBroadcast(m Msg) OnGossip(src Peer, m Msg) meshconn.Peer ReadFrom(b []byte) ... WriteTo(b []byte, addr Addr) ... incomingc chan Msg outgoingc chan Msg

Slide 61

Slide 61 text

weaveworks- type raftNode struct { node raft.Node entryc <-chan entry confchangec <-chan confchange commitc chan<- entry errorc chan<- error // storage? incoming <-chan msg // transport outgoing chan<- msg // transport // ... quitc chan struct{} }

Slide 62

Slide 62 text

weaveworks- The controller

Slide 63

Slide 63 text

weaveworks-

Slide 64

Slide 64 text

weaveworks- for { select { case <-ticker: c.node.Tick() case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }

Slide 65

Slide 65 text

weaveworks- for { select { case <-ticker: c.node.Tick() case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance case <-c.quitc: return } }

Slide 66

Slide 66 text

weaveworks-

Slide 67

Slide 67 text

weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance

Slide 68

Slide 68 text

weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance

Slide 69

Slide 69 text

weaveworks- case r := <-c.node.Ready(): // save entries // send messages // publish entries // advance

Slide 70

Slide 70 text

weaveworks- case r := <-c.node.Ready(): c.readySave(r) c.readySend(r) c.readyApply(r) c.readyAdvance()

Slide 71

Slide 71 text

weaveworks- case r := <-c.node.Ready(): if !c.readySave(r) { return } c.readySend(r) if !c.readyApply(r) { return } c.readyAdvance()

Slide 72

Slide 72 text

weaveworks-

Slide 73

Slide 73 text

weaveworks-

Slide 74

Slide 74 text

weaveworks-

Slide 75

Slide 75 text

weaveworks-

Slide 76

Slide 76 text

weaveworks-

Slide 77

Slide 77 text

weaveworks-

Slide 78

Slide 78 text

weaveworks-

Slide 79

Slide 79 text

weaveworks-

Slide 80

Slide 80 text

weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }

Slide 81

Slide 81 text

weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }

Slide 82

Slide 82 text

weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }

Slide 83

Slide 83 text

weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }

Slide 84

Slide 84 text

weaveworks- Drawing

Slide 85

Slide 85 text

weaveworks- Controller raft.Node

Slide 86

Slide 86 text

weaveworks- meshconn Controller incomingc outgoingc raft.Node Step

Slide 87

Slide 87 text

weaveworks- Transport meshconn Controller incomingc outgoingc raft.Node Step

Slide 88

Slide 88 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step

Slide 89

Slide 89 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step unreachablec ReportUnreachable

Slide 90

Slide 90 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange unreachablec confchangec ReportUnreachable

Slide 91

Slide 91 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc unreachablec ReportUnreachable confchangec

Slide 92

Slide 92 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec

Slide 93

Slide 93 text

weaveworks- Transport Router meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec ReportUnreachable confchangec

Slide 94

Slide 94 text

weaveworks- Transport Router Membership meshconn Controller incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 95

Slide 95 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 96

Slide 96 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 97

Slide 97 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 98

Slide 98 text

weaveworks-

Slide 99

Slide 99 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 100

Slide 100 text

weaveworks- type raftNode struct { node raft.Node incomingc <-chan raftpb.Message // from the transport unreachablec <-chan nodeID // from the transport confchangec <-chan raftpb.ConfChange // from the mesh proposalc <-chan []byte // from the state machine outgoingc chan<- raftpb.Message // to the transport entryc chan<- raftpb.Entry // to the demuxer snapshotc chan<- raftpb.Snapshot // to the state machine removedc chan<- struct{} // to calling context stopc chan struct{} // from the user terminatedc chan struct{} // to the user // storage? }

Slide 101

Slide 101 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 102

Slide 102 text

weaveworks- State & the API

Slide 103

Slide 103 text

weaveworks- State

Slide 104

Slide 104 text

weaveworks- State API?

Slide 105

Slide 105 text

weaveworks- State etcd V3

Slide 106

Slide 106 text

weaveworks-

Slide 107

Slide 107 text

weaveworks- State etcd V3
 KV

Slide 108

Slide 108 text

weaveworks-

Slide 109

Slide 109 text

weaveworks-

Slide 110

Slide 110 text

weaveworks-

Slide 111

Slide 111 text

weaveworks-

Slide 112

Slide 112 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 113

Slide 113 text

weaveworks-

Slide 114

Slide 114 text

weaveworks-

Slide 115

Slide 115 text

weaveworks-

Slide 116

Slide 116 text

weaveworks-

Slide 117

Slide 117 text

weaveworks-

Slide 118

Slide 118 text

weaveworks-

Slide 119

Slide 119 text

weaveworks- ¯\_(ϑ)_/¯

Slide 120

Slide 120 text

weaveworks- Demo

Slide 121

Slide 121 text

weaveworks- Conclusion & next steps

Slide 122

Slide 122 text

weaveworks- I've made a huge mistake.

Slide 123

Slide 123 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 124

Slide 124 text

weaveworks- Transport Router Membership meshconn Controller State incomingc outgoingc raft.Node Step ProposeConfChange ProposeEntry proposalc snapshotc entryc Demuxer Normal entries ConfChange entries unreachablec confchangec ReportUnreachable

Slide 125

Slide 125 text

weaveworks- The purpose of this project may be to serve as a warning to others.

Slide 126

Slide 126 text

weaveworks- github.com/weaveworks/mesh/tree/master/metcd

Slide 127

Slide 127 text

weaveworks- Thanks for listening! (I'll be at the bar.) (I assume there's a bar.)