membership lists at different members need not be consistent across the group at the same (causal) point in time. Stronger guarantees could be provided by augmenting the membership sub-system, e.g. a virtually-synchronous style membership can be provided through a sequencer process that checkpoints the membership list periodically. However, unlike the weakly consistent problem, strongly consistent specifications might have fundamental scalability limitations - SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol Abhinandan Das, Indranil Gupta, Ashish Motivala
of group size • Failure detection latency is independent of cluster size • Infection-style (Gossip) for membership updates ▪ Alive, Suspect, Dead Properties of SWIM
order messages ◦ Numbers start at 0 and increments when it receives information about it being suspected • Incarnation numbers are local, only the node can increment it’s own incarnation number • Incarnation numbers increment when a suspicion is received about ourselves but we are alive! Properties of SWIM
mechanism - Full state syncs over TCP periodically with random members - Separate messaging layer for member updates - Nodes will send out messages on their own for member updates periodically Memberlist Additions
from sending inaccurate suspect messages to reduce network traffic until local conditions improve - Useful in times of high local resource utilization, network partitions - Awareness score increases if the node suspects it is resource constrained
subsequent nodes get shorter timeframes. - Shortens suspicion timeout based of responses to reduce time failed node is in suspect state “We wait for the truth before we start spreading falsehoods” - Solomon Christoff, Golden Retriever
the cluster on their own behalf. - Snapshotting - Save state of a node - Network Coordinates - get Network Coordinates locally - KeyManager - Install, Uninstall keys - And much more! Serf Salesman: *slaps roof of Serf* “This bad boy has so many new functions”
of it’s timeout will request to be leader A node will typically reach the end of it’s timeout when it doesn’t get a message from the leader 0ms 7ms 40ms 150ms 51ms Vote for me please! Breakdown
three states at any given time: Follower Listening for heartbeats Candidate Polling for votes Leader Listening for incoming commands, sending out heartbeats to keep term alive
at most there is one leader per term. - Some terms can have no leaders “Terms identify obsolete information” - John Ousterhout - Leader’s log is seen as the truth, and is the most up to date log. Breakdown: Terms
receiving heartbeat from leader Request others to vote for you Becomes leader, send out heartbeats Somebody else becomes leader, become a follower Vote split, nobody wins. New term Breakdown: Leader Election
log has a higher term, higher index then the proposed-leaders log. 1 X = 3 1 X = 3 1 X = 3 1 X = 3 1 X = 3 2 Y = 8 2 Y = 8 2 Y = 8 2 Y = 8 INDEX Value INDEX Value Different color represents new term 2 Y = 8 3 Y = 8 3 N = 9 3 N = 9 3 N = 9 3 N = 9 Leader Election Vote for me please!
job of the consensus algorithm.” - Raft is designed around the log. Servers with inconsistent logs will never get elected as leader - Normal operation of Raft will repair inconsistencies Breakdown: Log Replication
3 2 Y = 8 2 Y = 8 3 N = 9 3 N = 9 Logs must persist through crashes Any committed entry is safe to execute in state machines A committed entry is replicated on the majority of servers 1 X = 3 2 Y = 8 3 N = 9 1 X = 3 2 Y = 8 3 N = 9 1 X = 3 2 Y = 8 3 N = 9 4 P = 6 4 P = 6 4 P = 6 4 P = 6 4 P = 6 5 L = 0 5 L = 0 5 L = 0 5 L = 0 6 R = 7 6 R = 7 6 R = 7 7 Z = 6 7 Z = 6 Committed Entries Breakdown
on every machine within a Consul cluster, it can be either a server or client. Server Typically a standalone instance that is involved in the Raft quorum and maintains state. Can communicate across datacenters. Client An agent that monitors an application* which is not apart of the Raft quorum and does not have state. Cannot communicate across datacenters.
- Implemented by a “simple in-memory database” based off Radix Trees - hashicorp/go-memdb - Stored on Consul Servers however can be accessed by both agents (clients or servers)
for Consul to follow. - Using the network coordinates we can provide failover for services based off geo location. - https://learn.hashicorp.com/consul/develo per-discovery/geo-failover
same in mem database as K/Vs, but different tables! - Health checks are configurable per service definition - HTTP - TCP - TTL - Docker - Script (Build your own!)
schristoff
shibuiwilliam
oikon48
ks91
yud0uhu
kenichinakamura
sakajunquality
sakamoto_582
hageyahhoo
lycorptech_jp
yamasaki696
sansantech
ahus1
lcolladotor
wjessup
destraynor
soudai
morganepeng
inesmontani
honnibal
kristinabergwall1
katarinadahlin
bryan
jakevdp
nmsamuel
