A hitchhiker’s guide to building distributed systems with Go

Transcript

1. A hitchhiker’s guide to building distributed systems with Go
3. Traffic • >1 billion adrequests a month • >12 micro services • <10ms strict SLA • ~200 million requests internal to services per day • Serve ads < 300ms (includes bidding on different ad servers) • ~200 million events per day • ~140GB of data ingested
4. So what’s the tech like? - Go, Go and Go - Redis for caching - Aerospike (for anything too serious) - Kafka/BigQuery for Data side - Python (ML/Image processing)
5. Motive of the talk • How Go makes it easy to build distributed systems • Implementing a simple distributed hash table • Advantages of Go
6. Distributed libraries in Go • etcd • consul • raft • pachyderm
7. Why Go? • Concurrency • Performance • Design • Static Type system • Fast compiler
8. Design • Package based approach • No cyclic import hell • Automatic memory management, through GC • Composition over inheritance (interfaces)
9. Concurrency/Scheduler • Lightweight goroutines • Can spawn millions of goroutines without a hitch • Fast scheduler • Better thread management • No blocks on syscall
10. Example: 
 Zizou (A simple cache) • Always use L1/L2 mechanism in our micro services (restricted by resources) • Most of data/configuration is stale for a short period • Helps in keeping latency low • Cores always used, RAM hardly utilized • Maps are not thread safe • How sharded structure helps • How spawning n goroutines for different shards is easy, can’t do in Python or any other language easily
11. type Cache struct { shards []*shard hash hasher shardMask uint64 } func (c *Cache) getShard(hashedKey uint64) (shard *shard) { return c.shards[hashedKey&c.shardMask] } func (c *Cache) Get(k string) (interface{}, bool) { hashedKey := c.hash.Sum64(k) shard := c.getShard(hashedKey) return shard.Get(k) } func (c *Cache) Set(k string, v interface{}, dur time.Duration) error { hashedKey := c.hash.Sum64(k) shard := c.getShard(hashedKey) return shard.Set(k, v, dur) } func newShard() *shard { return &shard{ data: make(map[string]item), } } func newShardWithSweeper(sweepTime time.Duration) *shard { nc := &shard{ data: make(map[string]item), } if sweepTime > 0 { runSweeper(nc, sweepTime) runtime.SetFinalizer(nc, stopSweeper) } return nc } func runSweeper(c *shard, dur time.Duration) { swp := newSweeper(dur) c.sweeper = swp go swp.Run(c) }
12. Interface • A collection of methods • A type can implement an interface by implementing the methods • Decouples implementation from the package that defines it • Makes testing easier, just mock the interface
13. type Storage interface { Get(string) ([]byte, error) Set(string, string) error Delete(string) error Between([]byte, []byte) ([]*internal.KV, error) MDelete(...string) error } type Transport interface { //RPC GetSuccessor(*internal.Node) (*internal.Node, error) FindSuccessor(*internal.Node, []byte) (*internal.Node, error) GetPredecessor(*internal.Node) (*internal.Node, error) Notify(*internal.Node, *internal.Node) error CheckPredecessor(*internal.Node) error SetPredecessor(*internal.Node, *internal.Node) error SetSuccessor(*internal.Node, *internal.Node) error //Storage GetKey(*internal.Node, string) (*internal.GetResponse, error) SetKey(*internal.Node, string, string) error DeleteKey(*internal.Node, string) error RequestKeys(*internal.Node, []byte, []byte) ([]*internal.KV, error) DeleteKeys(*internal.Node, []string) error }
14. type Node struct { storage Storage stMtx sync.RWMutex transport Transport tsMtx sync.RWMutex }
15. Shard please?
16. Consistent Hashing • In most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped because the mapping between the keys and the slots is defined by a modular operation. • Special kind of hashing such that when a hash table is resized, only K/n keys need to be remapped on average, where K is the number of keys, and n is the number of slots. • Maps the key to the same server, irrespective of the number of nodes • Provides Horizontal Scaling
17. Use case • Partitioning component of Amazon's storage system Dynamo • Data partitioning in Apache Cassandra • Akka's consistent hashing router • Riak, a distributed key-value database • Akamai Content Delivery Network • Client side load balancing
18. How to implement it on distributed machines practically?
19. Distributed Hash Table • Uses consistent hashing algorithm for key space partitioning • Decentralized, the nodes collectively form the system without any central coordination • Scalable, the system functions efficiently even with thousands or millions of nodes. • Fault Tolerant, the system is reliable (in some sense) even with nodes continuously joining, leaving, and failing • CAN, CHORD, PASTRY, TAPESTRY, KADEMLIA
20. Chord • Chord is a protocol and algorithm for a peer-to-peer distributed hash table. • A DHT stores key-value pairs by assigning keys to different nodes; a node will store the values for all the keys for which it is responsible. • Chord specifies how keys are assigned to nodes, and how a node can discover the value for a given key by first locating the node responsible for that key • Chord is based on consistent hashing • Chose to implement CHORD because of it’s simplicity
21. How Chord works?
23. // fingerEntry represents a single finger table entry type fingerEntry struct { Id []byte // ID hash of (n + 2^i) mod (2^m) Node *internal.Node } func newFingerTable(node *internal.Node, m int) fingerTable { ft := make([]*fingerEntry, m) for i := range ft { ft[i] = newFingerEntry(fingerID(node.Id, i, m), node) } return ft } // newFingerEntry returns an allocated new finger entry with the attributes set func newFingerEntry(id []byte, node *internal.Node) *fingerEntry { return &fingerEntry{ Id: id, Node: node, } } // Computes the offset by (n + 2^i) mod (2^m) func fingerID(n []byte, i int, m int) []byte { // Convert the ID to a bigint idInt := (&big.Int{}).SetBytes(n) // Get the offset two := big.NewInt(2) offset := big.Int{} offset.Exp(two, big.NewInt(int64(i)), nil) // Sum sum := big.Int{} sum.Add(idInt, &offset) // Get the ceiling ceil := big.Int{} ceil.Exp(two, big.NewInt(int64(m)), nil) // Apply the mod idInt.Mod(&sum, &ceil) // Add together return idInt.Bytes() }
24. Node join
25. • The following tasks should be done for a newly joined node n:
 
 1) Initialize node n (the predecessor and the finger table).
 
 2) Notify other nodes to update their predecessors and finger tables.
 
 3) The new node takes over its responsible keys from its successor.
26. type Node struct { *internal.Node cnf *Config predecessor *internal.Node predMtx sync.RWMutex successor *internal.Node succMtx sync.RWMutex shutdownCh chan struct{} fingerTable fingerTable ftMtx sync.RWMutex storage Storage stMtx sync.RWMutex transport Transport tsMtx sync.RWMutex lastStablized time.Time } func (n *Node) join(joinNode *internal.Node) error { // First check if node already present in the circle // Join this node to the same chord ring as parent var foo *internal.Node // // Ask if our id already exists on the ring. if joinNode != nil { remoteNode, err := n.findSuccessorRPC(joinNode, n.Id) if err != nil { return err } if isEqual(remoteNode.Id, n.Id) { return ERR_NODE_EXISTS } foo = joinNode } else { foo = n.Node } succ, err := n.findSuccessorRPC(foo, n.Id) if err != nil { return err } n.succMtx.Lock() n.successor = succ n.succMtx.Unlock() return nil }
27. Lookup • Compute the hash of the key • Check the finger table on the node to determine where the entry lies • The i{th} entry of node n will contain successor((n+2^{i-1}) mod 2^m) • Every time a node wants to look up a key k, it will pass the query to the closest successor or predecessor (depending on the finger table) of k in its finger table (the “largest” one on the circle whose ID is smaller than k), until a node finds out the key is stored in its immediate successor • O(logN) lookup time
29. Stabilize • A stabilization protocol is running periodically in the background which updates finger tables and successor pointers. The stabilization protocol works as follows:
 
 1) Stabilize(): n asks its successor for its predecessor p and decides whether p should be n‘s successor instead (this is the case if p recently joined the system).
 
 2) Notify(): notifies n‘s successor of its existence, so it can change its predecessor to n
 
 3) Fix_fingers(): updates finger tables
 
 4) check_predecessor(): Periodically checks in predecessor is alive
30. func NewNode(cnf *Config, joinNode *internal.Node) (*Node, error) { if err := cnf.Validate(); err != nil { return nil, err } node := &Node{ Node: new(internal.Node), shutdownCh: make(chan struct{}), cnf: cnf, storage: NewMapStore(cnf.Hash), } // Populate finger table node.fingerTable = newFingerTable(node.Node, cnf.HashSize) // Start RPC server transport, err := NewGrpcTransport(cnf) if err != nil { return nil, err } node.transport = transport internal.RegisterChordServer(transport.server, node) node.transport.Start() if err := node.join(joinNode); err != nil { return nil, err } // Peridoically stabilize the node. go func() { ticker := time.NewTicker(1 * time.Second) for { select { case <-ticker.C: node.stabilize() case <-node.shutdownCh: ticker.Stop() return } } }() // Peridoically fix finger tables. go func() { next := 0 ticker := time.NewTicker(100 * time.Millisecond) for { select { case <-ticker.C: next = node.fixFinger(next) case <-node.shutdownCh: ticker.Stop() return } } }() // Peridoically checkes whether predecessor has failed. go func() { ticker := time.NewTicker(10 * time.Second) for { select { case <-ticker.C: node.checkPredecessor() case <-node.shutdownCh: ticker.Stop() return } } }() return node, nil }
31. Shutdown • Notify successor to change its predecessor pointer to our predecessor. Do nothing if we are our own successor (i.e. we are the only node in the ring). • Transfer keys to the successor node
32. func (n *Node) Stop() { close(n.shutdownCh) // Notify successor to change its predecessor pointer to our predecessor. // Do nothing if we are our own successor (i.e. we are the only node in the // ring). n.succMtx.RLock() succ := n.successor n.succMtx.RUnlock() n.predMtx.RLock() pred := n.predecessor n.predMtx.RUnlock() if n.Node.Addr != succ.Addr && pred != nil { n.moveKeysFromLocal(pred, succ) predErr := n.setPredecessorRPC(succ, pred) succErr := n.setSuccessorRPC(pred, succ) fmt.Println("stop errors: ", predErr, succErr) } n.transport.Stop() }
33. Demo
34. • github.com/arriqaaq/chord • github.com/arriqaaq/zizou • https://blog.goodaudience.com/chord-building-a-dht- distributed-hash-table-in-golang-67c3ce17417b • https://hackernoon.com/consistent-hashing-with- bounded-loads-using-a-red-black-tree-b5aaf0d8540f
35. – Go Scheduler Life is short. Don’t “Panic”