A practical way to generate unique id in Go speaker := &Programmer{ Name: "Manh Dao Van", Github: "manhdaovan", Org: &Company{ Name: "Money Forward", StockCode: "TYO:3994", } }

Disclaimer All error handling, context params... of source code in this talk is reduced for representation the idea purpose, not production-ready source code. 2

What? ❖ Microservices --(needs)-> Idempotency --(needs)-> Request ID (Unique ID) ❖ Some constrain about this Request ID: ➢ Sortable is NOT required ➢ Universal uniqueness ➢ Be generated "fastly enough" per service instance ➢ Independent to any setting as much as possible ➢ Human readable and understandable about the request from requestID 3

Existing solutions - Snowflake(1*): - ID (64-bit) , sortable - (unused 1-bit) | (timestamp 41-bit) | (machine ID 10-bit) | (sequence number 36-bit) - Sonyflake(2*): - ID (64-bit) , sortable - (unused 1-bit) | (timestamp 39-bit) | (sequence number 8-bit) | (machine ID 16-bit) - Instagram’s method(3*): - ID (64-bit) , sortable - (timestamp 41-bit) | (shard ID 13-bit) | (sequence number 10-bit) - Pinterest’s method(4*): - ID (64-bit) , unsortable - (unused 2-bit) | (shard ID 16-bit) | (type ID 10-bit) | (local ID 36-bit) 4 Cons: Fixed info per app/db instance

Our solution - Architecture - Similar to Ticket Server(5*) algorithm - Request ID: string (non-fixed length) - IDPrefix (global unique id) + sequence number - Generated by and per app instance - IDPrefix = Service name + sequence number - Generated by IDManager Service - Flow: Service A -(requests)-> Service B -(requests) -> Service C - 1. Get IDPrefix from IDManager Service immediately after started up - Eg: “ServiceA-1” as IDPrefix for Service A, and “ServiceB-1” for Service B - 2, 3. Create ID basing on IDPrefix and request to downstream services. - Eg: “ServiceA-1-1” as Request ID from Service A to Service B - and “ServiceB-1-1” from Service B to Service C 5 Service A Service B Service C IDManager Service 1 1 2 3

Generate new Request ID: single goroutine case 6

Generate new Request ID: multiple goroutines case func (g *RequestIDGenerator) NextReqID() string { g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } 7 func (g *RequestIDGenerator) NextReqID() string { g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } GR1 GR2 timeline Data race

Solving data race - Method1: Using atomic operation - Eg: using “sync/atomic” - Method2: Using lock - Eg: using sync.Mutex / sync.RWMutex - Method3: Share data safety - Eg: using go channel 8

Generate new Request ID: using atomic operation func (g *RequestIDGenerator) NextReqID() string { if atomic.AddUint64(&g.seq, 1) == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, atomic.LoadUint64(&g.seq)) } 9 func (g *RequestIDGenerator) NextReqID() string { if atomic.AddUint64(&g.seq, 1) == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, atomic.LoadUint64(&g.seq)) } GR1 GR2 timeline Data race

Generate new Request ID: using lock var mu sync.Mutex // or sync.RWMutex func (g *RequestIDGenerator) NextReqID() string { mu.Lock() defer mu.Unlock() g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } 10 func (g *RequestIDGenerator) NextReqID() string { mu.Lock() defer mu.Unlock() g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } GR1 GR2 timeline Becomes 1-by-1

Generate new Request ID: share the data safety Some go channel patterns 11 Credit: https://youtu.be/YEKjSzIwAdA

Generate new Request ID: share the data safety Fan-out pattern 12 https://gist.github.com/manhdaovan/284bb7087a04cd0b797 4a776669f58ae

Generate new Request ID: share the data safety Fan-out pattern 13 GR1 GR2 timeline GR3 ID1, ID2, ID3,...,IDn g.preGenIDs() ID1, ID3, …, ID2, …, IDn g.NextID() g.NextID()

References - (1*) Snowflake: https://github.com/twitter-archive/snowflake - (2*) Sonyflake: https://github.com/sony/sonyflake - (3*) Instagram’s method: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c - (4*) Pinterest’s method: https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96c a6f - (5*) Flickr’s Ticket Server method: https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ 14

15 We are hiring Thank you for your attention!