A practical way to generate unique id in Go

Unique ID generation is variant and not so simple to fit with what the system wants. In this talk, I’ll talk about how my team designed and implemented a custom Unique ID generation method, that fits our requirements, and powers all service-to-service request’s id in my company.

Transcript

1. A practical way to generate unique id in Go speaker := &Programmer{ Name: "Manh Dao Van", Github: "manhdaovan", Org: &Company{ Name: "Money Forward", StockCode: "TYO:3994", } }
2. Disclaimer All error handling, context params... of source code in this talk is reduced for representation the idea purpose, not production-ready source code. 2
3. What? ❖ Microservices --(needs)-> Idempotency --(needs)-> Request ID (Unique ID) ❖ Some constrain about this Request ID: ➢ Sortable is NOT required ➢ Universal uniqueness ➢ Be generated "fastly enough" per service instance ➢ Independent to any setting as much as possible ➢ Human readable and understandable about the request from requestID 3
4. Existing solutions - Snowflake(1*): - ID (64-bit) , sortable - (unused 1-bit) | (timestamp 41-bit) | (machine ID 10-bit) | (sequence number 36-bit) - Sonyflake(2*): - ID (64-bit) , sortable - (unused 1-bit) | (timestamp 39-bit) | (sequence number 8-bit) | (machine ID 16-bit) - Instagram’s method(3*): - ID (64-bit) , sortable - (timestamp 41-bit) | (shard ID 13-bit) | (sequence number 10-bit) - Pinterest’s method(4*): - ID (64-bit) , unsortable - (unused 2-bit) | (shard ID 16-bit) | (type ID 10-bit) | (local ID 36-bit) 4 Cons: Fixed info per app/db instance
5. Our solution - Architecture - Similar to Ticket Server(5*) algorithm - Request ID: string (non-fixed length) - IDPrefix (global unique id) + sequence number - Generated by and per app instance - IDPrefix = Service name + sequence number - Generated by IDManager Service - Flow: Service A -(requests)-> Service B -(requests) -> Service C - 1. Get IDPrefix from IDManager Service immediately after started up - Eg: “ServiceA-1” as IDPrefix for Service A, and “ServiceB-1” for Service B - 2, 3. Create ID basing on IDPrefix and request to downstream services. - Eg: “ServiceA-1-1” as Request ID from Service A to Service B - and “ServiceB-1-1” from Service B to Service C 5 Service A Service B Service C IDManager Service 1 1 2 3
6. Generate new Request ID: single goroutine case 6
7. Generate new Request ID: multiple goroutines case func (g *RequestIDGenerator) NextReqID() string { g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } 7 func (g *RequestIDGenerator) NextReqID() string { g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } GR1 GR2 timeline Data race
8. Solving data race - Method1: Using atomic operation - Eg: using “sync/atomic” - Method2: Using lock - Eg: using sync.Mutex / sync.RWMutex - Method3: Share data safety - Eg: using go channel 8
9. Generate new Request ID: using atomic operation func (g *RequestIDGenerator) NextReqID() string { if atomic.AddUint64(&g.seq, 1) == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, atomic.LoadUint64(&g.seq)) } 9 func (g *RequestIDGenerator) NextReqID() string { if atomic.AddUint64(&g.seq, 1) == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, atomic.LoadUint64(&g.seq)) } GR1 GR2 timeline Data race
10. Generate new Request ID: using lock var mu sync.Mutex // or sync.RWMutex func (g *RequestIDGenerator) NextReqID() string { mu.Lock() defer mu.Unlock() g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } 10 func (g *RequestIDGenerator) NextReqID() string { mu.Lock() defer mu.Unlock() g.seq++ if g.seq == 0 { g.idPrefix = g.idMgrClient.GetNewIDPrefix("ServiceA") } return fmt.Sprintf("%s-%d", g.idPrefix, g.seq) } GR1 GR2 timeline Becomes 1-by-1
11. Generate new Request ID: share the data safety Some go channel patterns 11 Credit: https://youtu.be/YEKjSzIwAdA
12. Generate new Request ID: share the data safety Fan-out pattern 12 https://gist.github.com/manhdaovan/284bb7087a04cd0b797 4a776669f58ae
13. Generate new Request ID: share the data safety Fan-out pattern 13 GR1 GR2 timeline GR3 ID1, ID2, ID3,...,IDn g.preGenIDs() ID1, ID3, …, ID2, …, IDn g.NextID() g.NextID()
14. References - (1*) Snowflake: https://github.com/twitter-archive/snowflake - (2*) Sonyflake: https://github.com/sony/sonyflake - (3*) Instagram’s method: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c - (4*) Pinterest’s method: https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96c a6f - (5*) Flickr’s Ticket Server method: https://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ 14
15. 15 We are hiring Thank you for your attention!