Tackling contention: the monsters inside the `sync.Locker`

Go is all about parallelism and concurrency, but they don’t come for free. This talk is about measuring their contention price and being able to reduce it.

Transcript

1. The monsters in the Locker Tackling contention in Go
2. Who am I Security Enhancement for the Web @ Google Gopher since 2015 Strong interests in parallel programming since 2009 Roberto Clapis (@empijei)
3. Contention The competition between different processes to acquire a resource.
4. The problem Go makes it easy to write parallel and concurrent code. Parallel concurrent code might cause contention. Contention might cause performance loss.
5. Measure it Go has amazing measurement tools: • pprof • trace As scientists do
6. The tools: pprof pprof is very powerful and has many potential profiles. Just run your benchmarks with -cpuprofile=cpu.out Then run go tool pprof <exec.test> cpu.out Now type "web". Can also be collected live with minimum overhead and "net/http/pprof".
7. pprof with "cpuprofile" for i := 0; i < count; i++ { go func() { for j := 0; j < 100000; j++ { mu.Lock() a++ mu.Unlock() } }() }
8. The full output
9. pprof with "mutexprofile" ROUTINE ============= one.Contend.func1 in main.go 1.02ms (flat, cum) 4.13% of Total . 64: for j := 0; j < 100000; j++ { 1.02ms 65: mu.Lock() . 66: a++ . 67: mu.Unlock()
10. The tools: trace Like profiles traces can be captured live by importing "net/http/pprof". Traces can also be generated with go test -trace. go tool trace opens a browser (/trace only works with chrome). The blocking profile is still pprof.
11. Regions
12. Goroutines
13. /trace
14. Beware of Heisenberg's uncertainty principle If you try to measure contention too precisely, measurements will influence the runtime and will change the values.
15. Odd results You might end up measuring the measurements tools performance.
16. Warning Before you make any change on your code based on what follows you should measure. Never optimize for contention reduction unless you know it is your bottleneck. When you measure, beware of sample size: small samples will give wrong results.
17. Fix it • Reduce contention by changing the algorithm • Reduce contention by changing primitives As engineers do
18. A common pattern: multi-producer multi-consumer chan
19. A common pattern: double-bottleneck chan Shared, contended state
20. Channels Channels are mutex-protected structs. Channels are blocking, and if unbuffered every single operation will block. If many send/receive operations are executed in parallel they will fight for the lock. // runtime/chan.go type hchan struct { // Lots of unexported fields lock mutex }
21. Sharding: decentralize state If jobs take more or less the same time and if the queue is stable contention can be reduced and bandwidth can be increased by splitting work. chan chan
22. Batching: reduce n. of accesses to shared state If latency is not critical and if eventual consistency is enough contention can be reduced and bandwidth can be increased by sending batches of work over the channel. chan batch batch
23. Mutexes Mutex: faster than channels by a constant, but trade orchestration, readability and writability for the extra speed. That said Go mutexes are fair and fast. If a mutex becomes contended the same principles apply: ◦ Sharding (beware of Zipf law) ◦ Batching (beware of inconsistency) ◦ Shorten critical section to what is needed RWMutex: for a very high reads/writes ratio
24. Mutexes are fast func (m *Mutex) Lock() { if atomic.CompareAndSwapInt32(&m.state, 0, mutexLocked) { return // Handle contended case and starvation func (m *Mutex) Unlock() { new := atomic.AddInt32(&m.state, -mutexLocked) // Handle starvation
25. End of the safe zone This is where we abandon hope In some cases the previous suggestions might not be enough.
26. Intuition will not help much func ContendIntType() { var wg sync.WaitGroup wg.Add(cores) c := make([]Type, cores) for i := 0; i < cores; i++ { go func(i int) { defer wg.Done() for j := 0; j < ops; j++ { c[i] += Type(j) } }(i) } wg.Wait() } • Uint8 • Uint32 • Uint64 Which is faster?
27. False sharing ContendUint8-4 200 5808111 ns/op ContendUint32-4 1000 1075017 ns/op ContendUint64-4 2000 956336 ns/op
28. When addressing lock contention is not enough In the ideal case lock contention is the only contention that exists, but computers are real machines, not ideal machines. Enter cache contention. Sharing state across cores requires to go through layers of memory. Main memory Cache L3 Cache L2 Cache L2 Cache L1 Cache L1 Cache L1 Cache L1 Core Core Core Core
29. Real world "Reads" Acquiring a Read lock for a RWMutex requires to Write a counter. Even atomic adds are executed sequentially in the hardware. This is counterintuitive but modern hardware works this way. Main memory Cache L3 Cache L2 Cache L2 Cache L1 Cache L1 Cache L1 Cache L1 Core Core Core Core
30. Cache sharing: sync.Map sync.Map is functionally equivalent to a map with an RWMutex and it is cache friendly. Downside is that it gives less guarantees, less methods and loses type safety, so you should only use it if necessary. It is strongly advised to never use it directly but write type-safe wrappers for it. Package sync type Map func (m *Map) Delete(key interface{}) func (m *Map) Load(key interface{}) (val interface{}, ok bool) func (m *Map) Range(f func(key, value interface{}) bool) func (m *Map) Store(key, value interface{})
31. If a map is not what you need Sometimes you need to have something different than a map that you rarely write but frequently read. In those rare cases you can use the atomic package. If you deal with objects you'll need to also use unsafe.
32. What could possibly go wrong? Race detector will probably not detect your races. It is very hard to reason about the code. Very few people can. Even the experts on these topics have a hard time debugging this kind of code.
33. If you really need to Read: v := atomic.Load(addr) Write: for { oldValue := atomic.Load(addr) newValue := change(oldValue) if atomic.CompareAndSwap(addr, oldValue, newValue) { break } } Warning: do not just use Store or you'll get a nasty data race. Be also aware of the ABA problem.
34. A bit of history • sync.WaitGroup was rewritten from mutexes to atomics on Jul 2011. This introduced a bug that caused some random-looking memory corruptions. The bug was fixed on Apr 2014. • A bug in the parallel GC was introduced on Sep 2011 and fixed only on Jan 2014. runtime.parfordo(work.sweepfor); bufferList[m->helpgc].busy = 0; if(runtime.xadd(&work.ndone, +1) == work.nproc-1) runtime.notewakeup(&work.alldone);
35. The fix
36. Questions and, hopefully, answers! clap.page.link/contention Twitter: Roberto Clapis (@empijei)