Golang race detection – Neven Miculinić

Transcript

1. Golang race detection Neven Miculinić @nmiculinic
3. Counter Example
4. Counter example var Cnt int func Run(amount int) { for i := 0; i < amount; i++ { Cnt++ } }
5. Counter example func main() { wg := &sync.WaitGroup{} for i := 0; i < 10; i++ { wg.Add(1) go func() { Run(1000); wg.Done() }() } wg.Wait(); fmt.Println(Cnt) }
6. Famous examples
7. Dirty COW • Race condition in linux kernel • Consequences: privilege escalation
8. Therac-25 • Radiotherapy machine • Race condition between user input and system state • Consequences: 6+ accidents, 2+ deaths
9. Definitions
10. Data Race Concurrent access to memory location, one of which is a write
11. Concurrent access Concurrent access is one where event ordering isn’t known. There’s no happens-before relation
12. Happens before G1: 0 G2: 0 G1: 1 G1: 2 G2: 3 G2: 1 G2: 2
13. Happens before G1: 0 G2: 0 G1: 1 G1: 2 G2: 3 G2: 1 G2: 2 G1: 3 G2: 4 CHAN RECV CHAN SEND
14. Happens before G1: 0 G2: 0 G1: 1 G1: 2 G2: 3 G2: 1 G2: 2 G1: 3 G2: 4 MUTEX LOCK MUTEX UNLOCK
15. Typical examples
16. Race on loop counter func main() { var wg sync.WaitGroup wg.Add(5) for i := 0; i < 5; i++ { go func() { fmt.Println(i) // Not the 'i' you are looking for. wg.Done() }() } wg.Wait() }
17. Race on loop counter func main() { var wg sync.WaitGroup wg.Add(5) for i := 0; i < 5; i++ { go func(j int) { fmt.Println(j) wg.Done() }(i) } wg.Wait() }
18. Unprotected global var Cnt int func Inc(amount int) { Cnt += amount } func main() { go Inc(10) go Int(100) }
19. Unprotected global var Cnt int var m = &sync.Mutex{} func Inc(amount int) { m.Lock() defer m.Unlock() Cnt += amount }
20. Primitive unprotected variable var Cnt int64 func Inc(amount int64) { atomic.AddInt64(&Cnt, amount) }
21. Violating go memory model (aka being smart) var a string var done bool func setup() { a = "hello, world" done = true } func main() { go setup() for !done { } fmt.Print(a) }
22. Synchronization primitives • Sync package ◦ Mutex ◦ Atomics ◦ ... • channels
23. Detecting data races
24. Detecting race conditions • Go test -race • Go build -race • Go install -race
25. ThreadSanitizer v2 • Battle proven library • Runtime slowdown 2x-20x. • Memory overhead 5x-10x. • Requires CGO, 64bit OS • No false positives
26. Compiler instrumentation movq "".x+8(SP), AX incq (AX) func Inc(x *int) { *x++ }
27. func Inc(x *int) { *x++ } Compiler instrumentation movq "".x+32(SP), AX movq AX, (SP) call runtime.raceread(SB) movq "".x+32(SP), AX movq (AX), CX movq CX, ""..autotmp_4+8(SP) movq AX, (SP) call runtime.racewrite(SB) incq AX movq "".x+32(SP), CX movq AX, (CX)
28. func Inc(x *int) { *x++ } Compiler instrumentation ... call runtime.raceread(SB) ... call runtime.racewrite(SB) ...
29. How data race detection works?
30. Data race detection complexity Determining if arbitrary program contains potential data races is NP-hard. [Netzer&Miller 1990]
31. Race detection design choices • Static analysis • Dynamic analysis ◦ On-the-fly ◦ post-mortem
32. • Happens before • Lockset based • Hybrid models Dynamic race detection
33. Happens before Vector clocks
34. Vector clocks G1: 0 G2: 0 G1: 0 G2: 0 G1: 1 G2: 0 G1: 2 G2: 0 G1: 0 G2: 3 G1: 0 G2: 1 G1: 0 G2: 2
35. Vector clocks G1: 0 G2: 0 G1: 0 G2: 0 G1: 1 G2: 0 G1: 2 G2: 0 G1: 0 G2: 3 G1: 0 G2: 1 G1: 0 G2: 2 G1: 3 G2: 3 G1: 0 G2: 4 MUTEX LOCK MUTEX UNLOCK
36. Vector clocks • Partially ordered set • (4, 11) < (10, 12) → happens before • (12, 5) is concurrent to (7, 12)
37. DJIT+ We remember vector clocks for: • Each lock release L m = (t 0 , ... , t n ) • Last memory read, R x = (t 0 , …, t n ) • Last memory write W x = (t 0 , …, t n ) • Goroutine vector clock, C x = (t 0 , …, t n )
38. DJIT+ 4 0 4 0 5 0 5 0 5 0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 4 0 4 0 4 0 0 0 4 0 4 0 4 0 4 8 C 0 C 1 L m W x w(x) w(x) unlock (m) lock (m)
39. DJIT+ 4 0 4 0 4 0 4 0 4 0 0 8 0 8 0 8 0 8 0 8 0 0 0 0 0 0 0 0 0 0 0 0 4 0 4 0 4 0 RACE C 0 C 1 L m W x w(x) w(x)
40. FastTrack 4 0 4 0 5 0 5 0 5 0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 4 0 4 0 4 0 0@0 C 0 C 1 L m W x w(x) w(x) unlock (m) lock (m) 4@0 4@0 4@0 8@1
41. FastTrack 4 0 4 0 4 0 4 0 4 0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 0 0 0 0 0 0 0@0 C 0 C 1 L m R x r(x) r(x) 4@0 4@0 4@0 4 8
42. ThreadSanitizer v2
43. Shadow word pool Write goroutine 1 10@1 0:2 W 10 0 0 Memory mapped Fixed size pool 64 bits
44. Shadow word pool Read goroutine 2 10@1 0:2 W 20@2 4:8 R 5 20 0
45. Shadow word pool Read goroutine 3 10@1 0:2 W 20@2 4:8 R 40@3 0:4 R RACE found 5 0 40
46. Summary • What data race is • Most common mistakes • How Go’s -race works
47. Summary var Cnt int64 func Run(amount int) { for i := 0; i < amount; i++ { val := atomic.LoadInt64(&Cnt) val ++ atomic.StoreInt64(&Cnt, val) } }
48. Q & A @nmiculinic neven.miculinic@gmail.com
49. References • https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm • https://golang.org/doc/articles/race_detector.html • https://golang.org/ref/mem • ""go test -race" Under the Hood" by Kavya Joshi • https://blog.golang.org/race-detector • https://danluu.com/concurrency-bugs/ • ThreadSanitizer – data race detection in practice google paper • FastTrack: Efficient and Precise Dynamic Race Detection • AddressSanitizer/ThreadSanitizer for Linux Kernel and userspace. • [DJIT+ algo] MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs • On the Complexity of Event Ordering for Shared-Memory Parallel Program Executions (Netzer, Miller, 1990)
50. Appendix A - go -race tunables From: https://golang.org/doc/articles/race_detector.html#Options GORACE="option1=val1 option2=val2" The options are: • log_path (default stderr): • exitcode (default 66): • strip_path_prefix (default ""): • history_size (default 1). • halt_on_error (default 0):
51. Appendix B - compiler explorer Inspecting golang assembly generation: https://go.godbolt.org/
52. Appendix C - Shared-Variable Shenanigans From: Is Parallel Programming Hard, And, If So, What Can You Do About It?, Paul E. McKenney • Load/Store tearing • Load/Store fusing • Code reordering • Invented stores/loads