Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Golang race detection - Neven Miculinic, KrakenSystems

6e3ea86995d93d35c0fadf2694bca773?s=47 GoDays
January 30, 2019

Golang race detection - Neven Miculinic, KrakenSystems

Golang race detection - Neven Miculinic, KrakenSystems

6e3ea86995d93d35c0fadf2694bca773?s=128

GoDays

January 30, 2019
Tweet

Transcript

  1. Golang race detection Neven Miculinić @MiculinicNeven

  2. None
  3. Counter Example

  4. Counter example var Cnt int func Run(amount int) { for

    i := 0; i < amount; i++ { Cnt++ } }
  5. Counter example func main() { wg := &sync.WaitGroup{} for i

    := 0; i < 10; i++ { wg.Add(1) go func() { Run(1000); wg.Done() }() } wg.Wait(); fmt.Println(Cnt) }
  6. Famous examples

  7. Dirty COW • Race condition in linux kernel • Consequences:

    privilege escalation
  8. Therac-25 • Radiotherapy machine • Race condition between user input

    and system state • Consequences: 6+ accidents, 2+ deaths
  9. Definitions

  10. Data Race Concurrent access to memory location, one of which

    is a write
  11. Concurrent access Concurrent access is one where event ordering isn’t

    known. There’s no happens-before relation
  12. Happens before G1: 0 G2: 0 G1: 1 G1: 2

    G2: 3 G2: 1 G2: 2
  13. Happens before G1: 0 G2: 0 G1: 1 G1: 2

    G2: 3 G2: 1 G2: 2 G1: 3 G2: 4 CHAN RECV CHAN SEND
  14. Happens before G1: 0 G2: 0 G1: 1 G1: 2

    G2: 3 G2: 1 G2: 2 G1: 3 G2: 4 MUTEX LOCK MUTEX UNLOCK
  15. Typical examples

  16. Race on loop counter func main() { var wg sync.WaitGroup

    wg.Add(5) for i := 0; i < 5; i++ { go func() { fmt.Println(i) // Not the 'i' you are looking for. wg.Done() }() } wg.Wait() }
  17. Race on loop counter func main() { var wg sync.WaitGroup

    wg.Add(5) for i := 0; i < 5; i++ { go func(j int) { fmt.Println(j) wg.Done() }(i) } wg.Wait() }
  18. Unprotected global var Cnt int func Inc(amount int) { Cnt

    += amount } func main() { go Inc(10) go Int(100) }
  19. Unprotected global var Cnt int var m = &sync.Mutex{} func

    Inc(amount int) { m.Lock() defer m.Unlock() Cnt += amount }
  20. Primitive unprotected variable var Cnt int64 func Inc(amount int64) {

    atomic.AddInt64(&Cnt, amount) }
  21. Violating go memory model (aka being smart) var a string

    var done bool func setup() { a = "hello, world" done = true } func main() { go setup() for !done { } fmt.Print(a) }
  22. Synchronization primitives • Sync package ◦ Mutex ◦ Atomics ◦

    ... • channels
  23. Detecting race conditions

  24. Detecting race conditions • Go test -race • Go build

    -race • Go install -race
  25. ThreadSanitizer v2 • Battle proven library • Runtime slowdown 2x-20x.

    • Memory overhead 5x-10x. • Requires CGO, 64bit OS • No false positives
  26. Compiler instrumentation movq "".x+8(SP), AX incq (AX) func Inc(x *int)

    { *x++ }
  27. func Inc(x *int) { *x++ } Compiler instrumentation movq "".x+32(SP),

    AX movq AX, (SP) call runtime.raceread(SB) movq "".x+32(SP), AX movq (AX), CX movq CX, ""..autotmp_4+8(SP) movq AX, (SP) call runtime.racewrite(SB) incq AX movq "".x+32(SP), CX movq AX, (CX)
  28. func Inc(x *int) { *x++ } Compiler instrumentation ... call

    runtime.raceread(SB) ... call runtime.racewrite(SB) ...
  29. How race detection works?

  30. Race detection complexity Determining if arbitrary program contains potential data

    races is NP-hard. [Netzer&Miller 1990]
  31. Race detection design choices • Static analysis • Dynamic analysis

    ◦ On-the-fly ◦ post-mortem
  32. • Happens before • Lockset based • Hybrid models Dynamic

    race detection
  33. Happens before Vector clocks

  34. Vector clocks G1: 0 G2: 0 G1: 0 G2: 0

    G1: 1 G2: 0 G1: 2 G2: 0 G1: 0 G2: 3 G1: 0 G2: 1 G1: 0 G2: 2
  35. Vector clocks G1: 0 G2: 0 G1: 0 G2: 0

    G1: 1 G2: 0 G1: 2 G2: 0 G1: 0 G2: 3 G1: 0 G2: 1 G1: 0 G2: 2 G1: 3 G2: 3 G1: 0 G2: 4 MUTEX LOCK MUTEX UNLOCK
  36. Vector clocks • Partially ordered set • (4, 11) <

    (10, 12) → happens before • (12, 5) is concurrent to (7, 12)
  37. DJIT+ We remember vector clocks for: • Each lock release

    L m = (t 0 , ... , t n ) • Last memory read, R x = (t 0 , …, t n ) • Last memory write W x = (t 0 , …, t n ) • Goroutine vector clock, C x = (t 0 , …, t n )
  38. DJIT+ 4 0 4 0 5 0 5 0 5

    0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 4 0 4 0 4 0 0 0 4 0 4 0 4 0 4 8 C 0 C 1 L m W x w(x) w(x) unlock (m) lock (m)
  39. DJIT+ 4 0 4 0 4 0 4 0 4

    0 0 8 0 8 0 8 0 8 0 8 0 0 0 0 0 0 0 0 0 0 0 0 4 0 4 0 4 0 RACE C 0 C 1 L m W x w(x) w(x)
  40. FastTrack 4 0 4 0 5 0 5 0 5

    0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 4 0 4 0 4 0 0@0 C 0 C 1 L m W x w(x) w(x) unlock (m) lock (m) 4@0 4@0 4@0 8@1
  41. FastTrack 4 0 4 0 4 0 4 0 4

    0 0 8 0 8 0 8 4 8 4 8 0 0 0 0 0 0 0 0 0 0 0@0 C 0 C 1 L m R x r(x) r(x) 4@0 4@0 4@0 4 8
  42. ThreadSanitizer v2

  43. Shadow word pool Write goroutine 1 10@1 0:2 W 10

    0 0 Memory mapped Fixed size pool 64 bits
  44. Shadow word pool Read goroutine 2 10@1 0:2 W 20@2

    4:8 R 5 20 0
  45. Shadow word pool Read goroutine 3 10@1 0:2 W 20@2

    4:8 R 40@3 0:4 R RACE found 5 0 40
  46. Summary • What data race is • Most common mistakes

    • How Go’s -race works
  47. Summary var Cnt int64 func Run(amount int) { for i

    := 0; i < amount; i++ { val := atomic.LoadInt64(&Cnt) val ++ atomic.StoreInt64(&Cnt, val) } }
  48. Q & A @MiculinicNeven neven@krakensystems.co

  49. References • https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm • https://golang.org/doc/articles/race_detector.html • https://golang.org/ref/mem • ""go test

    -race" Under the Hood" by Kavya Joshi • https://blog.golang.org/race-detector • https://danluu.com/concurrency-bugs/ • ThreadSanitizer – data race detection in practice google paper • FastTrack: Efficient and Precise Dynamic Race Detection • AddressSanitizer/ThreadSanitizer for Linux Kernel and userspace. • [DJIT+ algo] MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs • On the Complexity of Event Ordering for Shared-Memory Parallel Program Executions (Netzer, Miller, 1990)