(Sync)testing Concurrent Code with Confidence

Testing concurrent code has always been a balance between tests being either hacky, flaky or slow. While the testing/synctest package made it possible for us to more effectively write idiomatic, easy-to-read tests for our concurrent programs that are both reliable and fast, it also came with a set of constraints.

In this talk, we’ll look at exactly what synctest can and cannot do, the underlying reasons behind its limitations, and how to avoid them. Finally, using a real-world example, we’ll see how proper use can push us towards robust system design we can feel confident about. In short, you’ll know precisely what to do to make the most of synctest, and why that will make your application code better.

Transcript

1. (Sync)testing Concurrent Code with Confidence Alexander Baygeldin @ Evil Martians
2. fi my rst quick & easy Go project
3. my rst quick & easy Go project fi new stack
4. my rst quick & easy Go projec fi deadlines new stack
5. my rst q easy Go fi SLOs deadlines new stack
6. Next steps: 1. Pray. 2. Do lots of testing.
7. Next steps: 1. Pray. 2. Do lots of testing. concurrency
8. I know how to test concurrent code!
9. I know how to test concurrent code...
10. ✨ testing/synctest ✨
11. “Non-deterministic tests have two problems, rstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite.” fi — Martin Fowler
12. func Greet(name string) string { greetings := []string{"Hello", "Hi"} idx := rand.Intn(len(greetings)) // Nondeterministic! return fmt.Sprintf("%s, %s!", greetings[idx], name) } func TestGreet(t *testing.T) { // Flaky! want := "Hi, Bob!" if got := Greet("Bob"); got != want { t.Fatalf("got %q, want %q", got, want) } } Why is my test nondeterministic? 1. Random number generation.
13. func Greet(name string) string { if time.Now().Hour() < 12 { // Nondeterministic! return "Good morning, " + name + "!" } return "Good afternoon, " + name + "!" } func TestGreet(t *testing.T) { // Flaky! want := "Good morning, Alice!" if got := Greet("Alice"); got != want { t.Fatalf("got %q, want %q", got, want) } } Why is my test nondeterministic? 1. Random number generation. 2. Unpredictable outside world.
14. func Greet(people map[string]struct{}) string { keys := []string{} for key := range people { // Nondeterministic! keys = append(keys, key) } return "Hi, " + strings.Join(keys, " & ") + "!" } func TestGreet(t *testing.T) { // Flaky! people := map[string]struct{}{ "Alice": {}, "Bob": {}, } want := "Hi, Alice & Bob!" fi if got := Greet(people); got != want { t.Fatalf("got %q, want %q", got, want) } } Why is my test nondeterministic? 1. Random number generation. 2. Unpredictable outside world. 3. Unspeci ed language behavior.
15. “A programmer had a problem. He thought to himself, "I know, I'll solve it with threads!". has Now problems. two he” — Lao Tzu, probably
16. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) go func() { batch := make([]string, 0, batchSize) for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } }() - - fi - } return out, func(userID int) { go func() { names < fetchUserName(userID) }() } Why is my test nondeterministic? 1. Random number generation. 2. Unpredictable outside world. 3. Unspeci ed language behavior. 4. Unreliable order of execution.
17. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) go func() { batch := make([]string, 0, batchSize) for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } }() - - fi - } return out, func(userID int) { go func() { names < fetchUserName(userID) }() } Why is my test nondeterministic? 1. Random number generation. 2. Unpredictable outside world. 3. Unspeci ed language behavior. 4. Unreliable order of execution.
18. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) go func() { batch := make([]string, 0, batchSize) for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } }() return out, func(userID int) { go func() { names < fetchUserName(userID) }() - - fi - } } Why is my test nondeterministic? 1. Random number generation. 2. Unpredictable outside world. 3. Unspeci ed language behavior. 4. Unreliable order of execution.
19. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) go func() { batch := make([]string, 0, batchSize) for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } }() return out, func(userID int) { go func() { names < fetchUserName(userID) }() - - - - } } func TestBatchGreeter(t *testing.T) { // Flaky! greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" } if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) }
20. flaky tests func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice time.Sleep(100 * time.Millisecond) greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" - time.Sleep if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }
21. flaky tests func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice time.Sleep(100 * time.Millisecond) greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" time.Sleep if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } } - eww! an ugly hack!
22. func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan string) // ... return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { defer close(done) names < }() - - - return done - - - - } } fetchUserName(userID) func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) go func() { < greet(1) // ID=1 is Alice < greet(2) // ID=2 is Bob }() want := "Hi, Alice & Bob!" if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }
23. func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan string) // ... return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { defer close(done) names < fetchUserName(userID) }() - - - - - - } } return done func TestBatchGreeterDoesNotGreet( t *testing.T, ) { greetings, greet := BatchGreeter(2) < greet(1) // ID=1 is Alice select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } }
24. func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan string) // ... return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { defer close(done) names < fetchUserName(userID) }() < greet(1) // ID=1 is Alice select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } } how do you know that something did not happen? - - - - - - } } return done func TestBatchGreeterDoesNotGreet( t *testing.T, ) { greetings, greet := BatchGreeter(2)
25. func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan string) // ... go func() { defer close(done) names < fetchUserName(userID) }() - - - return done - - - < greet(1) // ID=1 is Alice time.Sleep(100 * time.Millisecond) return out, func(userID int) < chan struct{} { done := make(chan struct{}) } } func TestBatchGreeterNotGreet(t *testing.T) { greetings, greet := BatchGreeter(2) select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } }
26. func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan string) // ... return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { defer close(done) names < fetchUserName(userID) }() - - - - - - - } } return done func TestBatchGreeterNotGreet(t *testing.T) { greetings, greet := BatchGreeter(2) < greet(1) // ID=1 is Alice select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) case < time.After(100 * time.Millisecond): } }
27. type request struct { name string done chan struct{} } func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan request) go func() { batch := make([]string, 0, batchSize) for request := range names { batch = append(batch, request.name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } close(request.done) } }() return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { names < request{ name: fetchUserName(userID), done: done, } }() - - - } - - - - } return done func TestBatchGreeterNotGreet(t *testing.T) { greetings, greet := BatchGreeter(2) < greet(1) // ID=1 is Alice select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } }
28. type request struct { name string done chan struct{} } func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan request) go func() { batch := make([]string, 0, batchSize) for request := range names { batch = append(batch, request.name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } close(request.done) } }() return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { names < request{ name: fetchUserName(userID), done: done, } }() - - - } - - - - } return done func TestBatchGreeterNotGreet(t *testing.T) { greetings, greet := BatchGreeter(2) < greet(1) // ID=1 is Alice select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } } how do you know it actually works?
29. type request struct { name string done chan struct{} } func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan request) go func() { batch := make([]string, 0, batchSize) for request := range names { batch = append(batch, request.name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } close(request.done) } }() return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { names < request{ name: fetchUserName(userID), done: done, } }() - - - - - - - } } - return done func TestBatchGreeterNotGreet(t *testing.T) { greetings, greet := BatchGreeter(2) select { case < greet(1): // ID=1 is Alice case < time.After(time.Second): t.Fatal("timed out waiting for Alice") } select { case got := < greetings: t.Fatalf("unexpected greeting: %q", got) default: } }
30. type request struct { name string done chan struct{} } func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan request) go func() { batch := make([]string, 0, batchSize) go func() { batch := make([]string, 0, batchSize) }() for request := range names { batch = append(batch, request.name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } }() return out, func(userID int) { go func() { names < fetchUserName(userID) }() } } return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { names < request{ name: fetchUserName(userID), done: done, } }() - - - - - - - } - } close(request.done) } return done
31. type request struct { name string done chan struct{} } func BatchGreeter(batchSize int) ( < chan string, func(int) < chan struct{}, ) { out := make(chan string) names := make(chan request) func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) go func() { batch := make([]string, 0, batchSize) }() for name := range names { batch = append(batch, name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } } go func() { batch := make([]string, 0, batchSize) was it worth it? for request := range names { batch = append(batch, request.name) if len(batch) == batchSize { out < "Hi, " + strings.Join(batch, " & ") + "!" batch = batch[:0] } 🤔🤔🤔 }() return out, func(userID int) { go func() { names < fetchUserName(userID) }() } } return out, func(userID int) < chan struct{} { done := make(chan struct{}) go func() { names < request{ name: fetchUserName(userID), done: done, } }() - - - - - - - } - } close(request.done) } return done
34. Instrumentation: 1. Complicated. 2. Fragile.
35. Instrumentation: time.Sleep: 1. Complicated. 1. 2. Fragile. Slow.
36. Quiesce nce [ k w i ˈ ɛ s ə n s ] n o u n The state of being at rest, quiet, still, inactive or motionless.
37. Q time
38. greet("Alice") Q time
39. greet("Alice") some computation happening Q Q time
40. greet("Alice") greet("Bob") some computation happening Q Q time
41. greet("Alice") greet("Bob") some computation happening Q some computation happening Q time Q
42. greet("Alice") greet("Bob") some computation happening - Q < greetings some computation happening Q time Q
43. time.Sleep(100 * time.Millisecond) greet("Alice") greet("Bob") some computation happening Q Q time
44. fl aky 👎 👎 less slow 👎 time.Sleep more
45. func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice greet(1) // ID=1 is Alice time.Sleep(100 * time.Millisecond) synctest.Wait() want := "Hi, Alice & Bob!" want := "Hi, Alice & Bob!" if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } } if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } } - - greet(2) // ID=2 is Bob greet(2) // ID=2 is Bob
46. func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice synctest.Wait() greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" - if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }
47. func TestBatchGreeter(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice synctest.Wait() greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" - if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }
48. func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice synctest.Wait() greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) - }
49. func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice synctest.Wait() greet(2) // ID=2 is Bob want := "Hi, Alice & Bob!" if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) - }
50. Quiesce nce* * s y n c t e s t s t y l e The state when all goroutines in the bubble are durably blocked.
51. Durably blocking operations • time.Sleep • sync.WaitGroup.Wait • sync.Cond.Wait • send/receive to/from a bubbled channel • send/receive to/from a nil channel • select{} • select where all cases are durably blocked synctest.Wait() returns only when all goroutines are blocked in one of these ways
52. Durably blocking operations • time.Sleep • sync.WaitGroup.Wait • sync.Cond.Wait • send/receive to/from a bubbled channel • send/receive to/from a nil channel • select{} • select where all cases are durably blocked synctest.Wait() returns only when all goroutines are blocked in one of these ways
53. Durably blocking operations • time.Sleep • sync.WaitGroup.Wait • sync.Cond.Wait • send/receive to/from a bubbled channel • send/receive to/from a nil channel • select{} • select where all cases are durably blocked synctest.Wait() returns only when all goroutines are blocked in one of these ways
54. Durably blocking operations • time.Sleep • sync.WaitGroup.Wait • sync.Cond.Wait • send/receive to/from a bubbled channel • send/receive to/from a nil channel • select{} • select where all cases are durably blocked synctest.Wait() returns only when all goroutines are blocked in one of these ways
55. Durably blocking operations • time.Sleep • sync.WaitGroup.Wait • sync.Cond.Wait • send/receive to/from a bubbled channel • send/receive to/from a nil channel • select{} • select where all cases are durably blocked synctest.Wait() returns only when all goroutines are blocked in one of these ways
56. func Test(t *testing.T, f func(*testing.T)) { bubble := &synctestBubble{ done: false, // Whether the main goroutine has exited waiting: false, // Whether a synctest.Wait call is pending running: 1, // Number of goroutines not durably blocked } currentGoroutine.bubble = bubble defer func() { currentGoroutine.bubble = nil }() go f(t) // Start the main goroutine! // ... }
57. func Test(t *testing.T, f func(*testing.T)) { bubble := &synctestBubble{ done: false, // Whether the main goroutine has exited waiting: false, // Whether a synctest.Wait call is pending running: 1, // Number of goroutines not durably blocked } currentGoroutine.bubble = bubble defer func() { currentGoroutine.bubble = nil }() go f(t) // Start the main goroutine! // ... }
58. func Test(t *testing.T, f func(*testing.T)) { bubble := &synctestBubble{ done: false, // Whether the main goroutine has exited waiting: false, // Whether a synctest.Wait call is pending running: 1, // Number of goroutines not durably blocked } currentGoroutine.bubble = bubble defer func() { currentGoroutine.bubble = nil }() go f(t) // Start the main goroutine! // ... }
59. When a durable operation happens, Go runtime updates the bubble! func Test(t *testing.T, f func(*testing.T)) { bubble := &synctestBubble{ done: false, // Whether the main goroutine has exited waiting: false, // Whether a synctest.Wait call is pending running: 1, // Number of goroutines not durably blocked } currentGoroutine.bubble = bubble defer func() { currentGoroutine.bubble = nil }() go f(t) // Start the main goroutine! // ... }
60. Non-durably blocking operations • syscalls, cgo calls, anything that isn’t Go fi • I/O ( les, pipes, network connections, etc.)
61. Proceed with caution ⚠ Using the network. ⚠ Interacting with external processes.
62. Proceed with caution ⚠ Using the network. ⚠ Interacting with external processes. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { // ... return out, func(userID int) { go func() { - - } } names < }() fetchUserName(userID)
63. Proceed with caution ⚠ Using the network. ⚠ Interacting with external processes. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { // ... return out, func(userID int) { go func() { } } names < }() fetchUserName(userID) - - just make sure this returns
64. func GreetServer(conn net.Conn) error { name, err := bufio.NewReader(conn).ReadString('!') if err != nil { return err } } _, err = fmt.Fprintf(conn, "Hi, %s!", name[:len(name)-1]) return err func GreetClient(conn net.Conn, name string) (string, error) { if _, err := fmt.Fprintf(conn, "%s!", name); err != nil { return "", err } } return bufio.NewReader(conn).ReadString('!')
65. func GreetServer(conn net.Conn) error { name, err := bufio.NewReader(conn).ReadString('!') if err != nil { return err } } _, err = fmt.Fprintf(conn, "Hi, %s!", name[:len(name)-1]) return err func GreetClient(conn net.Conn, name string) (string, error) { if _, err := fmt.Fprintf(conn, "%s!", name); err != nil { return "", err } } return bufio.NewReader(conn).ReadString('!') func TestGreetServerClient(t *testing.T) { synctest.Test(t, func(*testing.T) { serverConn, clientConn := net.Pipe() defer clientConn.Close() defer serverConn.Close() }) } // ... use a fake network connection!
66. Non-durably blocking operations • syscalls, cgo calls, anything that isn’t Go • I/O ( les, pipes, network connections, etc.) • sync.Mutex.Lock and sync.RWMutex.Lock ??? • Runtime instrumentation is not free. fi • Mutexes are not usually held for long periods of time.
67. type ThreadSafeGreeter struct { mu sync.Mutex } type ThreadSafeGreeter struct { mu fakemutex.Mutex }
68. //go:build !fakemutex package fakemutex import "sync" type ThreadSafeGreeter struct { mu sync.Mutex } type Mutex = sync.Mutex type ThreadSafeGreeter struct { mu fakemutex.Mutex }
69. //go:build !fakemutex package fakemutex import "sync" type ThreadSafeGreeter struct { mu sync.Mutex } type Mutex = sync.Mutex //go:build fakemutex package fakemutex type Mutex struct { /* ... */ } func (m *Mutex) Lock() { /* ... */ } func (m *Mutex) Unlock() { /* ... */ } func (m *Mutex) TryLock() bool { /* ... */ } type ThreadSafeGreeter struct { mu fakemutex.Mutex }
70. type Mutex struct { once sync.Once ch chan struct{} } func (m *Mutex) init() { m.once.Do(func() { m.ch := make(chan struct{}, 1) m.ch < struct{}{} }) } YO DAWG I HEARD YOU LIKE MUTEXES func (m *Mutex) Lock() { m.init() < m.ch } func (m *Mutex) Unlock() { m.init() select { case m.ch < struct{}{}: default: panic("sync: unlock of unlocked mutex") } } - - - // ... SO I PUT CHANNELS IN YOUR MUTEXES SO YOU CAN USE MUTEXES WHILE YOU USE MUTEXES
71. No-no's 🚫 Operating on a bubbled channel, timer, or ticker from outside the bubble. 🚫 Calling Add or Go on a bubbled sync.WaitGroup from outside the bubble. 🚫 Waking a bubbled goroutine blocked on Cond.Wait from outside the bubble.
72. No-no's 🚫 Operating on a bubbled channel, timer, or ticker from outside the bubble. func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) 🚫 Calling Add or Go on a bubbled greet(1) // ID=1 is Alice synctest.Wait() sync.WaitGroup from outside the bubble. greet(2) // ID=2 is Bob 🚫 Waking a bubbled goroutine blocked on want := "Hi, Alice & Bob!" Cond.Wait from outside the bubble. if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) - }
73. No-no's 🚫 Operating on a bubbled channel, timer, or ticker from outside the bubble. func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) 🚫 Calling Add or Go on a bubbled greet(1) // ID=1 is Alice synctest.Wait() sync.WaitGroup from outside the bubble. greet(2) // ID=2 is Bob 🚫 Waking a bubbled goroutine blocked on want := "Hi, Alice & Bob!" Cond.Wait from outside the bubble. if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) - }
74. No-no's 🚫 Operating on a bubbled channel, timer, or ticker from outside the bubble. func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) 🚫 Calling Add or Go on a bubbled greet(1) // ID=1 is Alice synctest.Wait() sync.WaitGroup from outside the bubble. greet(2) // ID=2 is Bob 🚫 Waking a bubbled goroutine blocked on want := "Hi, Alice & Bob!" Cond.Wait from outside the bubble. - 🚫 Leaking goroutines inside the bubble. if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) }
75. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice go func() { batch := make([]string, 0, batchSize) synctest.Wait() for name := range names { // ... } }() want := "Hi, Alice & Bob!" - - return out, func(userID int) { go func() { names < fetchUserName(userID) }() } } - func TestBatchGreeter(t *testing.T) { greet(2) // ID=2 is Bob if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) } panic: main bubble goroutine has exited but blocked goroutines remain! 😱 😱 😱
76. func BatchGreeter(batchSize int) ( < chan string, func(int), ) { out := make(chan string) names := make(chan string) synctest.Test(t, func(t *testing.T) { greetings, greet := BatchGreeter(2) greet(1) // ID=1 is Alice go func() { batch := make([]string, 0, batchSize) synctest.Wait() for name := range names { // ... } }() want := "Hi, Alice & Bob!" - - return out, func(userID int) { go func() { names < fetchUserName(userID) }() } } - func TestBatchGreeter(t *testing.T) { greet(2) // ID=2 is Bob if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) } }) } panic: main bubble goroutine has exited but blocked goroutines remain! 😱 😱 😱
77. func BatchGreeter(batchSize int) ( < chan string, func(int), func(), ) { out := make(chan string) names := make(chan string) func TestBatchGreeter(t *testing.T) { synctest.Test(t, func(t *testing.T) { greetings, greet, stop := BatchGreeter(2) greet(1) // ID=1 is Alice synctest.Wait() greet(2) // ID=2 is Bob go func() { batch := make([]string, 0, batchSize) stop() // All good now! for name := range names { // ... } want := "Hi, Alice & Bob!" }() - - - return out, func(userID int) { go func() { names < fetchUserName(userID) }() }, func() { close(names) } } }) } if got := < greetings; got != want { t.Fatalf("got %q, want %q", got, want) }
78. The Catch? 👍 Test isolation. 👍 Healthy test pyramid. 👍 Using channels for communication. ⏱ TIME! >synctest's limitations >look inside >good practices
79. THAT MOMENT WHEN synctest.Wait() RETURNS
80. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
81. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
82. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
83. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
84. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
85. func Test(t *testing.T, f func(*testing.T)) { // ... go f(t) for { bubble.waitUntilQuiescent() if bubble.waiting { bubble.resume() // synctest.Wait() returns continue } if !bubble.done && bubble.hasSomethingScheduled() { bubble.advanceTimeUntilSomethingHappens() continue } } break // ... }
86. “You can think of the bubble as simulating an in nitely fast computer: Any amount of computation takes no time.” fi — Damien Neil, author of synctest
87. func TestSynctestUnderstanding(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() mu.Lock() // OK! }) }
88. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
89. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
90. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
91. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
92. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
93. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
94. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
95. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Deadlock! }) }
96. func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Timelock 😱 }) }
97. Do not wait for a future event in a way that prevents it from ever happening! func TestSynctestUnderstanding(t *testing.T) { synctest.Test(t, func(t *testing.T) { var mu sync.Mutex go func() { mu.Lock() defer mu.Unlock() time.Sleep(10 * time.Millisecond) }() synctest.Wait() mu.Lock() // Timelock 😱 }) }
98. My life before synctest 1. I still needed fast feedback... 2. So, I did some refactoring! • How can I maximize coverage... • ...while minimizing time.Sleep?
99. T T B = Transport layer E = Execution policy B = Business logic E E E B B T B B B E B E T
100. E T E E T = Transport layer E = Execution policy B = Business logic T E T E B B B B B B B
101. E time.Sleep(...) T = Transport layer E = Execution policy B = Business logic T + E E T E T E B B B = B B B B still meh
102. T T E T = Transport layer E = Execution policy B = Business logic E E E B B B T B B E B B
103. T T E T = Transport layer E = Execution policy B = Business logic E E E B B B T B B E B B
104. T T E T = Transport layer E = Execution policy B = Business logic time.Sleep(...) + E E E B B B T B B = E B B acceptable
105. T T E T = Transport layer E = Execution policy B = Business logic synctest.Wait() + E E E B B B T B B = E B B 0.010s
106. T T E T = Transport layer E = Execution policy B = Business logic E E E B B B T B B E B B
107. T T E T = Transport layer E = Execution policy B = Business logic high cohesion E E E B B B T B B low coupling E B B
108. Lessons! • Testing with synctest is awesome: not complicated, not fragile, not slow. • A clean design tends to also be synctest-friendly! • If testing concurrent code feels painful, perhaps it's not you. It's the tooling.
109. Lessons! • Testing with synctest is awesome: not complicated, not fragile, not slow. • A clean design tends to also be synctest-friendly! • If testing concurrent code feels painful, perhaps it's not you. It's the tooling. I started building it for Ruby! github.com/baygeldin/synctest-rb
110. THANK YOU! evilmartians.com/chronicles github.com/baygeldin/synctest-rb