The secret life of a goroutine

Goroutines are an exciting part of the go language, but what are they? How are they created? How are they paused? How are they resumed?

There are many things that most people don’t know about the lifecycle of the goroutines. If you want to know more about them, this is your talk.

Transcript

2. How are goroutines born?
3. How are goroutines born?
4. The go scheduler
5. The pieces of the scheduler • Status (Idle, running, syscall, gcstop) • Current M • Runnable goroutines • Free gorotuines • Other metadata P (the Processor) src/runtime/runtime2.go:609
6. The pieces of the scheduler • Current goroutine • Current P • Other metadata M (the Machine) src/runtime/runtime2.go:526
7. The pieces of the scheduler • Idle Ms • Idle Ps • Global runnable goroutines • Global free goroutines • Other metadata Sched (the Scheduler) src/runtime/runtime2.go:766
8. The pieces of the scheduler • Stack (2048 bytes) • Program counter • Status • Current M • Wait reason • Other metadata. G (the Goroutine) src/runtime/runtime2.go:407
9. All Ms All Ps All Goroutines The whole picture src/runtime/runtime2.go
10. The birth of a goroutine
11. The goroutine lifecycle Idle Dead Runnable Dead Runnable src/runtime/proc.go:4241 (newproc)
12. All Ms All Ps All Goroutines The whole picture
13. All Ms All Ps All Goroutines The whole picture
14. All Ms All Ps All Goroutines The whole picture
15. All Ms All Ps All Goroutines The whole picture
16. All Ms All Ps All Goroutines The whole picture
17. All Ms All Ps All Goroutines The whole picture
18. All Ms All Ps All Goroutines The whole picture
19. All Ms All Ps All Goroutines The whole picture
20. All Ms All Ps All Goroutines The whole picture
21. All Ms All Ps All Goroutines The whole picture
22. All Ms All Ps All Goroutines The whole picture
23. All Ms All Ps All Goroutines The whole picture
24. All Ms All Ps All Goroutines The whole picture
25. All Ms All Ps All Goroutines The whole picture
26. The life of a goroutine
27. The life of a goroutine Runnable Running Waiting Preempted Copystack Syscall
28. Runnable to running • Scheduler runs • Find a goroutine to run • Assigns it to the M • Mark as Running • Executes the code src/runtime/proc.go:3331 (schedule)
29. Running to waiting • Park itself • Detach from M • Run the scheduler src/runtime/proc.go:364 (gopark)
30. waitReasonZero // "" waitReasonGCAssistMarking // "GC assist marking" waitReasonIOWait // "IO wait" waitReasonChanReceiveNilChan // "chan receive (nil chan)" waitReasonChanSendNilChan // "chan send (nil chan)" waitReasonDumpingHeap // "dumping heap" waitReasonGarbageCollection // "garbage collection" waitReasonGarbageCollectionScan // "garbage collection scan" waitReasonPanicWait // "panicwait" waitReasonSelect // "select" waitReasonSelectNoCases // "select (no cases)" waitReasonGCAssistWait // "GC assist wait" waitReasonGCSweepWait // "GC sweep wait" waitReasonGCScavengeWait // "GC scavenge wait" waitReasonChanReceive // "chan receive" waitReasonChanSend // "chan send" waitReasonFinalizerWait // "finalizer wait" waitReasonForceGCIdle // "force gc (idle)" waitReasonSemacquire // "semacquire" waitReasonSleep // "sleep" waitReasonSyncCondWait // "sync.Cond.Wait" waitReasonSyncMutexLock // "sync.Mutex.Lock" waitReasonSyncRWMutexRLock // "sync.RWMutex.RLock" waitReasonSyncRWMutexLock // "sync.RWMutex.Lock" waitReasonTraceReaderBlocked // "trace reader (blocked)" waitReasonWaitForGCCycle // "wait for GC cycle" waitReasonGCWorkerIdle // "GC worker (idle)" waitReasonGCWorkerActive // "GC worker (active)" waitReasonPreempted // "preempted" waitReasonDebugCall // "debug call" waitReasonGCMarkTermination // "GC mark termination" waitReasonStoppingTheWorld // "stopping the world" Running to waiting src/runtime/runtime2.go:14
31. Running to waiting • GC • Mutex • Semaphore • Channel • Sleep • IO
32. Running to syscall and to running or runnable • On every syscall • entersyscall is executed (moving to Sycall state) • The syscall is executed • exitsyscall is executed (moving back to Running/Runnable) src/runtime/proc.go:3825 (entersyscall) src/runtime/proc.go:3920 (exitsyscall)
33. Running to copystack and back • More stack needed • Change Running to Copystack • Grow the stack • Change back to Running src/runtime/stack.go:964 (newstack)
34. Waiting to runnable • goready is called • Is added to the queue • Try to get a P src/runtime/proc.go:390 (goready)
35. Waiting to runnable • Reactivate a list of goroutines • Mark all of them as runnable • Wakes up all the Ps needed • Add them to the queues src/runtime/proc.go:3255 (injectglist)
36. Waiting to runnable • Change to Waiting • Donʼt need to wait • Change to Runnable • And then to Running rights away src/runtime/proc.go:3487 (park_m)
37. Waiting to runnable • Finding a goroutine • Check in the netpoll • Or wake up tasks for mark assist src/runtime/proc.go:2672 (findRunnable)
38. Running to preempt, waiting and runnable • Preempt flag is set • Change to preempted • Next GC change to waiting • Do the GC scan • Change to Runnable • Add it back to the queue src/runtime/preempt.go:104 (suspendG) src/runtime/preempt.go:257 (resumeG)
39. The goroutine lifecycle Runnable Running Waiting Preempted Copystack Syscall
40. Examples
41. channel The channel example data src/runtime/chan.go:160
42. The channel example channel data src/runtime/chan.go:160
43. The channel example channel data src/runtime/chan.go:457
44. The channel example channel data ready data src/runtime/chan.go:615
45. channel The channel example src/runtime/chan.go:615
46. channel data channel The channel example data channel data channel data ready data channel
47. create The waitgroup example add 3 Spawn Wait Group src/sync/waitgroup.go:43
48. wait The waitgroup example Wait Group src/sync/waitgroup.go:91
49. The waitgroup example Wait Group
50. The waitgroup example Wait Group
51. The waitgroup example Wait Group Ready Done src/sync/waitgroup.go:43
52. The waitgroup example Wait Group
53. Wait Group Wait Group Wait Group create The waitgroup example Add 3 Spawn wait Ready Done Wait Group
54. The death of a goroutine
55. The death of a goroutine • A goroutine finish it work • Change state to Dead • Set most of the goroutine values to zero • Disconnect the goroutine from the M • Add the goroutine to the free list of the P • Call the scheduler src/runtime/proc.go:3616 (goexit0)
56. The whole life
57. The whole picture All Ms All Ps All Goroutines
58. The whole lifecycle Runnable Running Waiting Preempted Copystack Syscall Dead Idle
59. Summary
60. Illustrations • CC-BY • Created by Laura Pareja • http://laurapareja.com
61. What is missing? • The garbage collector • The netpoll • Cgo • Mark assist • Sysmon
62. References • The Go source code • Illustrated Tales of Go Runtime Scheduler: https://www.youtube.com/watch?v=KxOwt6z0FvY • Scheduling In Go: https://www.ardanlabs.com/blog/2018/08/scheduling-in-g o-part1.html
63. Conclusions