Goroutine Pool in 17LIVE

Golang Taipei Gathering #56 x 旋轉拍賣
https://www.meetup.com/golang-taipei-meetup/events/277374228/

# 講者
Viney Shih @ 17Live

# 題目
那些年我們追的 Goroutine Pool

# 議程摘要
Go 語言的開發者一定對 goroutine 不陌生，它非常的簡單容易操作，也能快速達到高併發的效果，另一方面它的 stack 很小，相較於傳統的 thread，context switch 速度更快。然而沒有限制的使用它也會造成系統負擔。

本次主題會分享 17LIVE 在 2018 年曾經面對的開播通知問題，進而發展自己的 goroutine pool 的心路歷程。

Transcript

1. Goroutine Pool in 17LIVE Viney Shih, Sr. Engineering Manager viney@17.live
2. Agenda • System issues in 2018 • Analysis and requirement • Goroutine Pool (Goroutines) in 17LIVE • Conclusion • Q&A
3. System issues in 2018 • Push notification (PN) issue for live stream - Delayed PN - Lost PN
4. System issues in 2018 (cont.) • Complex process during sending PN - Personal preferences - Streamer’s blacklist - System rules - Millions of fans
5. System overview App LB goapi notif notif notif notif … LB
7. Analysis • Memory limitation - The default starting size for a goroutine's stack in 1.4 is 2048 bytes - On 4GB memory, this limits the max number of goroutines to less than 2 million
8. Analysis (cont.) • System performance - Allocate the stacks and destroy them high frequently - Adjust the stack size dynamically by factor 2 (this is how go runtime works), and copy the previous in the new one
9. Requirements • Limit the maximum number of goroutines • Recycle and reuse goroutines • Need a queue to be the buffer of goroutines
10. Manage goroutines
11. • Queue - Buffered channel - Unbuffered channel Scale-Per-Request Autoscaling
12. Schedule()
13. • Recycler
14. Example
15. Conclusion • Goroutines in 17LIVE save the PN issue significantly - No PN lost for hot streamers - Views receive PN within two minutes • We also do other optimization with goroutines - One time batch jobs: Batch
16. • Github (https://github.com/viney-shih/goroutines) • Medium (https://medium.com/17media- tech/%E9%82%A3%E4%BA%9B%E5%B9%B4%E6%88%91%E5%80%91%E8 %BF%BD%E7%9A%84-goroutine-pool-e8d211757ee)