Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Goroutine Pool in 17LIVE

98f8e6125c052c4ac3fcc94f97919371?s=47 Viney
April 27, 2021

Goroutine Pool in 17LIVE

Golang Taipei Gathering #56 x 旋轉拍賣
https://www.meetup.com/golang-taipei-meetup/events/277374228/

# 講者
Viney Shih @ 17Live

# 題目
那些年我們追的 Goroutine Pool

# 議程摘要
Go 語言的開發者一定對 goroutine 不陌生,它非常的簡單容易操作,也能快速達到高併發的效果,另一方面它的 stack 很小,相較於傳統的 thread,context switch 速度更快。然而沒有限制的使用它也會造成系統負擔。

本次主題會分享 17LIVE 在 2018 年曾經面對的開播通知問題,進而發展自己的 goroutine pool 的心路歷程。

98f8e6125c052c4ac3fcc94f97919371?s=128

Viney

April 27, 2021
Tweet

Transcript

  1. Goroutine Pool in 17LIVE Viney Shih, Sr. Engineering Manager viney@17.live

  2. Agenda • System issues in 2018 • Analysis and requirement

    • Goroutine Pool (Goroutines) in 17LIVE • Conclusion • Q&A
  3. System issues in 2018 • Push notification (PN) issue for

    live stream - Delayed PN - Lost PN
  4. System issues in 2018 (cont.) • Complex process during sending

    PN - Personal preferences - Streamer’s blacklist - System rules - Millions of fans
  5. System overview App LB goapi notif notif notif notif …

    LB
  6. None
  7. Analysis • Memory limitation - The default starting size for

    a goroutine's stack in 1.4 is 2048 bytes - On 4GB memory, this limits the max number of goroutines to less than 2 million
  8. Analysis (cont.) • System performance - Allocate the stacks and

    destroy them high frequently - Adjust the stack size dynamically by factor 2 (this is how go runtime works), and copy the previous in the new one
  9. Requirements • Limit the maximum number of goroutines • Recycle

    and reuse goroutines • Need a queue to be the buffer of goroutines
  10. Manage goroutines

  11. • Queue - Buffered channel - Unbuffered channel Scale-Per-Request Autoscaling

  12. Schedule()

  13. • Recycler

  14. Example

  15. Conclusion • Goroutines in 17LIVE save the PN issue significantly

    - No PN lost for hot streamers - Views receive PN within two minutes • We also do other optimization with goroutines - One time batch jobs: Batch
  16. • Github (https://github.com/viney-shih/goroutines) • Medium (https://medium.com/17media- tech/%E9%82%A3%E4%BA%9B%E5%B9%B4%E6%88%91%E5%80%91%E8 %BF%BD%E7%9A%84-goroutine-pool-e8d211757ee)