Goroutine Patterns

5b4d39ac5dbb89a5cb09d118a0264661?s=47 koya fukushi
August 22, 2017
2k

Goroutine Patterns

5b4d39ac5dbb89a5cb09d118a0264661?s=128

koya fukushi

August 22, 2017
Tweet

Transcript

  1. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 1/52 Goroutine Patterns 2017/8/18 #cm_osaka Koya

    Fukushi Eureka, Inc. Developer
  2. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 2/52 Introduction

  3. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 3/52 What is the Goroutine? https://golang.org/doc/e

    ective_go.html#goroutines (https://golang.org/doc/e ective_go.html#goroutines) They're called goroutines because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
  4. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 4/52 What is the Goroutine? Google

    翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語(スレッド、コルーチン、プロセスなど)が不正確な意味を伝えるため 、それら はgoroutine と呼ばれています。 ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチン と同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかり ません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレー ジを割り当て(および解放)することで成長します。
  5. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 5/52 What is the Goroutine? Google

    翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語(スレッド、コルーチン、プロセスなど)が不正確な意味を伝えるため、それら はgoroutine と呼ばれています。 ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチン と同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかり ません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレー ジを割り当て(および解放)することで成長します。
  6. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 6/52 What is the Goroutine? 既存の並列プログラミングのモデルとは別物

    OS のスレッド上で多重化されている 詳しくは morsmachine.dk/go-scheduler (https://morsmachine.dk/go-scheduler) blog.nindalf.com/how-goroutines-work/ (http://blog.nindalf.com/how-goroutines-work/)
  7. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 7/52 What is the Goroutine? はじめの一歩

    package main import ( "fmt" "time" ) func main() { for i := 0; i < 10; i++ { go func(i int) { fmt.Println("Hello, 大阪!", i) }(i) } time.Sleep(time.Second) } Run
  8. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 8/52 Communicating

  9. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 9/52 Communicating https://golang.org/doc/e ective_go.html#sharing (https://golang.org/doc/e ective_go.html#sharing)

    Do not communicate by sharing memory; instead, share memory by communicating. Google 翻訳 (Google%E7%BF%BB%E8%A8%B3) メモリを共有して通信してはいけません。かわりに、通信によってメモリを共有してくださ い。
  10. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 10/52 Communicating Go の並行プログラミングモデルは CSP(Communicating sequential

    processes) がルーツ channel を介してメッセージを受け渡し(メッセージパッシング)
  11. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 11/52 Communicating package main import "fmt"

    func main() { ch := make(chan int) const n = 10 for i := 0; i < n; i++ { go func(i int) { ch <- i }(i) } for i := 0; i < n; i++ { fmt.Println(" 会 ", <-ch) } } Run
  12. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 12/52 Patterns

  13. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 13/52 Returns a channel channel は第一級オブジェクト

    channel は関数の戻り値にできる func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } c := Say(" 串 食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run
  14. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 14/52 Multiplexing 複数のchannel を使用する このままだとchannel 毎に同期的

    func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } daruma := Say(" 串 食 ") tengu := Say(" 串 食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-daruma) fmt.Printf("You say: %s\n", <-tengu) } fmt.Println("疲 !") Run
  15. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 15/52 Fan-in talks.golang.org/2012/concurrency.slide#28 (https://talks.golang.org/2012/concurrency.slide#28) 複数のchanne を1つにまとめる

    完全に非同期的 func FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { c <- <-input1 } }() go func() { for { c <- <-input2 } }() return c } daruma := Say(" 串 食 ") tengu := Say(" 串 食 ") c := FanIn(daruma, tengu) for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run
  16. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 16/52 Select 複数のchannel からの入出力を制御 同時の場合ランダムな順で実行 入出力がない場合ブロック

    default 句があり、入出力が検知できなかったときに、default 句が直ちに実行される select { case v1 := <-c1: fmt.Printf("received %v from c1\n", v1) case v2 := <-c2: fmt.Printf("received %v from c2\n", v1) case c3 <- 23: fmt.Printf("sent %v to c3\n", 23) default: fmt.Printf("no one was ready to communicate\n") }
  17. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 17/52 Fan-in Again 一つのgoroutine にまとめる func

    FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { select { case s := <-input1: c <- s case s := <-input2: c <- s } } }() return c } Run
  18. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 18/52 Daisy-chain talks.golang.org/2012/concurrency.slide#40 (https://talks.golang.org/2012/concurrency.slide#40) package main

    import "fmt" func f(left, right chan int) { left <- 1 + <-right } func main() { const n = 10000 leftmost := make(chan int) right := leftmost left := leftmost for i := 0; i < n; i++ { right = make(chan int) go f(left, right) left = right } go func(c chan int) { c <- 1 }(right) fmt.Println(<-leftmost) } Run
  19. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 19/52 Restricted Concurrency goroutine の最小のスタックサイズは 2048

    バイトととても軽量
  20. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 20/52 Restricted Concurrency だからといって闇雲にgo ステートメントで多重化していると… for

    i := 0; ; i++ { go Log(i) time.Sleep(time.Nanosecond) } func Log(i int) { fmt.Fprintf(ioutil.Discard, "%d", i) time.Sleep(5 * time.Second) }
  21. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 21/52 Restricted Concurrency 2017/08/12 21:33:32 Mem

    = 1740800B 2017/08/12 21:33:32 Goroutines: 33 2017/08/12 21:33:33 Mem = 233392376B 2017/08/12 21:33:33 Goroutines: 83218 2017/08/12 21:33:34 Mem = 439296272B 2017/08/12 21:33:34 Goroutines: 158467 2017/08/12 21:33:35 Mem = 677046464B 2017/08/12 21:33:35 Goroutines: 243806 2017/08/12 21:33:36 Mem = 923726016B 2017/08/12 21:33:36 Goroutines: 336502
  22. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 22/52 Restricted Concurrency 際限なくリソースが消費される goroutine の数を制御する必要がある

  23. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 23/52 Bu ered Channels make でchannel

    を作成するときに、第2引数にchannel のlength を指定することができる (Bu ered Channels) 空のときに送信しようとするとブロックする 一杯のときに受信しようとするとブロックする ch := make(chan int, 100) tour.golang.org/concurrency/3 (https://tour.golang.org/concurrency/3)
  24. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 24/52 Restricted Concurrency ticket 方式 func

    makeTickets(n int) chan bool { var tickets = make(chan bool, n) for i := 0; i < n; i++ { tickets <- true } return tickets } const maxNum = 50 tickets := makeTickets(maxNum) for i := 0; ; i++ { i := i <-tickets go func() { Log(i) tickets <- true }() time.Sleep(time.Nanosecond) } Run
  25. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 25/52 Restricted Concurrency 2017/08/12 20:21:40 Mem

    = 1740800B 2017/08/12 20:21:40 Goroutines: 31 2017/08/12 20:21:41 Mem = 1740800B 2017/08/12 20:21:41 Goroutines: 52 2017/08/12 20:21:42 Mem = 1740800B 2017/08/12 20:21:42 Goroutines: 52 2017/08/12 20:21:43 Mem = 1740800B 2017/08/12 20:21:43 Goroutines: 52 2017/08/12 20:21:44 Mem = 1740800B 2017/08/12 20:21:44 Goroutines: 52 2017/08/12 20:21:45 Mem = 1740800B 2017/08/12 20:21:45 Goroutines: 52 ※ 2 つ多いのは、main 関数とトレース用の関数
  26. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 26/52 Restricted Concurrency worker 方式 for

    i := 0; ; i++ { c <- i } const maxNum = 50 var c = make(chan int) for i := 0; i < maxNum; i++ { go func() { for { Log(<-c) } }() } Run
  27. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 27/52 Restricted Concurrency 2017/08/12 21:41:11 Mem

    = 1740800B 2017/08/12 21:41:11 Goroutines: 52 2017/08/12 21:41:12 Mem = 1740800B 2017/08/12 21:41:12 Goroutines: 52 2017/08/12 21:41:13 Mem = 1740800B 2017/08/12 21:41:13 Goroutines: 52 2017/08/12 21:41:14 Mem = 1740800B 2017/08/12 21:41:14 Goroutines: 52 2017/08/12 21:41:15 Mem = 1740800B 2017/08/12 21:41:15 Goroutines: 52 2017/08/12 21:41:16 Mem = 1740800B 2017/08/12 21:41:16 Goroutines: 52
  28. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 28/52 Wait 実行結果は? package main import

    "fmt" func main() { for i := 0; i < 5; i++ { i := i go fmt.Printf("hogehoge %d\n", i) } } Run
  29. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 29/52 WaitGroup goroutine が終わるのを待つための構造体 Add で待機するgoroutine

    分のカウンタをセット Done でカウンタをデクリメント Wait でカウンタが0 になるまでブロック var wg sync.WaitGroup wg.Add(10) wg.Done() wg.Wait()
  30. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 30/52 Wait package main import (

    "fmt" "sync" ) func main() { wg := new(sync.WaitGroup) for i := 0; i < 5; i++ { i := i wg.Add(1) go func() { fmt.Printf("hogehoge %d\n", i) wg.Done() }() } wg.Wait() } Run
  31. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 31/52 Goroutine Leak goroutine はGC の対象にならない

    自分で開放する必要がある for i := 0; i < 50; i++ { go LaunchWorker(c) } func LaunchWorker(c chan int) { for { select { case v <- c: // do something } } }
  32. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 32/52 Quit Channel var ( c

    = make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }
  33. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 33/52 Timeout golang.org/pkg/time/#After (https://golang.org/pkg/time/#After) d が経過した後にtime.Time

    を送信するchannel を返す func After(d Duration) <-chan Time
  34. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 34/52 Timeout select の中で作成すると各処理に対してtimeout を設定できる func

    DoSomething(c chan string) { for { select { case v := <-c: fmt.Println(v) case <-time.After(500 * time.Millisecond): fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  35. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 35/52 Timeout select の外で作成するとループ全体に対してtimeout を設定できる func

    DoSomething(c chan string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  36. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 36/52 Ticker golang.org/pkg/time/#Tick (https://golang.org/pkg/time/#Tick) d が経過する毎にtime.Time

    を送信するchannel を返す c := time.Tick(1 * time.Minute) for now := range c { fmt.Printf("%v %s\n", now, statusUpdate()) }
  37. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 37/52 Ticker 基本形 func TickTack() <-chan

    time.Time { c := make(chan time.Time) go func() { tick := time.Tick(100 * time.Millisecond) for now := range tick { c <- now } }() return c } c := TickTack() for i := 0; i < 10; i++ { fmt.Println(<-c) } Run
  38. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 38/52 Ticker 複数Ticker func TickTack() <-chan

    Time { c := make(chan Time) go func() { tick0, tick1 := time.Tick(100*time.Millisecond), time.Tick(200*time.Millisecond) for { select { case t := <-tick0: c <- Time{t, "interval 100ms"} case t := <-tick1: c <- Time{t, "interval 200ms"} } } }() return c } c := TickTack() for i := 0; i < 10; i++ { t := <-c fmt.Println(t.msg, t.t) } Run
  39. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 39/52 Context golang.org/pkg/context/ (https://golang.org/pkg/context/) タイムアウトやキャンセル用のシグナルを運ぶ構造体 go1.7

    から標準パッケージの仲間入り
  40. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 40/52 Quit Channel Again var (

    c = make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }
  41. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 41/52 Quit Channel Again var (

    c = make(chan int) ctx, cancel = context.WithCancel(context.Background()) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(ctx, c) } cancel() fmt.Println("That was refreshing") func LaunchWorker(ctx context.Context, c chan int) { for { select { case <-c: // do something case <-ctx.Done(): return } } }
  42. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 42/52 Timeout Again func DoSomething(c chan

    string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  43. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 43/52 Timeout Again func DoSomething(ctx context.Context,

    c chan string) { for { select { case v := <-c: fmt.Println(v) case <-ctx.Done(): fmt.Println("timed out") return } } } c := make(chan string, 10) ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) defer cancel() go DoSomething(ctx, c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  44. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 44/52 Shared Memory メモリ共有で通信したいときもある Singleton 、Con

    g などなど channel はちょっと大げさだな、、
  45. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 45/52 Race Condition 単純にメモリを共有するとレースコンディションが起きる package main

    var n = 0 func main() { for i := 0; i < 1000; i++ { go func() { n++ }() } } Run
  46. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 46/52 Race Condition $ go run

    -race goroutine_patterns/race_condition.go ================== WARNING: DATA RACE Read at 0x000000101b38 by goroutine 6: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x3d Previous write at 0x000000101b38 by goroutine 5: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x59 Goroutine 6 (running) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 Goroutine 5 (finished) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 ================== Found 1 data race(s) exit status 66
  47. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 47/52 Mutex golang.org/pkg/sync/#Mutex (https://golang.org/pkg/sync/#Mutex) 排他ロック用の構造体 m

    := new(sync.Mutex) m.Lock() u = NewUser() m.Unlock()
  48. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 48/52 Mutex package main import "sync"

    var n = 0 func main() { m := new(sync.Mutex) for i := 0; i < 1000; i++ { go func() { m.Lock() n++ m.Unlock() }() } } Run
  49. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 49/52 Summary Returns a channel Fan-in

    Daisy-chain Restricted Concurrency Wait (WaitGroup) Quit Channel Timeout Ticker Context Mutex
  50. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 50/52 Links Advanced Go Concurrency Patterns

    talks.golang.org/2013/advconc.slide (https://talks.golang.org/2013/advconc.slide) Go Concurrency Patterns talks.golang.org/2012/chat.slide (https://talks.golang.org/2012/chat.slide) Go Concurrency Patterns: Pipelines and cancellation blog.golang.org/pipelines (https://blog.golang.org/pipelines) LearnConcurrency github.com/golang/go/wiki/LearnConcurrency (https://github.com/golang/go/wiki/LearnConcurrency)
  51. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 51/52 Thank you Koya Fukushi Eureka,

    Inc. Developer https://github.com/Kooooya/ (https://github.com/Kooooya/)
  52. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 52/52