Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Goroutine Patterns

koya fukushi
August 22, 2017
2.9k

Goroutine Patterns

koya fukushi

August 22, 2017
Tweet

Transcript

  1. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 3/52 What is the Goroutine? https://golang.org/doc/e

    ective_go.html#goroutines (https://golang.org/doc/e ective_go.html#goroutines) They're called goroutines because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
  2. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 4/52 What is the Goroutine? Google

    翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語(スレッド、コルーチン、プロセスなど)が不正確な意味を伝えるため 、それら はgoroutine と呼ばれています。 ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチン と同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかり ません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレー ジを割り当て(および解放)することで成長します。
  3. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 5/52 What is the Goroutine? Google

    翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語(スレッド、コルーチン、プロセスなど)が不正確な意味を伝えるため、それら はgoroutine と呼ばれています。 ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチン と同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかり ません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレー ジを割り当て(および解放)することで成長します。
  4. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 6/52 What is the Goroutine? 既存の並列プログラミングのモデルとは別物

    OS のスレッド上で多重化されている 詳しくは morsmachine.dk/go-scheduler (https://morsmachine.dk/go-scheduler) blog.nindalf.com/how-goroutines-work/ (http://blog.nindalf.com/how-goroutines-work/)
  5. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 7/52 What is the Goroutine? はじめの一歩

    package main import ( "fmt" "time" ) func main() { for i := 0; i < 10; i++ { go func(i int) { fmt.Println("Hello, 大阪!", i) }(i) } time.Sleep(time.Second) } Run
  6. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 9/52 Communicating https://golang.org/doc/e ective_go.html#sharing (https://golang.org/doc/e ective_go.html#sharing)

    Do not communicate by sharing memory; instead, share memory by communicating. Google 翻訳 (Google%E7%BF%BB%E8%A8%B3) メモリを共有して通信してはいけません。かわりに、通信によってメモリを共有してくださ い。
  7. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 11/52 Communicating package main import "fmt"

    func main() { ch := make(chan int) const n = 10 for i := 0; i < n; i++ { go func(i int) { ch <- i }(i) } for i := 0; i < n; i++ { fmt.Println(" 会 ", <-ch) } } Run
  8. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 13/52 Returns a channel channel は第一級オブジェクト

    channel は関数の戻り値にできる func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } c := Say(" 串 食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run
  9. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 14/52 Multiplexing 複数のchannel を使用する このままだとchannel 毎に同期的

    func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } daruma := Say(" 串 食 ") tengu := Say(" 串 食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-daruma) fmt.Printf("You say: %s\n", <-tengu) } fmt.Println("疲 !") Run
  10. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 15/52 Fan-in talks.golang.org/2012/concurrency.slide#28 (https://talks.golang.org/2012/concurrency.slide#28) 複数のchanne を1つにまとめる

    完全に非同期的 func FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { c <- <-input1 } }() go func() { for { c <- <-input2 } }() return c } daruma := Say(" 串 食 ") tengu := Say(" 串 食 ") c := FanIn(daruma, tengu) for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run
  11. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 16/52 Select 複数のchannel からの入出力を制御 同時の場合ランダムな順で実行 入出力がない場合ブロック

    default 句があり、入出力が検知できなかったときに、default 句が直ちに実行される select { case v1 := <-c1: fmt.Printf("received %v from c1\n", v1) case v2 := <-c2: fmt.Printf("received %v from c2\n", v1) case c3 <- 23: fmt.Printf("sent %v to c3\n", 23) default: fmt.Printf("no one was ready to communicate\n") }
  12. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 17/52 Fan-in Again 一つのgoroutine にまとめる func

    FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { select { case s := <-input1: c <- s case s := <-input2: c <- s } } }() return c } Run
  13. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 18/52 Daisy-chain talks.golang.org/2012/concurrency.slide#40 (https://talks.golang.org/2012/concurrency.slide#40) package main

    import "fmt" func f(left, right chan int) { left <- 1 + <-right } func main() { const n = 10000 leftmost := make(chan int) right := leftmost left := leftmost for i := 0; i < n; i++ { right = make(chan int) go f(left, right) left = right } go func(c chan int) { c <- 1 }(right) fmt.Println(<-leftmost) } Run
  14. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 21/52 Restricted Concurrency 2017/08/12 21:33:32 Mem

    = 1740800B 2017/08/12 21:33:32 Goroutines: 33 2017/08/12 21:33:33 Mem = 233392376B 2017/08/12 21:33:33 Goroutines: 83218 2017/08/12 21:33:34 Mem = 439296272B 2017/08/12 21:33:34 Goroutines: 158467 2017/08/12 21:33:35 Mem = 677046464B 2017/08/12 21:33:35 Goroutines: 243806 2017/08/12 21:33:36 Mem = 923726016B 2017/08/12 21:33:36 Goroutines: 336502
  15. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 23/52 Bu ered Channels make でchannel

    を作成するときに、第2引数にchannel のlength を指定することができる (Bu ered Channels) 空のときに送信しようとするとブロックする 一杯のときに受信しようとするとブロックする ch := make(chan int, 100) tour.golang.org/concurrency/3 (https://tour.golang.org/concurrency/3)
  16. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 24/52 Restricted Concurrency ticket 方式 func

    makeTickets(n int) chan bool { var tickets = make(chan bool, n) for i := 0; i < n; i++ { tickets <- true } return tickets } const maxNum = 50 tickets := makeTickets(maxNum) for i := 0; ; i++ { i := i <-tickets go func() { Log(i) tickets <- true }() time.Sleep(time.Nanosecond) } Run
  17. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 25/52 Restricted Concurrency 2017/08/12 20:21:40 Mem

    = 1740800B 2017/08/12 20:21:40 Goroutines: 31 2017/08/12 20:21:41 Mem = 1740800B 2017/08/12 20:21:41 Goroutines: 52 2017/08/12 20:21:42 Mem = 1740800B 2017/08/12 20:21:42 Goroutines: 52 2017/08/12 20:21:43 Mem = 1740800B 2017/08/12 20:21:43 Goroutines: 52 2017/08/12 20:21:44 Mem = 1740800B 2017/08/12 20:21:44 Goroutines: 52 2017/08/12 20:21:45 Mem = 1740800B 2017/08/12 20:21:45 Goroutines: 52 ※ 2 つ多いのは、main 関数とトレース用の関数
  18. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 26/52 Restricted Concurrency worker 方式 for

    i := 0; ; i++ { c <- i } const maxNum = 50 var c = make(chan int) for i := 0; i < maxNum; i++ { go func() { for { Log(<-c) } }() } Run
  19. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 27/52 Restricted Concurrency 2017/08/12 21:41:11 Mem

    = 1740800B 2017/08/12 21:41:11 Goroutines: 52 2017/08/12 21:41:12 Mem = 1740800B 2017/08/12 21:41:12 Goroutines: 52 2017/08/12 21:41:13 Mem = 1740800B 2017/08/12 21:41:13 Goroutines: 52 2017/08/12 21:41:14 Mem = 1740800B 2017/08/12 21:41:14 Goroutines: 52 2017/08/12 21:41:15 Mem = 1740800B 2017/08/12 21:41:15 Goroutines: 52 2017/08/12 21:41:16 Mem = 1740800B 2017/08/12 21:41:16 Goroutines: 52
  20. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 29/52 WaitGroup goroutine が終わるのを待つための構造体 Add で待機するgoroutine

    分のカウンタをセット Done でカウンタをデクリメント Wait でカウンタが0 になるまでブロック var wg sync.WaitGroup wg.Add(10) wg.Done() wg.Wait()
  21. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 30/52 Wait package main import (

    "fmt" "sync" ) func main() { wg := new(sync.WaitGroup) for i := 0; i < 5; i++ { i := i wg.Add(1) go func() { fmt.Printf("hogehoge %d\n", i) wg.Done() }() } wg.Wait() } Run
  22. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 31/52 Goroutine Leak goroutine はGC の対象にならない

    自分で開放する必要がある for i := 0; i < 50; i++ { go LaunchWorker(c) } func LaunchWorker(c chan int) { for { select { case v <- c: // do something } } }
  23. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 32/52 Quit Channel var ( c

    = make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }
  24. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 34/52 Timeout select の中で作成すると各処理に対してtimeout を設定できる func

    DoSomething(c chan string) { for { select { case v := <-c: fmt.Println(v) case <-time.After(500 * time.Millisecond): fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  25. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 35/52 Timeout select の外で作成するとループ全体に対してtimeout を設定できる func

    DoSomething(c chan string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  26. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 37/52 Ticker 基本形 func TickTack() <-chan

    time.Time { c := make(chan time.Time) go func() { tick := time.Tick(100 * time.Millisecond) for now := range tick { c <- now } }() return c } c := TickTack() for i := 0; i < 10; i++ { fmt.Println(<-c) } Run
  27. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 38/52 Ticker 複数Ticker func TickTack() <-chan

    Time { c := make(chan Time) go func() { tick0, tick1 := time.Tick(100*time.Millisecond), time.Tick(200*time.Millisecond) for { select { case t := <-tick0: c <- Time{t, "interval 100ms"} case t := <-tick1: c <- Time{t, "interval 200ms"} } } }() return c } c := TickTack() for i := 0; i < 10; i++ { t := <-c fmt.Println(t.msg, t.t) } Run
  28. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 40/52 Quit Channel Again var (

    c = make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }
  29. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 41/52 Quit Channel Again var (

    c = make(chan int) ctx, cancel = context.WithCancel(context.Background()) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(ctx, c) } cancel() fmt.Println("That was refreshing") func LaunchWorker(ctx context.Context, c chan int) { for { select { case <-c: // do something case <-ctx.Done(): return } } }
  30. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 42/52 Timeout Again func DoSomething(c chan

    string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  31. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 43/52 Timeout Again func DoSomething(ctx context.Context,

    c chan string) { for { select { case v := <-c: fmt.Println(v) case <-ctx.Done(): fmt.Println("timed out") return } } } c := make(chan string, 10) ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) defer cancel() go DoSomething(ctx, c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run
  32. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 46/52 Race Condition $ go run

    -race goroutine_patterns/race_condition.go ================== WARNING: DATA RACE Read at 0x000000101b38 by goroutine 6: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x3d Previous write at 0x000000101b38 by goroutine 5: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x59 Goroutine 6 (running) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 Goroutine 5 (finished) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 ================== Found 1 data race(s) exit status 66
  33. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 48/52 Mutex package main import "sync"

    var n = 0 func main() { m := new(sync.Mutex) for i := 0; i < 1000; i++ { go func() { m.Lock() n++ m.Unlock() }() } } Run
  34. 8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 50/52 Links Advanced Go Concurrency Patterns

    talks.golang.org/2013/advconc.slide (https://talks.golang.org/2013/advconc.slide) Go Concurrency Patterns talks.golang.org/2012/chat.slide (https://talks.golang.org/2012/chat.slide) Go Concurrency Patterns: Pipelines and cancellation blog.golang.org/pipelines (https://blog.golang.org/pipelines) LearnConcurrency github.com/golang/go/wiki/LearnConcurrency (https://github.com/golang/go/wiki/LearnConcurrency)