Goroutine Patterns

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 1/52 Goroutine Patterns 2017/8/18 #cm_osaka Koya
Fukushi Eureka, Inc. Developer

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 2/52 Introduction

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 3/52 What is the Goroutine? https://golang.org/doc/e
ective_go.html#goroutines (https://golang.org/doc/e ective_go.html#goroutines) They're called goroutines because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 4/52 What is the Goroutine? Google
翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語（スレッド、コルーチン、プロセスなど）が不正確な意味を伝えるため、それらはgoroutine と呼ばれています。ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチンと同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかりません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレージを割り当て（および解放）することで成長します。

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 5/52 What is the Goroutine? Google
翻訳 (Google%E7%BF%BB%E8%A8%B3) 既存の用語（スレッド、コルーチン、プロセスなど）が不正確な意味を伝えるため、それらはgoroutine と呼ばれています。ゴルーチンにはシンプルなモデルがあります。これは同じアドレス空間内の他のゴルーチンと同時に実行される関数です。軽量で、スタックスペースの割り当てよりもコストがかかりません。また、スタックのサイズは小さいので、安価であり、必要に応じてヒープストレージを割り当て（および解放）することで成長します。

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 6/52 What is the Goroutine? 既存の並列プログラミングのモデルとは別物
OS のスレッド上で多重化されている詳しくは morsmachine.dk/go-scheduler (https://morsmachine.dk/go-scheduler) blog.nindalf.com/how-goroutines-work/ (http://blog.nindalf.com/how-goroutines-work/)

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 7/52 What is the Goroutine? はじめの一歩
package main import ( "fmt" "time" ) func main() { for i := 0; i < 10; i++ { go func(i int) { fmt.Println("Hello, 大阪!", i) }(i) } time.Sleep(time.Second) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 8/52 Communicating

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 9/52 Communicating https://golang.org/doc/e ective_go.html#sharing (https://golang.org/doc/e ective_go.html#sharing)
Do not communicate by sharing memory; instead, share memory by communicating. Google 翻訳 (Google%E7%BF%BB%E8%A8%B3) メモリを共有して通信してはいけません。かわりに、通信によってメモリを共有してください。

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 10/52 Communicating Go の並行プログラミングモデルは CSP(Communicating sequential
processes) がルーツ channel を介してメッセージを受け渡し（メッセージパッシング）

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 11/52 Communicating package main import "fmt"
func main() { ch := make(chan int) const n = 10 for i := 0; i < n; i++ { go func(i int) { ch <- i }(i) } for i := 0; i < n; i++ { fmt.Println(" 会 ", <-ch) } } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 12/52 Patterns

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 13/52 Returns a channel channel は第一級オブジェクト
channel は関数の戻り値にできる func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } c := Say(" 串食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 14/52 Multiplexing 複数のchannel を使用するこのままだとchannel 毎に同期的
func Say(msg string) <-chan string { // 送信専用 channel 返 c := make(chan string) go func() { for i := 0; ; i++ { c <- fmt.Sprintf(`%s %d`, msg, i) time.Sleep(100 * time.Millisecond) } }() return c } daruma := Say(" 串食 ") tengu := Say(" 串食 ") for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-daruma) fmt.Printf("You say: %s\n", <-tengu) } fmt.Println("疲 !") Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 15/52 Fan-in talks.golang.org/2012/concurrency.slide#28 (https://talks.golang.org/2012/concurrency.slide#28) 複数のchanne を１つにまとめる
完全に非同期的 func FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { c <- <-input1 } }() go func() { for { c <- <-input2 } }() return c } daruma := Say(" 串食 ") tengu := Say(" 串食 ") c := FanIn(daruma, tengu) for i := 0; i < 5; i++ { fmt.Printf("You say: %s\n", <-c) } fmt.Println("疲 !") Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 16/52 Select 複数のchannel からの入出力を制御同時の場合ランダムな順で実行入出力がない場合ブロック
default 句があり、入出力が検知できなかったときに、default 句が直ちに実行される select { case v1 := <-c1: fmt.Printf("received %v from c1\n", v1) case v2 := <-c2: fmt.Printf("received %v from c2\n", v1) case c3 <- 23: fmt.Printf("sent %v to c3\n", 23) default: fmt.Printf("no one was ready to communicate\n") }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 17/52 Fan-in Again 一つのgoroutine にまとめる func
FanIn(input1, input2 <-chan string) <-chan string { c := make(chan string) go func() { for { select { case s := <-input1: c <- s case s := <-input2: c <- s } } }() return c } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 18/52 Daisy-chain talks.golang.org/2012/concurrency.slide#40 (https://talks.golang.org/2012/concurrency.slide#40) package main
import "fmt" func f(left, right chan int) { left <- 1 + <-right } func main() { const n = 10000 leftmost := make(chan int) right := leftmost left := leftmost for i := 0; i < n; i++ { right = make(chan int) go f(left, right) left = right } go func(c chan int) { c <- 1 }(right) fmt.Println(<-leftmost) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 19/52 Restricted Concurrency goroutine の最小のスタックサイズは 2048
バイトととても軽量

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 20/52 Restricted Concurrency だからといって闇雲にgo ステートメントで多重化していると… for
i := 0; ; i++ { go Log(i) time.Sleep(time.Nanosecond) } func Log(i int) { fmt.Fprintf(ioutil.Discard, "%d", i) time.Sleep(5 * time.Second) }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 21/52 Restricted Concurrency 2017/08/12 21:33:32 Mem
= 1740800B 2017/08/12 21:33:32 Goroutines: 33 2017/08/12 21:33:33 Mem = 233392376B 2017/08/12 21:33:33 Goroutines: 83218 2017/08/12 21:33:34 Mem = 439296272B 2017/08/12 21:33:34 Goroutines: 158467 2017/08/12 21:33:35 Mem = 677046464B 2017/08/12 21:33:35 Goroutines: 243806 2017/08/12 21:33:36 Mem = 923726016B 2017/08/12 21:33:36 Goroutines: 336502

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 22/52 Restricted Concurrency 際限なくリソースが消費される goroutine の数を制御する必要がある

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 23/52 Bu ered Channels make でchannel
を作成するときに、第２引数にchannel のlength を指定することができる (Bu ered Channels) 空のときに送信しようとするとブロックする一杯のときに受信しようとするとブロックする ch := make(chan int, 100) tour.golang.org/concurrency/3 (https://tour.golang.org/concurrency/3)

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 24/52 Restricted Concurrency ticket 方式 func
makeTickets(n int) chan bool { var tickets = make(chan bool, n) for i := 0; i < n; i++ { tickets <- true } return tickets } const maxNum = 50 tickets := makeTickets(maxNum) for i := 0; ; i++ { i := i <-tickets go func() { Log(i) tickets <- true }() time.Sleep(time.Nanosecond) } Run

= 1740800B 2017/08/12 20:21:40 Goroutines: 31 2017/08/12 20:21:41 Mem = 1740800B 2017/08/12 20:21:41 Goroutines: 52 2017/08/12 20:21:42 Mem = 1740800B 2017/08/12 20:21:42 Goroutines: 52 2017/08/12 20:21:43 Mem = 1740800B 2017/08/12 20:21:43 Goroutines: 52 2017/08/12 20:21:44 Mem = 1740800B 2017/08/12 20:21:44 Goroutines: 52 2017/08/12 20:21:45 Mem = 1740800B 2017/08/12 20:21:45 Goroutines: 52 ※ 2 つ多いのは、main 関数とトレース用の関数

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 26/52 Restricted Concurrency worker 方式 for
i := 0; ; i++ { c <- i } const maxNum = 50 var c = make(chan int) for i := 0; i < maxNum; i++ { go func() { for { Log(<-c) } }() } Run

= 1740800B 2017/08/12 21:41:11 Goroutines: 52 2017/08/12 21:41:12 Mem = 1740800B 2017/08/12 21:41:12 Goroutines: 52 2017/08/12 21:41:13 Mem = 1740800B 2017/08/12 21:41:13 Goroutines: 52 2017/08/12 21:41:14 Mem = 1740800B 2017/08/12 21:41:14 Goroutines: 52 2017/08/12 21:41:15 Mem = 1740800B 2017/08/12 21:41:15 Goroutines: 52 2017/08/12 21:41:16 Mem = 1740800B 2017/08/12 21:41:16 Goroutines: 52

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 28/52 Wait 実行結果は？ package main import
"fmt" func main() { for i := 0; i < 5; i++ { i := i go fmt.Printf("hogehoge %d\n", i) } } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 29/52 WaitGroup goroutine が終わるのを待つための構造体 Add で待機するgoroutine
分のカウンタをセット Done でカウンタをデクリメント Wait でカウンタが0 になるまでブロック var wg sync.WaitGroup wg.Add(10) wg.Done() wg.Wait()

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 30/52 Wait package main import (
"fmt" "sync" ) func main() { wg := new(sync.WaitGroup) for i := 0; i < 5; i++ { i := i wg.Add(1) go func() { fmt.Printf("hogehoge %d\n", i) wg.Done() }() } wg.Wait() } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 31/52 Goroutine Leak goroutine はGC の対象にならない
自分で開放する必要がある for i := 0; i < 50; i++ { go LaunchWorker(c) } func LaunchWorker(c chan int) { for { select { case v <- c: // do something } } }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 32/52 Quit Channel var ( c
= make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 33/52 Timeout golang.org/pkg/time/#After (https://golang.org/pkg/time/#After) d が経過した後にtime.Time
を送信するchannel を返す func After(d Duration) <-chan Time

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 34/52 Timeout select の中で作成すると各処理に対してtimeout を設定できる func
DoSomething(c chan string) { for { select { case v := <-c: fmt.Println(v) case <-time.After(500 * time.Millisecond): fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 35/52 Timeout select の外で作成するとループ全体に対してtimeout を設定できる func
DoSomething(c chan string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 36/52 Ticker golang.org/pkg/time/#Tick (https://golang.org/pkg/time/#Tick) d が経過する毎にtime.Time
を送信するchannel を返す c := time.Tick(1 * time.Minute) for now := range c { fmt.Printf("%v %s\n", now, statusUpdate()) }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 37/52 Ticker 基本形 func TickTack() <-chan
time.Time { c := make(chan time.Time) go func() { tick := time.Tick(100 * time.Millisecond) for now := range tick { c <- now } }() return c } c := TickTack() for i := 0; i < 10; i++ { fmt.Println(<-c) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 38/52 Ticker 複数Ticker func TickTack() <-chan
Time { c := make(chan Time) go func() { tick0, tick1 := time.Tick(100*time.Millisecond), time.Tick(200*time.Millisecond) for { select { case t := <-tick0: c <- Time{t, "interval 100ms"} case t := <-tick1: c <- Time{t, "interval 200ms"} } } }() return c } c := TickTack() for i := 0; i < 10; i++ { t := <-c fmt.Println(t.msg, t.t) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 39/52 Context golang.org/pkg/context/ (https://golang.org/pkg/context/) タイムアウトやキャンセル用のシグナルを運ぶ構造体 go1.7
から標準パッケージの仲間入り

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 40/52 Quit Channel Again var (
c = make(chan int) quit = make(chan bool) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(c, quit) } for i := 0; i < n; i++ { quit <- true } func LaunchWorker(c chan int, quit chan bool) { for { select { case <-c: // do something case <-quit: return } } }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 41/52 Quit Channel Again var (
c = make(chan int) ctx, cancel = context.WithCancel(context.Background()) ) const n = 50 for i := 0; i < n; i++ { go LaunchWorker(ctx, c) } cancel() fmt.Println("That was refreshing") func LaunchWorker(ctx context.Context, c chan int) { for { select { case <-c: // do something case <-ctx.Done(): return } } }

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 42/52 Timeout Again func DoSomething(c chan
string) { timeout := time.After(500 * time.Millisecond) for { select { case v := <-c: fmt.Println(v) case <-timeout: fmt.Println("timed out") return } } } c := make(chan string, 10) go DoSomething(c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 43/52 Timeout Again func DoSomething(ctx context.Context,
c chan string) { for { select { case v := <-c: fmt.Println(v) case <-ctx.Done(): fmt.Println("timed out") return } } } c := make(chan string, 10) ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) defer cancel() go DoSomething(ctx, c) rand.Seed(time.Now().UnixNano()) for i := 0; i < 10; i++ { c <- " 会 " SleepRandam(200) } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 44/52 Shared Memory メモリ共有で通信したいときもある Singleton 、Con
g などなど channel はちょっと大げさだな、、

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 45/52 Race Condition 単純にメモリを共有するとレースコンディションが起きる package main
var n = 0 func main() { for i := 0; i < 1000; i++ { go func() { n++ }() } } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 46/52 Race Condition $ go run
-race goroutine_patterns/race_condition.go ================== WARNING: DATA RACE Read at 0x000000101b38 by goroutine 6: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x3d Previous write at 0x000000101b38 by goroutine 5: main.main.func1() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:8 +0x59 Goroutine 6 (running) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 Goroutine 5 (finished) created at: main.main() /.../go/src/github.com/Kooooya/slides/goroutine_patterns/race_condition.go:9 +0x51 ================== Found 1 data race(s) exit status 66

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 47/52 Mutex golang.org/pkg/sync/#Mutex (https://golang.org/pkg/sync/#Mutex) 排他ロック用の構造体 m
:= new(sync.Mutex) m.Lock() u = NewUser() m.Unlock()

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 48/52 Mutex package main import "sync"
var n = 0 func main() { m := new(sync.Mutex) for i := 0; i < 1000; i++ { go func() { m.Lock() n++ m.Unlock() }() } } Run

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 49/52 Summary Returns a channel Fan-in
Daisy-chain Restricted Concurrency Wait (WaitGroup) Quit Channel Timeout Ticker Context Mutex

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 50/52 Links Advanced Go Concurrency Patterns
talks.golang.org/2013/advconc.slide (https://talks.golang.org/2013/advconc.slide) Go Concurrency Patterns talks.golang.org/2012/chat.slide (https://talks.golang.org/2012/chat.slide) Go Concurrency Patterns: Pipelines and cancellation blog.golang.org/pipelines (https://blog.golang.org/pipelines) LearnConcurrency github.com/golang/go/wiki/LearnConcurrency (https://github.com/golang/go/wiki/LearnConcurrency)

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 51/52 Thank you Koya Fukushi Eureka,
Inc. Developer https://github.com/Kooooya/ (https://github.com/Kooooya/)

8/22/2017 Goroutine Patterns http://go-talks.appspot.com/github.com/Kooooya/slides/goroutine_patterns.slide#30 52/52

Goroutine Patterns

Goroutine Patterns

More Decks by koya fukushi

Featured

Transcript