Pub/Sub with concurrent Go(Un)ConferenceʢGo͋Μ͜ʣLTେձ 6kg @takashabe

whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • ޿ࠂ഑৴ϓϩμΫτΛGoͰ࡞͍ͬͯ·͢ • Goྺ3೥͘Β͍ • ޷͖ͳඪ४ύοέʔδ͸ `go` ύοέʔδͰ͢ • Macͷ`g`Ωʔ͕յΕ͍ͯͯGo͕ଧͪͮΒͯ͘ࠔ͍ͬͯ·͢

Cloud Pub/Sub Λ࢖ͬͯಘͨ Subscriber Workerͷ஌ݟΛ঺հ

Cloud Pub/Sub is Կ • GCPͷpub/subαʔϏε • Subscriberͷ࣮૷ʹ͍͔ͭ͘ύλʔϯ͕͋Δ • Pull • worker͕pub/sub APIΛcallͯ͠ϝοηʔδΛ ϑΣον͢Δ • Push • worker͕HTTPαʔόͱͯ͠ಈ࡞͢Δ • pub/sub͕ϝοηʔδΛPOSTϦΫΤετ͢Δ

Cloud Pub/Sub is Կ • GCPͷpub/subαʔϏε • Subscriberͷ࣮૷ʹ͍͔ͭ͘ύλʔϯ͕͋Δ • Pull • worker͕pub/sub APIΛcallͯ͠ϝοηʔδΛ ϑΣον͢Δ • Push • worker͕HTTPαʔόͱͯ͠ಈ࡞͢Δ • pub/sub͕ϝοηʔδΛPOSTϦΫΤετ͢Δ

Cloud Pub/Sub is Կ https://speakerdeck.com/hlts2/sub-worker-framework- implementation?slide=6 ΑΓҾ༻

ࣄྫ 1 1ϝοηʔδͷॲཧʹ͕͔͔࣌ؒͬͯ٧·Δ

1ϝοηʔδͷॲཧʹ͕͔͔࣌ؒͬͯ٧·Δ • 1ϝοηʔδ౰ͨΓͷॲཧ͕࣌ؒ௕͍ • ͔ͭಛఆͷSubscriber worker͕େྔͷϝοηʔδΛॲ ཧ͠Α͏ͱͯ͠͠·͏

• Receive Settingͷ࠷దԽ • goroutineͰฒྻॲཧ

Receive Settingͷ࠷దԽ https://github.com/golang/dep/blob/master/docs/assets/DigbyShadows.png ΑΓҾ༻ • σϑΥϧτ • MaxOutstandingMessages: 1000 • NumGoroutines: 1 →1 worker͕େྔͷϝοηʔδΛอ࣋ͯ͠͠·͏ type ReceiveSettings struct { ... MaxOutstandingMessages int // ϝϞϦ಺ʹஔ͍͓ͯ͘ϝοηʔδ਺ NumGoroutines int // ϝοηʔδΛॲཧ͢Δworker goroutine }

goroutineͰฒྻॲཧ • IOό΢ϯυͳॲཧͳͲɺฒྻԽग़དྷΔͱ͜ΖΛͻͨ͢ ΒฒྻԽ

ࣄྫ2 ͋ΔଐੑΛχΞϦΞϧλΠϜͰҰճ͚ͩߋ৽ ͍ͨ͠

͋ΔଐੑΛχΞϦΞϧλΠϜͰҰճ͚ͩߋ৽͍ͨ͠ • ޿ࠂ͕ݟΒΕͨ࣌ɺͦͷ޿ࠂʹඥͮ͘ଐੑΛߋ৽ • ݫີʹຖճߋ৽͢Δඞཁ͸ͳ͍ • 1෼͝ͱʹߋ৽͢ΔΑ͏ʹͨ͠ • σʔλετΞͷ࠷ऴߋ৽೔࣌Ͱ੍ޚ • ॲཧதʹෳ਺ϦΫΤετ͕͋Δͱແବͳߋ৽͕૸Δ

golang.org/x/sync/singleflight Ͱॏෳߋ৽Λආ͚Δ

golang.org/x/sync/singleflight https://play.golang.org/p/h_lCoqJDp9s package main import ( "fmt" "sync" "time" "golang.org/x/sync/singleflight" ) func main() { var ( sg singleflight.Group wg sync.WaitGroup ) for i := 0; i < 10; i++ { wg.Add(1) go func() { // "key" ͝ͱʹಉ࣮࣌ߦΛߜΔ now, _, shared := sg.Do("key", func() (interface{}, error) { time.Sleep(time.Millisecond * 100) return time.Now(), nil }) fmt.Printf("%v: shared: %t\n", now, shared) wg.Done() }() } wg.Wait() }

golang.org/x/sync/singleflight 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true 2009-11-10 23:00:00.1 +0000 UTC m=+0.100000001: shared: true time.Now() ͷ݁Ռ͕ἧ͍ͬͯΔ

ࣄྫ3 όʔετੑ͕͋ΔSubscriberͷAPIίʔϧͰ όοΫΤϯυ͕ࢮ͵

όʔετੑ͕͋ΔSubscriberͷAPIίʔϧͰόοΫΤϯυ͕ࢮ͵ • ϝοηʔδΛड͚ͯόοΫΤϯυͷgRPCαʔόʹϦΫ ΤετΛૹΔαʔϏε • Publishྔ͕֎෦αʔϏεґଘ • ීஈ͸3,000RPSఔ౓ • ͨ·ʹ20,000RPS͘Β͍

golang.org/x/time/rate ͰBackpressureΛ࣮૷͢Δ

golang.org/x/time/rate https://play.golang.org/p/HsnbmPfUZ_A package main import ( "context" "fmt" "sync" "time" "golang.org/x/time/rate" ) func main() { limit := rate.Every(time.Second / 1) // τʔΫϯͷཷ·Δ଎౓ limiter := rate.NewLimiter(limit, 1) // limitΛݩʹlimiterΛੜ੒ɻୈ2Ҿ਺͸τʔΫϯͷ࠷େஷଂྔ(burst) var wg sync.WaitGroup wg.Add(1) go func() { for i := 0; i < 10; i++ { limiter.Wait(context.Background()) fmt.Println(time.Now()) } wg.Done() }() wg.Wait() }

golang.org/x/time/rate 2009-11-10 23:00:00 +0000 UTC m=+0.000000001 2009-11-10 23:00:01 +0000 UTC m=+1.000000001 2009-11-10 23:00:02 +0000 UTC m=+2.000000001 2009-11-10 23:00:03 +0000 UTC m=+3.000000001 2009-11-10 23:00:04 +0000 UTC m=+4.000000001 2009-11-10 23:00:05 +0000 UTC m=+5.000000001 2009-11-10 23:00:06 +0000 UTC m=+6.000000001 2009-11-10 23:00:07 +0000 UTC m=+7.000000001 2009-11-10 23:00:08 +0000 UTC m=+8.000000001 2009-11-10 23:00:09 +0000 UTC m=+9.000000001 େମ1ඵͣͭॲཧ͞Ε͍ͯΔ

·ͱΊ • Cloud Pub/SubΛGoͰ࢖ͬͯૺ۰ͨ͠ඇಉظॲཧܥͷ ࣄྫʹ͍ͭͯ঺հͨ͠ • Pub/Subʹݶͬͨ࿩Ͱ͸ͳ͍ • `golang.org/x/…` ύοέʔδ͸syncΛ͸͡Ίɺศརͳ ΋ͷ͕ଟ͍ͷͰͨ·ʹ८ճ͢Δͱָ͍͠ • golang.org/x/net/netutil ͱ͔΋޷͖