Subee: Pub/Sub Worker Framework Implementation

talked on Go Conference 2019 Spring
https://gocon.jp/2019-spring-sessions#c2-s-subee-pubsub-worker-framework-implementation

Transcript

1. Subee Pub/Sub Worker Framework Implementation Go Conference Spring - @hlts2
2. $ whoami • Hiroto Funakoshi • @hlts2 • ۦ͚ग़͠ͷ৽ଔΤϯδχΞ • αʔϏεͷΞʔΩςΫνϟ΍ج൫։ൃʹڵຯ͕͋Δ
3. $ whoami • Hiroto Funakoshi • @hlts2 • ۦ͚ग़͠ͷ৽ଔΤϯδχΞ • αʔϏεͷΞʔΩςΫνϟ΍ج൫։ൃʹڵຯ͕͋Δ WantedlyΠϯλʔϯͰ։ൃͨ͠OSSʹ͍ͭͯൃද͠·͢
4. ໨࣍ • Pub/Subͱ͸ʁ • Subeeͷ঺հ • Subeeͷར༻ํ๏
5. Pub/Sub?
6. Pub/Sub • ඇಉظϝοηʔδϯάύϥμΠϜͷҰछ • ϝοηʔδૹ৴ऀ͕ड৴ऀΛ૝ఆͤͣʹ
 ϝοηʔδΛૹΕΔΑ͏ϓϩάϥϜ͞Εͨ΋ͷ
 
 
 
 
 
 Go Publisher
 ʢProducerʣ Ruby Python Go Go Subscriber
 ʢConsumerʣ Pub/Sub
 ʢBrokerʣ
7. ͳͥඞཁʁ
8. Ϣʔεέʔε https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
9. Ϣʔεέʔε https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
10. Ϣʔεέʔε https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
11. Ϣʔεέʔε https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios QueueʢඇಉظAPIʣ
12. Ϣʔεέʔε https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios Πϕϯτ఻ൖ
13. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server
14. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server HTTP or gRPC Push ґཔ
15. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Push ґཔ × 100ສ HTTP or gRPC
16. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Push ґཔ × 100ສ HTTP or gRPC Push Server͕൵໐Λ͋͛Δ
17. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub
18. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub Push ґཔ × 100ສ
19. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub ࣗ෼ͷϖʔεͰॲཧ Push ґཔ × 100ສ
20. ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub ࣗ෼ͷϖʔεͰॲཧ Push ґཔ × 100ສ Push ServerͷෛՙΛ཈͑Δ͜ͱ͕ग़དྷΔ
21. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service
22. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service ProfileΛߋ৽ HTTP or gRPC
23. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service Profile͕ߋ৽ HTTP or gRPC ґଘ͢ΔαʔϏεΛ֮͑Δͷ͕େม
24. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service
25. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service ProfileΛߋ৽
26. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service ProfileΛߋ৽
27. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service ProfileΛߋ৽ υϝΠϯڥք͕Ͱ͖Δ
28. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service ProfileΛߋ৽ υϝΠϯڥք͕Ͱ͖Δ ґଘ͢ΔαʔϏεΛ஌Βͳͯ͘΋͍͍
29. ࣮૷ྫʢCloud Pub/Subʣ ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
30. ࣮૷ྫʢCloud Pub/Subʣ ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
31. ࣮૷ྫʢCloud Pub/Subʣ ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) • ϝοηʔδΛਖ਼͘͠ॲཧग़དྷͨࣄΛPub/Subʹ఻͑Δ ‣ Pub/Sub͸ϝοηʔδΛ࡟আ͢Δ
 ɹɹɹɹϝοηʔδ͕࠶ૹ͞Εͳ͍
32. ࣮૷ྫʢCloud Pub/Subʣ ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) • ϝοηʔδΛਖ਼͘͠ॲཧग़དྷͳ͔ͬͨࣄΛPub/Subʹ఻͑Δ ‣ Pub/Sub͸ϝοηʔδΛ࡟আ͠ͳ͍
 ɹɹɹɹɹɹɹϝοηʔδ͕࠶ૹ͞ΕΔ
33. Pub/Subಋೖͷ՝୊఺ • Worker࣮૷͕ඞཁ ‣ ୯Ұɾෳ਺ϝοηʔδΛॲཧ ‣ Consume࿙Εͳ͘Graceful Shutdown ‣ ϝτϦΫεͱΤϥʔϨϙʔςΟϯά • ຊ࣭Ͱ͸ͳ͍ίʔυʢج൫ͷ࣮૷ʣ͕૿͑Δ • ϚΠΫϩαʔϏεΛΨϯΨϯ࡞Δ૊৫Ͱ͸ 
 Workerͷίʔυ͕ڞ௨Խग़དྷ͍ͯͳ͍ͱࣅͨΑ͏ͳίʔυ͕ࢄཚ
34. Pub/Subಋೖͷ՝୊఺ • Worker࣮૷͕ඞཁ ‣ ୯Ұɾෳ਺ϝοηʔδΛॲཧ ‣ Consume࿙Εͳ͘Graceful Shutdown ‣ ϝτϦΫεͱΤϥʔϨϙʔςΟϯά • ຊ࣭Ͱ͸ͳ͍ίʔυʢج൫ͷ࣮૷ʣ͕૿͑Δ • ϚΠΫϩαʔϏεΛΨϯΨϯ࡞Δ૊৫Ͱ͸ 
 Workerͷίʔυ͕ڞ௨Խग़དྷ͍ͯͳ͍ͱࣅͨΑ͏ͳίʔυ͕ࢄཚ Կ౓΋࣮૷͢Δͷʁ
35. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { msg.Ack() return } if err := update(post); err != nil { msg.Nack() return } msg.Ack() })
36. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { msg.Ack() return } if err := update(post); err != nil { msg.Nack() return } msg.Ack() }) LoggerΛ͍Ε͍ͨʂʂ
37. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer log.Printf("End consume process. time: %v", time.Since(start)) var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
38. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer log.Printf("End consume process. time: %v", time.Since(start)) var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
39. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer log.Printf("End consume process. time: %v", time.Since(start)) var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) Panic RecoverೖΕ͍ͨʂʂ
40. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
41. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
42. ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) Error ReportingೖΕ͍ͨʂʂ
43. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
44. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
45. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { log.Printf("Revoery from panic: %v", p) } }() var post Post if err := json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err := update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) StatsΛऔΓ͍ͨʂʂ
46. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
47. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
48. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() }) ෳ਺ϝοηʔδΛॲཧ͍ͨ͠
50. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
51. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
52. ͦ΋ͦ΋୭͕࣮૷͢Δͷʁ
53. ΞϓϦέʔγϣϯΤϯδχΞʁ
54. Want to
55. ͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱ͸ͳ͍෦෼ʣ
56. ͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱ͸ͳ͍෦෼ʣ Worker࣮૷
 ʴ
 ΞϓϦέʔγϣϯ࣮૷
57. ͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱ͸ͳ͍෦෼ʣ Worker࣮૷
 ʴ
 ΞϓϦέʔγϣϯ࣮૷ ελʔτμογϡ͕஗͘ͳΔ
58. ཉ͔ͬͨ͠΋ͷ • ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
 ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳ࢓૊Έ
 
 
 
 
 
 

59. ཉ͔ͬͨ͠΋ͷ • ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
 ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳ࢓૊Έ
 
 
 
 
 
 
 ΞϓϦέʔγϣϯ࣮૷ ج൫࣮૷ ج൫ͱΞϓϦέʔγϣϯίʔυ͕෼཭Ͱ͖͍ͯΔੈք
60. ཉ͔ͬͨ͠΋ͷ • ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
 ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳ࢓૊Έ
 
 
 
 
 
 
 ΞϓϦέʔγϣϯ࣮૷ ج൫࣮૷ ج൫ͱΞϓϦέʔγϣϯίʔυ͕෼཭Ͱ͖͍ͯΔੈք
61. ཉ͔ͬͨ͠΋ͷ • ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
 ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳ࢓૊Έ
 
 
 
 
 
 
 ΞϓϦέʔγϣϯ࣮૷ ج൫࣮૷ ج൫ͱΞϓϦέʔγϣϯίʔυ͕෼཭Ͱ͖͍ͯΔੈք ࡞Δͧ
62. github.com/wantedly/subee
63. • ಡΈํ͸αϏΟʙʢα΂ΣʙͰ͸ͳ͍…ʣ • Go੡ • WantedlyͰ̑ͭͷϚΠΫϩαʔϏεͰಈ͍͍ͯΔ • Pub/Sub WorkerΛॻ͔ͣʹϝοηʔδॲཧΛ
 Մೳʹ͢ΔWorkerϑϨʔϜϫʔΫ Subscribe + bee = Subee
64. Subeeಋೖͷൺֱʢલʣ <=====> Worker Code Application Code Application layer ΞϓϦέʔγϣϯίʔυͱج൫͕ີ݁߹ Pub/Sub
65. Subeeಋೖͷൺֱʢޙʣ Worker Code Application Code Framework Application ΞϓϦέʔγϣϯίʔυͱج൫ͷ੾Γ཭͠ <=====> Pub/Sub
66. Subeeಋೖͷൺֱʢޙʣ Worker Code Application Code Framework Application ΞϓϦέʔγϣϯίʔυͱج൫ͷ੾Γ཭͠ <=====> Pub/Sub
67. ಋೖલͷίʔυ
68. raven.SetDSN("https://<key>:<secret>@sentry.io/<project>") ctx := context.Background() err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { start := time.Now() log.Println("Receive message") log.Printf("Start consume process") statsHandler.HandleProcess(ctx, &Receive{ BeginTime: start, }) defer func() { log.Printf("End consume process. time: %v", time.Since(start)) if p := recover(); p != nil { raven.CaptureError(err, nil) log.Printf("Revoery from panic: %v", p) } }() consumeBeginTime := time.Now() var err error ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{}) defer func() { statsHandler.HandleProcess(ctx, &ConsumeEnd{ BeginTime: consumeBeginTime, EndTime: time.Now(), Error: err, }) statsHandler.HandleProcess(ctx, &End{ MsgCount: 1, BeginTime: start, EndTime: time.Now(), }) } var post Post if err = json.Unmarshal(msg.Data, &post); err != nil { raven.CaptureError(err, nil) log.Printf("could not decode message data: %#v", msg) msg.Ack() return } if err = update(post); err != nil { raven.CaptureError(err, nil) log.Printf("[ID %d] could not update: %v", post.Id, err) msg.Nack() return } msg.Ack() })
69. ಋೖޙͷίʔυ
70. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx); type consumer struct {} func NewConsumer() subee.Consumer { return new(consumer) } func (c *consumer) Consume(ctx context.Context, msg subee.Message) error { return nil }
71. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx); type consumer struct {} func NewConsumer() subee.Consumer { return new(consumer) } func (c *consumer) Consume(ctx context.Context, msg subee.Message) error { return nil } ConsumerΛੜ੒
72. type consumer struct {} func NewConsumer() subee.Consumer { return new(consumer) } func (c *consumer) Consume(ctx context.Context, msg subee.Message) error { return nil } subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx); Frameworkʹొ࿥͢Δ
73. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
74. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
75. Subscriberͷੜ੒ • Pub/Sub͔ΒϝοηʔδΛऔಘ͢Δ੹຿Λ࣋ͭ • Cloud Pub/SubͷSubscriber࣮૷͸
 ϑϨʔϜϫʔΫʹ༻ҙ • ઃఆ஋ΛຒΊΔ͚ͩͰ
 Subscriberͷར༻͕Մೳ • ॊೈͳઃఆΛ͢ΔͨΊʹ
 Function OptionͰΧελϚΠζՄೳ subscriber, err := cloudpubsub.CreateSubscriber(ctx, “cloudpub-app", "subscription-id") if err != nil { panic(err) }
76. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
77. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
78. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
79. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx); type consumer struct {} func NewConsumer() subee.Consumer { return new(consumer) } func (c *consumer) Consume(ctx context.Context, msg subee.Message) error { return nil }
80. Consumerͷੜ੒ • ୯ҰϝοηʔδΛॲཧ͢Δ
 ੹຿Λ࣋ͭ • ΞϓϦέʔγϣϯΤϯδχΞ͕
 ࣮ࡍʹ࣮૷͢Δ෦෼ type consumer struct {} func NewConsumer() subee.Consumer { return new(consumer) } func (c *consumer) Consume(ctx context.Context, msg subee.Message) error { return nil }
81. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
82. BatchConsumerͷੜ੒ • ෳ਺ϝοηʔδΛॲཧ͢Δ
 ੹຿Λ࣋ͭ • ΞϓϦέʔγϣϯΤϯδχΞ͕
 ࣮ࡍʹ࣮૷͢Δ෦෼ subee.BatchConsumerFunc( func(ctx context.Context, msg []subee.Message) error { return nil }, )
83. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
84. ACK/NACKͷ੍ޚ • ConsumerɾBatchConsumerͷ໭Γ஋Ͱ
 ACK/NACK͞ΕΔ ‣ ໭Γ஋͕Τϥʔͷ৔߹ɺNACK͞ΕΔ
85. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
86. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
87. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
88. InterceptorɾStatsHandlerͷੜ੒ • Interceptor ‣ Consumeલޙʹհೖ
 Ͱ͖Δ ‣ Logger, Recovery͸
 Frameworkʹ༻ҙ • StatsHandler ‣ ܭଌܥͷͨΊʹར༻ subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ), subee.WithStatsHandler( NewStatshandler(), )
89. ܭଌܥͷͨΊͷStatsHandler Channel͔ΒϝοηʔδΛड͚औΔ ϝοηʔδΛChannelʹྲྀ͢ Pub/Sub͔ΒϝοηʔδΛऔಘ Consume։࢝લ Consume։࢝ Consumeऴྃ औΓग़͔͠ΒConsume·ͰͷҰ࿈͕ऴྃ ࣌ܥྻ
90. ܭଌܥͷͨΊͷStatsHandler Channel͔ΒϝοηʔδΛड͚औΔ ϝοηʔδΛChannelʹྲྀ͢ Pub/Sub͔ΒϝοηʔδΛऔಘ Consume։࢝લ Consume։࢝ Consumeऴྃ औΓग़͔͠ΒConsume·ͰͷҰ࿈͕ऴྃ ࣌ܥྻ
91. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
92. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
93. subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId") if err != nil { panic(err) } engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ) subee.WithStatsHandler( NewStatshandler(), ), ) engine.Start(ctx);
94. Engine • Subscribeͨ͠ϝοηʔδΛ
 Consumerʹ఻ൖ ‣ ୯ҰϝοηʔδΛ఻ൖ ‣ Bufferingͯ͠ෳ਺ϝοηʔδΛ఻ൖ • Graceful Shutdown engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ), subee.WithStatsHandler( NewStatshandler(), ), )
95. Engine • Subscribeͨ͠ϝοηʔδΛ
 Consumerʹ఻ൖ ‣ ୯ҰϝοηʔδΛ఻ൖ ‣ Bufferingͯ͠ෳ਺ϝοηʔδΛ఻ൖ • Graceful Shutdown engine := subee.New( subscriber, NewConsumer(), NewStatshandler(), subee.WithConsumerInterceptors( subee_zap.ConsumerInterceptor(zap.Nop()), subee_recovery.ConsumerInterceptor( func(ctx context.Context, p interface{}) error { honybadger.Notify(p) return nil }, ), ), subee.WithStatsHandler( NewStatshandler(), ), ) InterfaceΛຬͨ͢ObjectΛొ࿥͢Δ͚ͩʂʂ
 ͋ͱ͸ྑ͍ײ͡ʹ΍ͬͯ͘ΕΔ
96. Pub/Sub Consumer Interceptor Queue Subscriber ack ack receive ack (Business Logic) Engine
97. ·ͱΊ • Subee: Pub/Sub Worker Framework Implementation ‣ WorkerपΓͷෳࡶ͞ΛӅṭ ‣ ΞϓϦέʔγϣϯΤϯδχΞ͕ຊ࣭తͳ࣮૷ʹूதͰ͖Δ https://github.com/wantedly/subee
98. github.com/wantedly/subee