Slide 1
Slide 1 text
Subee
Pub/Sub Worker Framework Implementation
Go Conference Spring - @hlts2
Slide 2
Slide 2 text
$ whoami
• Hiroto Funakoshi
• @hlts2
• ۦ͚ग़͠ͷ৽ଔΤϯδχΞ
• αʔϏεͷΞʔΩςΫνϟج൫։ൃʹڵຯ͕͋Δ
Slide 3
Slide 3 text
$ whoami
• Hiroto Funakoshi
• @hlts2
• ۦ͚ग़͠ͷ৽ଔΤϯδχΞ
• αʔϏεͷΞʔΩςΫνϟج൫։ൃʹڵຯ͕͋Δ
WantedlyΠϯλʔϯͰ։ൃͨ͠OSSʹ͍ͭͯൃද͠·͢
Slide 4
Slide 4 text
࣍
• Pub/Subͱʁ
• Subeeͷհ
• Subeeͷར༻ํ๏
Slide 5
Slide 5 text
Pub/Sub?
Slide 6
Slide 6 text
Pub/Sub
• ඇಉظϝοηʔδϯάύϥμΠϜͷҰछ
• ϝοηʔδૹ৴ऀ͕ड৴ऀΛఆͤͣʹ
ϝοηʔδΛૹΕΔΑ͏ϓϩάϥϜ͞Εͨͷ
Go
Publisher
ʢProducerʣ
Ruby
Python
Go
Go
Subscriber
ʢConsumerʣ
Pub/Sub
ʢBrokerʣ
Slide 7
Slide 7 text
ͳͥඞཁʁ
Slide 8
Slide 8 text
Ϣʔεέʔε
https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
Slide 9
Slide 9 text
Ϣʔεέʔε
https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
Slide 10
Slide 10 text
Ϣʔεέʔε
https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
Slide 11
Slide 11 text
Ϣʔεέʔε
https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
QueueʢඇಉظAPIʣ
Slide 12
Slide 12 text
Ϣʔεέʔε
https://cloud.google.com/pubsub/docs/overview?hl=ja#scenarios
Πϕϯτൖ
Slide 13
Slide 13 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Slide 14
Slide 14 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
HTTP or gRPC
Push ґཔ
Slide 15
Slide 15 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Push ґཔ × 100ສ
HTTP or gRPC
Slide 16
Slide 16 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Push ґཔ × 100ສ
HTTP or gRPC
Push Server͕൵໐Λ͋͛Δ
Slide 17
Slide 17 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Pub/Sub
Slide 18
Slide 18 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Pub/Sub
Push ґཔ × 100ສ
Slide 19
Slide 19 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Pub/Sub
ࣗͷϖʔεͰॲཧ
Push ґཔ × 100ສ
Slide 20
Slide 20 text
ϢʔεέʔεʢඇಉظAPIʣ
API Server Push Server
Pub/Sub
ࣗͷϖʔεͰॲཧ
Push ґཔ × 100ສ
Push ServerͷෛՙΛ͑Δ͜ͱ͕ग़དྷΔ
Slide 21
Slide 21 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
B Service
C Service
Slide 22
Slide 22 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
B Service
C Service
ProfileΛߋ৽
HTTP or gRPC
Slide 23
Slide 23 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
B Service
C Service
Profile͕ߋ৽
HTTP or gRPC
ґଘ͢ΔαʔϏεΛ֮͑Δͷ͕େม
Slide 24
Slide 24 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
Pub/Sub
B Service
C Service
Slide 25
Slide 25 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
Pub/Sub
B Service
C Service
ProfileΛߋ৽
Slide 26
Slide 26 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
Pub/Sub
B Service
C Service
ProfileΛߋ৽
Slide 27
Slide 27 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
Pub/Sub
B Service
C Service
ProfileΛߋ৽
υϝΠϯڥք͕Ͱ͖Δ
Slide 28
Slide 28 text
ϢʔεέʔεʢΠϕϯτൖʣ
User Service
A Service
Pub/Sub
B Service
C Service
ProfileΛߋ৽
υϝΠϯڥք͕Ͱ͖Δ
ґଘ͢ΔαʔϏεΛΒͳ͍͍ͯ͘
Slide 29
Slide 29 text
࣮ྫʢCloud Pub/Subʣ
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 30
Slide 30 text
࣮ྫʢCloud Pub/Subʣ
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 31
Slide 31 text
࣮ྫʢCloud Pub/Subʣ
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
• ϝοηʔδΛਖ਼͘͠ॲཧग़དྷͨࣄΛPub/Subʹ͑Δ
‣ Pub/SubϝοηʔδΛআ͢Δ
ɹɹɹɹϝοηʔδ͕࠶ૹ͞Εͳ͍
Slide 32
Slide 32 text
࣮ྫʢCloud Pub/Subʣ
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
• ϝοηʔδΛਖ਼͘͠ॲཧग़དྷͳ͔ͬͨࣄΛPub/Subʹ͑Δ
‣ Pub/SubϝοηʔδΛআ͠ͳ͍
ɹɹɹɹɹɹɹϝοηʔδ͕࠶ૹ͞ΕΔ
Slide 33
Slide 33 text
Pub/Subಋೖͷ՝
• Worker࣮͕ඞཁ
‣ ୯ҰɾෳϝοηʔδΛॲཧ
‣ Consume࿙Εͳ͘Graceful Shutdown
‣ ϝτϦΫεͱΤϥʔϨϙʔςΟϯά
• ຊ࣭Ͱͳ͍ίʔυʢج൫ͷ࣮ʣ͕૿͑Δ
• ϚΠΫϩαʔϏεΛΨϯΨϯ࡞Δ৫Ͱ
Workerͷίʔυ͕ڞ௨Խग़དྷ͍ͯͳ͍ͱࣅͨΑ͏ͳίʔυ͕ࢄཚ
Slide 34
Slide 34 text
Pub/Subಋೖͷ՝
• Worker࣮͕ඞཁ
‣ ୯ҰɾෳϝοηʔδΛॲཧ
‣ Consume࿙Εͳ͘Graceful Shutdown
‣ ϝτϦΫεͱΤϥʔϨϙʔςΟϯά
• ຊ࣭Ͱͳ͍ίʔυʢج൫ͷ࣮ʣ͕૿͑Δ
• ϚΠΫϩαʔϏεΛΨϯΨϯ࡞Δ৫Ͱ
Workerͷίʔυ͕ڞ௨Խग़དྷ͍ͯͳ͍ͱࣅͨΑ͏ͳίʔυ͕ࢄཚ
Կ࣮͢Δͷʁ
Slide 35
Slide 35 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
msg.Ack()
return
}
if err := update(post); err != nil {
msg.Nack()
return
}
msg.Ack()
})
Slide 36
Slide 36 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
msg.Ack()
return
}
if err := update(post); err != nil {
msg.Nack()
return
}
msg.Ack()
})
LoggerΛ͍Ε͍ͨʂʂ
Slide 37
Slide 37 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer log.Printf("End consume process. time: %v", time.Since(start))
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 38
Slide 38 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer log.Printf("End consume process. time: %v", time.Since(start))
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 39
Slide 39 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer log.Printf("End consume process. time: %v", time.Since(start))
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Panic RecoverೖΕ͍ͨʂʂ
Slide 40
Slide 40 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 41
Slide 41 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 42
Slide 42 text
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Error ReportingೖΕ͍ͨʂʂ
Slide 43
Slide 43 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 44
Slide 44 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 45
Slide 45 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
log.Printf("Revoery from panic: %v", p)
}
}()
var post Post
if err := json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err := update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
StatsΛऔΓ͍ͨʂʂ
Slide 46
Slide 46 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 47
Slide 47 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 48
Slide 48 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
ෳϝοηʔδΛॲཧ͍ͨ͠
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 51
Slide 51 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 52
Slide 52 text
ͦͦ୭͕࣮͢Δͷʁ
Slide 53
Slide 53 text
ΞϓϦέʔγϣϯΤϯδχΞʁ
Slide 54
Slide 54 text
Want to
Slide 55
Slide 55 text
͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ
ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱͳ͍෦ʣ
Slide 56
Slide 56 text
͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ
ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱͳ͍෦ʣ
Worker࣮
ʴ
ΞϓϦέʔγϣϯ࣮
Slide 57
Slide 57 text
͍ͪૣ͘ΞϓϦέʔγϣϯΛϢʔβʔʹಧ͚͍ͨ
ແବͳࣄʹ࣌ؒΛ͖ͨ͘͞ͳ͍ʢຊ࣭Ͱͳ͍෦ʣ
Worker࣮
ʴ
ΞϓϦέʔγϣϯ࣮
ελʔτμογϡ͕͘ͳΔ
Slide 58
Slide 58 text
ཉ͔ͬͨ͠ͷ
• ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳΈ
Slide 59
Slide 59 text
ཉ͔ͬͨ͠ͷ
• ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳΈ
ΞϓϦέʔγϣϯ࣮
ج൫࣮
ج൫ͱΞϓϦέʔγϣϯίʔυ͕Ͱ͖͍ͯΔੈք
Slide 60
Slide 60 text
ཉ͔ͬͨ͠ͷ
• ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳΈ
ΞϓϦέʔγϣϯ࣮
ج൫࣮
ج൫ͱΞϓϦέʔγϣϯίʔυ͕Ͱ͖͍ͯΔੈք
Slide 61
Slide 61 text
ཉ͔ͬͨ͠ͷ
• ج൫ʢe.g. Workerʣ͕Ͳ͏ͳ͍ͬͯΔ͔ҙࣝͤͣʹ
ΞϓϦέʔγϣϯίʔυʹूதͰ͖ΔΑ͏ͳΈ
ΞϓϦέʔγϣϯ࣮
ج൫࣮
ج൫ͱΞϓϦέʔγϣϯίʔυ͕Ͱ͖͍ͯΔੈք
࡞Δͧ
Slide 62
Slide 62 text
github.com/wantedly/subee
Slide 63
Slide 63 text
• ಡΈํαϏΟʙʢαΣʙͰͳ͍…ʣ
• Go
• WantedlyͰ̑ͭͷϚΠΫϩαʔϏεͰಈ͍͍ͯΔ
• Pub/Sub WorkerΛॻ͔ͣʹϝοηʔδॲཧΛ
Մೳʹ͢ΔWorkerϑϨʔϜϫʔΫ
Subscribe + bee = Subee
Slide 64
Slide 64 text
Subeeಋೖͷൺֱʢલʣ
<=====>
Worker Code
Application Code
Application layer
ΞϓϦέʔγϣϯίʔυͱج൫͕ີ݁߹
Pub/Sub
Slide 65
Slide 65 text
Subeeಋೖͷൺֱʢޙʣ
Worker Code
Application Code
Framework
Application
ΞϓϦέʔγϣϯίʔυͱج൫ͷΓ͠
<=====>
Pub/Sub
Slide 66
Slide 66 text
Subeeಋೖͷൺֱʢޙʣ
Worker Code
Application Code
Framework
Application
ΞϓϦέʔγϣϯίʔυͱج൫ͷΓ͠
<=====>
Pub/Sub
Slide 67
Slide 67 text
ಋೖલͷίʔυ
Slide 68
Slide 68 text
raven.SetDSN("https://:@sentry.io/")
ctx := context.Background()
err := subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
start := time.Now()
log.Println("Receive message")
log.Printf("Start consume process")
statsHandler.HandleProcess(ctx, &Receive{
BeginTime: start,
})
defer func() {
log.Printf("End consume process. time: %v", time.Since(start))
if p := recover(); p != nil {
raven.CaptureError(err, nil)
log.Printf("Revoery from panic: %v", p)
}
}()
consumeBeginTime := time.Now()
var err error
ctx = statsHandler.TagProcess(ctx, &ConsumeBeginTag{})
defer func() {
statsHandler.HandleProcess(ctx, &ConsumeEnd{
BeginTime: consumeBeginTime,
EndTime: time.Now(),
Error: err,
})
statsHandler.HandleProcess(ctx, &End{
MsgCount: 1,
BeginTime: start,
EndTime: time.Now(),
})
}
var post Post
if err = json.Unmarshal(msg.Data, &post); err != nil {
raven.CaptureError(err, nil)
log.Printf("could not decode message data: %#v", msg)
msg.Ack()
return
}
if err = update(post); err != nil {
raven.CaptureError(err, nil)
log.Printf("[ID %d] could not update: %v", post.Id, err)
msg.Nack()
return
}
msg.Ack()
})
Slide 69
Slide 69 text
ಋೖޙͷίʔυ
Slide 70
Slide 70 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
type consumer struct {}
func NewConsumer() subee.Consumer {
return new(consumer)
}
func (c *consumer) Consume(ctx context.Context, msg subee.Message) error {
return nil
}
Slide 71
Slide 71 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
type consumer struct {}
func NewConsumer() subee.Consumer {
return new(consumer)
}
func (c *consumer) Consume(ctx context.Context, msg subee.Message) error {
return nil
}
ConsumerΛੜ
Slide 72
Slide 72 text
type consumer struct {}
func NewConsumer() subee.Consumer {
return new(consumer)
}
func (c *consumer) Consume(ctx context.Context, msg subee.Message) error {
return nil
}
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Frameworkʹొ͢Δ
Slide 73
Slide 73 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 74
Slide 74 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 75
Slide 75 text
Subscriberͷੜ
• Pub/Sub͔ΒϝοηʔδΛऔಘ͢ΔΛ࣋ͭ
• Cloud Pub/SubͷSubscriber࣮
ϑϨʔϜϫʔΫʹ༻ҙ
• ઃఆΛຒΊΔ͚ͩͰ
Subscriberͷར༻͕Մೳ
• ॊೈͳઃఆΛ͢ΔͨΊʹ
Function OptionͰΧελϚΠζՄೳ
subscriber, err := cloudpubsub.CreateSubscriber(ctx,
“cloudpub-app",
"subscription-id")
if err != nil {
panic(err)
}
Slide 76
Slide 76 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 77
Slide 77 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 78
Slide 78 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 79
Slide 79 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
type consumer struct {}
func NewConsumer() subee.Consumer {
return new(consumer)
}
func (c *consumer) Consume(ctx context.Context, msg subee.Message) error {
return nil
}
Slide 80
Slide 80 text
Consumerͷੜ
• ୯ҰϝοηʔδΛॲཧ͢Δ
Λ࣋ͭ
• ΞϓϦέʔγϣϯΤϯδχΞ͕
࣮ࡍʹ࣮͢Δ෦
type consumer struct {}
func NewConsumer() subee.Consumer {
return new(consumer)
}
func (c *consumer) Consume(ctx context.Context, msg subee.Message) error {
return nil
}
Slide 81
Slide 81 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 82
Slide 82 text
BatchConsumerͷੜ
• ෳϝοηʔδΛॲཧ͢Δ
Λ࣋ͭ
• ΞϓϦέʔγϣϯΤϯδχΞ͕
࣮ࡍʹ࣮͢Δ෦
subee.BatchConsumerFunc(
func(ctx context.Context, msg []subee.Message) error {
return nil
},
)
Slide 83
Slide 83 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 84
Slide 84 text
ACK/NACKͷ੍ޚ
• ConsumerɾBatchConsumerͷΓͰ
ACK/NACK͞ΕΔ
‣ Γ͕Τϥʔͷ߹ɺNACK͞ΕΔ
Slide 85
Slide 85 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 86
Slide 86 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 87
Slide 87 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 88
Slide 88 text
InterceptorɾStatsHandlerͷੜ
• Interceptor
‣ Consumeલޙʹհೖ
Ͱ͖Δ
‣ Logger, Recovery
Frameworkʹ༻ҙ
• StatsHandler
‣ ܭଌܥͷͨΊʹར༻
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
),
subee.WithStatsHandler(
NewStatshandler(),
)
Slide 89
Slide 89 text
ܭଌܥͷͨΊͷStatsHandler
Channel͔ΒϝοηʔδΛड͚औΔ
ϝοηʔδΛChannelʹྲྀ͢
Pub/Sub͔ΒϝοηʔδΛऔಘ
Consume։࢝લ
Consume։࢝
Consumeऴྃ
औΓग़͔͠ΒConsume·ͰͷҰ࿈͕ऴྃ
࣌ܥྻ
Slide 90
Slide 90 text
ܭଌܥͷͨΊͷStatsHandler
Channel͔ΒϝοηʔδΛड͚औΔ
ϝοηʔδΛChannelʹྲྀ͢
Pub/Sub͔ΒϝοηʔδΛऔಘ
Consume։࢝લ
Consume։࢝
Consumeऴྃ
औΓग़͔͠ΒConsume·ͰͷҰ࿈͕ऴྃ
࣌ܥྻ
Slide 91
Slide 91 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 92
Slide 92 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 93
Slide 93 text
subscriber, err := cloudpubsub.CreateSubscriber(ctx, "projectId", "subscriptionId")
if err != nil {
panic(err)
}
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
)
subee.WithStatsHandler(
NewStatshandler(),
),
)
engine.Start(ctx);
Slide 94
Slide 94 text
Engine
• Subscribeͨ͠ϝοηʔδΛ
Consumerʹൖ
‣ ୯ҰϝοηʔδΛൖ
‣ Bufferingͯ͠ෳϝοηʔδΛൖ
• Graceful Shutdown
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
),
subee.WithStatsHandler(
NewStatshandler(),
),
)
Slide 95
Slide 95 text
Engine
• Subscribeͨ͠ϝοηʔδΛ
Consumerʹൖ
‣ ୯ҰϝοηʔδΛൖ
‣ Bufferingͯ͠ෳϝοηʔδΛൖ
• Graceful Shutdown
engine := subee.New(
subscriber,
NewConsumer(),
NewStatshandler(),
subee.WithConsumerInterceptors(
subee_zap.ConsumerInterceptor(zap.Nop()),
subee_recovery.ConsumerInterceptor(
func(ctx context.Context, p interface{}) error {
honybadger.Notify(p)
return nil
},
),
),
subee.WithStatsHandler(
NewStatshandler(),
),
)
InterfaceΛຬͨ͢ObjectΛొ͢Δ͚ͩʂʂ
͋ͱྑ͍ײ͡ʹͬͯ͘ΕΔ
Slide 96
Slide 96 text
Pub/Sub
Consumer
Interceptor
Queue
Subscriber
ack
ack
receive
ack
(Business Logic)
Engine
Slide 97
Slide 97 text
·ͱΊ
• Subee: Pub/Sub Worker Framework Implementation
‣ WorkerपΓͷෳࡶ͞ΛӅṭ
‣ ΞϓϦέʔγϣϯΤϯδχΞ͕ຊ࣭తͳ࣮ʹूதͰ͖Δ
https://github.com/wantedly/subee
Slide 98
Slide 98 text
github.com/wantedly/subee