`cloud.google.com/go/pubsub` internal

talked on https://golangtokyo.connpass.com/event/129067/

Transcript

1. ©2019 Wantedly, Inc. `cloud.google.com/go/pubsub` internal golang.tokyo #24 May 20, 2019 - Masayuki Izumi
2. ©2019 Wantedly, Inc. $ whoami @izumin5210 Wantedly, Inc. ‣ Wantedly People - Application Engineer (Go, Ruby, Web Frontend) ‣ Interested in… - Microservices - Developer Productivity - Developer Experience
3. ©2019 Wantedly, Inc. Talk abst and desc
4. ©2019 Wantedly, Inc. HPPHMFDMPVEHP
   ͱ
   ఆٛݩ΁ͷδϟϯϓ͕࢖͑Δ؀ڥ
   ͕͋ΔͱɺൃදΛΑΓָ͓͠Έ͍͚ͨͩ·͢ git clone https://github.com/googleapis/google-cloud-go.git cd google-cloud-go git checkout v0.39.0 export GO111MODULE=on go mod download
5. ©2019 Wantedly, Inc. 1VC4VCɾϝοηʔδΩϡʔ
   Έ͍ͨͳ΍ͭ
   ͔ͭͬͨ͜ͱ͋Δͻͱʔ
6. ©2019 Wantedly, Inc. Pub/Sub • ඇಉظϝοηʔδϯάύϥμΠϜͷҰछ • ϝοηʔδૹ৴ऀ͕ड৴ऀΛ૝ఆͤͣʹ
 ϝοηʔδΛૹΕΔΑ͏ϓϩάϥϜ͞Εͨ΋ͷ
 
 
 
 
 
 Go Publisher
 ʢProducerʣ Ruby Python Go Go Subscriber
 ʢConsumerʣ Pub/Sub
 ʢBrokerʣ @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO
7. ©2019 Wantedly, Inc. @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Push ґཔ × 100ສ HTTP or gRPC ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub ࣗ෼ͷϖʔεͰॲཧ Push ґཔ × 100ສ "1*
   αʔό͕ϖʔεΛѲΔͷͰɺ
   8PSLFS
   ଆ͕ϫʔΫϩʔυʹ଱͖͑Εͳ͍ 8PSLFS
   ͸ࣗ෼ͷϖʔεͰॲཧΛ͢ΔͷͰམͪͳ͍͠ɺ
   ඇಉظͳͷͰ"1*
   αʔόʹ΋Өڹ͕ग़ͳ͍
8. ©2019 Wantedly, Inc. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B Service C Service ProfileΛߋ৽ υϝΠϯڥք͕Ͱ͖Δ ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service ProfileΛߋ৽ HTTP or gRPC @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ج൫αʔϏε͕຤୺ͷϚΠΫϩαʔϏεʹ
 ґଘ͞Ε͍ͯΔ͜ͱΛ஌Βͳ͍ͱ͍͚ͳ͍ ຤୺ͷαʔϏε͸ࣗ෼͕୭ʹґଘ͍ͯ͠Δ͔Λ஌͍ͬͯΔ͕ɺ
   ج൫αʔϏε͸୭ʹґଘ͞Ε͍ͯΔ͔Λ஌Βͳͯ͘ࡁΉ
9. ©2019 Wantedly, Inc. client, err := pubsub.NewClient(ctx, "project-id") if err != nil { // ... } defer client.Close() // topic リソースは事前につくっておく topic := client.Topic("my-topic") defer topic.Stop() topic.Publish(ctx, &pubsub.Message{ Data: string("Hello, Workd!"), }) IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX
10. ©2019 Wantedly, Inc. IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX client, err := pubsub.NewClient(ctx, "project-id") if err != nil { // ... } defer client.Close() // subscription リソースは事前につくっておく subscription := client.Subscription("my-subscription") subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) error { // ... return nil })
11. ©2019 Wantedly, Inc. "1*ɾ࢖͍ํ͸௒γϯϓϧ
    Ͱ΋಺෦Ͱ͸͔ͳΓ͍Ζ͍Ζ΍͍ͬͯΔ
12. ©2019 Wantedly, Inc. ৴པੑͱύϑΥʔϚϯε͕ཁٻ͞ΕΔ
    ϛυϧ΢ΣΞͷΫϥΠΞϯτ
13. ©2019 Wantedly, Inc. ͦͷ಺෦Λ࣮ࡍʹݟͯΈ·͠ΐ͏
    ϥΠϒίʔυϦʔσΟϯά͢Δͧʂʂ
14. ©2019 Wantedly, Inc. Publishing
15. ©2019 Wantedly, Inc. entrypoint: `func (*pubsub.Topic) Publish(context.Context, *pubsub.Message)` QVCTVCUPQJDHP
    - 1VCMJTIJOH FOUSZQPJOU
16. ©2019 Wantedly, Inc. google.golang.org/api/support/bundler func NewBundler(itemExample interface{}, handler func(interface{})) *Bundler func (b *Bundler) Add(item interface{}, size int) error func (b *Bundler) AddWait(ctx context.Context, item interface{}, size int) error func (b *Bundler) Flush() 1VCMJTIJOH CVOEMFS ΞΠςϜΛཷΊࠐΜͰɺմ͝ͱʹॲཧ͢Δ
    ʢσʔλ਺
    σʔλ༰ྔ
    ࣌ؒ
    ͳͲͰ
    GMVTI
    ͢Δ৚݅ΛܾΊΒΕΔʣ
17. ©2019 Wantedly, Inc. google.golang.org/api/support/bundler t.bundler = bundler.NewBundler(&bundledMessage{}, func(items interface{}) { // TODO(jba): use a context detached from the one passed to NewClient. ctx := context.TODO() if timeout != 0 { var cancel func() ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } t.publishMessageBundle(ctx, items.([]*bundledMessage)) }) 1VCMJTIJOH CVOEMFS (PPHMF
    $MPVE
    1VC4VC
    ͷ
    QVCMJTIFS
    ͸ɺ
    CVOEMFS
    ʹཷΊࠐΜͰඇಉظʹϝοηʔδΛૹ৴͍ͯ͠Δ ./pubsub/topic.go L412
18. ©2019 Wantedly, Inc. github.com/googleapis/gax-go gax.WithRetry(func() gax.Retryer { return gax.OnCodes([]codes.Code{ codes.Aborted, codes.Canceled, codes.Internal, codes.ResourceExhausted, codes.Unavailable, codes.Unknown, }, gax.Backoff{ Initial: 100 * time.Millisecond, Max: 60000 * time.Millisecond, Multiplier: 1.3, }) }), }, 1VCMJTIJOH HJUIVCDPNHPPHMFBQJTHBYHP H31$
    Λར༻ͨ͠
    "1*
    
    DBMM
    ͷϦτϥΠઓུΛӅṭ͢Δύοέʔδ
    QVCTVCBQJWQVCMJTIFIS@DMJFOUHP
    -
19. ©2019 Wantedly, Inc. Subscribing
20. ©2019 Wantedly, Inc. entrypoint: `func (*pubsub.Subscription) Receive(context.Context, func(context.Context, *pubsub.Message)) error` QVCTVCTVCTDSJQUJPOHP
    - 4VCTDSJCJOH FOUSZQPJOU
21. ©2019 Wantedly, Inc. golang.org/x/sync/errgroup group, gctx := errgroup.WithContext(ctx) for i := 0; i < numGoroutines; i++ { group.Go(func() error { return s.receive(gctx, po, fc, f) }) } return group.Wait() 1VCMJTIJOH FSSHSPVQ AFSSHSPVQA
    Λར༻͠ɺϝοηʔδΛड৴͢ΔͨΊͷ
    XPSLFS
    HPSPVUJOF
    ΛੜΈग़͍ͯ͠Δ
    ./pubsub/subscription.go L547
22. ©2019 Wantedly, Inc. 4VCTDSJCJOH "DL
    
    /BDL A QVCTVC.FTTBHF "DL A
    ͕ݺ͹ΕΔͱ
    "DL
    ଴ͪϝοηʔδͷεϥΠεʹ
    BQQFOE
    ͨ͋͠ͱɺ
    GMPX
    DPOUSPMMFS
    ʹͦΕΛ஌ΒͤͯΩϡʔʹۭ͖Λ࡞Δ msg.doneFunc = func(ackID string, ack bool, receiveTime time.Time) { defer fc.release(msgLen) old(ackID, ack, receiveTime) } func (m *Message) Ack() { m.done(true) } func (m *Message) Nack() { m.done(false) } func (m *Message) done(ack bool) { if m.calledDone { return } m.calledDone = true m.doneFunc(m.ackID, ack, m.receiveTime) } func (it *messageIterator) done(ackID string, ack bool, receiveTime time.Time) { it.ackTimeDist.Record(int(time.Since(receiveTime) / time.Second)) it.mu.Lock() defer it.mu.Unlock() delete(it.keepAliveDeadlines, ackID) if ack { it.pendingAcks[ackID] = true } else { it.pendingNacks[ackID] = true } it.checkDrained() } QVCTVCJUFSBUPSHP
    - QVCTVCTVCTDSJQUJPOHP
    - QVCTVCNFTTBHFHP
    -
23. ©2019 Wantedly, Inc. ʜΈ͍ͨͳײ͡ͷ͜ͱΛɺࣾ಺ษڧձͰ΍ͬͯ·͢ .JDSPTFSWJDFT
    .POEBZ
    
    IUUQTXXXXBOUFEMZDPNQSPKFDUT
    (PQIFST
    $PEF
    3FBEJOH
    1BSUZ
    
    Πϕϯτϖʔδ͕ͳ͍ͷͰ੠͔͚͍ͯͩ͘͞