Pro Yearly is on sale from $80 to $50! »

`cloud.google.com/go/pubsub` internal

`cloud.google.com/go/pubsub` internal

9eed44f137609e6ce3b6f1e14f80b9e1?s=128

Masayuki Izumi

May 20, 2019
Tweet

Transcript

  1. ©2019 Wantedly, Inc. `cloud.google.com/go/pubsub` internal golang.tokyo #24 May 20, 2019

    - Masayuki Izumi
  2. ©2019 Wantedly, Inc. $ whoami @izumin5210 Wantedly, Inc. ‣ Wantedly

    People - Application Engineer (Go, Ruby, Web Frontend) ‣ Interested in… - Microservices - Developer Productivity - Developer Experience
  3. ©2019 Wantedly, Inc. Talk abst and desc

  4. ©2019 Wantedly, Inc. HPPHMFDMPVEHPͱఆٛݩ΁ͷδϟϯϓ͕࢖͑Δ؀ڥ ͕͋ΔͱɺൃදΛΑΓָ͓͠Έ͍͚ͨͩ·͢ git clone https://github.com/googleapis/google-cloud-go.git cd google-cloud-go

    git checkout v0.39.0 export GO111MODULE=on go mod download
  5. ©2019 Wantedly, Inc. 1VC4VCɾϝοηʔδΩϡʔΈ͍ͨͳ΍ͭ ͔ͭͬͨ͜ͱ͋Δͻͱʔ

  6. ©2019 Wantedly, Inc. Pub/Sub • ඇಉظϝοηʔδϯάύϥμΠϜͷҰछ • ϝοηʔδૹ৴ऀ͕ड৴ऀΛ૝ఆͤͣʹ
 ϝοηʔδΛૹΕΔΑ͏ϓϩάϥϜ͞Εͨ΋ͷ
 


    
 
 
 
 Go Publisher
 ʢProducerʣ Ruby Python Go Go Subscriber
 ʢConsumerʣ Pub/Sub
 ʢBrokerʣ @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO
  7. ©2019 Wantedly, Inc. @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon

    2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Push ґཔ × 100ສ HTTP or gRPC ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub ࣗ෼ͷϖʔεͰॲཧ Push ґཔ × 100ສ "1*αʔό͕ϖʔεΛѲΔͷͰɺ 8PSLFSଆ͕ϫʔΫϩʔυʹ଱͖͑Εͳ͍ 8PSLFS͸ࣗ෼ͷϖʔεͰॲཧΛ͢ΔͷͰམͪͳ͍͠ɺ ඇಉظͳͷͰ"1*αʔόʹ΋Өڹ͕ग़ͳ͍
  8. ©2019 Wantedly, Inc. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B

    Service C Service ProfileΛߋ৽ υϝΠϯڥք͕Ͱ͖Δ ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service ProfileΛߋ৽ HTTP or gRPC @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ج൫αʔϏε͕຤୺ͷϚΠΫϩαʔϏεʹ
 ґଘ͞Ε͍ͯΔ͜ͱΛ஌Βͳ͍ͱ͍͚ͳ͍ ຤୺ͷαʔϏε͸ࣗ෼͕୭ʹґଘ͍ͯ͠Δ͔Λ஌͍ͬͯΔ͕ɺ ج൫αʔϏε͸୭ʹґଘ͞Ε͍ͯΔ͔Λ஌Βͳͯ͘ࡁΉ
  9. ©2019 Wantedly, Inc. client, err := pubsub.NewClient(ctx, "project-id") if err

    != nil { // ... } defer client.Close() // topic リソースは事前につくっておく topic := client.Topic("my-topic") defer topic.Stop() topic.Publish(ctx, &pubsub.Message{ Data: string("Hello, Workd!"), }) IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX
  10. ©2019 Wantedly, Inc. IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX client, err := pubsub.NewClient(ctx, "project-id") if

    err != nil { // ... } defer client.Close() // subscription リソースは事前につくっておく subscription := client.Subscription("my-subscription") subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) error { // ... return nil })
  11. ©2019 Wantedly, Inc. "1*ɾ࢖͍ํ͸௒γϯϓϧ Ͱ΋಺෦Ͱ͸͔ͳΓ͍Ζ͍Ζ΍͍ͬͯΔ

  12. ©2019 Wantedly, Inc. ৴པੑͱύϑΥʔϚϯε͕ཁٻ͞ΕΔ ϛυϧ΢ΣΞͷΫϥΠΞϯτ

  13. ©2019 Wantedly, Inc. ͦͷ಺෦Λ࣮ࡍʹݟͯΈ·͠ΐ͏ ϥΠϒίʔυϦʔσΟϯά͢Δͧʂʂ

  14. ©2019 Wantedly, Inc. Publishing

  15. ©2019 Wantedly, Inc. entrypoint: `func (*pubsub.Topic) Publish(context.Context, *pubsub.Message)` QVCTVCUPQJDHP- 1VCMJTIJOH

    FOUSZQPJOU
  16. ©2019 Wantedly, Inc. google.golang.org/api/support/bundler func NewBundler(itemExample interface{}, handler func(interface{})) *Bundler

    func (b *Bundler) Add(item interface{}, size int) error func (b *Bundler) AddWait(ctx context.Context, item interface{}, size int) error func (b *Bundler) Flush() 1VCMJTIJOH CVOEMFS ΞΠςϜΛཷΊࠐΜͰɺմ͝ͱʹॲཧ͢Δ ʢσʔλ਺ σʔλ༰ྔ ࣌ؒͳͲͰGMVTI͢Δ৚݅ΛܾΊΒΕΔʣ
  17. ©2019 Wantedly, Inc. google.golang.org/api/support/bundler t.bundler = bundler.NewBundler(&bundledMessage{}, func(items interface{}) {

    // TODO(jba): use a context detached from the one passed to NewClient. ctx := context.TODO() if timeout != 0 { var cancel func() ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } t.publishMessageBundle(ctx, items.([]*bundledMessage)) }) 1VCMJTIJOH CVOEMFS (PPHMF$MPVE1VC4VCͷQVCMJTIFS͸ɺCVOEMFSʹཷΊࠐΜͰඇಉظʹϝοηʔδΛૹ৴͍ͯ͠Δ ./pubsub/topic.go L412
  18. ©2019 Wantedly, Inc. github.com/googleapis/gax-go gax.WithRetry(func() gax.Retryer { return gax.OnCodes([]codes.Code{ codes.Aborted,

    codes.Canceled, codes.Internal, codes.ResourceExhausted, codes.Unavailable, codes.Unknown, }, gax.Backoff{ Initial: 100 * time.Millisecond, Max: 60000 * time.Millisecond, Multiplier: 1.3, }) }), }, 1VCMJTIJOH HJUIVCDPNHPPHMFBQJTHBYHP H31$Λར༻ͨ͠"1*DBMMͷϦτϥΠઓུΛӅṭ͢Δύοέʔδ QVCTVCBQJWQVCMJTIFIS@DMJFOUHP-
  19. ©2019 Wantedly, Inc. Subscribing

  20. ©2019 Wantedly, Inc. entrypoint: `func (*pubsub.Subscription) Receive(context.Context, func(context.Context, *pubsub.Message)) error`

    QVCTVCTVCTDSJQUJPOHP- 4VCTDSJCJOH FOUSZQPJOU
  21. ©2019 Wantedly, Inc. golang.org/x/sync/errgroup group, gctx := errgroup.WithContext(ctx) for i

    := 0; i < numGoroutines; i++ { group.Go(func() error { return s.receive(gctx, po, fc, f) }) } return group.Wait() 1VCMJTIJOH FSSHSPVQ AFSSHSPVQAΛར༻͠ɺϝοηʔδΛड৴͢ΔͨΊͷXPSLFSHPSPVUJOFΛੜΈग़͍ͯ͠Δ ./pubsub/subscription.go L547
  22. ©2019 Wantedly, Inc. 4VCTDSJCJOH "DL/BDL A QVCTVC.FTTBHF "DL A͕ݺ͹ΕΔͱ "DL଴ͪϝοηʔδͷεϥΠεʹBQQFOEͨ͋͠ͱɺ

    GMPXDPOUSPMMFSʹͦΕΛ஌ΒͤͯΩϡʔʹۭ͖Λ࡞Δ msg.doneFunc = func(ackID string, ack bool, receiveTime time.Time) { defer fc.release(msgLen) old(ackID, ack, receiveTime) } func (m *Message) Ack() { m.done(true) } func (m *Message) Nack() { m.done(false) } func (m *Message) done(ack bool) { if m.calledDone { return } m.calledDone = true m.doneFunc(m.ackID, ack, m.receiveTime) } func (it *messageIterator) done(ackID string, ack bool, receiveTime time.Time) { it.ackTimeDist.Record(int(time.Since(receiveTime) / time.Second)) it.mu.Lock() defer it.mu.Unlock() delete(it.keepAliveDeadlines, ackID) if ack { it.pendingAcks[ackID] = true } else { it.pendingNacks[ackID] = true } it.checkDrained() } QVCTVCJUFSBUPSHP- QVCTVCTVCTDSJQUJPOHP- QVCTVCNFTTBHFHP-
  23. ©2019 Wantedly, Inc. ʜΈ͍ͨͳײ͡ͷ͜ͱΛɺࣾ಺ษڧձͰ΍ͬͯ·͢ .JDSPTFSWJDFT.POEBZIUUQTXXXXBOUFEMZDPNQSPKFDUT (PQIFST$PEF3FBEJOH1BSUZΠϕϯτϖʔδ͕ͳ͍ͷͰ੠͔͚͍ͯͩ͘͞