`cloud.google.com/go/pubsub` internal

`cloud.google.com/go/pubsub` internal

9eed44f137609e6ce3b6f1e14f80b9e1?s=128

Masayuki Izumi

May 20, 2019
Tweet

Transcript

  1. 2.

    ©2019 Wantedly, Inc. $ whoami @izumin5210 Wantedly, Inc. ‣ Wantedly

    People - Application Engineer (Go, Ruby, Web Frontend) ‣ Interested in… - Microservices - Developer Productivity - Developer Experience
  2. 6.

    ©2019 Wantedly, Inc. Pub/Sub • ඇಉظϝοηʔδϯάύϥμΠϜͷҰछ • ϝοηʔδૹ৴ऀ͕ड৴ऀΛ૝ఆͤͣʹ
 ϝοηʔδΛૹΕΔΑ͏ϓϩάϥϜ͞Εͨ΋ͷ
 


    
 
 
 
 Go Publisher
 ʢProducerʣ Ruby Python Go Go Subscriber
 ʢConsumerʣ Pub/Sub
 ʢBrokerʣ @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO
  3. 7.

    ©2019 Wantedly, Inc. @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon

    2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Push ґཔ × 100ສ HTTP or gRPC ϢʔεέʔεʢඇಉظAPIʣ API Server Push Server Pub/Sub ࣗ෼ͷϖʔεͰॲཧ Push ґཔ × 100ສ "1*αʔό͕ϖʔεΛѲΔͷͰɺ 8PSLFSଆ͕ϫʔΫϩʔυʹ଱͖͑Εͳ͍ 8PSLFS͸ࣗ෼ͷϖʔεͰॲཧΛ͢ΔͷͰམͪͳ͍͠ɺ ඇಉظͳͷͰ"1*αʔόʹ΋Өڹ͕ग़ͳ͍
  4. 8.

    ©2019 Wantedly, Inc. ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service Pub/Sub B

    Service C Service ProfileΛߋ৽ υϝΠϯڥք͕Ͱ͖Δ ϢʔεέʔεʢΠϕϯτ఻ൖʣ User Service A Service B Service C Service ProfileΛߋ৽ HTTP or gRPC @hlts2, "Subee: Pub/Sub Worker Framework Implementation", GoCon 2019 Spring IUUQTTQFBLFSEFDLDPNIMUTTVCXPSLFSGSBNFXPSLJNQMFNFOUBUJPO ج൫αʔϏε͕຤୺ͷϚΠΫϩαʔϏεʹ
 ґଘ͞Ε͍ͯΔ͜ͱΛ஌Βͳ͍ͱ͍͚ͳ͍ ຤୺ͷαʔϏε͸ࣗ෼͕୭ʹґଘ͍ͯ͠Δ͔Λ஌͍ͬͯΔ͕ɺ ج൫αʔϏε͸୭ʹґଘ͞Ε͍ͯΔ͔Λ஌Βͳͯ͘ࡁΉ
  5. 9.

    ©2019 Wantedly, Inc. client, err := pubsub.NewClient(ctx, "project-id") if err

    != nil { // ... } defer client.Close() // topic リソースは事前につくっておく topic := client.Topic("my-topic") defer topic.Stop() topic.Publish(ctx, &pubsub.Message{ Data: string("Hello, Workd!"), }) IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX
  6. 10.

    ©2019 Wantedly, Inc. IUUQTDMPVEHPPHMFDPNQVCTVCEPDTPWFSWJFX client, err := pubsub.NewClient(ctx, "project-id") if

    err != nil { // ... } defer client.Close() // subscription リソースは事前につくっておく subscription := client.Subscription("my-subscription") subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) error { // ... return nil })
  7. 16.

    ©2019 Wantedly, Inc. google.golang.org/api/support/bundler func NewBundler(itemExample interface{}, handler func(interface{})) *Bundler

    func (b *Bundler) Add(item interface{}, size int) error func (b *Bundler) AddWait(ctx context.Context, item interface{}, size int) error func (b *Bundler) Flush() 1VCMJTIJOH CVOEMFS ΞΠςϜΛཷΊࠐΜͰɺմ͝ͱʹॲཧ͢Δ ʢσʔλ਺ σʔλ༰ྔ ࣌ؒͳͲͰGMVTI͢Δ৚݅ΛܾΊΒΕΔʣ
  8. 17.

    ©2019 Wantedly, Inc. google.golang.org/api/support/bundler t.bundler = bundler.NewBundler(&bundledMessage{}, func(items interface{}) {

    // TODO(jba): use a context detached from the one passed to NewClient. ctx := context.TODO() if timeout != 0 { var cancel func() ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } t.publishMessageBundle(ctx, items.([]*bundledMessage)) }) 1VCMJTIJOH CVOEMFS (PPHMF$MPVE1VC4VCͷQVCMJTIFS͸ɺCVOEMFSʹཷΊࠐΜͰඇಉظʹϝοηʔδΛૹ৴͍ͯ͠Δ ./pubsub/topic.go L412
  9. 18.

    ©2019 Wantedly, Inc. github.com/googleapis/gax-go gax.WithRetry(func() gax.Retryer { return gax.OnCodes([]codes.Code{ codes.Aborted,

    codes.Canceled, codes.Internal, codes.ResourceExhausted, codes.Unavailable, codes.Unknown, }, gax.Backoff{ Initial: 100 * time.Millisecond, Max: 60000 * time.Millisecond, Multiplier: 1.3, }) }), }, 1VCMJTIJOH HJUIVCDPNHPPHMFBQJTHBYHP H31$Λར༻ͨ͠"1*DBMMͷϦτϥΠઓུΛӅṭ͢Δύοέʔδ QVCTVCBQJWQVCMJTIFIS@DMJFOUHP-
  10. 21.

    ©2019 Wantedly, Inc. golang.org/x/sync/errgroup group, gctx := errgroup.WithContext(ctx) for i

    := 0; i < numGoroutines; i++ { group.Go(func() error { return s.receive(gctx, po, fc, f) }) } return group.Wait() 1VCMJTIJOH FSSHSPVQ AFSSHSPVQAΛར༻͠ɺϝοηʔδΛड৴͢ΔͨΊͷXPSLFSHPSPVUJOFΛੜΈग़͍ͯ͠Δ ./pubsub/subscription.go L547
  11. 22.

    ©2019 Wantedly, Inc. 4VCTDSJCJOH "DL/BDL A QVCTVC.FTTBHF "DL A͕ݺ͹ΕΔͱ "DL଴ͪϝοηʔδͷεϥΠεʹBQQFOEͨ͋͠ͱɺ

    GMPXDPOUSPMMFSʹͦΕΛ஌ΒͤͯΩϡʔʹۭ͖Λ࡞Δ msg.doneFunc = func(ackID string, ack bool, receiveTime time.Time) { defer fc.release(msgLen) old(ackID, ack, receiveTime) } func (m *Message) Ack() { m.done(true) } func (m *Message) Nack() { m.done(false) } func (m *Message) done(ack bool) { if m.calledDone { return } m.calledDone = true m.doneFunc(m.ackID, ack, m.receiveTime) } func (it *messageIterator) done(ackID string, ack bool, receiveTime time.Time) { it.ackTimeDist.Record(int(time.Since(receiveTime) / time.Second)) it.mu.Lock() defer it.mu.Unlock() delete(it.keepAliveDeadlines, ackID) if ack { it.pendingAcks[ackID] = true } else { it.pendingNacks[ackID] = true } it.checkDrained() } QVCTVCJUFSBUPSHP- QVCTVCTVCTDSJQUJPOHP- QVCTVCNFTTBHFHP-