メルペイにおける、マイクロサービスに寄り添うログ収集基盤 / Microservices-frendly Data Pipeline

A LT slide for builderscon 2018

Transcript

1. Microservices-frendly
   Data Pipeline
   merpay DataPlatform
   @syucream
   

   View Slide

2. Microservices in merpay
   ServiceA ServiceB ServiceC
   App App App
   

   View Slide

3. Data sources vs usecases
   D-service
   B-service
   A-service
   C-service
   KPI Analytics
   Fraud Detection
   Credit Scoring
   Funnel Analytics
   ML system
   Customer Support
   

   View Slide

4. microservice -A
   microservice -B
   microservice -C
   Data
   Pipeline
   datauser -A
   datauser-B
   BigQuery
   BigQuery
   Event
   Log
   Event
   Log
   Event
   Log
   DB
   DB
   DB
   batch
   transfer
   stream
   transfer
   batch
   transfer
   batch
   transfer
   stream
   transfer
   stream
   transfer
   BqLoad
   Tool
   Cloud
   Storage
   ToGCS
   Tool?
   Cloud
   Dataflow
   BqLoad
   Tool
   Publish message Subscribe message
   merpay DataPipeline
   

   View Slide

5. Batch (prototype)
   microservice-B
   merpay-dataplatform
   Data User - A
   microservice-A
   Cloud
   Pub/Sub
   BigQuery
   Cloud
   Functions
   data mart
   change
   notification
   Pub/Sub trigger
   (BqLoad path)
   BqLoad
   Cloud
   SQL
   data lake
   Cloud
   Storage
   Cloud
   Spanner
   microservice-C
   Cloud
   Datastore
   Data Pipeline
   

   View Slide

6. Stream (prototype)
   Microservice platform team
   Kubernetes cluster
   A-service B-service
   merpay-dataplatform
   Logging
   Cloud
   Pub/Sub
   Cloud
   Dataflow
   BigQuery
   stdout via
   logging library
   Sink to Pub/Sub
   Subscribe
   Data User - A
   BigQuery
   DWH
   Streaming Insert
   

   View Slide

7. Schema Registory
   ● Pre-define log schema in ProtocolBuffer
   ○ It’s popular in mercari/merpay
   ○ There’s are some useful protoc plugins
   ● Manage .proto files on GitHub
   

   View Slide

8. Remaining issues
   ● Batch
   ○ Scalability
   ● Stream
   ○ Reliability of google-fluentd DaemonSet
   ● Schema Registory
   ○ Who does defining/reviewing the schemas?
   

   View Slide