Data sources vs usecases D-service B-service A-service C-service KPI Analytics Fraud Detection Credit Scoring Funnel Analytics ML system Customer Support
microservice -A microservice -B microservice -C Data Pipeline datauser -A datauser-B BigQuery BigQuery Event Log Event Log Event Log DB DB DB batch transfer stream transfer batch transfer batch transfer stream transfer stream transfer BqLoad Tool Cloud Storage ToGCS Tool? Cloud Dataflow BqLoad Tool Publish message Subscribe message merpay DataPipeline
Batch (prototype) microservice-B merpay-dataplatform Data User - A microservice-A Cloud Pub/Sub BigQuery Cloud Functions data mart change notification Pub/Sub trigger (BqLoad path) BqLoad Cloud SQL data lake Cloud Storage Cloud Spanner microservice-C Cloud Datastore Data Pipeline
Schema Registory ● Pre-define log schema in ProtocolBuffer ○ It’s popular in mercari/merpay ○ There’s are some useful protoc plugins ● Manage .proto files on GitHub