Data sources vs usecases D-service B-service A-service C-service KPI Analytics Fraud Detection Credit Scoring Funnel Analytics ML system Customer Support
microservice -A microservice -B microservice -C Data Pipeline datauser -A datauser-B BigQuery BigQuery Event Log Event Log Event Log DB DB DB batch transfer stream transfer batch transfer batch transfer stream transfer stream transfer BqLoad Tool Cloud Storage ToGCS Tool? Cloud Dataflow BqLoad Tool Publish message Subscribe message merpay DataPipeline
microservice -A microservice -B microservice -C Data Pipeline datauser -A datauser-B BigQuery BigQuery Event Log Event Log Event Log DB DB DB batch transfer stream transfer batch transfer batch transfer stream transfer stream transfer BqLoad Tool Cloud Storage ToGCS Tool? Cloud Dataflow BqLoad Tool Publish message Subscribe message merpay DataPipeline 1. マイクロサービスは DataPipeline へ配送 2. データ利用者は DataPipeline から受け取り
batch microservice-B merpay-dataplatform Data User - A microservice-A Cloud Pub/Sub BigQuery Cloud Functions data mart change notification Pub/Sub trigger (BqLoad path) BqLoad Cloud SQL data lake Cloud Storage Cloud Spanner microservice-C Cloud Datastore Data Pipeline