Creating Stream DataPipeline on GCP Using Apache Beam

5e70fc45fb6bcca87416c112a9e7e679?s=47 Shu Suzuki
September 12, 2019

Creating Stream DataPipeline on GCP Using Apache Beam

We built a scalable and flexible stream data pipeline for our microservices
on Google Cloud Platform (GCP), using Cloud Pub/Sub, Google Cloud Storage, BigQuery, and Cloud Dataflow, using Apache Beam. The stream data pipeline is working on the production system for Mercari, one of the biggest C2C e-commerce services in Japan. The pipeline currently accepts logs from 5+ microservices, and the number will increase soon.

5e70fc45fb6bcca87416c112a9e7e679?s=128

Shu Suzuki

September 12, 2019
Tweet