Production-ready stream data pipeline in Merpay, Inc

4ab3fec3e82ddb19bcadd93ef909a443?s=47 Ryo Okubo
September 12, 2019

Production-ready stream data pipeline in Merpay, Inc

This is a slide for https://www.apachecon.com/acna19/s/#/scheduledEvent/1329

--

We’ve started to provide our stream based data pipeline by using Google Cloud Dataflow and Apache Beam since 2018 fall. It collects event logs from microservices running on GKE, then transforms and forwards the logs to GCS and BigQuery to use for analytics, etc. As you know, implementing and operating streaming jobs are challenging. We’re encountered various issues during that time.
I’d like to share our knowledge on development and operation perspective. There are 3 topics in the development part, 1) Implementing stream jobs with using spotify/scio, 2) How to debug the jobs, especially having DynamicDestination, 3) How to load testing, to ensure our jobs stable. And topics in the next part, 1) How to deploy new jobs safely(with avoiding data loss), 2) How to monitor the jobs and surrounding systems, and misc.

4ab3fec3e82ddb19bcadd93ef909a443?s=128

Ryo Okubo

September 12, 2019
Tweet