Data processing, workflow and us ~How to manage automated jobs~

Transcript

1. Data processing, workflow and us ~How to manage automated jobs~ SRE-SET Automation Night Mercari SRE team :: @syu_cream
2. whoami • @syu_cream • Mercari SRE newbie :) • Develop/maintain Data processing systems and some middlewares • My interests: ◦ Go, mruby, and automation
3. My concerns... • Data processing is already automated. ◦ Fluentd, Embulk and some tools … ◦ Shell scripts, cron … • How can we improve job management?
4. Agenda • BigQuery log uploader ◦ Previous status and issues ◦ Job management w/ digdag • Statistics analyser ◦ Previous status and issues ◦ Re-implementation w/ Cloud Dataflow ◦ Job management w/ Airflow
5. BigQuery log uploader
6. Mercari log analysis infrastructure See details at https://speakerdeck.com/cubicdaiya/mercari-data-analysis-infrastructure
7. Deep dive to around BigQuery • Receive forwarded logs by Fluentd • Manage uploading job by cron ◦ Split and compress large log files ◦ Upload logs to GCS ◦ Load logs from GCS to BigQuery
8. Issues... • It was hard to retry failed job ◦ We catch slack alert when it fails ◦ But we need to reproduce some cmds • It didn’t visualize job status • Uploading logic becomes complex ◦ Related to BQ resource limit ▪ Compressed JSON has 4GB upper limit ◦ So we had split large file before uploading
9. Improve Job w/ digdag • We start to use digdag, a simple workflow engine. ◦ https://www.digdag.io/ • And Split log files in Fluentd ◦ By adjusting buffer_chunk_limit
10. Improvements • We can retry failed job. • We can view job progress. • Splitting logic is offloaded. ◦ So uploading logic become simple :)
11. Statistics analyser
12. Mercari basic statistics analyzer • Collect some data from DB and logs • Process by Node.js • Report basic KPIs by slack and email
13. Issues... • It’s hard to maintain … ◦ Analyser become very complex :( ◦ Memory usage is very high • It’s performance is poor. ◦ Single thread ◦ Sequential processing
14. Reimplement statistics analyser • Replace w/ Cloud Dataflow ◦ https://cloud.google.com/dataflow/ ◦ Full-managed ETL service ◦ It can offload machine resource issues. • Reimplement analyser logic by Scala ◦ Using https://github.com/spotify/scio ◦ Performance is good and it’s auto-scaled ◦ Job status is visualized automatically
15. Job management w/ Airflow • I’m trying Apache Airflow ◦ https://airflow.apache.org/ • It can define complex job relations. ◦ Job dependencies ◦ Waiting datasource ◦ Reporting via e-mail/slack
16. Improvements • Memory usage is offloaded • Job status is visualized • Each job become retryable • Slack/email reporting is integrated
17. Summary • I’ve tried to improve for data processing ◦ w/ workflow engines ◦ And full-managed services
18. We’re hiring • ソフトウェアエンジニア（Backend System） ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/4245 • ソフトウェアエンジニア（Site Reliability） ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/4246 • ソフトウェアエンジニア（MySQL Reliability） ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/6415 • … and who loves automation!
19. References • KPI に関わる数値の集計処理を Cloud Dataflow に置 き換えている話 ◦ http://tech.mercari.com/entry/2017/11/02/142931 • メルカリのデータ分析基盤の紹介〜BigQuery周辺の 話〜 ◦ http://tech.mercari.com/entry/2017/12/09/103000