Data processing, workflow and us ~How to manage automated jobs~

4ab3fec3e82ddb19bcadd93ef909a443?s=47 Ryo Okubo
December 12, 2017

Data processing, workflow and us ~How to manage automated jobs~

4ab3fec3e82ddb19bcadd93ef909a443?s=128

Ryo Okubo

December 12, 2017
Tweet

Transcript

  1. Data processing, workflow and us ~How to manage automated jobs~

    SRE-SET Automation Night Mercari SRE team :: @syu_cream
  2. whoami • @syu_cream • Mercari SRE newbie :) • Develop/maintain

    Data processing systems and some middlewares • My interests: ◦ Go, mruby, and automation
  3. My concerns... • Data processing is already automated. ◦ Fluentd,

    Embulk and some tools … ◦ Shell scripts, cron … • How can we improve job management?
  4. Agenda • BigQuery log uploader ◦ Previous status and issues

    ◦ Job management w/ digdag • Statistics analyser ◦ Previous status and issues ◦ Re-implementation w/ Cloud Dataflow ◦ Job management w/ Airflow
  5. BigQuery log uploader

  6. Mercari log analysis infrastructure See details at https://speakerdeck.com/cubicdaiya/mercari-data-analysis-infrastructure

  7. Deep dive to around BigQuery • Receive forwarded logs by

    Fluentd • Manage uploading job by cron ◦ Split and compress large log files ◦ Upload logs to GCS ◦ Load logs from GCS to BigQuery
  8. Issues... • It was hard to retry failed job ◦

    We catch slack alert when it fails ◦ But we need to reproduce some cmds • It didn’t visualize job status • Uploading logic becomes complex ◦ Related to BQ resource limit ▪ Compressed JSON has 4GB upper limit ◦ So we had split large file before uploading
  9. Improve Job w/ digdag • We start to use digdag,

    a simple workflow engine. ◦ https://www.digdag.io/ • And Split log files in Fluentd ◦ By adjusting buffer_chunk_limit
  10. Improvements • We can retry failed job. • We can

    view job progress. • Splitting logic is offloaded. ◦ So uploading logic become simple :)
  11. Statistics analyser

  12. Mercari basic statistics analyzer • Collect some data from DB

    and logs • Process by Node.js • Report basic KPIs by slack and email
  13. Issues... • It’s hard to maintain … ◦ Analyser become

    very complex :( ◦ Memory usage is very high • It’s performance is poor. ◦ Single thread ◦ Sequential processing
  14. Reimplement statistics analyser • Replace w/ Cloud Dataflow ◦ https://cloud.google.com/dataflow/

    ◦ Full-managed ETL service ◦ It can offload machine resource issues. • Reimplement analyser logic by Scala ◦ Using https://github.com/spotify/scio ◦ Performance is good and it’s auto-scaled ◦ Job status is visualized automatically
  15. Job management w/ Airflow • I’m trying Apache Airflow ◦

    https://airflow.apache.org/ • It can define complex job relations. ◦ Job dependencies ◦ Waiting datasource ◦ Reporting via e-mail/slack
  16. Improvements • Memory usage is offloaded • Job status is

    visualized • Each job become retryable • Slack/email reporting is integrated
  17. Summary • I’ve tried to improve for data processing ◦

    w/ workflow engines ◦ And full-managed services
  18. We’re hiring • ソフトウェアエンジニア(Backend System) ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/4245 • ソフトウェアエンジニア(Site

    Reliability) ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/4246 • ソフトウェアエンジニア(MySQL Reliability) ◦ https://open.talentio.com/1/c/mercari/requisitions/ detail/6415 • … and who loves automation!
  19. References • KPI に関わる数値の集計処理を Cloud Dataflow に置 き換えている話 ◦ http://tech.mercari.com/entry/2017/11/02/142931

    • メルカリのデータ分析基盤の紹介〜BigQuery周辺の 話〜 ◦ http://tech.mercari.com/entry/2017/12/09/103000