We catch slack alert when it fails ◦ But we need to reproduce some cmds • It didn’t visualize job status • Uploading logic becomes complex ◦ Related to BQ resource limit ▪ Compressed JSON has 4GB upper limit ◦ So we had split large file before uploading
◦ Full-managed ETL service ◦ It can offload machine resource issues. • Reimplement analyser logic by Scala ◦ Using https://github.com/spotify/scio ◦ Performance is good and it’s auto-scaled ◦ Job status is visualized automatically