Introduction to ML Pipelines

機械学習パイプラインの概念について説明した資料です。

## Reference

### ML Pipelines for Software Engineers

GigaOm-Delivering on the Vision of MLOps - Microsoft Azure https://azure.microsoft.com/ja-jp/resources/gigaom-delivering-on-the-vision-of-mlops/
Sculley, D. and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Fran\c{c}ois and Dennison, Dan (2015) Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28 (NIPS 2015) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Denis Baylor and Kevin Haas and Konstantinos Katsiapis and Sammy Leong and Rose Liu and Clemens Menwald and Hui Miao and Neoklis Polyzotis and Mitchell Trott and Martin Zinkevich, Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform, 2019 {USENIX} Conference on Operational Machine Learning (OpML 19) (2019) https://www.usenix.org/conference/opml19/presentation/baylor
Daniel Papasian and Todd Underwood, How ML Breaks: A Decade of Outages for One Large ML Pipeline, USENIX Association 2020 https://www.usenix.org/conference/opml20/presentation/papasian

### Architecture of ML Pipelines

MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
The TFX User Guide - TensorFlow https://www.tensorflow.org/tfx/guide
The ExampleGen TFX Pipeline Component - TensorFlow https://www.tensorflow.org/tfx/guide/examplegen
Akshay Naresh Modi and Chiu Yuen Koo and Chuan Yu Foo and Clemens Mewald and Denis M. Baylor and Eric Breck and Heng-Tze Cheng and Jarek Wilkiewicz and Levent Koc and Lukasz Lew and Martin A. Zinkevich and Martin Wicke and Mustafa Ispir and Neoklis Polyzotis and Noah Fiedel and Salem Elie Haykal and Steven Whang and Sudip Roy and Sukriti Ramesh and Vihan Jain and Xin Zhang and Zakaria Haque TFX: A TensorFlow-Based Production-Scale Machine Learning Platform, KDD 2017 (2017) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Data preprocessing for machine learning: options and recommendations - https://cloud.google.com/architecture/data-preprocessing-for-ml-with-tf-transform-pt1
Basic text classification - TensorFlow Core https://www.tensorflow.org/tutorials/keras/text_classification
The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow - Spotify Engineering https://engineering.atspotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/
ML Metadata - TensorFlow https://www.tensorflow.org/tfx/guide/mlmd
Building Standard TensorFlow ModelServer - TFX https://www.tensorflow.org/tfx/serving/serving_advanced

### How to Design Your ML Pipelines

機械学習基盤 Hekatoncheir の取り組み【DeNA TechCon 2021】/techcon2021-12 https://speakerdeck.com/dena_tech/techcon2021-12

Transcript

1. View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide