Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to ML Pipelines

Introduction to ML Pipelines


## Reference

### ML Pipelines for Software Engineers

GigaOm-Delivering on the Vision of MLOps - Microsoft Azure https://azure.microsoft.com/ja-jp/resources/gigaom-delivering-on-the-vision-of-mlops/
Sculley, D. and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Fran\c{c}ois and Dennison, Dan (2015) Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28 (NIPS 2015) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Denis Baylor and Kevin Haas and Konstantinos Katsiapis and Sammy Leong and Rose Liu and Clemens Menwald and Hui Miao and Neoklis Polyzotis and Mitchell Trott and Martin Zinkevich, Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform, 2019 {USENIX} Conference on Operational Machine Learning (OpML 19) (2019) https://www.usenix.org/conference/opml19/presentation/baylor
Daniel Papasian and Todd Underwood, How ML Breaks: A Decade of Outages for One Large ML Pipeline, USENIX Association 2020 https://www.usenix.org/conference/opml20/presentation/papasian

### Architecture of ML Pipelines

MLOps: 機械学習における継続的デリバリーと自動化のパイプライン - Google Cloud https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
The TFX User Guide - TensorFlow https://www.tensorflow.org/tfx/guide
The ExampleGen TFX Pipeline Component - TensorFlow https://www.tensorflow.org/tfx/guide/examplegen
Akshay Naresh Modi and Chiu Yuen Koo and Chuan Yu Foo and Clemens Mewald and Denis M. Baylor and Eric Breck and Heng-Tze Cheng and Jarek Wilkiewicz and Levent Koc and Lukasz Lew and Martin A. Zinkevich and Martin Wicke and Mustafa Ispir and Neoklis Polyzotis and Noah Fiedel and Salem Elie Haykal and Steven Whang and Sudip Roy and Sukriti Ramesh and Vihan Jain and Xin Zhang and Zakaria Haque TFX: A TensorFlow-Based Production-Scale Machine Learning Platform, KDD 2017 (2017) https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Data preprocessing for machine learning: options and recommendations - https://cloud.google.com/architecture/data-preprocessing-for-ml-with-tf-transform-pt1
Basic text classification - TensorFlow Core https://www.tensorflow.org/tutorials/keras/text_classification
The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow - Spotify Engineering https://engineering.atspotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/
ML Metadata - TensorFlow https://www.tensorflow.org/tfx/guide/mlmd
Building Standard TensorFlow ModelServer - TFX https://www.tensorflow.org/tfx/serving/serving_advanced

### How to Design Your ML Pipelines

機械学習基盤 Hekatoncheir の取り組み【DeNA TechCon 2021】/techcon2021-12 https://speakerdeck.com/dena_tech/techcon2021-12

Asei Sugiyama

June 04, 2021

More Decks by Asei Sugiyama

Other Decks in Technology


  1. View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide